CN118555361A - System and method for managing spatial states and display modes in a multi-user communication session - Google Patents

Abstract

The present disclosure relates to systems and methods for managing spatial states and display modes in a multi-user communication session. Some examples of the present disclosure relate to systems and methods for facilitating the display of content and avatars in a multi-communication session including a first electronic device and a second electronic device. In some examples, a first electronic device presents a computer-generated environment including an avatar and a first object corresponding to a user of a second electronic device, wherein the computer-generated environment is presented based on a first set of display parameters satisfying a first set of criteria, the first set of display parameters including spatial parameters for the user of the second electronic device, spatial parameters for the first object, and display mode parameters for the first object. In response to detecting a change in one or more display parameters in the first set of display parameters, the first electronic device updates the presentation of the computer-generated environment according to the one or more changes in the first set of display parameters.

Description

System and method for managing spatial state and display mode in a multi-user communication session

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 63/487,244 filed on February 27, 2023, U.S. Provisional Application No. 63/505,522 filed on June 1, 2023, U.S. Provisional Application No. 63/515,080 filed on July 21, 2023, U.S. Provisional Application No. 63/587,448 filed on October 2, 2023, and U.S. Patent Application No. 18/423,187 filed on January 25, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes.

Technical Field

The present invention generally relates to systems and methods for managing the spatial state and display mode of avatars within a multi-user communication session.

Background Art

Some computer graphics environments provide two-dimensional and/or three-dimensional environments in which at least some of the objects displayed for user observation are virtual and computer-generated. In some examples, the three-dimensional environment is presented by multiple devices communicating in a multi-user communication session. In some examples, an avatar (e.g., a representation) of each user participating in the multi-user communication session (e.g., via a computing device) is displayed in the three-dimensional environment of the multi-user communication session. In some examples, content can be shared in the three-dimensional environment for viewing and interaction by multiple users participating in the multi-user communication session.

Summary of the invention

Some examples of the present disclosure relate to systems and methods for facilitating the display of content and avatars in multiple communication sessions. In some examples, a first electronic device is in a communication session with a second electronic device, wherein the first electronic device and the second electronic device are configured to present a computer-generated environment. In some examples, the first electronic device presents a computer-generated environment including an avatar and a first object corresponding to a user of the second electronic device, wherein the computer-generated environment is presented based on a first set of display parameters satisfying a first set of standards. In some examples, the first set of display parameters includes space parameters for the user of the second electronic device, space parameters for the first object, and display mode parameters for the first object. In some examples, when displaying a computer-generated environment including an avatar and a first object corresponding to a user of the second electronic device, the first electronic device detects a change in one or more display parameters in the first set of display parameters. In some embodiments, in response to detecting a change, according to a determination that a change in the one or more display parameters in the first set of display parameters makes the first set of display parameters satisfy a second set of standards different from the first set of standards, the first electronic device updates the presentation of the computer-generated environment according to one or more changes in the first set of display parameters. In some examples, the first electronic device moves the first object or changes the display state of the first object in the computer-generated environment. In some examples, the first electronic device moves an avatar corresponding to a user of the second electronic device or ceases display of the avatar in the computer-generated environment. In some examples, based on a determination that a change in the one or more display parameters in the first set of display parameters does not cause the first set of display parameters to satisfy the second set of criteria, the first electronic device maintains presentation of the computer-generated environment based on the first set of display parameters satisfying the first set of criteria.

In some examples, if spatial live is enabled, the spatial parameters for the first object define the spatial template for the first object as the first spatial template, and/or the first object is displayed in a non-exclusive mode in the computer-generated environment in the communication session, then the first set of display parameters meets the first set of criteria. In some examples, if spatial live is disabled, the spatial parameters for the first object define the spatial template as the second spatial template, and/or the first object is displayed in an exclusive mode in the computer-generated environment in the communication session, then the first set of display parameters meets the second set of criteria.

A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

FIG. 1 illustrates an electronic device presenting an extended reality environment according to some examples of the present disclosure.

FIG2 shows a block diagram of an exemplary architecture of a system according to some examples of the present disclosure.

3 illustrates an example of a multi-user communication session including a first electronic device and a second electronic device according to some examples of the present disclosure.

4 illustrates a block diagram of an exemplary architecture of a communication application configured to facilitate multi-user communication sessions according to some examples of the present disclosure.

5A-5F illustrate exemplary interactions within a multi-user communication session according to some examples of the present disclosure.

6A-6L illustrate exemplary interactions within a multi-user communication session according to some examples of the present disclosure.

7A-7B show flow diagrams illustrating exemplary processes for displaying content within a multi-user communication session based on one or more display parameters, according to some examples of the present disclosure.

DETAILED DESCRIPTION

Some examples of the present disclosure relate to systems and methods for facilitating the display of content and avatars in multiple communication sessions. In some examples, a first electronic device is in a communication session with a second electronic device, wherein the first electronic device and the second electronic device are configured to present a computer-generated environment. In some examples, the first electronic device presents a computer-generated environment including an avatar and a first object corresponding to a user of the second electronic device, wherein the computer-generated environment is presented based on a first set of display parameters satisfying a first set of standards. In some examples, the first set of display parameters includes space parameters for the user of the second electronic device, space parameters for the first object, and display mode parameters for the first object. In some examples, when displaying a computer-generated environment including an avatar and a first object corresponding to a user of the second electronic device, the first electronic device detects a change in one or more display parameters in the first set of display parameters. In some embodiments, in response to detecting a change, according to a determination that a change in the one or more display parameters in the first set of display parameters makes the first set of display parameters satisfy a second set of standards different from the first set of standards, the first electronic device updates the presentation of the computer-generated environment according to one or more changes in the first set of display parameters. In some examples, the first electronic device moves the first object or changes the display state of the first object in the computer-generated environment. In some examples, the first electronic device moves an avatar corresponding to a user of the second electronic device or ceases display of the avatar in the computer-generated environment. In some examples, based on a determination that a change in the one or more display parameters in the first set of display parameters does not cause the first set of display parameters to satisfy the second set of criteria, the first electronic device maintains presentation of the computer-generated environment based on the first set of display parameters satisfying the first set of criteria.

In some examples, if spatial live is enabled, the spatial parameters for the first object define the spatial template for the first object as the first spatial template, and/or the first object is displayed in a non-exclusive mode in the computer-generated environment in the communication session, then the first set of display parameters meets the first set of criteria. In some examples, if spatial live is disabled, the spatial parameters for the first object define the spatial template as the second spatial template, and/or the first object is displayed in an exclusive mode in the computer-generated environment in the communication session, then the first set of display parameters meets the second set of criteria.

In some examples, a spatial group or state in a multi-user communication session represents a spatial arrangement/template indicating the locations of users and content located in the spatial group. In some examples, users in the same spatial group within a multi-user communication session experience spatial reality according to the spatial arrangement of the spatial group. In some examples, when a user of a first electronic device is in a first spatial group in a multi-user communication session and a user of a second electronic device is in a second spatial group in the multi-user communication session, the users experience spatial reality localized for their respective spatial groups. In some examples, when a user of a first electronic device and a user of a second electronic device are grouped into separate spatial groups or states within a multi-user communication session, if the first electronic device and the second electronic device return to the same operating state, the user of the first electronic device and the user of the second electronic device are regrouped into the same spatial group within the multi-user communication session.

In some examples, displaying content in a three-dimensional environment while in a multi-user communication session may include interaction with one or more user interface elements. In some examples, a user's gaze may be tracked by an electronic device as an input for targeting selectable options/indicators within a corresponding user interface element displayed in the three-dimensional environment. For example, gaze may be used to identify one or more options/indicators selected as a target using another selection input. In some examples, hand tracking input detected via an input device that communicates with the electronic device may be used to select a corresponding option/indicator. In some examples, objects displayed in the three-dimensional environment may be moved and/or reoriented in the three-dimensional environment based on movement input detected via the input device.

Figure 1 shows an electronic device 101 that presents an extended reality (XR) environment (e.g., a computer-generated environment) according to some examples of the present disclosure. In some examples, the electronic device 101 is a handheld device or a mobile device, such as a tablet, a laptop, a smart phone, or a head-mounted display. An example of the device 101 is described below with reference to the architectural block diagram of Figure 2. As shown in Figure 1, the electronic device 101, the table 106, and the coffee cup 152 are located in a physical environment 100. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, the electronic device 101 may be configured to capture an image of the physical environment 100 including the table 106 and the coffee cup 152 (shown in the field of view of the electronic device 101). In some examples, in response to a trigger, the electronic device 101 may be configured to display a virtual object 110 (e.g., two-dimensional virtual content) in a computer-generated environment (e.g., represented by a rectangle shown in FIG. 1 ) that does not exist in the physical environment 100 but is displayed in the computer-generated environment on top of (e.g., anchored to) a computer-generated representation 106' of a real-world table 106. For example, in response to detecting a flat surface of the table 106 in the physical environment 100, the virtual object 110 may be displayed on the surface of the computer-generated representation 106' of the table in the computer-generated environment, next to a computer-generated representation 152' of a real-world coffee cup 152 displayed via the device 101.

It should be understood that the virtual object 110 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or three-dimensional virtual objects) in a three-dimensional computer-generated environment. For example, a virtual object may represent an application or user interface displayed in a computer-generated environment. In some examples, a virtual object may represent content corresponding to an application and/or displayed via a user interface in a computer-generated environment. In some examples, the virtual object 110 is optionally configured to be interactive and responsive to user input, so that a user can virtually touch, tap, move, rotate, or otherwise interact with the virtual object. In some examples, the virtual object 110 may be displayed in a three-dimensional computer-generated environment within a multi-user communication session ("multi-user communication session", "communication session"). In some such examples, as described in more detail below, the virtual object 110 may be viewable and/or configured to be interactive and responsive to multiple users and/or user input provided by multiple users, respectively. In addition, it should be understood that the 3D environment (or 3D virtual object) described herein may be a representation of a 3D environment (or three-dimensional virtual object) projected or presented at an electronic device.

In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and/or a digital video player application.

FIG2 shows a block diagram of an exemplary architecture of a system 201 according to some examples of the present disclosure. In some examples, the system 201 includes multiple devices. For example, the system 201 includes a first electronic device 260 and a second electronic device 270, wherein the first electronic device 260 and the second electronic device 270 communicate with each other. In some examples, the first electronic device 260 and the second electronic device 270 are portable devices, such as mobile phones, smart phones, tablet computers, laptop computers, auxiliary devices that communicate with another device, etc.

As shown in FIG2 , the first electronic device 260 optionally includes various sensors (e.g., one or more hand tracking sensors 202A, one or more position sensors 204A, one or more image sensors 206A, one or more touch-sensitive surfaces 209A, one or more motion and/or orientation sensors 210A, one or more eye tracking sensors 212A, one or more microphones 213A or other audio sensors, etc.), one or more display generation components 214A, one or more speakers 216A, one or more processors 218A, one or more memories 220A, and/or communication circuits 222A. In some examples, the second device 270 optionally includes various sensors (e.g., one or more hand tracking sensors 202B, one or more position sensors 204B, one or more image sensors 206B, one or more touch-sensitive surfaces 209B, one or more motion and/or orientation sensors 210B, one or more eye tracking sensors 212B, one or more microphones 213B or other audio sensors, etc.), one or more display generation components 214B, one or more speakers 216, one or more processors 218B, one or more memories 220B, and/or communication circuits 222B. One or more communication buses 208A and 208B are optionally used for communication between the above components of devices 260 and 270, respectively. First electronic device 260 and second electronic device 270 optionally communicate via a wired or wireless connection between the two devices (eg, via communication circuits 222A-222B).

The communication circuits 222A, 222B optionally include circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and/or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuits 222A, 222B optionally include circuits for using near field communication (NFC) and/or short-range communication such as A circuit for communication.

Processors 218A, 218B include one or more general purpose processors, one or more graphics processors, and/or one or more digital signal processors. In some examples, memory 220A, 220B is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processors 218A, 218B to perform the techniques, processes, and/or methods described below. In some examples, memory 220A, 220B may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and/or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, or the like.

In some examples, display generation components 214A, 214B include a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation components 214A, 214B include multiple displays. In some examples, display generation components 214A, 214B may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, devices 260 and 270 include touch-sensitive surfaces 209A and 209B, respectively, for receiving user inputs such as tap inputs and swipe inputs or other gestures. In some examples, display generation components 214A, 214B and touch-sensitive surfaces 209A, 209B form a touch-sensitive display (e.g., a touch screen integrated with devices 260 and 270, respectively, or a touch screen external to devices 260 and 270, respectively, that communicates with devices 260 and 270).

Devices 260 and 270 optionally include image sensors 206A and 206B, respectively. Image sensors 206A/206B optionally include one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and/or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. Image sensors 206A/206B also optionally include one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensors 206A/206B also optionally include one or more cameras configured to capture the movement of physical objects in the real-world environment. Image sensors 206A/206B also optionally include one or more depth sensors configured to detect the distance of physical objects from device 260/270. In some examples, information from one or more depth sensors may allow a device to identify an object in a real-world environment and distinguish it from other objects in the real-world environment. In some examples, one or more depth sensors may allow a device to determine the texture and/or topography of an object in a real-world environment.

In some examples, devices 260 and 270 use CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around devices 260 and 270. In some examples, image sensors 206A/206B include a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in a real-world environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, devices 260/270 use image sensors 206A/206B to detect the position and orientation of devices 260/270 and/or display generation components 214A/214B in a real-world environment. For example, devices 260/270 use image sensors 206A/206B to track the position and orientation of display generation components 214A/214B relative to one or more fixed objects in a real-world environment.

In some examples, device 260/270 includes microphone 213A/213B or other audio sensor. Device 260/270 uses microphone 213A/213B to detect sounds from the user and/or the user's real-world environment. In some examples, microphone 213A/213B includes a microphone array (multiple microphones) that optionally operate in concert to identify ambient noise or locate sound sources in the space of the real-world environment.

In some examples, the device 260/270 includes a location sensor 204A/204B for detecting the location of the device 260/270 and/or the display generating component 214A/214B. For example, the location sensor 204A/204B may include a GPS receiver that receives data from one or more satellites and allows the device 260/270 to determine the absolute location of the device in the physical world.

In some examples, the device 260/270 includes an orientation sensor 210A/210B for detecting the orientation and/or movement of the device 260/270 and/or the display generation component 214A/214B. For example, the device 260/270 uses the orientation sensor 210A/210B to track changes in the position and/or orientation of the device 260/270 and/or the display generation component 214A/214B, such as changes relative to physical objects in the real world environment. The orientation sensor 210A/210B optionally includes one or more gyroscopes and/or one or more accelerometers.

In some examples, device 260/270 includes hand tracking sensor 202A/202B and/or eye tracking sensor 212A/212B. Hand tracking sensor 202A/202B is configured to track the position/location of one or more parts of a user's hand and/or the movement of one or more parts of a user's hand relative to an extended reality environment, relative to display generation component 214A/214B and/or relative to another defined coordinate system. Eye tracking sensor 212A/212B is configured to track the position and movement of a user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and/or relative to display generation component 214A/214B. In some examples, hand tracking sensor 202A/202B and/or eye tracking sensor 212A/212B is implemented together with display generation component 214A/214B. In some examples, hand tracking sensors 202A/202B and/or eye tracking sensors 212A/212B are implemented separately from display generation components 214A/214B.

In some examples, the hand tracking sensors 202A/202B may use image sensors 206A/206B (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture three-dimensional information from the real world including one or more hands (e.g., one or more hands of a human user). In some examples, the hands may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206A/206B are positioned relative to the user to define a field of view and interaction space of the image sensors 206A/206B, in which the finger/hand positions, orientations, and/or movements captured by the image sensors are used as input (e.g., to distinguish from the user's idle hands or other hands of other people in the real-world environment). Tracking fingers/hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

In some examples, eye tracking sensors 212A/212B include at least one eye tracking camera (e.g., an infrared (IR) camera) and/or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus/gaze can be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by corresponding eye tracking camera/illumination source.

Device 260/270 and system 201 are not limited to the components and configurations of FIG. 2, but may include fewer components, other components, or additional components in a variety of configurations. In some examples, system 201 may be implemented in a single device. One or more people using system 201 are optionally referred to herein as one or more users of the device. Now turn your attention to the exemplary simultaneous display of a three-dimensional environment on a first electronic device (e.g., corresponding to device 260) and a second electronic device (e.g., corresponding to device 270). As described below, a first electronic device may communicate with a second electronic device in a multi-user communication session. In some examples, an avatar (e.g., representation) of a user of a first electronic device may be displayed in a three-dimensional environment at a second electronic device, and an avatar of a user of a second electronic device may be displayed in a three-dimensional environment at a first electronic device. In some examples, a user of a first electronic device and a user of a second electronic device may be associated with the same spatial state in a multi-user communication session. In some examples, an interaction (or other type of interaction) with content in a three-dimensional environment when a first electronic device and a second electronic device are in a multi-user communication session may cause a user of the first electronic device and a user of the second electronic device to become associated with different spatial states in a multi-user communication session.

3 shows an example of a multi-user communication session including a first electronic device 360 and a second electronic device 370 according to some examples of the present disclosure. In some examples, the first electronic device 360 may present a three-dimensional environment 350A, and the second electronic device 370 may present a three-dimensional environment 350B. The first electronic device 360 and the second electronic device 370 may be similar to the device 101 or 260/270, and/or may be head-mounted systems/devices and/or projection-based systems/devices (including hologram-based systems/devices) configured to generate and present a three-dimensional environment, such as a head-up display (HUD), a head-mounted display (HMD), a window with integrated display capabilities, a display formed as a lens designed to be placed on a person's eye (e.g., similar to a contact lens), respectively. In the example of Figure 3, the first user optionally wears the first electronic device 360 and the second user optionally wears the second electronic device 370, so that the three-dimensional environment 350A/350B can be defined by the X, Y, and Z axes viewed from the perspective of the electronic devices (e.g., a viewpoint associated with the electronic devices 360/370 (which can be, for example, a head-mounted display)).

As shown in FIG3 , the first electronic device 360 may be in a first physical environment including a table 306 and a window 309. Therefore, the three-dimensional environment 350A presented using the first electronic device 360 optionally includes a captured portion of the physical environment surrounding the first electronic device 360, such as a representation 306' of the table and a representation 309' of the window. Similarly, the second electronic device 370 may be in a second physical environment different from (e.g., separate from) the first physical environment, the second physical environment including a floor lamp 307 and a coffee table 308. Therefore, the three-dimensional environment 350B presented using the second electronic device 370 optionally includes a captured portion of the physical environment surrounding the second electronic device 370, such as a representation 307' of the floor lamp and a representation 308' of the coffee table. In addition, the three-dimensional environments 350A and 350B may include representations of the floor, ceiling, and walls of the rooms in which the first electronic device 360 and the second electronic device 370 are respectively located.

As described above, in some examples, the first electronic device 360 is optionally in a multi-user communication session with the second electronic device 370. For example, the first electronic device 360 and the second electronic device 370 (e.g., via the communication circuit 222A/222B) are configured to present a shared three-dimensional environment 350A/350B including one or more shared virtual objects (e.g., content such as images, videos, audio, etc., representations of user interfaces of applications, etc.). As used herein, the term "shared three-dimensional environment" refers to a three-dimensional environment that is independently presented, displayed, and/or visible at two or more electronic devices, via which content, applications, data, etc. can be shared and/or presented to users of the two or more electronic devices. In some examples, when the first electronic device 360 is in a multi-user communication session with the second electronic device 370, an avatar corresponding to a user of one electronic device is optionally displayed in a three-dimensional environment displayed via another electronic device. For example, as shown in FIG. 3, at the first electronic device 360, an avatar 315 corresponding to a user of the second electronic device 370 is displayed in a three-dimensional environment 350A. Similarly, at the second electronic device 370, an avatar 317 corresponding to the user of the first electronic device 360 is displayed in the three-dimensional environment 350B.

In some examples, the presentation of the avatar 315/317 as part of the shared three-dimensional environment is optionally accompanied by an audio effect corresponding to the voice of the user of the electronic device 370/360. For example, the avatar 315 displayed in the three-dimensional environment 350A using the first electronic device 360 is optionally accompanied by an audio effect corresponding to the voice of the user of the second electronic device 370. In some such examples, when the user of the second electronic device 370 speaks, the user's voice can be detected by the second electronic device 370 (e.g., via microphone 213B) and transmitted to the first electronic device 360 (e.g., via communication circuit 222B/222A), so that the detected voice of the user of the second electronic device 370 can be presented to the user of the first electronic device 360 as audio in the three-dimensional environment 350A (e.g., using speaker 216A). In some examples, the audio effect corresponding to the voice of the user of the second electronic device 370 can be spatialized so that it appears to the user of the first electronic device 360 as emanating from the location of the avatar 315 in the shared three-dimensional environment 350A (e.g., despite being output from the speaker of the first electronic device 360). Similarly, the avatar 317 displayed in the three-dimensional environment 350B using the second electronic device 370 is optionally accompanied by an audio effect corresponding to the voice of the user of the first electronic device 360. In some such examples, when the user of the first electronic device 360 speaks, the user's voice may be detected by the first electronic device 360 (e.g., via microphone 213A) and transmitted to the second electronic device 370 (e.g., via communication circuitry 222A/222B), so that the detected voice of the user of the first electronic device 360 may be presented to the user of the second electronic device 370 as audio in the three-dimensional environment 350B (e.g., using speaker 216B). In some examples, the audio effect corresponding to the voice of the user of the first electronic device 360 may be spatialized so that it appears to the user of the second electronic device 370 as emanating from the location of the avatar 317 in the shared three-dimensional environment 350B (e.g., despite being output from the speaker of the first electronic device 360).

In some examples, when in a multi-user communication session, the avatars 315/317 are displayed in the three-dimensional environment 350A/350B with respective orientations corresponding to and/or based on the orientations of the electronic devices 360/370 (and/or the users of the electronic devices 360/370) in the physical environment surrounding the electronic devices 360/370. For example, as shown in FIG3 , in the three-dimensional environment 350A, the avatar 315 is optionally facing the viewpoint of the user of the first electronic device 360, and in the three-dimensional environment 350B, the avatar 317 is optionally facing the viewpoint of the user of the second electronic device 370. When a particular user moves the electronic device (and/or himself) in the physical environment, the user's viewpoint changes according to the movement, which may therefore also change the orientation of the user's avatar in the three-dimensional environment. For example, referring to Figure 3, if the user of the first electronic device 360 looks to the left in the three-dimensional environment 350A, causing the first electronic device 360 to rotate to the left (e.g., counterclockwise) (e.g., a corresponding amount), the user of the second electronic device 370 will see the avatar 317 corresponding to the user of the first electronic device 360 rotate to the right (e.g., clockwise) relative to the viewpoint of the user of the second electronic device 370 based on the movement of the first electronic device 360.

Additionally, in some examples, the viewpoint of the three-dimensional environment 350A/350B and/or the position of the viewpoint of the three-dimensional environment 350A/350B optionally changes based on movement of the electronic device 360/370 (e.g., by a user of the electronic device 360/370) while in a multi-user communication session. For example, if the first electronic device 360 moves closer to the representation 306' of the table and/or the avatar 315 while in a communication session (e.g., because the user of the first electronic device 360 moves forward in the physical environment surrounding the first electronic device 360), the viewpoint of the three-dimensional environment 350A will change accordingly, making the representation 306' of the table, the representation 309' of the window, and the avatar 315 appear larger in the field of view. In some examples, each user may interact with three-dimensional environment 350A/350B independently, such that changes in the viewpoint of three-dimensional environment 350A and/or interaction of first electronic device 360 with virtual objects in three-dimensional environment 350A optionally do not affect content shown in three-dimensional environment 350B at second electronic device 370, and vice versa.

In some examples, the avatar 315/317 is a representation of a user of the electronic device 370/360 (e.g., a full body rendering). In some examples, the avatar 315/317 is a representation of a portion of a user of the electronic device 370/360 (e.g., a rendering of a head, a face, a head and a torso, etc.). In some examples, the avatar 315/317 is a user-personalized, user-selected, and/or user-created representation displayed in the three-dimensional environment 350A/350B that represents the user of the electronic device 370/360. It should be understood that although the avatars 315/317 shown in FIG. 3 correspond to full body representations of users of the electronic devices 370/360, respectively, alternative avatars such as those described above may be provided.

As described above, when the first electronic device 360 and the second electronic device 370 are in a multi-user communication session, the three-dimensional environment 350A/350B can be a shared three-dimensional environment presented using the electronic device 360/370. In some examples, in a multi-user communication session, content viewed by one user at one electronic device can be shared with another user at another electronic device. In some such examples, the content can be experienced (e.g., viewed and/or interacted with) by two users (e.g., via their respective electronic devices) in a shared three-dimensional environment (e.g., the content is shared content in the three-dimensional environment). For example, as shown in FIG. 3 , the three-dimensional environment 350A/350B includes a shared virtual object 310 (e.g., which is optionally a three-dimensional virtual sculpture), which is associated with a respective application (e.g., a content creation application) and can be viewed by two users and interacted with by two users. As shown in FIG. 3 , the shared virtual object 310 can be displayed together with a grabber affordance (e.g., a manipulation bar) 335 that can be selected to initiate movement of the shared virtual object 310 within the three-dimensional environment 350A/350B.

In some examples, the three-dimensional environment 350A/350B includes non-shared content that is private to one user in the multi-user communication session. For example, in FIG. 3, the first electronic device 360 is displaying a private application window 330 (e.g., a private object) in the three-dimensional environment 350A, which is optionally an object that is not shared between the first electronic device 360 and the second electronic device 370 in the multi-user communication session. In some examples, the private application window 330 may be associated with a corresponding application (e.g., such as a media player application, a web browsing application, a messaging application, etc.) operating on the first electronic device 360. Because the private application window 330 is not shared with the second electronic device 370, the second electronic device 370 optionally displays a representation 330 of the private application window in the three-dimensional environment 350B. As shown in FIG. 3, in some examples, the representation 330 of the private application window may be a faded, blocked, discolored, and/or translucent representation of the private application window 330, which prevents the user of the second electronic device 370 from viewing the content of the private application window 330.

In addition, in some examples, the virtual object 310 corresponds to an object of a first type, and the private application window 330 corresponds to an object of a second type different from the object of the first type. In some examples, the object type is determined based on the orientation of the shared object in the shared three-dimensional environment. For example, the first type of object is an object with a horizontal orientation relative to the viewpoint of the user of the electronic device in the shared three-dimensional environment. As shown in FIG. 3, as similarly discussed above, the shared virtual object 310 is optionally a virtual sculpture with a volume and/or horizontal orientation relative to the viewpoint of the user of the first electronic device 360 and the second electronic device 370 in the three-dimensional environment 350A/350B. Therefore, as discussed above, the shared virtual object 310 is an object of the first type. On the other hand, the second type of object is an object with a vertical orientation relative to the viewpoint of the user of the electronic device in the shared three-dimensional environment. For example, in FIG. 3, as similarly discussed above, the shared virtual object 310 (e.g., a private application window) is a two-dimensional object with a vertical orientation relative to the viewpoint of the user of the first electronic device 360 and the second electronic device 370 in the three-dimensional environment 350A/350B. Thus, as described above, private application window 330 (and thus the representation of private application window 330) is an object of the second type. In some examples, as described in more detail later, the object type indicates a spatial template for the user in the shared three-dimensional environment that determines where avatar 315/317 is spatially positioned relative to the object in the shared three-dimensional environment.

In some examples, the user of the first electronic device 360 and the user of the second electronic device 370 share the same spatial state 340 within the multi-user communication session. In some examples, the spatial state 340 can be a baseline (e.g., a first or default) spatial state within the multi-user communication session. For example, when the user of the first electronic device 360 and the user of the second electronic device 370 initially join the multi-user communication session, the user of the first electronic device 360 and the user of the second electronic device 370 are automatically (and initially, as discussed in more detail below) associated with (e.g., grouped into) the spatial state 340 within the multi-user communication session. In some examples, when the users are in the spatial state 340 as shown in FIG. 3, the user of the first electronic device 360 and the user of the second electronic device 370 have a first spatial arrangement (e.g., a first spatial template) within the shared three-dimensional environment, as represented by the positions of the ellipses 315A (e.g., corresponding to the user of the second electronic device 370) and 317A (e.g., corresponding to the user of the first electronic device 360). For example, the user of the first electronic device 360 and the user of the second electronic device 370 (including objects displayed in the shared three-dimensional environment) have spatial reality within the spatial state 340. In some examples, spatial reality requires a consistent spatial arrangement between the user (or its representation) and the virtual object. For example, the distance between the viewpoint of the user of the first electronic device 360 and the avatar 315 corresponding to the user of the second electronic device 360 may be the same as the distance between the viewpoint of the user of the second electronic device 370 and the avatar 317 corresponding to the user of the first electronic device 370. As described herein, if the position of the viewpoint of the user of the first electronic device 360 moves, the avatar 317 corresponding to the user of the first electronic device 360 moves in the three-dimensional environment 350B according to the movement of the position of the user's viewpoint relative to the viewpoint of the user of the second electronic device 370. In addition, if the user of the first electronic device 360 performs an interaction on the shared virtual object 310 (e.g., moves the virtual object 310 in the three-dimensional environment 350A), the second electronic device 370 changes the display of the shared virtual object 310 in the three-dimensional environment 350B according to the interaction (e.g., moves the virtual object 310 in the three-dimensional environment 350B).

It should be understood that in some examples, more than two electronic devices can be communicatively linked in a multi-user communication session. For example, in a situation where three electronic devices are communicatively linked in a multi-user communication session, a first electronic device will display two avatars corresponding to the users of the other two electronic devices, rather than just one avatar. Therefore, it should be understood that the various processes and exemplary interactions described herein with reference to a first electronic device 360 and a second electronic device 370 in a multi-user communication session are optionally applicable to situations where more than two electronic devices are communicatively linked in a multi-user communication session.

In some examples, it may be advantageous to selectively control the display of content and avatars corresponding to users of electronic devices that are communicatively linked in a multi-user communication session. As described above, content displayed and/or shared in a three-dimensional environment when multiple users are in a multi-user communication session may be associated with a corresponding application that provides data for displaying content in a three-dimensional environment. In some examples, a communication application for facilitating a multi-user communication session may be provided (e.g., provided locally on each electronic device or remotely via a server (e.g., a wireless communication terminal) that communicates with each electronic device). In some such examples, the communication application receives data from the corresponding application and sets/defines one or more display parameters based on the data that controls the display of content in a three-dimensional environment. In addition, in some examples, the one or more display parameters control the display of avatars corresponding to users of the electronic device in a three-dimensional environment within a multi-user communication session. For example, data corresponding to the spatial state of each user in a multi-user communication session and/or data indicating user interactions in a multi-user communication session also set/define the one or more display parameters for a multi-user communication session, as discussed herein. An exemplary architecture of a communication application is provided in FIG. 4, as discussed in more detail below.

FIG4 shows a block diagram of an exemplary architecture of a communication application configured to facilitate a multi-user communication session according to some examples of the present disclosure. In some examples, as shown in FIG4, a communication application 488 may be configured to operate on an electronic device 401 (e.g., corresponding to the electronic device 101 in FIG1). In some examples, the communication application 488 may be configured to operate at a server (e.g., a wireless communication terminal) that communicates with the electronic device 401. In some examples, as discussed below, the communication application 488 may facilitate a multi-user communication session including multiple electronic devices (e.g., including the electronic device 401) (such as the first electronic device 360 and the second electronic device 370 described above with reference to FIG3).

In some examples, as shown in FIG. 4 , the communication application 488 is configured to communicate with one or more auxiliary applications 470. In some examples, as discussed in more detail below, the communication application 488 and one or more auxiliary applications 470 transmit and exchange data and other high-level information via a spatial coordinator application program interface (API) 462. As used herein, an API may define one or more parameters passed between a calling application and other software codes (e.g., an operating system, a library routine, a function) that provide services, provide data, or perform operations or calculations. The API may be implemented as one or more calls in a program code, which are based on the calling convention defined in the API specification document and send or receive one or more parameters through a parameter list or other structure. The parameter may be a constant, a key, a data structure, a target, a target class, a variable, a data type, a pointer, an array, a list, or another call. API calls and parameters may be implemented in any programming language. A programming language may define the vocabulary and calling convention that a programmer will use to access the functions that support the API. In some examples, an API call may report to an application the ability of a device to run an application, such as input capability, output capability, processing capability, power capability, communication capability, etc.

In some examples, as shown in FIG. 4 , the scene integration service 466 is configured to receive application data 471 from one or more auxiliary applications 470. For example, as previously discussed with reference to FIG. 3 , a virtual object (e.g., including content) may be displayed in a shared three-dimensional environment within a multi-user communication session. In some examples, a virtual object may be associated with one or more corresponding applications (such as one or more auxiliary applications 470). In some examples, the application data 471 includes information corresponding to the appearance of the virtual object, the interactive features of the virtual object (e.g., whether the object can be moved, selected, etc.), the positioning information of the virtual object (e.g., the placement of the virtual object within the shared three-dimensional environment), etc. In some examples, as discussed in more detail below, the scene integration service 466 utilizes the application data 471 to generate and define one or more display parameters for one or more virtual objects associated with one or more auxiliary applications 470, wherein the one or more display parameters control the display of the one or more virtual objects in the shared three-dimensional environment. In some examples, as shown in FIG. 4 , the application data 471 is received via the scene integration service 466.

Additionally, in some examples, as shown in FIG. 4 , the scene integration service 466 is configured to utilize scene data 485. In some examples, the scene data 485 includes information corresponding to a physical environment (e.g., a real-world environment) captured via one or more sensors of the electronic device 401 (e.g., via the image sensor 206A/206B in FIG. 2 ) (e.g., the real-world environment discussed above with reference to FIG. 3 ). For example, the scene data 485 includes information corresponding to one or more features of the physical environment, such as the appearance of the physical environment, including the location of objects within the physical environment (e.g., objects that form part of the physical environment, optionally excluding virtual objects), the size of the physical environment, the behavior of objects within the computer-generated environment (e.g., background objects, such as background users, pets, vehicles, etc.), etc. In some examples, the scene integration service 466 receives the scene data 485 from the outside (e.g., from an operating system of the electronic device 401). In some examples, the scene data 485 can be provided to the one or more auxiliary applications in the form of context data 473. For example, context data 473 enables one or more auxiliary applications 470 to interpret the physical environment surrounding the virtual object described above, which is optionally included as a pass-through in the shared three-dimensional environment.

In some examples, communication application 488 and/or one or more auxiliary applications 470 are configured to receive user input data 481A (e.g., from an operating system of electronic device 401). For example, user input data 481A may correspond to user input detected via one or more input devices in communication with electronic device 401, such as contact-based input detected via a physical input device (e.g., touch-sensitive surface 209A/209B in FIG. 2) or gesture-based and/or gaze-based input detected via a sensor device (e.g., hand tracking sensor 202A/202B, orientation sensor 210A/210B, and/or eye tracking sensor 212A/212B). In some examples, user input data 481A includes information corresponding to input directed to one or more virtual objects displayed in a shared three-dimensional environment and associated with one or more auxiliary applications 470. For example, user input data 481A includes information corresponding to input for interacting directly with a virtual object, such as moving a virtual object in a shared three-dimensional environment, or information corresponding to input for causing a virtual object to be displayed (e.g., launching one or more auxiliary applications 470). In some examples, user input data 481A includes information corresponding to input directed to a shared three-dimensional environment displayed at electronic device 401. For example, user input data 481A includes information corresponding to input for moving (e.g., rotating and/or shifting) a viewpoint of a user of electronic device 401 in a shared three-dimensional environment.

In some examples, as described above, the spatial coordinator API 462 is configured to define one or more display parameters according to which the shared three-dimensional environment (e.g., including virtual objects and avatars) is displayed at the electronic device 401. In some examples, as shown in FIG. 4, the spatial coordinator API 462 includes an application space state determiner 464 (e.g., optionally a sub-API and/or a first function, such as a space template preference API/function), which provides (e.g., defines) space state parameters for one or more auxiliary applications 470. In some examples, the space state parameters for the one or more auxiliary applications are provided via application space state data 463, as discussed in more detail below. In some examples, the space state parameters for one or more auxiliary applications 470 define a space template for the one or more auxiliary applications 470. For example, the space state parameters for the respective applications define a spatial arrangement of one or more participants in a multi-user communication session relative to a virtual object (e.g., such as the virtual object 310 or the private application window 330 in FIG. 3) displayed in the shared three-dimensional environment, as discussed in more detail below with reference to FIGS. 5A to 5F. In some examples, as shown in FIG. 4 , the application space state determiner 464 defines space state parameters for one or more auxiliary applications 470 based on space state request data 465 received from one or more auxiliary applications 470. In some examples, the space state request data 465 includes information corresponding to a request to display virtual objects associated with one or more auxiliary applications 470 in a specific space state (e.g., spatial arrangement) in a shared three-dimensional environment within a multi-user communication session. In some examples, the space state request data 465 includes information indicating a default (e.g., baseline) space template according to which content associated with one or more auxiliary applications 470 (e.g., including one or more virtual objects) and one or more avatars corresponding to one or more users in the multi-user communication session are arranged. For example, as discussed below with reference to FIGS. 5A to 5F , content shared as “primary” content in a three-dimensional environment (e.g., media-based content such as videos, music, podcasts, and/or image-based content presented to users for viewing/consumption) may be displayed by default in a side-by-side space template, and other types of content (e.g., private windows or two-dimensional representations of users) may be displayed by default in a circular (e.g., conversation) space template. Additionally, in some examples, defining a spatial template for one or more auxiliary applications 470 includes establishing a spatial separation between one or more virtual objects associated with the one or more auxiliary applications 470 and one or more participants in a multi-user communication session. For example, the application spatial state determiner 464 is configured to define distances between adjacent avatars corresponding to users in the multi-user communication session and/or distances between one or more avatars and virtual objects (e.g., application windows) within a corresponding spatial template (e.g., where such distances may be different values or the same value), as described in more detail below with reference to FIGS. 5A to 5F . In some examples, the separation distance is automatically determined (e.g., set to a predefined value) by the communication application 488 (e.g., via the application spatial state determiner 464). Alternatively, in some examples, the separation distance is determined based on information provided by the one or more auxiliary applications 470. For example, the spatial state request data 465 provided to the application spatial state determiner 464 includes information indicating a specified or requested value for the spatial separation discussed above.

In some examples, as discussed below with reference to FIGS. 5A-5F , the spatial state parameters determined for one or more auxiliary applications 470 may or may not correspond to the spatial state requested by the one or more auxiliary applications 470. In some examples, as described in more detail with reference to FIGS. 5A-5F , changes in the spatial state parameters of one or more auxiliary applications 470 may cause changes in the spatial template within the multi-user communication session. For example, the application spatial state determiner 464 may change the spatial state parameters for the one or more auxiliary applications 470 in response to a change in the display state of a virtual object associated with the one or more auxiliary applications 470 (e.g., transmitted from the one or more auxiliary applications 470 via the spatial state request data 465). In some examples, the spatial state parameters for the one or more auxiliary applications 470 may also represent whether a particular application in the one or more auxiliary applications 470 supports spatial live (e.g., is compatible with the rules of spatial live). For example, if the application is an audio-based application (e.g., a phone call application), the application spatial state determiner 464 optionally does not define a spatial template for a virtual object associated with the application.

In some examples, as shown in FIG. 4 , the spatial coordinator API 462 includes a participant spatial state determiner 468 (e.g., optionally a sub-API and/or a second function, such as a participant spatial state API/function) that provides (e.g., defines) spatial state parameters for users of the electronic device 401. In some examples, the spatial state parameters for the user are provided via the user spatial state data 467, as discussed in more detail below. In some examples, the spatial state parameters for the user define the enablement of spatial live within a multi-user communication session. For example, the spatial state parameters for the user of the electronic device 401 define whether the avatar corresponding to the user of the electronic device maintains the spatial live of the avatar corresponding to the second user (e.g., and/or virtual object) of the second electronic device within the multi-user communication session, as described similarly above with reference to FIG. 3 . In some examples, if the number of participants (e.g., users) within the multi-user communication session is below a threshold number of participants (e.g., less than four, five, six, eight, or ten participants), spatial live is enabled for the multi-user communication session. In some examples, for the case where the number of participants in a multi-user communication session is or reaches a number greater than a threshold number, spatial live is not enabled for the multi-user communication session. In some examples, as discussed in more detail below with reference to Figures 5A to 5F, if the spatial parameters for the user define spatial live as enabled, the electronic device 401 displays an avatar corresponding to the user in the multi-user communication session in a shared three-dimensional environment. In some examples, if the spatial parameters of the user define spatial live as disabled, the electronic device 401 abandons displaying an avatar corresponding to the user in a shared three-dimensional environment (e.g., displaying a two-dimensional representation instead), as discussed below with reference to Figures 5A to 5F. In some examples, as shown in Figure 4, the participant spatial state determiner 468 optionally defines spatial state parameters for the user based on user input data 481B. In some examples, the user input data 481B may include information corresponding to a user input that explicitly disables or enables spatial live in a multi-user communication session. For example, as described in more detail below with reference to FIGS. 5A-5F , user input data 481B includes information corresponding to a user input for activating an audio-only mode (eg, which disables spatial live).

In some examples, as shown in FIG. 4 , the spatial coordinator API 462 includes a display mode determiner 472 (e.g., optionally a sub-API and/or a third function, such as supporting a stage spatial live API/function) that provides (e.g., defines) display mode parameters. In some examples, the display mode parameters are provided via display mode data 469, as discussed in more detail below. In some examples, the display mode parameters control whether a particular experience in a multi-user communication session is exclusive or non-exclusive (e.g., windowed). For example, the display mode parameters define whether users who are viewing/experiencing content in a shared three-dimensional environment within a multi-user communication session share the same spatial state (e.g., an exclusive state or a non-exclusive state) when viewing/experiencing the content, as similarly described above with reference to FIG. 3 . In some examples, as similarly described above with reference to FIG. 3 , spatial live is enabled for participants in a multi-user communication session that share the same spatial state. In some examples, as shown in FIG. 4 , the display mode determiner 472 may define the display mode parameters based on input data 483. In some examples, input data 483 includes information corresponding to a user input corresponding to a request to change a display mode of a virtual object displayed in a shared three-dimensional environment. For example, as described below with reference to Figures 6A to 6I, input data 483 may include information indicating that a user has provided input for causing a virtual object to be displayed in an exclusive state in a multi-user communication session, which causes display mode determiner 472 to define the display mode parameter as exclusive. As described in more detail with reference to Figures 6A to 6I, in some examples, one or more auxiliary applications 470 may provide an indication of a change in the level of exclusivity, which optionally disables spatial live until all users in the multi-user communication session are again in the same spatial state.

In some examples, as shown in FIG. 4 , the spatial coordinator API 462 transmits display stage data 477 (e.g., via a display mode determiner 472) to one or more auxiliary applications 470. In some examples, the display stage data 477 includes information corresponding to whether a stage or scenery is applied to a spatial template/arrangement (e.g., as described above), according to which virtual objects associated with one or more auxiliary applications 470 (e.g., and avatars) are displayed in a shared three-dimensional environment in a multi-user communication session. For example, applying a stage or scenery to a spatial template/arrangement of virtual objects represents whether a participant viewing/experiencing the virtual object maintains spatial liveness (and therefore whether an avatar corresponding to the participant is displayed), as described similarly above. In some examples, as described in more detail below with reference to FIGS. 6A to 6I , the stage is aligned to the determined spatial template/arrangement of the virtual object. For example, the spatial template/arrangement defined by the participant spatial state determiner 468 may represent a specific location within the stage where an avatar is displayed. In some examples, as described herein, a stage may provide that a given virtual object associated with one or more auxiliary applications 470 may be displayed in any of an experience-first display mode representing a non-exclusive stage or set (e.g., such as displaying content at a predefined position within the stage) and an exclusive display mode representing an exclusive stage or set (e.g., such as displaying content at a position offset from a predefined position), as described in more detail below with reference to Figures 6A to 6I.

Additionally, in some examples, the display stage data 477 includes information corresponding to a stage offset value for a given virtual object associated with one or more auxiliary applications 470. For example, as described in more detail below with reference to FIGS. 6A-6I , the location at which a virtual object is displayed in a shared three-dimensional environment may differ from a predetermined placement location of the virtual object within the stage (e.g., based on the stage offset value). In some examples, one or more auxiliary applications 470 utilize the display stage data 477 as context for generating the application data 471 and/or spatial state request data 465 discussed above. Specifically, as discussed by way of example in FIGS. 6A-6I , transmitting the display stage data 477 to one or more auxiliary applications 470 provides one or more auxiliary applications 470 with information regarding whether one or more users in a multi-user communication session are viewing content in an exclusive display mode (e.g., which determines where and/or how content is displayed in a three-dimensional environment relative to a particular spatial template/arrangement) and/or whether spatial live is enabled in the multi-user communication session (e.g., whether an avatar is displayed).

In some examples, as shown in FIG. 4 , the spatial coordinator API 462 transmits the display mode update data 475 to one or more auxiliary electronic devices (e.g., to a communication application 488 running locally on one or more auxiliary electronic devices) (e.g., optionally via the scene integration service 466 or directly via the display mode determiner 472). In some examples, as described in more detail with reference to FIGS. 6A to 6I , electronic devices communicatively linked in a multi-user communication session may implement an “auto-follow” behavior to maintain users in the multi-user communication session within the same spatial state (and thus maintain spatial liveness within the multi-user communication session). In some such examples, the display mode update data 475 may be used as a command or other instruction to cause the one or more auxiliary electronic devices to automatically follow the electronic device 401 if the electronic device 401 enters an exclusive display mode in the multi-user communication session (e.g., according to the display mode parameters discussed above).

In some examples, as shown in FIG. 4 , application space state data 463, user space state data 467, and display mode data 469 may be received by scene integration service 466 of spatial coordinator API 462. In some examples, scene integration service 466 generates display data 487 based on the one or more display parameters included in application space state data 463, user space state data 467, and/or display mode data 469 discussed above. In some examples, display data 487 generated by scene integration service 466 includes commands/instructions for displaying one or more virtual objects and/or avatars in a shared three-dimensional environment within a multi-user communication session. For example, display data 487 includes information about the appearance of virtual objects displayed in the shared three-dimensional environment (e.g., generated based on application data 471), the location of virtual objects displayed in the shared three-dimensional environment, the location of avatars (or two-dimensional representations of users) displayed in the shared three-dimensional environment, and/or other features/characteristics of the shared three-dimensional environment. In some examples, display data 487 is transmitted from communication application 488 to an operating system of electronic device 401 for display at a display in communication with electronic device 401 , as similarly shown in FIG. 3 .

The communication application 488 is not limited to the components and configurations of FIG. 4 , but may include fewer components, other components, or additional components in a variety of configurations. In addition, the above process is exemplary, and it should be understood that more, fewer, or different operations may be performed using the above components and/or using fewer, other, or additional components in a variety of configurations. Attention is now directed to an exemplary interaction of the above operations of the communication application 488 within a multi-user communication session.

Figures 5A to 5F illustrate exemplary interactions within a multi-user communication session according to some examples of the present disclosure. In some examples, when a first electronic device 560 is in a multi-user communication session with a second electronic device 570 (and a third electronic device (not shown for ease of illustration)), a three-dimensional environment 550A is presented using the first electronic device 560, and a three-dimensional environment 550B is presented using the second electronic device 570. In some examples, electronic devices 560/570 optionally correspond to the electronic devices 360/370 discussed above. In some examples, three-dimensional environment 550A/550B includes a captured portion of the physical environment in which electronic device 560/570 is located. For example, three-dimensional environment 550A includes a table (e.g., representation 506' of the table) and a window (e.g., representation 509' of the window), and three-dimensional environment 550B includes a coffee table (e.g., representation 508' of the coffee table) and a floor lamp (e.g., representation 507' of the floor lamp). In some examples, the three-dimensional environment 550A/550B optionally corresponds to the three-dimensional environment 350A/350B described above with reference to FIG3. As described above, the three-dimensional environment also includes avatars 517/515/519 corresponding to users of the first electronic device 360, the second electronic device 370, and the third electronic device, respectively. In some examples, the avatars 515/517 optionally correspond to the avatars 315/317 described above with reference to FIG3.

As similarly described above with reference to FIG. 3 , users of first electronic device 560, second electronic device 570, and third electronic device may share a spatial state 540 (e.g., a baseline spatial state) within a multi-user communication session (e.g., represented by the placement of ellipses 515A, 517A, and 519A within the circle representing spatial state 540 in FIG. 5A ). In some examples, spatial state 540 optionally corresponds to spatial state 340 discussed above with reference to FIG. 3 . As similarly described above, when a user of the first electronic device 560, a user of the second electronic device 570, and a user of the third electronic device are in a spatial state 540 within a multi-user communication session, the users have a first (e.g., predefined) spatial arrangement in a shared three-dimensional environment (e.g., represented by the positions of ellipses 515A, 517A, and 519A in the circle representing the spatial state 540 in FIG. 5A and/or the distances between the ellipses), such that the first electronic device 560, the second electronic device 570, and the third electronic device (not shown) maintain a consistent spatial relationship (e.g., spatial reality) between the positions of the user's viewpoints (e.g., which correspond to the positions of the ellipses 515A, 517A, and 519A in the circle representing the spatial state 540) and the virtual content at each electronic device.

As shown in FIG5A , the first electronic device 560 optionally displays an application window 530 associated with a corresponding application (e.g., an application that can be configured to display content in the three-dimensional environment 550A, such as a video player application) running on the first electronic device 560. For example, as shown in FIG5A , the application window 530 optionally displays video content (e.g., corresponding to a movie, a television series, or other video clip) visible to a user of the first electronic device 560. In some examples, the application window 530 is displayed with a grabber bar indication 535 (e.g., a control bar) that can be selected to initiate movement of the application window 530 within the three-dimensional environment 550A. In addition, as shown in FIG5A , the application window may include a playback control 556 that can be selected to control playback of the video content displayed in the application window 530 (e.g., rewind the video content, pause the video content, fast forward through the video content, etc.).

In some examples, the application window 530 can be a shared virtual object in a shared three-dimensional environment. For example, as shown in FIG. 5A , the application window 530 can also be displayed in a three-dimensional environment 550B at the second electronic device 570. As shown in FIG. 5A , the application window 530 can be displayed together with the grabber bar indicator 535 and the playback control 556 discussed above. In some examples, because the application window 530 is a shared virtual object, the application window 530 (and the video content of the application window 530) can also be visible to the user of the third electronic device (not shown). As previously discussed above, in FIG. 5A , the user of the first electronic device 560, the user of the second electronic device 570, and the user of the third electronic device (not shown) can share a spatial state (e.g., a baseline spatial state) 540 within a multi-user communication session. Therefore, as shown in FIG. 5A , when the first spatial state 540 is shared, the user (e.g., represented by ellipses 515A, 519A, and 517A) maintains a spatial live with the application window 530 (represented by a line in a circle representing the spatial state 540) within the shared three-dimensional environment.

As previously discussed with reference to FIG. 3 , an object displayed in a shared three-dimensional environment may have an orientation that defines the type of object. For example, an object may be a vertically oriented object (e.g., an object of the first type) or a horizontally oriented object (e.g., an object of the second type). As shown in FIG. 5A , the application window 530 is optionally a vertically oriented object in the three-dimensional environment 550A/550B (e.g., relative to the viewpoints of the user of the first electronic device 560, the user of the second electronic device 570, and the user of the third electronic device). As discussed above with reference to FIG. 4 , the application window 530 is displayed in the three-dimensional environment 550A/550B in a spatial state (e.g., a default spatial state) based on the object type (e.g., object orientation) of the application window 530 (e.g., determined by the application spatial state determiner 464). Alternatively, as discussed above, in some examples, the application window 530 is displayed in the three-dimensional environment 550A/550B in a spatial state corresponding to a selected (e.g., specified) spatial state (e.g., a spatial state that is not necessarily based on the object type but is marked as a preferred spatial state by the application associated with the application window 530). As shown in Fig. 5A, in some examples, because application window 530 is a vertically oriented object in three-dimensional environment 550A/550B, the avatars are arranged in a first spatial arrangement/template relative to application window 530. For example, as shown, the avatars are arranged in a side-by-side spatial arrangement/template, as reflected in first spatial state 540, such that at first electronic device 560, avatar 515 and avatar 519 are located near/next to (e.g., on the left side) the viewpoint of the user of first electronic device 560, and at second electronic device 570, avatar 519 and avatar 517 are located near/next to (e.g., on the right side) the viewpoint of the user of second electronic device 570.

In addition, in some examples, as shown in FIG. 5A , when the avatars are arranged according to the first spatial template discussed above, adjacent avatars may be separated by a first distance 557A (e.g., measured from the center of one avatar to the center of an adjacent avatar or corresponding to the gap between adjacent avatars). For example, as shown in FIG. 5A , avatar 515 represented by ellipse 515A is separated from avatar 519 represented by ellipse 519A by a first spatial separation corresponding to first distance 557A. In addition, in some examples, as shown in FIG. 5A , the avatars may be separated from application window 530 by a second distance 559A (e.g., measured from each avatar to the center of application window 530). In some examples, first distance 557A is different from (e.g., less than) second distance 559A. As described above with reference to FIG. 4 , the separation spacing in the first spatial template (e.g., the values of first distance 557A and/or second distance 559A) is automatically determined by first electronic device 560 and second electronic device 570 (determined together or determined separately (e.g., automatically selected by communication application 488 in FIG. 4 running on the electronic devices)). Alternatively, as described above with reference to FIG. 4 , the separation spacing in the first space template is selected by an application associated with application window 530 (eg, via space state request data 465 in FIG. 4 ).

Thus, if a horizontally oriented shared object is displayed in the three-dimensional environment 550A/550B, the avatar may be arranged relative to the object in a second spatial arrangement/template that is different from the first spatial arrangement/template discussed above. For example, as shown in FIG. 5B , a shared virtual tray 555 having a horizontal orientation may be displayed in the shared three-dimensional environment. In some examples, as shown in FIG. 5B , the virtual tray 555 may be displayed with a virtual cup 552 (e.g., disposed on top of the virtual tray 555) and a gripper bar enable representation 535 that may be selected to initiate movement of the virtual tray 555 in the three-dimensional environment 550A/550B. In some examples, as shown in FIG. 5B , when an object of a second type (e.g., a horizontally oriented object) is displayed in the shared three-dimensional environment, the avatar is displayed relative to the object in a second spatial arrangement/template. For example, as shown in FIG. 5B , because virtual tray 555 is a horizontally oriented object (e.g., and optionally a volumetric object), the avatars are arranged in a circular arrangement relative to virtual tray 555 as indicated in spatial state 540, such that at first electronic device 560, from the viewpoint of the user of first electronic device 560, avatar 515 is located on the left side of virtual tray 555 and avatar 519 is located on the right side of virtual tray 555, and at second electronic device 570, from the viewpoint of the user of second electronic device 570, avatar 519 is located behind virtual tray 555 and avatar 517 is located on the right side of virtual tray 555. In addition, in some examples, as shown in FIG. 5B , when the avatars are arranged in the second spatial template, adjacent avatars may be separated by a third distance 557B. For example, as shown in FIG. 5B , avatar 515 represented by ellipse 515A is separated from avatar 517 represented by ellipse 517A by a second spatial separation corresponding to third distance 557B. Additionally, in some examples, as shown in FIG5B , the avatar may be separated from the virtual tray 555 by a fourth distance 559B. In some examples, the third distance 557B is different from (e.g., less than) or equal to the fourth distance 559B. Additionally, in some examples, the spatial separation provided in the first spatial template discussed above with reference to FIG5A may be different from the spatial separation provided in the second spatial template shown in FIG5B (e.g., due to differences in object types and/or differences in applications).

In some examples, referring back to FIG. 4 , the space coordinator API 462 determines a space template/arrangement for virtual objects in a shared three-dimensional environment based on the space state request data 465 received from one or more auxiliary applications 470. In some examples, the space state request data 465 includes information corresponding to the requested space template/arrangement, and optionally a change in the application state of one or more auxiliary applications 470. For example, as described above and shown in FIG. 5B , the virtual tray 555 may include a virtual cup 552 located at the top of the virtual tray 555. In some examples, the corresponding application associated with the virtual tray 555 may change state (e.g., automatically or in response to user input) so that the display of the virtual tray 555 changes in the three-dimensional environment 550A/550B as shown in FIG. 5C . For example, as shown in FIG. 5C , the first electronic device 560 and the second electronic device 570 transition from displaying the virtual cup 552 at the top of the virtual tray 555 to displaying a representation of the virtual cup 552 (e.g., an enlarged two-dimensional representation) in the window 532 in the three-dimensional environment 550A/550B. In some examples, as shown in Figure 5C, when the display state of the corresponding application changes, the spatial arrangement of the avatar relative to the virtual object optionally also changes. For example, in Figure 5C, the display of the virtual cup 552 within the window 532 causes a change in the object type in the three-dimensional environment 550A/550B (e.g., from a horizontal orientation to a vertical orientation), which causes the spatial arrangement/template to also change, so that the avatars 515/519/517 in the spatial state 540 in Figure 5B change from being in a circular spatial arrangement to being in a side-by-side spatial arrangement as shown in Figure 5C.

In some examples, the spatial arrangement of the avatars 515/519/517 may not necessarily be based on the object type of the virtual object displayed in the shared three-dimensional environment. For example, as discussed above, when a vertically oriented object such as an application window 530 is displayed in the three-dimensional environment 550A/550B, the avatars 515/519/517 may be displayed in a side-by-side spatial arrangement, and when a horizontally oriented object such as a virtual tray 555 is displayed, the avatars 515/519/517 may be displayed in a circular spatial arrangement. However, in some instances, the corresponding application may request (e.g., via the spatial state request data 465 in FIG. 4) a spatial arrangement for the object that is different from the specification discussed above. For example, as shown in FIG. 5D, in some examples, although a horizontally oriented object (e.g., a virtual tray 555) is displayed in the three-dimensional environment 550A/550B, the avatars 515/519/517 are arranged in a side-by-side spatial arrangement/template relative to the horizontally oriented object. Thus, as discussed above, the application spatial state determiner 464 of Figure 4 may define spatial state parameters for a virtual object based on a requested spatial template/arrangement (provided via spatial state request data 465), which spatial state parameters control an avatar (e.g., avatar 515/519/517) relative to the spatial template/arrangement of the virtual object.

In some examples, when the avatars 515/519/517 are arranged in the side-by-side spatial arrangement shown in FIG. 5D (e.g., or other spatial arrangements, such as the circular spatial arrangement in FIG. 5B ) relative to the virtual tray 555 (e.g., the horizontally oriented object), the center point of the virtual tray 555 (e.g., or other horizontally oriented object) may be positioned at (e.g., aligned to) the center position in the spatial state 540. For example, as indicated in the spatial state 540 in FIG. 5D , the center position in the spatial state 540 is indicated by circle 551. However, in some examples, as shown in FIG. 5D , the virtual tray 555 may be positioned at a front-facing surface or side of the virtual tray 555 relative to the center position represented by circle 551, rather than at the center of the virtual tray 555. For example, as previously shown in FIG5B , when the virtual tray 555 is displayed in the shared three-dimensional environment and the avatars 515/519/517 are arranged in a circular spatial arrangement, the virtual tray 555 is aligned/anchored to a center position represented by circle 551 at the center of the virtual tray 555 (e.g., a center point in the horizontal body of the virtual tray 555). As shown in FIG5D , in some examples, when the virtual tray 555 is presented in the shared three-dimensional environment and the avatars 515/519/517 are arranged in a side-by-side spatial arrangement as discussed above, the virtual tray 555 (e.g., or other horizontally oriented object) is aligned to a center position represented by circle 551 at a forward-facing side of the virtual tray 555, as shown in the spatial state 540 in FIG5D , so that the forward-facing surface of the virtual tray 555 is centered in front of the viewpoint of the user of the third electronic device (not shown) corresponding to the avatar 519. For example, one advantage of anchoring the virtual tray 555 (e.g., or other horizontally oriented object) to the center position of the spatial state 540 at the forward-facing side of the virtual tray 555 is to avoid presenting the virtual tray 555 in a manner such that the forward-facing side of the virtual tray 555 intersects the viewpoint of the user and/or avatar 515/519/517 (e.g., when the size of the virtual tray 555 is large enough to traverse the distance between the center of the template and the location of the avatar). Instead, as discussed above, the forward-facing side of the virtual tray 555 is positioned at the center position represented by the circle 551 so that the virtual tray 555 visually appears to extend backward in space in the shared three-dimensional environment (e.g., rather than visually appearing to extend forward in space toward the user's viewpoint). It should be noted that in some examples, anchoring the virtual tray 555 (or more generally, a horizontally oriented object) to a center position in the spatial state 540 at the forward-facing side of the virtual tray 555 is optionally applied only to a side-by-side spatial arrangement of avatars 515/519/517 (e.g., rather than the circular spatial arrangement discussed above and shown in FIG. 5B ).

In some examples, while side-by-side spatial arrangements are allowed for vertically oriented objects and horizontally oriented objects in the three-dimensional environment 550A/550B within a multi-user communication session, this may not necessarily be the case for circular spatial arrangements. For example, a communication application (e.g., 488 in FIG. 4 ) that facilitates a multi-user communication session may limit/prevent the use of circular spatial arrangements for vertically oriented objects, such as application window 530. Specifically, circular spatial arrangements of avatars 515/517/519 may be prevented when displaying vertically oriented objects because the vertically oriented objects are optionally two-dimensional objects (e.g., planar objects) with content displayed on a forward-facing surface of the vertically oriented objects. Enabling circular spatial arrangements in such instances may cause the viewpoint of one or more users in the multi-user communication session to be positioned (e.g., behind a vertically oriented object) in such a manner that content displayed in the forward-facing surface of the vertically oriented object is obscured or completely out of view, which would degrade the user experience. In some such examples, referring to FIG. 4 , if display of vertically oriented objects is initiated (e.g., via application data 471) and one or more auxiliary applications 470 transmit a request to display the vertically oriented objects in a circular spatial arrangement (e.g., via spatial state request data 465), the spatial coordinator API 462 overrides the request to display the vertically oriented objects in a circular spatial arrangement (e.g., via application spatial state determiner 464) and causes the vertically oriented objects to be displayed in the side-by-side spatial arrangement discussed above.

As previously described herein and as shown in Figures 5A to 5E, in some examples, when a user within a multi-user communication session is experiencing spatial live (e.g., because spatial live is enabled), an avatar corresponding to the user is displayed in a shared three-dimensional environment. For example, as shown in Figure 5E and as previously discussed, because spatial live is enabled in a multi-user communication session, when a shared application window 530 is displayed in a three-dimensional environment 550A/550B, an avatar 515/519/517 corresponding to the user participating in the multi-user communication session is displayed in the three-dimensional environment 550A/550B. Referring back to Figure 4, as previously discussed, the determination of whether spatial live is enabled in a multi-user communication session is performed by the participant spatial state determiner 468 of the spatial coordinator API 462. In some examples, the participant spatial state determiner 468 determines whether spatial live is enabled based on the number of participants in the multi-user communication session. In some examples, if the number of participants is within a threshold number of participants (such as 3, 4, 5, 6, or 8 participants), spatial live is enabled, and if the number of participants is greater than the threshold number of participants, spatial live is not enabled. As shown in Figure 5E, there are currently three participants in the multi-user communication session, which is within the threshold number discussed above. Therefore, in Figure 5E, spatial live is enabled in the multi-user communication session, and avatars 515/519/517 are displayed in a shared three-dimensional environment, as shown.

In some examples, if the number of participants within a multi-user communication session increases to a number greater than the threshold number of participants discussed above, spatial live is optionally disabled in the multi-user communication session. For example, in FIG. 5F, three additional users have joined the multi-user communication session (e.g., three additional electronic devices communicate with the first electronic device 560, the second electronic device 570, and the third electronic device), so that there are now a total of six participants, as indicated in the spatial state 540. In some examples, because the number of participants is greater than the threshold number discussed above, the communication application that facilitates the multi-user communication session disables spatial live for the multi-user communication session. Specifically, with reference to FIG. 4, the participant spatial state determiner 468 determines that when three additional users (e.g., represented by ellipses 541, 543, and 545 in FIG. 5F) join the multi-user communication session (optionally communicated via user input data 481B), the total number of participants exceeds the threshold number of participants, and spatial live is therefore disabled (e.g., communicated via user spatial state data 467). Therefore, because spatial live is disabled in the multi-user communication session, the avatars corresponding to the users in the multi-user communication session are no longer displayed in the shared three-dimensional environment 550A/550B. For example, as shown in FIG. 5F , the first electronic device 560 stops displaying the avatars 515/519 and displays a canvas 525 that includes representations (e.g., two-dimensional images, video streams, or other graphics) of the users in the multi-communication session (e.g., users other than the users of the first electronic device 560), including a representation 515A of the user of the second electronic device 570, a representation 519A of the user of the third electronic device, and representations 541A/543A/545A of the additional users. Similarly, as shown in FIG. 5F , the second electronic device 570 optionally stops displaying the avatars 517/519 and displays a canvas 525 that includes representations of the users in the multi-communication session (e.g., users other than the users of the second electronic device 570). In some examples, when spatial live is disabled in the multi-user communication session in FIG5F , first electronic device 560 presents audio of other users in the multi-user communication session (e.g., voice or other audio detected via microphones of the users' respective electronic devices), as indicated by audio bubble 514, and second electronic device 570 presents audio of users among the other users in the multi-communication session, as indicated by audio bubble 512. In some examples, the audio of the users of the electronic devices may be spatialized, presented in mono, or presented in stereo.

In addition, in FIG. 5F , when spatial live is disabled in the multi-user communication session, the three-dimensional environment 550A/550B is no longer a real shared environment. For example, referring back to FIG. 4 , when the participant spatial state determiner 468 defines spatial live as disabled (e.g., via the user spatial state data 467 ), the spatial coordinator API 462 no longer displays the application window 530 according to the spatial arrangement/template defined by the application spatial state determiner 464. Therefore, as shown in FIG. 5F , the application window 530 is optionally no longer displayed in the two three-dimensional environments 550A/550B, so that the application window 530 is no longer a shared experience within the multi-user communication session. In some such examples, as shown in FIG. 5F , the application window 530 is redisplayed as a window private to the user of the first electronic device 560 (e.g., because the user of the first electronic device 560 optionally initially launched and shared the application window 530). 3, at the second electronic device 570, the three-dimensional environment 550B includes a representation of the application window 530" that no longer includes the content of the application window 530 (eg, does not include the video content discussed previously).

Thus, as described above, providing an API (e.g., spatial coordinator API 462 of FIG. 4 ) that facilitates communication between a communication application and one or more corresponding applications enables virtual objects (e.g., such as application window 530 or virtual tray 555) to be displayed in a shared three-dimensional environment in a manner that complies with spatial real-time rules and enables the virtual objects to be clearly displayed to all users in a multi-user communication session, which is an advantage. Now focus on further exemplary interactions within a multi-user communication session.

Figures 6A to 6L illustrate exemplary interactions within a multi-user communication session according to some examples of the present disclosure. In some examples, a first electronic device 660 and a second electronic device 670 may be communicatively linked in a multi-user communication session, as shown in Figure 6A. In some examples, when the first electronic device 660 is in a multi-user communication session with the second electronic device 670, the first electronic device 660 is used to present a three-dimensional environment 650A, and the second electronic device 670 is used to present a three-dimensional environment 650B. In some examples, the electronic device 660/670 optionally corresponds to the electronic device 560/570 discussed above and/or the electronic device 360/370 in Figure 3. In some examples, the three-dimensional environment 650A/650B includes a captured portion of the physical environment in which the electronic device 660/670 is located. For example, the three-dimensional environment 650A includes a window (e.g., a representation 609' of the window), and the three-dimensional environment 650B includes a coffee table (e.g., a representation 608' of the coffee table) and a floor lamp (e.g., a representation 607' of the floor lamp). In some examples, the three-dimensional environment 650A/650B optionally corresponds to the three-dimensional environment 550A/550B described above and/or the three-dimensional environment 350A/350B in Figure 3. As described above, the three-dimensional environment also includes an avatar 615/617 corresponding to the user of the electronic device 670/660. In some examples, the avatar 615/617 optionally corresponds to the avatar 515/517 described above and/or the avatar 315/317 in Figure 3.

As shown in FIG. 6A , the first electronic device 660 and the second electronic device 670 optionally display a user interface object 636 associated with a corresponding application (e.g., an application that can be configured to display content corresponding to a game (“Game A”) in a three-dimensional environment 650A/650B, such as a video game application) running on the electronic device 660/670. In some examples, as shown in FIG. 6A , the user interface object 636 may include a selectable option 623A that can be selected to initiate display of shared exclusive content (e.g., immersive interactive content) associated with Game A. In some examples, as shown in FIG. 6A , the user interface object 636 is shared between a user of the first electronic device 660 and a user of the second electronic device 670. In addition, as shown in FIG. 6A , the second electronic device 670 is displaying a virtual object 633 including a user interface A, which is private to the user of the second electronic device 670, as previously discussed herein. For example, only the user of the second electronic device 670 can view and/or interact with the user interface of the virtual object 633. Thus, as similarly described above, the three-dimensional environment 650A displayed at the first electronic device 660 includes a representation of the virtual object 633 that does not include a user interface (e.g., user interface A) for the virtual object 633 displayed at the second electronic device 670. Additionally, as shown in FIG6A , the virtual object 633 is optionally displayed with a gripper bar enable representation 635 that can be selected to initiate movement of the virtual object 633 within the three-dimensional environment 650B.

As previously discussed herein, in FIG. 6A , a user of a first electronic device 660 and a user of a second electronic device 670 may share the same spatial state (e.g., a baseline spatial state) 640 within a multi-user communication session. In some examples, the first spatial state 640 optionally corresponds to the spatial state 540 discussed above and/or the spatial state 340 discussed above with reference to FIG. 3 . As similarly described above, when the user of the first electronic device 660, the user of the second electronic device 670, and the user of a third electronic device (not shown) share the first spatial state 640 within the multi-user communication session, the users experience the spatial reality (e.g., represented by the position of ellipses 615A and 617A in the circle representing the spatial state 640 in FIG. 6A and/or the distance between the ellipses) in the shared three-dimensional environment, so that the first electronic device 660, the second electronic device 670, and the third electronic device (not shown) maintain a consistent spatial relationship between the position of the user's viewpoint (e.g., which corresponds to the position of the avatar 617/615 in the three-dimensional environment 650A/650B) and the virtual content (e.g., virtual object 633) at each electronic device.

6A , when the virtual object 633 is displayed in the three-dimensional environment 650B, the second electronic device 670 detects a selection input 672A directed to the selectable option 623A. For example, when the gaze of the user of the second electronic device 670 is directed to the selectable option 623A, the second electronic device 670 detects a pinch input (e.g., a pinch input in which the index finger and thumb of the user's hand make contact), a tap or touch input (e.g., provided by the index finger of the hand), a verbal command, or some other direct or indirect input.

In some examples, as shown in FIG6B , in response to detecting a selection of selectable option 623A, the second electronic device 670 initiates a process for displaying shared content (e.g., a shared immersive experience) in a shared three-dimensional environment within the multi-user communication session. In some examples, as shown in FIG6B , initiating a process for displaying shared content includes transitioning to an exclusive display of the three-dimensional environment 650B. For example, in response to detecting a selection of selectable option 623A to cause exclusive content display, the second electronic device 670 stops display of all other content in the three-dimensional environment 650B, such as the virtual object 633 in FIG6A .

In some examples, as shown in FIG6B , electronic devices in a multi-user communication session may implement an “auto-follow” behavior to maintain the users in the multi-user communication session in the same spatial state (and thus maintain spatial reality within the multi-user communication session). For example, when a user of the second electronic device 670 selects the selectable option 623A to display exclusive content associated with game A in FIG6A , the second electronic device 670 may transmit (e.g., directly or indirectly) to the first electronic device 660 one or more commands for causing the first electronic device 660 to automatically follow the second electronic device 670. Specifically, referring to FIG4 , the selection of the selectable option 623A to cause the second electronic device 670 to transition to an exclusive display mode is optionally transmitted to the spatial coordinator API 462 in the form of input data 483, and the one or more commands transmitted to the first electronic device 660 are optionally in the form of display mode update data 475, similar to what was previously discussed above. As shown in FIG6B , in response to receiving the one or more commands transmitted by the second electronic device 670, the first electronic device 660 also transitions to an exclusive display mode and stops displaying other content (e.g., such as representation 633" in FIG6A ). Thus, the user of the first electronic device 660 and the user of the second electronic device 670 remain associated with the same spatial state and spatial live remains enabled within the multi-user communication session.

In some examples, the content of the video game application discussed above being displayed in the shared three-dimensional environment may be associated with a spatial template/arrangement. For example, as previously discussed herein, with reference to FIG. 4, the application spatial state determiner 464 of the spatial coordinator API 462 defines application spatial state parameters, which define the spatial template for the content associated with the video game application to be displayed. Therefore, in some examples, as shown in FIG. 6B, when the first electronic device 660 and the second electronic device 670 enter the exclusive display mode, the position of the avatar 615/617 is rearranged/shifted in the shared three-dimensional environment according to the determined spatial template/arrangement, as similar to that discussed above. In addition, because the content associated with the video game application is exclusive content (e.g., such as immersive interactive content), the stage 648 is associated with the determined spatial template/arrangement of the content in the shared three-dimensional environment for the multi-user communication session. In some examples, as shown in FIG. 6B, the stage 648 is aligned with the spatial arrangement/template defined for the avatar 615/617 (e.g., according to the application spatial state parameters previously described above). 6B , the defined spatial arrangement/template assigns positions or “seats” for display avatars 615/617 within stage 648 (and where the viewpoints of users of electronic devices 660/670 are spatially located within the multi-user communication session). In some examples, as similarly discussed above, stage 648 can be an exclusive stage such that display of shared content associated with game A is visible and interactable only to those users sharing the same spatial state (e.g., first spatial state 640).

In some examples, as previously mentioned herein, the display of shared content within stage 648 within a multi-user communication session may be user-first or experience-first. In some examples, the experience-first display of shared content within stage 648 will cause the shared content to be displayed at a predefined location within stage 648, such as at the center of stage 648 (e.g., and/or at the average location of the users' seats within stage 648). In some examples, as shown in FIG6C , the user-first display of shared content within the stage causes the shared content to be displayed at a location offset from a predefined location 649 (e.g., the center) of stage 648. In some examples, the user-first display of shared content enables a separate version of the shared content (e.g., a separate user interface or object) to be displayed to each user within the multi-user communication session, rather than a single shared content visible to all users within the multi-user communication session. For example, as shown in FIG6C , a first placement location 651-1 is determined for a user of second electronic device 670 (e.g., in front of avatar 615) and a second placement location 651-2 is determined for a user of first electronic device 660 (e.g., in front of avatar 617) within stage 648. In some examples, referring back to FIG4 , the stage offset values from which first placement location 651-1 and second placement location 651-2 are calculated are based on display stage data 477 discussed previously.

In some examples, as shown in Figure 6D, displaying shared content associated with the video game application discussed above includes displaying a game user interface including a plurality of interactive objects 655. As described above and shown in Figure 6D, the plurality of interactive objects 655 are optionally displayed at a first placement location 651-1 and a second placement location 651-2 within the stage 648. In some examples, as shown in Figure 6D, because the display of the plurality of interactive objects 655 is user-first, as discussed above, a separate version of the plurality of interactive objects 655 is provided to each user in the multi-user communication session.

Alternatively, as described above, in some examples, when in an exclusive display mode, the shared content may be displayed as experience-first. For example, in FIG. 6E , the shared three-dimensional environment may alternatively include an application window 630 that is shared between a user of a first electronic device 660, a user of a second electronic device 670, and a user of a third electronic device (not shown) that are communicatively linked within a multi-user communication session. In some examples, as shown in FIG. 6E , the application window 630 optionally displays video content (e.g., corresponding to a movie, a TV series, or other video clip) visible to a user of the first electronic device 660, a user of the second electronic device 670, and a user of the third electronic device. In some examples, the application window 630 is displayed with a grabber bar indication 635 (e.g., a manipulation bar), which may be selected to initiate movement of the application window 630 within the three-dimensional environment 650A/650B. Additionally, as shown in FIG. 6E , the application window may include playback controls 656 that may be selected to control playback of the video content displayed in the application window 630 (eg, rewind the video content, pause the video content, fast forward through the video content, etc.).

As previously discussed herein, in FIG6E , users of the first electronic device 660, the second electronic device 670, and the third electronic device (not shown) may share the same first spatial state (e.g., baseline spatial state) 640 within a multi-user communication session. As similarly described above, when users of the first electronic device 660, the second electronic device 670, and the third electronic device (not shown) share the first spatial state 640 within the multi-user communication session, the users experience a spatial reality (e.g., represented by the positions of ellipses 615A, 617A, and 619A within the circle representing the spatial state 640 in FIG6E and/or the distances between the ellipses) in a shared three-dimensional environment, such that the first electronic device 660, the second electronic device 670, and the third electronic device (not shown) maintain a consistent spatial relationship between the position of the user's viewpoint (e.g., which corresponds to the position of the avatar 617/615/619 within the circle representing the spatial state 640) and the virtual content (e.g., application window 630) at each electronic device. As shown in FIG. 6E , when in the first spatial state 640 , the users (eg, represented by their avatars 615 , 619 , and 617 ) are positioned side by side with the front-facing surface of the application window 630 facing toward the users.

In some examples, the video content of the application window 630 is displayed in a window mode in a shared three-dimensional environment. For example, as shown in FIG. 6E, the video content displayed in the three-dimensional environment is constrained/restricted by the size of the application window 630. In some examples, the video content of the application window 630 may alternatively be displayed in a full-screen mode in the three-dimensional environment. As used herein, the display of video content in a "full-screen mode" in a three-dimensional environment 650A/650B optionally refers to the display of video content in a corresponding size and/or with corresponding visual emphasis in the three-dimensional environment 650A/650B. For example, the electronic device 660/670 may display the video content in a three-dimensional environment 650A/650B with a size larger than the size of the application window 630 displaying the video content (e.g., 1.2x, 1.4x, 1.5x, 2x, 2.5x, or 3x). In addition, for example, the video content may be displayed with a visual emphasis larger than the representation of other virtual objects and/or physical objects displayed in the three-dimensional environment 650A/650B. As described in more detail below, when the video content is displayed in full screen mode, the captured portion of the physical environment surrounding the electronic device may become faded and/or dimmed in the three-dimensional environment. As shown in Figure 6E, the application window 630 in the three-dimensional environment 650A/650B may include a selectable option 626 that can be selected to cause the video content of the application window 630 to be displayed in full screen mode.

6E , the user of the first electronic device 660 is optionally providing a selection input 672B directed to the selectable option 626 in the application window 630. For example, while the gaze of the user of the first electronic device 660 is directed to the selectable option 626, the first electronic device 660 detects a pinch input (e.g., a pinch input in which the user's index finger and thumb make contact), a tap or touch input (e.g., provided by the user's index finger), a verbal command, or some other direct or indirect input.

In some examples, in response to receiving the selection input 672B, the first electronic device 660 displays the video content in full screen mode in the three-dimensional environment 650A, as shown in FIG6F, including converting the display of the application window to an exclusive display mode, as discussed similarly above. For example, as shown in FIG6F, when the first electronic device 660 displays the video content in full screen mode, the first electronic device 660 increases the size of the application window 630 that is displaying the video content. In addition, in some examples, as shown in FIG6F, the electronic device in the multi-user communication session can implement the automatic following behavior discussed above to maintain the users in the multi-user communication session in the same spatial state. For example, when the user of the first electronic device 660 activates the full screen mode in FIG6E, the first electronic device 660 can transmit (e.g., directly or indirectly) one or more commands for causing the second electronic device 670 and the third electronic device to automatically follow the first electronic device 660 to the second electronic device 670 and the third electronic device (not shown). As shown in Figure 6F, in some examples, in response to receiving one or more commands transmitted by the first electronic device 660, the second electronic device 670 and the third electronic device display the video content of the application window 630 in full screen mode, as discussed above.

In addition, in some examples, when the electronic device 660/670 is converted to displaying the video content of the application window 630 in an exclusive full-screen mode, the stage 648 is applied to the side-by-side space template defined for the avatar 615/617/619, as shown in Figure 6F. In some examples, as discussed similarly above, the stage 648 can be an experience-first stage. As shown in Figure 6F, the application window 630 is parked (e.g., positioned) at a predefined position 649 (e.g., a center position) within the stage 648 in the multi-user communication session (e.g., so that when the full-screen mode is active, the application window 630 is no longer movable in the three-dimensional environment 650A/650B). In addition, in some examples, when presenting video content in full-screen mode, the first electronic device 660 and the second electronic device 670 visually weaken the importance of the display of the representation of the captured portion of the physical environment surrounding the first electronic device 660 and the second electronic device 670. For example, as shown in FIG. 6F , the representation of window 609' and the representation of the floor, ceiling, and walls surrounding the first electronic device 660 may be visually weakened in importance (e.g., faded, dimmed, or adjusted to be translucent or transparent) in the three-dimensional environment 650A, and the representation of the floor lamp 607', the representation of the coffee table 608', and the representation of the floor, ceiling, and walls surrounding the second electronic device 670 may be visually weakened in importance in the three-dimensional environment 650B, so that the user's attention is primarily drawn to the video content in the enlarged application window 630.

In some examples, the electronic device within the multi-user communication session may alternatively not implement the automatic following behavior discussed above. For example, specific content displayed in a three-dimensional environment may prevent one or more electronic devices in the electronic device from implementing the automatic following behavior. For example, in Figure 6G, when the first electronic device 660 is transformed to display the video content of the application window 630 in full screen mode in response to the selection input 672B of Figure 6E, the third electronic device automatically follows the first electronic device 660, but the second electronic device 670 does not automatically follow. In some examples, when the third electronic device automatically follows the first electronic device 660, the user of the third electronic device joins the user of the first electronic device 660 to enter the second spatial state 661, as shown in Figure 6G. For example, because both the first electronic device 660 and the third electronic device (not shown) are displaying video content in full screen mode, the first electronic device 660 and the third electronic device operate in the same spatial state 661 within the multi-user communication session, as previously discussed herein. In addition, as shown in FIG6G , the user of the first electronic device 660 (e.g., represented by an ellipse 617A in a circle representing the spatial state 661) and the user of the third electronic device (e.g., represented by an ellipse 619A in a circle representing the spatial state 661) are arranged in a new spatial arrangement/template when in the second spatial state 661. For example, as shown in the circle representing the spatial state 661 in FIG6G , the user of the first electronic device 660 (e.g., represented by an ellipse 617A) and the user of the third electronic device (e.g., represented by an ellipse 619A) are positioned side by side in front of the application window 630 in full screen mode.

As previously mentioned above, the user of the second electronic device 670 optionally does not automatically follow the first electronic device 660 to join in watching the video content in full screen mode. Specifically, in some examples, due to the display of the private window 662 in the three-dimensional environment 650B, the second electronic device 670 does not automatically follow the first electronic device 660. For example, when the user of the first electronic device 660 provides input to display the video content of the application window 630 in full screen mode in Figure 6E, the user of the second electronic device 670 is watching the private window 662, as shown in Figure 6G. In such instances, it may not be desirable to make the second electronic device 670 automatically follow the first electronic device 660, because such operations will cause the private window 662 to stop displaying in the three-dimensional environment 650B (e.g., without user consent). Therefore, in some examples, when the private window 662 remains displayed in the three-dimensional environment 650B, the second electronic device 670 does not automatically follow the first electronic device 660, as previously discussed. In some examples, because the second electronic device 670 does not automatically follow the first electronic device 660, the second electronic device 670 operates in a different state from the first electronic device 660 and the third electronic device, which allows the user of the second electronic device 670 (e.g., represented by the ellipse 615A in the circle representing the spatial state 640) to remain in the first spatial state 640. In addition, as shown in FIG. 6G, the user of the second electronic device is arranged in a new spatial arrangement/template within the first spatial state 640. For example, as shown in FIG. 6G by the circle representing the spatial state 640, the user of the third electronic device is centrally positioned within the first spatial state 640 relative to the application window 630.

As shown in Figure 6G, because the user of the second electronic device 670 is no longer in the same spatial state as the user of the first electronic device 660 and the user of the third electronic device (not shown), the three-dimensional environment 650A/650B is no longer a real shared environment. Therefore, the second electronic device 670 stops displaying the avatar 617 corresponding to the user of the first electronic device 660 and the avatar 619 corresponding to the user of the third electronic device (not shown). In some examples, as shown in Figure 6G, because the user of the first electronic device 660 and the user of the third electronic device share the same second spatial state 661, the avatars 617/619 corresponding to the users of the first electronic device 660 and the third electronic device remain displayed. For example, as shown in Figure 6G, the first electronic device 660 stops displaying the avatar 615 corresponding to the user of the second electronic device 670, but maintains the display of the avatar 619 corresponding to the user of the third electronic device (not shown) in the three-dimensional environment 650A.

In some examples, as shown in FIG. 6G, the second electronic device 670 replaces the display of the avatar 617/619 with a two-dimensional representation corresponding to the user of the other electronic device. For example, as shown in FIG. 6G, the second electronic device 670 displays the first two-dimensional representation 625 and the second two-dimensional representation 627 in the three-dimensional environment 650B. In some examples, as discussed similarly above, the two-dimensional representation 625/627 includes an image, video, or other rendering representing the user of the first electronic device 660 and the user of the third electronic device. Similarly, the first electronic device 660 replaces the display of the avatar 615 corresponding to the user of the second electronic device 670 with a two-dimensional representation corresponding to the user of the second electronic device 670. For example, as shown in FIG. 6G, the first electronic device 660 displays a two-dimensional representation 629, which optionally includes an image, video, or other rendering representing the user of the second electronic device 670. As shown in FIG. 6G, the first electronic device 660 may display the two-dimensional representation 629 in a predetermined area of the display of the first electronic device 660. For example, as shown in FIG. 6G, the first electronic device 660 displays the two-dimensional representation 629 in the top/upper area of the display. In some examples, the second electronic device 670 displays the two-dimensional representations 625/627 corresponding to the users of the first electronic device 660 and the third electronic device relative to the application window 630. For example, as shown in FIG6G , the second electronic device 670 displays the two-dimensional representations 625/627 together with (e.g., adjacent to) the application window 630 in the three-dimensional environment 650B.

As similarly described above, the display of the avatar 615/617/619 in the three-dimensional environment 650A/650B is optionally accompanied by the presentation of an audio effect corresponding to the voice of each of the users of the three electronic devices, which in some examples may be spatialized so that the audio appears to the users of the three electronic devices as emanating from the position of the avatar 615/617/619 in the three-dimensional environment 650A/650B. In some examples, as shown in FIG6G , when the avatar 615 stops being displayed in the three-dimensional environment 650A at the first electronic device 660, the first electronic device 660 maintains the presentation of the audio of the user of the second electronic device 670, as indicated by the audio bubble 616. Similarly, when the avatar 617/619 stops being displayed in the three-dimensional environment 650B at the second electronic device 670, the second electronic device 670 maintains the presentation of the audio of the users of the first electronic device 660 and the third electronic device, as indicated by the audio bubble 612/614. However, in some examples, the audio of the users of the electronic devices may no longer be spatialized and may be presented in mono or stereo instead. Therefore, even though the avatar 617/619 is no longer displayed in the three-dimensional environment 650B and the avatar 615 is no longer displayed in the three-dimensional environment 650A, the users of the three electronic devices may continue to communicate (e.g., verbally) because the first electronic device 660, the second electronic device 670, and the third electronic device (not shown) are still in the multi-user communication session. In other examples, the audio of the users of the electronic devices may be spatialized so that the audio appears to emanate from their respective two-dimensional representations 625/627/629.

As previously mentioned herein, in some examples, although the users of the three electronic devices are associated with independent spatial states within the multi-user communication session, the users experience a spatial reality that is localized based on the spatial state associated with each user. For example, as previously discussed, the display of content in the three-dimensional environment 650A at the first electronic device 660 (and subsequent interaction with the content) can be independent of the display of content in the three-dimensional environment 650B at the second electronic device 670, although the content of the application windows can still be synchronized (e.g., the same portion of video content (e.g., movie or television program content) is played back in application windows across the first electronic device 660 and the second electronic device 670).

In some examples, as shown in FIG6H, the second electronic device 670 may no longer display the private window 662. For example, when the private window 662 is displayed in FIG6G and after the user of the third electronic device joins the user of the first electronic device 660 to watch the video content of the application window 630 in full screen mode, the second electronic device 670 detects an input for closing the private window 662 (e.g., such as the selection of the close option 663 in FIG6G). In some examples, in FIG6H, because the private window 662 is no longer displayed in the three-dimensional environment 650B, there is no longer an obstacle to joining the first electronic device 660 and the third electronic device to watch the video content of the application window 630 in full screen exclusive mode. In some examples, when the second electronic device 670 determines that the private window 662 is no longer displayed in the three-dimensional environment 650B, the second electronic device 670 operates the invitation from the first electronic device 660 to join (e.g., automatically follow) the first electronic device 660 to watch the video content in full screen. In some examples, such actions include displaying an indication of a user input prompting for synchronizing the display of the shared video content.

For example, as shown in FIG6H, when the private window 662 stops being displayed in the three-dimensional environment 650B, the second electronic device 670 displays a notification element 620 corresponding to an invitation to view video content in full screen mode in the three-dimensional environment 650B. For example, as shown in FIG6H, the notification element 620 includes a first option 621 that can be selected to cause the second electronic device 670 to display the video content of the application window 630 in full screen mode, and a second option 622 that can be selected to cause the second electronic device 670 to close the notification element 620 (and continue to display the application window 630, as shown in FIG6H). In some examples, the notification element 620 is displayed in the three-dimensional environment 650B in an alternative manner. For example, the notification element 620 may be displayed on a two-dimensional representation 627 corresponding to the user of the first electronic device 660 and/or may be displayed as a message (e.g., "Join me to view content in full screen") within the two-dimensional representation 627 including selectable options 621 and 622.

6H , the user of the second electronic device 670 is optionally providing a selection input 672C related to the first option 621 in the notification element 634 in the three-dimensional environment 650B. For example, when the gaze of the user of the second electronic device 670 is related to the first option 621, the second electronic device 670 optionally detects a pinch input, a touch or tap input, a verbal command, or some other direct or indirect input.

In some examples, in response to detecting the selection input 672C, the second electronic device 670 optionally presents the video content of the application window 630 in the three-dimensional environment 650B in full screen mode, as shown in FIG6I. For example, as similarly described above, the second electronic device 670 may increase the size of the application window 630 in the three-dimensional environment 650B so that the video content is displayed with greater visual prominence in the three-dimensional environment 650B. In addition, as discussed above, the second electronic device 670 may dock the application window 630 (e.g., position the application window in a fixed position (e.g., a center position)) in the three-dimensional environment 650B (e.g., so that when the full screen mode is active, the application window 630 is no longer movable in the three-dimensional environment 650B). In addition, in some examples, when presenting the video content in full screen mode, the second electronic device 670 may visually weaken the importance of the representation of the captured portion of the physical environment surrounding the second electronic device 670. For example, as shown in FIG6I , the representation 608' of the coffee table, the representation 607' of the floor lamp, and the representations of the floor, ceiling, and walls surrounding the second electronic device 670 may be visually diminished in importance (e.g., faded, dimmed, or made translucent or transparent) in the three-dimensional environment 650B so that attention is primarily drawn to the video content of the application window 630 in full screen mode.

In some examples, instead of displaying a notification (e.g., such as notification element 620) corresponding to an invitation from first electronic device 660 to join in viewing the video content of application window 630 in full screen mode as discussed above with reference to FIG. 6H , second electronic device 670 may automatically follow first electronic device 660 (e.g., without user input) as discussed above with reference to FIG. 6H . For example, in FIG. 6H , when private window 662 is no longer displayed in three-dimensional environment 650B, second electronic device 670 optionally automatically transitions to displaying the video content of application window 630 in full screen mode after a threshold amount of time (e.g., 0.5, 1, 2, 3, 5, 8, 10 minutes, etc. after private window 662 is no longer displayed). In some examples, if second electronic device 670 detects user input, such as launching another private application in three-dimensional environment 650B, before the threshold amount of time that prevents display of virtual content in full screen mode has passed, second electronic device 670 abandons joining first electronic device 660 and second electronic device 670 in entering full screen display mode.

In some examples, as described similarly above, when the second electronic device 670 joins the first electronic device 660 and the third electronic device (not shown) to watch video content in full screen mode (as shown in FIG. 6I), the users of the three electronic devices become associated with the same spatial state within the multi-user communication session again. For example, as shown in FIG. 6I, because the first electronic device 660, the second electronic device 670, and the third electronic device (not shown) are displaying video content in full screen mode, the three electronic devices share the same spatial state within the multi-user communication session, as previously discussed herein. In addition, as shown in FIG. 6I, the user of the first electronic device 660 (e.g., represented by the ellipse 617A in the circle representing the spatial state 661), the user of the second electronic device 670 (e.g., represented by the ellipse 615A in the circle representing the spatial state 661), and the user of the third electronic device (e.g., represented by the ellipse 619A in the circle representing the spatial state 661) are arranged in the second spatial state 661 with a new spatial arrangement/template (e.g., compared to the spatial arrangement/template shown in FIG. 6H). For example, as shown by the circle representing the spatial state 661 in Figure 6I, the user of the first electronic device 660 (e.g., represented by ellipse 617A) and the user of the third electronic device (e.g., represented by ellipse 619A) are shifted to the right when in the second spatial state 661 to illustrate the placement of the user of the second electronic device 670 (e.g., represented by ellipse 615A).

In addition, in some examples, as previously described herein, when the user of the second electronic device 670 joins the user of the first electronic device 660 and the user of the third electronic device (not shown) to enter the second spatial state 661, as shown in FIG6I, the three electronic devices redisplay the avatars 615/617/619 in the three-dimensional environment 650A/650B. For example, as shown in FIG6I, the first electronic device 660 stops display of the two-dimensional representation 629 (e.g., from FIG6H), and redisplays the avatar 615 corresponding to the user of the second electronic device 670 in the three-dimensional environment 650A based on the spatial arrangement/template defined when in the second spatial state 661 (e.g., the avatar 615/619 is displayed to the left of the viewpoint of the user of the first electronic device 660). Similarly, as shown, the second electronic device 670 stops displaying the two-dimensional representation 625/627 (e.g., from FIG. 6H ), and redisplays the avatar 617 corresponding to the user of the first electronic device 660 and the avatar 619 corresponding to the user of the third electronic device in the three-dimensional environment 650B based on the spatial arrangement/template defined when in the second spatial state 661 (e.g., the avatar 617/619 is displayed to the right of the viewpoint of the user of the second electronic device 670). Thus, as an advantage, the disclosed methods and APIs provide a shared and unobstructed viewing experience for multiple users in a communication session that takes into account individual user interactions with shared and private content and individual display states of the users in a three-dimensional environment.

In some examples, when the user of the first electronic device 660, the user of the second electronic device 670, and the user of the third electronic device (not shown) are in the second spatial state 661, as shown in FIG6I, if one of the users provides an input for stopping displaying the video content in full screen mode, the user can be caused to leave the second spatial state 661 (e.g., and no longer watch the video content in full screen mode). For example, as shown in FIG6I, the application window 630 includes an exit option 638, which can be selected to redisplay the video content in the window mode discussed above, and as similarly shown in FIG6E (e.g., and stop displaying the video content of the application window 630 in full screen mode). In FIG6I, when the video content of the application window 630 is displayed in full screen mode in the three-dimensional environment 650A/650B, the first electronic device 660 detects a selection input 672D (e.g., an air pinch gesture, an air tap or touch gesture, a gaze dwell, a verbal command, etc.) pointing to the exit option 638 provided by the user of the first electronic device 660.

In some examples, in response to detecting a selection of the exit option 638, as shown in FIG6J, the first electronic device 660 stops displaying the video content of the application window 630 in full screen mode and redisplays the video content in window mode, as discussed similarly above with reference to FIG6E. In some examples, when the first electronic device 660 stops displaying the video content in full screen mode in response to the selection input 672D, other electronic devices in the multi-user communication session can implement the automatic follow-up behavior discussed above to maintain the users in the multi-user communication session in the same spatial state (e.g., the first spatial state 640). Therefore, in some examples, as shown in FIG6J and as discussed similarly above, when the first electronic device 660 redisplays the video content of the application window 630 in window mode, the second electronic device 670 (and the third electronic device (not shown)) also stops displaying the video content of the application window 630 in full screen mode and redisplays the video content in window mode.

In some examples, when the user of the first electronic device 660, the user of the second electronic device 670, and the user of the third electronic device (not shown) are in the second spatial state 661, one of the users may cause the video content of the application window 630 to no longer be displayed in full screen mode (e.g., temporarily) without causing the other users to no longer view the video content in full screen mode (e.g., without implementing the automatic follow behavior discussed above). For example, if one of the electronic devices detects an input corresponding to a request to view private content at the electronic device, the electronic devices in the multi-user communication session do not implement the automatic follow behavior. In FIG6K, the second electronic device 670 optionally receives an indication of an incoming message (e.g., a text message, a voice message, an email, etc.) associated with a messaging application, which causes the second electronic device 670 to display a message notification 646 in the three-dimensional environment 650B. As shown in FIG6K, when the second electronic device 670 displays the message notification 646 in the three-dimensional environment 650B, the second electronic device 670 does not stop displaying the video content of the application window 630 in full screen mode.

In FIG. 6K , when the message notification 646 is displayed in the three-dimensional environment 650B, the second electronic device 670 detects a selection input 672E (e.g., an air pinch gesture, an air tap or touch gesture, a gaze dwell, a verbal command, etc.) provided by the user of the second electronic device 670 pointing to the message notification 646. Alternatively, in some examples, the second electronic device 670 detects a selection of a button (e.g., a physical button of the second electronic device 670) or other options in the three-dimensional environment 650B for launching a messaging application associated with the message notification 646. In some examples, as shown in FIG. 6L , in response to detecting a selection of the message notification 646 (or similar input), the second electronic device 670 displays a message window 664 associated with the messaging application discussed above. In addition, in some examples, when the second electronic device 670 displays the message window 664 as a private application window, the second electronic device 670 stops displaying the video content of the application window 630 in full screen mode and redisplays the video content in window mode, as discussed similarly above. In some examples, because the second electronic device 670 no longer displays the video content of the application window 630 in full screen mode, the second electronic device 670 operates in a different state from the first electronic device 660 and the third electronic device, which causes the user of the second electronic device 670 (e.g., represented by the ellipse 615A in the circle representing the spatial state 640) to be placed in the first spatial state 640 discussed previously above.

As shown in FIG6L, because the user of the second electronic device 670 is no longer in the same spatial state as the user of the first electronic device 660 and the user of the third electronic device (not shown), the three-dimensional environment 650A/650B is no longer a real shared environment. Therefore, the second electronic device 670 stops displaying the avatar 617 corresponding to the user of the first electronic device 660 and the avatar 619 corresponding to the user of the third electronic device (not shown). In addition, as shown in FIG6L, the first electronic device 660 stops displaying the avatar 615 corresponding to the user of the second electronic device 670, but maintains the display of the avatar 619 corresponding to the user of the third electronic device (not shown) in the three-dimensional environment 650A. In some examples, as shown in FIG6L, the second electronic device 670 replaces the display of the avatar 617/619 with the two-dimensional representation corresponding to the user of other electronic devices. For example, as shown in FIG6L, the second electronic device 670 displays the first two-dimensional representation 625 and the second two-dimensional representation 627 in the three-dimensional environment 650B, as previously discussed above.

In some examples, as shown in FIG6L, when the second electronic device 670 stops displaying the video content of the application window 630 in full screen mode, the first electronic device 660 and the third electronic device (not shown) abandon the automatic follow behavior discussed above. In some examples, the first electronic device 660 and the third electronic device abandon the automatic follow behavior because the launch of the private content (e.g., the message window 664) is not interpreted as an input for actively stopping the display of the video content in full screen mode (e.g., such as the selection of the exit option 638 in FIG6I). Instead, the launch of the private content is interpreted as a temporary separation from the second space state 661 (e.g., a temporary interaction with the private content). Thus, in Figure 6L, if the user of the second electronic device 670 provides input for stopping display of the message window 664, such as via selection of the close option 663, or if the user of the second electronic device 670 provides a selection of option 621 in the notification element 620, the second electronic device 670 will redisplay the video content of the application window 630 in full screen mode, so that the user of the second electronic device 670 is again in the same spatial state (e.g., second spatial state 661) as the user of the first electronic device 660 and the user of the third electronic device, as similarly shown in Figure 6K.

It should be understood that the above examples of exclusive display of video content in full screen mode are similarly applicable to other exclusive immersive experiences. For example, the above interactions are applicable to immersive environments, such as a virtual environment that occupies a particular user's field of view and provides the user with six degrees of freedom of movement within a particular virtual environment. It should also be noted that in some examples, additional and/or alternative factors affect the determination of whether spatial live is enabled for a particular spatial state within a multi-user communication session. For example, when a user transitions to an exclusive display mode, spatial live (e.g., the avatar is no longer displayed) may be disabled when viewing particular content in an exclusive display mode, even though the users share the same spatial state (e.g., because the content can only be viewed with three degrees of freedom of movement (e.g., roll, pitch, and yaw rotations) and/or the stage provided for the content when displaying the content in an exclusive display mode is not large enough to accommodate user movement).

In addition, it should be understood that the examples shown and described herein are merely exemplary, and additional and/or alternative elements for interacting with illustrative content can be provided in a three-dimensional environment. It should be understood that the appearance, shape, form, and size of each of the various user interface elements and objects shown and described herein are exemplary, and alternative appearances, shapes, forms, and/or sizes can be provided. For example, a virtual object representing an application window (e.g., windows 330, 530, 662, and 630) can be provided in an alternative shape (such as a circular shape, a triangular shape, etc.) other than a rectangular shape. In some examples, various optional options described herein (e.g., option 623A, option 626, and/or options 621 and 622), user interface objects (e.g., virtual object 633), control elements (e.g., playback controls 556 or 656), etc. can be verbally selected and/or interacted with via a user verbal command (e.g., a "select option" verbal command). Additionally or alternatively, in some examples, various options, user interface elements, control elements, etc. described herein can be selected and/or manipulated via user input received through one or more separate input devices in communication with the electronic device. For example, selection input can be received via a physical input device in communication with the electronic device (such as a mouse, trackpad, keyboard, etc.).

7A to 7B show a flowchart of an exemplary process for displaying content within a multi-user communication session based on one or more display parameters according to some examples of the present disclosure. In some examples, process 700 starts at a first electronic device that communicates with a display, one or more input devices, and a second electronic device. In some examples, the first electronic device and the second electronic device are optionally head-mounted displays similar to or corresponding to the device 260/270 of FIG. 2, respectively. As shown in FIG. 7A, in some examples, at 702, when in a communication session with a second electronic device, the first electronic device presents a computer-generated environment including an avatar and a first object corresponding to a user of the second electronic device via a display, wherein the computer-generated environment is presented based on a first set of display parameters that meet a first set of criteria. For example, as shown in FIG. 6A, the first electronic device 660 displays a three-dimensional environment 650A including an avatar 615 and a user interface object 636 corresponding to a user of the second electronic device 670, and the second electronic device 670 displays a three-dimensional environment 650B including an avatar 617 and a user interface object 636 corresponding to a user of the first electronic device 660.

In some examples, the first set of display parameters includes: spatial parameters for the user of the second electronic device at 704; spatial parameters for the first object at 706; and display mode parameters for the first object at 708. For example, as described above with reference to FIG. 4 , the spatial parameters for the user of the second electronic device define whether spatial live is enabled for the communication session, the spatial parameters for the first object define a spatial template/arrangement of an avatar corresponding to the user of the second electronic device and the first object in the computer-generated environment (e.g., if spatial live is enabled), and the display mode parameters for the first object define whether display of the first object (and/or content associated with the first object) is exclusive or non-exclusive (e.g., and whether a stage is associated with display of the first object in the computer-generated environment). In some examples, if spatial live is enabled (e.g., spatial parameters for a user of the second electronic device are set to “true” (e.g., or some other indicative value, such as “1”)), the spatial parameters for the first object define the spatial template as a first spatial template (e.g., a side-by-side spatial template, as shown in FIG. 5A ), and/or the first object is displayed in a non-exclusive mode in a computer-generated environment in the communication session (e.g., no stage is provided in the computer-generated environment, as similarly shown in FIG. 6A ), then the first set of display parameters satisfies the first set of criteria.

In some examples, at 710, when displaying a computer-generated environment including an avatar corresponding to a user of the second electronic device and a first object, the first electronic device detects a change in one or more display parameters in the first set of display parameters. For example, as shown in FIG6A, the second electronic device 670 detects a selection input 672A pointing to a selectable option 623A of a user interface object 636, which can be selected to display content associated with the user interface object 636, or as shown in FIG5B, detects a change in the display state of a virtual tray 555 in a three-dimensional environment 550A/550B. In some examples, at 712, in response to detecting the change, at 714, based on a determination that the change in the one or more display parameters in the first set of display parameters causes the first set of display parameters to meet a second set of criteria different from the first set of criteria, the first electronic device updates the presentation of the computer-generated environment via the display based on one or more changes in the first set of display parameters. In some examples, if spatial live is disabled (e.g., the spatial parameters for the user of the second electronic device are set to “false” (e.g., or some other indicative value, such as “0”)), the spatial parameters for the first object define the spatial template as a second spatial template (e.g., a circular spatial template, as shown in FIG. 5B ), and/or the first object is displayed in an exclusive mode in a computer-generated environment in the communication session (e.g., providing a stage in the computer-generated environment, as similarly shown in FIG. 6B ), then the first set of display parameters satisfies the second set of criteria.

In some examples, at 716, the first electronic device updates the display of the first object in the computer-generated environment. For example, as shown in FIG. 5C, the virtual cup 552 is displayed in a window state in the three-dimensional environment 550A/550B, or as shown in FIG. 6G, the video content of the application window 630 is displayed in an exclusive full-screen mode in the three-dimensional environment 650A. In some examples, at 718, the first electronic device updates the display of the avatar corresponding to the user of the second electronic device in the computer-generated environment. For example, as shown in FIG. 5C, the avatar 515/517/519 is aligned to a new spatial template (e.g., a side-by-side spatial template) in the three-dimensional environment 550A/550B, or as shown in FIG. 6G, the avatar 619/617 stops being displayed in the three-dimensional environment 650B.

In some examples, as shown in Figure 7B, at 720, based on the determination that the change in the one or more display parameters in the first set of display parameters does not cause the first set of display parameters to satisfy the second set of criteria, the first electronic device maintains the presentation of the computer-generated environment based on the first set of display parameters satisfying the first set of criteria. For example, as shown in Figure 6F, when the first electronic device 660 transitions to displaying the video content of the application window 630 in the three-dimensional environment 650A in full screen mode, the second electronic device 670 automatically follows the first electronic device 660 so that the video content of the application window 630 is also displayed in the three-dimensional environment 650B in full screen mode, which causes the avatar 615/617/619 to remain displayed (e.g., because the spatial live is still enabled).

It should be understood that process 700 is an example, and more, less or different operations may be performed in the same or different order. In addition, the operations in the above process 700 are optionally implemented by running one or more functional modules in an information processing device such as a general-purpose processor (e.g., as described with respect to FIG. 2) or a dedicated chip and/or by other components of FIG. 2.

Therefore, based on the above, some examples of the present disclosure relate to a method, which includes: at a first electronic device that communicates with a display, one or more input devices, and a second electronic device: when in a communication session with the second electronic device, presenting a computer-generated environment including an avatar corresponding to a user of the second electronic device and a first object via the display, wherein the computer-generated environment is presented based on a first set of display parameters satisfying a first set of criteria, the first set of display parameters including spatial parameters for the user of the second electronic device, spatial parameters for the first object, and display mode parameters for the first object; when displaying the computer-generated environment including the avatar corresponding to the user of the second electronic device and the first object, detecting one or more display parameters in the first set of display parameters changes; and in response to detecting changes in the one or more display parameters in the first group of display parameters: based on a determination that the changes in the one or more display parameters in the first group of display parameters cause the first group of display parameters to satisfy a second group of standards that is different from the first group of standards, updating the presentation of the computer-generated environment via the display based on the one or more changes in the first group of display parameters, including: updating the display of the first object in the computer-generated environment, and updating the display of an avatar corresponding to a user of the second electronic device in the computer-generated environment; and based on a determination that the changes in the one or more display parameters in the first group of display parameters do not cause the first group of display parameters to satisfy the second group of standards, maintaining the presentation of the computer-generated environment based on the first group of display parameters satisfying the first group of standards.

Additionally or alternatively, in some examples, according to the determination of enabling spatial live for the communication session, the spatial parameters for the user of the second electronic device meet the first set of criteria. Additionally or alternatively, in some examples, the determination of enabling spatial live for the communication session is based on the determination that the number of users in the communication session is within the threshold number of users. Additionally or alternatively, in some examples, according to the determination of disabling spatial live for the communication session, the spatial parameters for the user of the second electronic device meet the second set of criteria. Additionally or alternatively, in some examples, the determination of disabling spatial live for the communication session is based on the determination that the number of users in the communication session is greater than the threshold number of users. Additionally or alternatively, in some examples, updating the display of the avatar corresponding to the user of the second electronic device in the computer-generated environment includes replacing the display of the avatar corresponding to the user of the second electronic device with a two-dimensional representation of the user of the second electronic device. Additionally or alternatively, in some examples, the spatial parameters for the first object define the spatial relationship between the first object, the avatar corresponding to the user of the second electronic device, and the viewpoint of the user of the first electronic device, wherein the avatar corresponding to the user of the second electronic device is displayed at a predefined position in the computer-generated environment. Additionally or alternatively, in some examples, spatial parameters for the first object satisfy a first set of criteria based on a determination that a predefined location is adjacent to a viewpoint of a user of the first electronic device. Additionally or alternatively, in some examples, display mode parameters for the first object satisfy a first set of criteria based on a determination that the first object is displayed in a non-exclusive mode in a computer-generated environment. Additionally or alternatively, in some examples, display mode parameters for the first object satisfy a second set of criteria based on a determination that the first object is displayed in an exclusive mode in a computer-generated environment.

Some examples of the present disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing any of the above methods.

Some examples of the present disclosure relate to a non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the above methods.

Some examples of the present disclosure relate to an electronic device comprising: one or more processors; a memory; and a device for performing any of the above methods.

Some examples of the present disclosure relate to an information processing apparatus for use in an electronic device, the information processing apparatus including means for performing any one of the above methods.

For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible based on the above teachings. The examples are selected and described in order to best illustrate the principles of the invention and its practical application, so that other persons skilled in the art can best use the invention and the various described examples with various modifications suitable for the specific purposes contemplated.

Claims (20)

1. A method comprising: At a first electronic device in communication with a display, one or more input devices, and a second electronic device: while in a communication session with the second electronic device, presenting, via the display, a computer-generated environment including an avatar corresponding to a user of the second electronic device and a first object, wherein the computer-generated environment is presented based on a first set of display parameters satisfying a first set of criteria, the first set of display parameters comprising: spatial parameters for the user of the second electronic device; spatial parameters for said first object; and display mode parameters for said first object; detecting a change in one or more display parameters of the first set of display parameters while displaying the computer-generated environment including the avatar corresponding to the user of the second electronic device and the first object; and In response to detecting a change in the one or more display parameters in the first set of display parameters: Based on a determination that a change to the one or more display parameters in the first set of display parameters causes the first set of display parameters to satisfy a second set of criteria that is different from the first set of criteria, updating a presentation of the computer-generated environment via the display based on the change to the one or more display parameters in the first set of display parameters, comprising: updating a display of the first object in the computer-generated environment; and updating a display of the avatar corresponding to the user of the second electronic device in the computer-generated environment; and Based on a determination that the change in the one or more of the first set of display parameters does not cause the first set of display parameters to satisfy the second set of criteria, maintaining presentation of the computer-generated environment based on the first set of display parameters satisfying the first set of criteria. 2. The method of claim 1, wherein based on a determination that spatial live is enabled for the communication session, the spatial parameters for the user of the second electronic device satisfy the first set of criteria. 3. The method of claim 2, wherein the determination to enable spatial live for the communication session is based on a determination that a number of users in the communication session is within a threshold number of users. 4. The method of any one of claims 1 to 3, wherein based on the determination to disable spatial live for the communication session, the spatial parameters for the user of the second electronic device meet the second set of criteria. 5. The method of claim 4, wherein the determination to disable spatial live for the communication session is based on a determination that a number of users in the communication session is greater than a threshold number of users. 6. A method according to claim 4, wherein updating the display of the avatar corresponding to the user of the second electronic device in the computer-generated environment includes replacing the display of the avatar corresponding to the user of the second electronic device with a two-dimensional representation of the user of the second electronic device. 7. A method according to any one of claims 1 to 6, wherein the spatial parameters for the first object define a spatial relationship between the first object, the avatar corresponding to the user of the second electronic device, and a viewpoint of the user of the first electronic device, wherein the avatar corresponding to the user of the second electronic device is displayed at a predetermined position in the computer-generated environment. 8. The method of claim 7, wherein the spatial parameters for the first object satisfy the first set of criteria based on a determination that the predetermined location is proximate to the viewpoint of the user of the first electronic device. 9. A method according to claim 7, wherein based on a determination that the predetermined position is along a line across the viewpoint of the user of the first electronic device, the spatial parameters for the first object satisfy the first set of criteria, and the first object is positioned at a position on the line between the viewpoint and the predetermined position. 10. The method of claim 7, wherein based on the determination that a change to the one or more display parameters in the first set of display parameters causes the first set of display parameters to satisfy the second set of criteria, updating the presentation of the computer-generated environment based on a change to the one or more display parameters in the first set of display parameters comprises: The spatial relationship between the first object, the avatar corresponding to the user of the second electronic device, and a viewpoint of the user of the first electronic device is updated. 11. The method of any one of claims 1 to 10, wherein the display mode parameters for the first object satisfy the first set of criteria based on a determination that the first object is displayed in the computer-generated environment in a non-exclusive mode. 12. The method of claim 11, wherein displaying the first object in the non-exclusive mode comprises displaying the first object as a shared object that is shared between a user of the first electronic device and the user of the second electronic device in the computer-generated environment. 13. The method of any one of claims 1 to 12, wherein the display mode parameters for the first object satisfy the second set of criteria based on a determination that the first object is to be displayed in an exclusive mode in the computer-generated environment. 14. The method of claim 13, wherein displaying the first object in the exclusive mode comprises displaying the first object as a private object, the private object being private to a user of the first electronic device, wherein the first object is displayed in a full screen mode in the computer-generated environment. 15. A method according to claim 13, wherein displaying the first object in the exclusive mode includes displaying the first object as a shared object, the shared object is shared between a user of the first electronic device and the user of the second electronic device, and the first object is displayed in full screen mode in the computer-generated environment. 16. The method according to claim 15, further comprising: In response to detecting a change in the one or more display parameters in the first set of display parameters: Based on the determination that a change in one or more display parameters in the first set of display parameters causes the first set of display parameters to satisfy the second set of criteria, data corresponding to a change in the display of the first object by the user of the first electronic device in the computer-generated environment is sent, wherein the data causes the second electronic device to display an option that is selectable to display the first object in the full screen mode in the second electronic device in the computer-generated environment. 17. The method of claim 13, wherein updating display of the avatar corresponding to the user of the second electronic device in the computer-generated environment comprises ceasing display of the avatar in the computer-generated environment. 18. The method of claim 13, wherein: Based on a determination that the change in the one or more display parameters in the first set of display parameters such that the first set of display parameters satisfies the second set of criteria corresponds to a change in the spatial parameters for the first object: Updating the display of the first object in the computer-generated environment includes changing a position of the first object in the computer-generated environment based on the change in the spatial parameter for the first object. 19. A first electronic device, comprising: one or more processors; Memory; and One or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any one of the methods according to claims 1 to 18. 20. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of a first electronic device, cause the first electronic device to perform any one of the methods according to claims 1 to 18.