JP7537016B2 - DEVICE, METHOD AND GRAPHICAL USER INTERFACE FOR INTERACTING WITH A THREE-DIM - Google Patents

Description

(Related Applications)
This application is a continuation of U.S. Patent Application No. 17/483,730, filed September 23, 2021, which claims priority to U.S. Provisional Patent Application No. 63/083,816, filed September 25, 2020, each of which is incorporated by reference in its entirety herein.

(Technical field)
The present disclosure generally relates to computer systems having a display generating component, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display, and one or more input devices that provide computer-generated reality (CGR) experiences.

The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays for computer systems and other electronic computing devices are used to interact with the virtual/augmented reality environment. Exemplary virtual elements include digital images, videos, text, icons, and virtual objects such as buttons and other graphics.

However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback to perform actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulation of virtual objects is complex and error-prone create a significant cognitive burden for users and detract from the experience of the virtual/augmented reality environment. In addition, the methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices.

Therefore, there is a need for a computer system having improved methods and interfaces for providing a user with a computer-generated experience that makes interaction with the computer system more efficient and intuitive for the user. The above-mentioned deficiencies and other problems associated with user interfaces for computer systems having a display generating component and one or more input devices are reduced or eliminated by the disclosed systems, methods, and user interfaces. Such systems, methods, and interfaces optionally complement or replace conventional systems, methods, and user interfaces that provide a user with a computer-generated reality experience. Such methods and interfaces reduce the number, extent, and/or type of inputs from a user by helping the user understand the connection between the inputs provided and the device response to those inputs, thereby producing a more efficient human-machine interface.

According to some embodiments, a method is performed in a computer system in communication with a first display generation component and one or more first input devices. The method includes: displaying a first user interface object in a first view of the three-dimensional environment that is at least partially shared between a first user and a second user, the first user interface object being displayed with a first set of appearance characteristics at a first position within the first view of the three-dimensional environment; and detecting a first user input provided by the first user while displaying the first user interface object with the first set of appearance characteristics at the first position within the first view of the three-dimensional environment, the first user input being directed toward the first user interface object. The method further includes, in response to detecting a first user input directed to the first user interface object, performing a first operation on the first user interface object in accordance with the first user input in accordance with a determination that a second user is not currently interacting with the first user interface object, and displaying a visual indication that the first user interface object is unavailable for interaction with the first user in accordance with a determination that a second user is currently interacting with the first user interface object, where displaying the visual indication includes changing at least one of an appearance of the first user interface object or a position of the first user interface object within the first view of the three-dimensional environment, and canceling performing the first operation on the first user interface object in accordance with the first user input.

According to some embodiments, a method is executed in a computer system in communication with a first display generation component and one or more first input devices, and includes: displaying a first view of the three-dimensional environment including a first user interface object representing a first object in a second physical environment different from the first physical environment while a first user is at a first location in the first physical environment, the first user interface object corresponding to a first viewpoint associated with the first location in the first physical environment, wherein individual positions of the first user interface object in the three-dimensional environment correspond to individual locations of the first object in the second physical environment in a first manner; detecting at least one of a movement of the first user in the first physical environment and a movement of the first object in the second physical environment; displaying a second view of the three-dimensional environment corresponding to the second viewpoint in response to detecting at least one of the movement of the first user in the first physical environment and the movement of the first object in the second physical environment; and displaying the first user interface object in the second view of the three-dimensional environment. Displaying the first user interface object in the second view of the three dimensional environment includes: displaying the first user interface object at a first display position in the second view of the three dimensional environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment, which corresponds to the individual location of the first object in the second physical environment in a first manner, is more than a threshold distance from the individual position in the three dimensional environment corresponding to a second viewpoint associated with the second view of the three dimensional environment; and displaying the first user interface object at a second display position in the second view of the three dimensional environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment, which corresponds to the individual location of the first object in the second physical environment in a first manner, is less than a threshold distance from the individual position in the three dimensional environment corresponding to the second viewpoint associated with the second view of the three dimensional environment, the second display position being offset from the individual position of the first user interface object in the three dimensional environment.

According to some embodiments, a method is executed on a computer system in communication with a first display generation component and one or more first input devices, and includes displaying a first computer-generated experience at a first immersion level; receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level; in response to receiving the biometric data corresponding to the first user, in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, displaying the first computer-generated experience at a second immersion level, where the first computer-generated experience displayed at the second immersion level occupies a greater portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level; and in accordance with a determination that the biometric data corresponding to the first user does not satisfy the first criterion, continuing to display the first computer-generated experience at the first immersion level.

According to some embodiments, a method is executed in a computer system in communication with a first display generation component and one or more first input devices, the method comprising: displaying a first view of the physical environment including a first representation of a first portion of the physical environment; detecting a first user input while displaying the first view of the physical environment, the first user input corresponding to a request to activate a first type of computer-generated sensory adjustment of two or more types of computer-generated sensory adjustment; and in response to detecting the first user input, displaying a second view of the physical environment, the second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment. and, while displaying the second view of the physical environment, detecting a second user input corresponding to a request to activate a second type of computer-generated sensory adjustment of the two or more types of computer-generated sensory adjustments that is different from the first type of computer-generated sensory adjustments; and, in response to detecting the second user input, displaying a third view of the physical environment, the third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having a first display characteristic adjusted for the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustments and a second display characteristic adjusted for the second representation of the physical environment in accordance with the second type of computer-generated sensory adjustments.

According to some embodiments, a method is executed in a computer system in communication with a first display generation component and one or more first input devices, and includes: displaying a first view of a three-dimensional environment including a first representation of a first portion of a physical environment; detecting movement of a first user from a first location of the physical environment to a second location while displaying the first view of the three-dimensional environment including the first representation of the first portion of the physical environment; and in response to detecting the movement of the first user from the first location to the second location and in accordance with a determination that the movement to the second location satisfies a first criterion, the first criterion including a first requirement that the second location corresponds to a location associated with a first type of exercise, and in accordance with a determination that the first criterion is satisfied, displaying a second view of the three-dimensional environment, the second view of the three-dimensional environment including a first virtual content set corresponding to the first type of exercise. displaying a second view of the three-dimensional environment, in which the first set of virtual content replaces at least a portion of the second representation of the second portion of the physical environment including the second location; and in accordance with a determination that the movement to the second location satisfies a second criterion different from the first criterion, the second criterion including a second requirement that the second location corresponds to a location associated with a second type of exercise, the second type of exercise being different from the first type of exercise, displaying a third view of the three-dimensional environment, the third view of the three-dimensional environment including a second set of virtual content corresponding to the second type of exercise, the second set of virtual content being different from the first set of virtual content, and the second set of virtual content replacing at least a portion of the third representation of the third portion of the physical environment including the second location.

According to some embodiments, a computer system includes or is in communication with a display generating component (e.g., a display, projector, head mounted display, etc.), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more haptic output generators, one or more processors, and a memory that stores one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions that perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by a computer system having a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface on a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, a memory, and one or more processors executing one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, which elements are updated in response to input as described in any of the methods described herein. According to some embodiments, a computer system includes a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein. According to some embodiments, an information processing apparatus for use in a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, and optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, includes means for performing or causing to be performed any of the operations of the methods described herein.

Thus, computer systems having display generation components are provided with improved methods and interfaces for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments, thereby increasing the effectiveness, efficiency, and safety and satisfaction of such computer systems. Such methods and interfaces may complement or replace conventional methods for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments.

It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, specification, and claims. Furthermore, it should be noted that the language used herein has been selected solely for ease of reading and explanation, and not to define or limit the subject matter of the present invention.

For a better understanding of the various embodiments described, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

FIG. 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

FIG. 2 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of a CGR experience to a user, according to some embodiments.

FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

FIG. 1 is a block diagram illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared among two or more users, according to some embodiments. FIG. 1 is a block diagram illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared among two or more users, according to some embodiments. FIG. 1 is a block diagram illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared among two or more users, according to some embodiments.

FIG. 1 is a block diagram illustrating methods of displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movements in a first physical environment and the representations of the physical objects move according to movements of the physical objects in a second physical environment different from the first physical environment, and where a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion. FIG. 1 is a block diagram illustrating methods of displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movements in a first physical environment and the representations of the physical objects move according to movements of the physical objects in a second physical environment different from the first physical environment, and where a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion. FIG. 1 is a block diagram illustrating methods of displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movement in a first physical environment and the representations of the physical objects move according to movement of the physical objects in a second physical environment different from the first physical environment, and where a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion.

FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments. FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments. FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments. FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments.

FIG. 1 is a block diagram illustrating the aggregation of effects of multiple types of sensory adjustments provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments. FIG. 1 is a block diagram illustrating the aggregation of effects of multiple types of sensory adjustments provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments. FIG. 1 is a block diagram illustrating the aggregation of effects of multiple types of sensory adjustments provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

A block diagram illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with a determination that a portion of the physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise, according to some embodiments. A block diagram illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with a determination that a portion of the physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise, according to some embodiments. A block diagram illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with a determination that a portion of the physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise, according to some embodiments.

1 is a flowchart of a method for supporting interaction with user interface objects in a computer-generated three-dimensional environment shared among two or more users, according to some embodiments.

1 is a flowchart of a method for displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movements in a first physical environment and the representations of the physical objects move according to movements of the physical objects in a second physical environment different from the first physical environment, and where a change in the manner in which the representations are displayed is triggered in response to a spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion. 1 is a flowchart of a method for displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movements in a first physical environment and the representations of the physical objects move according to movements of the physical objects in a second physical environment different from the first physical environment, and where a change in the manner in which the representations are displayed is triggered in response to a spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion.

1 is a flowchart of a method for altering an immersion level for displaying an environment of a computer-generated experience according to changes in biometric data of a user received by a computer system, according to some embodiments.

1 is a flowchart of a method for aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

1 is a flowchart of a method for selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with some embodiments, pursuant to a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to the particular type of exercise.

The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user in some embodiments.

The systems, methods, and GUIs described herein improve user interface interaction with virtual/augmented reality environments in multiple ways.

In some embodiments, the computer system allows multiple users to have the right to access a first user interface object displayed in a three-dimensional environment, but prevents a user from accessing the first user interface object while another user is interacting with the first user interface object. When displaying a view of the three-dimensional environment including the first user interface object via a first display generation component used by the first user, the computer system detects a first user input directed at the first user interface object. In response to detecting the first user input, the computer system performs a first action on the first user interface object corresponding to the first user input, or displays a visual indication that the first user interface object is not available for interaction with the first user and cancels the execution of the first action, depending on whether the first user interface object is currently available for interaction with the first user. The computer system provides a visual indication and aborts execution of the first action pursuant to a determination that another user currently controls the first user interface object (e.g., another user is interacting with the first user interface object, interacting with the first user interface object in a manner that precludes simultaneous interaction of the first user, and/or locks the first user interface object for the type of action the first user is attempting to perform, etc.). In some embodiments, displaying the visual indication includes moving the first user interface object within the view of the three-dimensional environment shown to the first user to maintain a preset distance between the first user interface object and a close representation of the first user's hand. In some embodiments, displaying the visual indication includes modifying a visual appearance of the first user interface object within the view of the three-dimensional environment shown to the first user. In some embodiments, when the first user interface object is released by the controlling user to the first user (e.g., by a throw gesture, a toss gesture, etc.), the computer system rotates the first user interface object such that the first user interface object is displayed at a preset orientation relative to a viewpoint of a currently displayed view of the three-dimensional environment shown to the first user. In some embodiments, the computer system controls access to the first user interface object by displaying a representation of the first user interface object at or near a position of a representation of a portion of the first user (e.g., a representation of the first user's hand, within arm's reach of a virtual position of the user's face, etc.). Displaying a visual indication in a view of the three-dimensional environment displayed via a display generation component used by the first user in response to the first user attempting to interact with the first user interface object that indicates that the first user interface object is not available for interaction with the first user provides intuitive and timely feedback when an interaction is attempted and reduces unnecessary visual clutter in the view of the three-dimensional environment. Additionally, there is no need to display the same visual indication to other users sharing the environment with the first user, reducing user confusion and improving the efficiency of the man-machine interface.

In some embodiments, a computer system displays a view of a three-dimensional environment including a representation of a physical object (e.g., a second user, an animal, a moving drone, etc.) located in a physical environment different from the physical environment of a first user (and a first display generating component used by the first user to view the three-dimensional environment). The computer system optionally moves a viewpoint corresponding to a currently displayed view of the three-dimensional environment according to the movement of the first user (and/or the first display generating component) in the physical environment. The computer system determines a position and a movement path of a representation of the physical object in the three-dimensional environment based on the location and movement path of the physical object in the physical environment. The computer system utilizes a first type of correspondence relationship (e.g., a mapping relationship and a transformation relationship, optionally various mapping relationships and transformation relationships for the viewpoint, the physical object, the first user, etc.) between the positions in the three-dimensional environment and the locations in the respective physical environments (e.g., the physical environment of the first user and the first display generating component, the physical environment of the physical object, etc.). In some circumstances (e.g., due to movement of the first user and/or movement of the physical object, etc.), a position of the representation of the physical object, when the position(s) is determined using a first type of correspondence between a position in the three dimensional environment and a location in the physical environment, would be within a threshold distance (e.g., an arm's length, three feet, a user-specified distance, etc.) of the position shown via the first display generating component. Under such conditions, the computer system displays the representation of the physical object at an adjusted position that is offset from the position determined based on the first type of correspondence. In some embodiments, the adjusted position is determined based on a second type of correspondence that is different from the first type of correspondence to ensure that the adjusted position remains more than a threshold distance from the position of the viewpoint of the currently displayed view of the three dimensional environment shown via the first display generating component. The computer system continues to use the second type of correspondence to determine the adjusted position of the representation of the physical object until the unadjusted position calculated based on the first type of correspondence is more than a threshold distance away from the position of the viewpoint of the currently displayed view of the three dimensional environment shown via the first display generating component. By monitoring the relative distance between the position of the representation of the physical object and the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generating component, the computer can timely adjust the display position of the representation of the physical object, thereby avoiding visual collisions between the viewpoint and the representation of the physical object. This improves the user's visual experience and reduces user confusion and errors when the user interacts with the three-dimensional environment.

In some embodiments, the computer system modifies the immersion level at which a computer-generated experience (e.g., visual experience, audiovisual experience, virtual reality experience, augmented reality experience, etc.) is presented to the user according to biometric data corresponding to the user. For example, after the computer-generated experience is initiated, the computer system may detect changes in the biometric data (e.g., heart rate, blood pressure, respiratory rate, etc.) corresponding to the user when the user, for example, actively or under the influence of the computer-generated content, adjusts his/her physical and emotional state. According to the changes in the biometric data with respect to respective sets of pre-defined criteria associated with various immersion levels, the computer system increases or decreases the immersion level at which the computer-generated experience is presented to the user by modifying the visual prominence (including, for example, spatial extent, visual depth, saturation, visual contrast, etc.) of the virtual content relative to the visual prominence of the representation of the physical environment (e.g., by enhancing the complexity, spatial extent, and/or visual characteristics of the virtual content and/or reducing the visual clarity, blur radius, opacity, saturation, etc. of the representation of the physical environment). Adjusting the level of immersion at which a computer-generated experience is provided to a user based on changes in biometric data corresponding to the user helps the computer system provide smoother transitions between low and high immersion level experiences that better correspond to the user's perceptual state with respect to the computer-generated experience, thereby reducing user confusion and improving the effectiveness of the computer-generated experience.

In some embodiments, the computer system provides multiple types of sensory modulation features that enhance the user's ability to perceive various aspects of the physical environment that may not be readily perceived without the assistance of special equipment or the computer system. Instead of only allowing the user to use a single type of sensory modulation feature when viewing a portion of the physical environment at a time, the computer system aggregates the effects of two or more types of sensory enhancement features on a representation of a portion of the physical environment such that features and characteristics present in a portion of the physical environment that were previously hidden within the view of the physical environment provided by the computer system may be revealed. By allowing the effects of multiple types of sensory modulation features to be aggregated on a representation of the same portion of the physical environment and presented in a view of the three-dimensional environment that includes the representation of the portion of the physical environment, the user is able to better perceive and understand the physical environment, improving the usefulness of the computer-generated view of the physical environment.

In some embodiments, the computer system displays virtual content (e.g., virtual scenery, visual and functional enhancements to exercise equipment, user interfaces, health and scoreboards, etc.) corresponding to the individual type of exercise pursuant to a determination that the physical location represented in the view of the three-dimensional environment is associated with the individual type of exercise. For example, as the user and the display generating component move through the real-world location, the virtual content shown in the view of the three-dimensional environment is adjusted to correspond to the type of exercise associated with the current location of the user and the display generating component. In some embodiments, if the location is associated with multiple types of exercise, the computer system selects a type of exercise from the multiple types of exercise associated with the location based on other contextual information (e.g., the user's movement, the user's engagement with objects at the location, etc.) and displays visual content corresponding to the selected type of exercise. Automatically selecting and/or modifying virtual content based on the individual type of exercise associated with the user and the location of the display generating component creates a more efficient human-machine interface by narrowing the number, range, and/or nature of inputs from the user to achieve a desired result (e.g., selecting virtual content appropriate for a type of exercise, initiating a particular mode of exercise, etc.).

1-6 illustrate an exemplary computer system for providing a CGR experience to a user. FIGS. 7A-7C are block diagrams illustrating interactions with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments. FIGS. 7D-7F are block diagrams illustrating a method of displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of the three-dimensional environment, where, according to some embodiments, the viewpoint moves according to a user's movement in a first physical environment, the representations of the physical objects move according to a movement of the physical objects in a second physical environment different from the first physical environment, and a change in the manner of displaying the representations is triggered in response to a spatial relationship between the representations of the physical objects and the viewpoint meeting a pre-set criterion. FIGS. 7G-7J are block diagrams illustrating a change in the immersion level of displaying an environment of a computer-generated experience according to a change in biometric data of a user received by the computer system, according to some embodiments. FIGS. 7K-7M are block diagrams illustrating aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment including a representation of a physical environment, according to some embodiments. 7N-7P are block diagrams illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment pursuant to a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise, according to some embodiments. The user interfaces of FIGS. 7A-7P are used to illustrate the processes of FIGS. 8-12, respectively.

In some embodiments, as shown in FIG. 1, the CGR experience is provided to a user through an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted or handheld device).

When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and/or with which the user can interact (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and/or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following is a subset of these terms:

Physical environment: The physical environment refers to the physical world that people can sense and/or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can sense and/or interact with the physical environment directly through their senses, such as through sight, touch, hearing, taste, and smell.

Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via electronic systems. In CGR, a subset of a person's body movements or representations thereof are tracked, and one or more properties of one or more virtual objects simulated within the CGR environment are adjusted accordingly to behave with at least one law of physics. For example, a CGR system may detect the rotation of a person's head and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to the property(s) of the virtual object(s) in the CGR environment may be made in response to representations of body movements (e.g., voice commands). A person may sense and/or interact with a CGR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create an audio environment with 3D or spatial breadth, providing the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and/or interact with only audio objects.

Examples of CGR include virtual reality and mixed reality.

Virtual reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and/or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and/or interact with the virtual objects in the VR environment through a simulation of the person's presence in the computer-generated environment and/or through a simulation of a subset of the person's physical movements in the computer-generated environment.

Mixed reality: A mixed reality (MR) environment refers to a simulation environment designed to incorporate sensory input from the physical environment or representations thereof in addition to including computer-generated sensory input (e.g., virtual objects) as opposed to a VR environment, which is designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between but not including a complete physical environment at one end and a virtual reality environment at the other end. In some MR environments, the computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track location and/or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

Examples of mixed reality include augmented reality and augmented virtuality.

Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, whereby a person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, the video of the physical environment shown on the opaque display is referred to as "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, such that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, such that the altered portion is a modified version that represents the original captured image but is non-photorealistic. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

Hardware: There are many different types of electronic systems that allow a person to sense and/or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and/or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or semi-transparent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, a transparent or semi-transparent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. A projection system may also be configured to project virtual objects, e.g., as holograms or as physical surfaces, into the physical environment. In some embodiments, the controller 110 is configured to manage and regulate the user's CGR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and/or hardware. The controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical setting/environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to a display generation component 120 (e.g., an HMD, a display, a projector, a touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within or shares the same physical housing or support structure as one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and/or one or more of the peripheral devices 195.

In some embodiments, the display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of a CGR experience) to a user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and/or hardware. The display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, the functionality of the controller 110 is provided by and/or combined with the display generation component 120.

According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and/or physically present in the scene 105.

In some embodiments, the display generation component is worn on a part of the user's body (e.g., on one's head, on one's hand, etc.). Thus, the display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present CGR content, where the user holds a device with a display directed to the user's field of view and a camera directed to the scene 105. In some embodiments, the handheld device is optionally disposed in a housing worn on the user's head. In some embodiments, the handheld device is optionally disposed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing an interaction with CGR content triggered based on an interaction occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD in which the interaction occurs in the space in front of the HMD and the response of the CGR content is displayed via the HMD. Similarly, a user interface showing an interaction with CGR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD in which the interaction occurs in the space in front of the HMD and the response of the CGR content is displayed via the HMD.

While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

FIG. 2 is a block diagram of an example of a controller 110 according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 may include one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input/output (I/O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.1 ... 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar type interface), one or more programming (e.g., I/O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

In some embodiments, one or more communication buses 204 include circuitry for interconnecting and controlling communications between system components. In some embodiments, one or more I/O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or the non-transitory computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

The operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, the CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

In some embodiments, the data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation component 120 of FIG. 1, and optionally one or more of the input devices 125, the output devices 155, the sensors 190, and/or the peripheral devices 195. To that end, in various embodiments, the data acquisition unit 242 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

In some embodiments, the tracking unit 244 is configured to map the scene 105 and at least the display generation component 120 relative to the scene 105 of FIG. 1, and optionally track the position/location of one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. To that end, in various embodiments, the tracking unit 244 includes instructions and/or logic therefor, as well as heuristics and metadata therefor. In some embodiments, the tracking unit 244 includes a hand tracking unit 243 and/or an eye tracking unit 245. In some embodiments, the hand tracking unit 243 is configured to track the position/location of one or more parts of the user's hand and/or the movement of one or more parts of the user's hand relative to the scene 105 of FIG. 1, relative to the display generation component 120, and/or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 243 is described in more detail below with respect to FIG. 4. In some embodiments, eye tracking unit 245 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and/or the user (e.g., the user's hands)) or relative to CGR content displayed via display generation component 120. Eye tracking unit 245 is described in more detail below with respect to FIG. 5.

In some embodiments, the adjustment unit 246 is configured to manage and adjust the CGR experience presented to the user by the display generation component 120, and optionally by one or more of the output devices 155 and/or the peripheral devices 195. To that end, in various embodiments, the adjustment unit 246 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. To that end, in various embodiments, the data transmission unit 248 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

Furthermore, FIG. 2 is intended more to illustrate the functionality of various features that may be present in a particular embodiment, as opposed to a structural overview of the embodiments described herein. As will be recognized by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in FIG. 2 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary by implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

3 is a block diagram of an example of a display generation component 120 according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input/output (I/O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and/or similar types of interfaces), one or more programming (e.g., I/O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and/or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communication between the system components. In some embodiments, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

In some embodiments, the one or more CGR displays 312 are configured to provide a CGR experience to the user. In some embodiments, the one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), MEMS, and/or similar display types. In some embodiments, the one or more CGR displays 312 correspond to a waveguide display, such as diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, the one or more CGR displays 312 can present MR or VR content. In some embodiments, the one or more CGR displays 312 can present MR or VR content.

In some embodiments, the one or more image sensors 314 are configured to capture image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as eye tracking cameras). In some embodiments, the one or more image sensors 314 are configured to capture image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as hand tracking cameras). In some embodiments, the one or more image sensors 314 are configured to face forward to capture image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and/or the like.

Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or the non-transitory computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of FIG. 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

In some embodiments, the CGR presentation unit 344 is configured to present the CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally to one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. To that end, in various embodiments, the data transmission unit 348 includes instructions and/or logic therefor, as well as heuristics and metadata therefor.

Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located in separate computing devices.

Furthermore, FIG. 3 is intended more to illustrate the functionality of various features that may be present in a particular implementation, as opposed to a structural overview of the embodiments described herein. As will be recognized by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in FIG. 3 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary by implementation, and in some embodiments will depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

4 is a schematic diagram of an exemplary embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (FIG. 1) is controlled by the hand tracking unit 243 (FIG. 2) to track the position/location of one or more parts of the user's hand and/or the movement of one or more parts of the user's hand relative to the scene 105 of FIG. 1 (e.g., relative to parts of the physical environment surrounding the user, relative to the display generation component 120, or relative to parts of the user (e.g., the user's face, eyes, or head), and/or relative to a coordinate system defined relative to the user's hand. In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and/or color cameras, etc.) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or images of the entire body, and can have either zoom capabilities or dedicated sensors with high magnification to capture images of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving the hand 408 and changing the hand posture.

In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any kind of beacon, sensor, or other marker. It gives the depth coordinate of a point in the scene relative to a predefined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, z axes such that the depth coordinate of a point in the scene corresponds to the z component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps including the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and/or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D locations of the user's wrist joints and fingertips.

The software can also analyze hand and/or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with the motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. The program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and/or gesture information.

In some embodiments, the software may be downloaded in electronic form to the controller 110, for example over a network, or may alternatively be provided on a tangible non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in FIG. 4 as a separate unit from the image sensor 440, by way of example, some or all of the processing functions of the controller may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit within the housing of the hand tracking device 402, or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or with any other suitable computerized device, such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

4 further includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and the wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing gray levels as depth increases. The controller 110 processes these depth values to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

4 also shows a schematic of the hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In FIG. 4, the skeleton 414 is superimposed on the hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, palm center, end of hand connecting to wrist, etc.) and optionally the wrist or arm connected to the hand are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine the hand gesture being performed by the hand or the current state of the hand, according to some embodiments.

5 shows an exemplary embodiment of the eye tracking device 130 (FIG. 1). In some embodiments, the eye tracking device 130 is controlled by the eye tracking unit 245 (FIG. 2) to track the position and movement of the user's gaze relative to the scene 105 or relative to the CGR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, if the display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed in a wearable frame, the head-mounted device includes both a component for generating CGR content for viewing by the user and a component for tracking the user's gaze relative to the CGR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, if the display generation component is a handheld device or a CGR chamber, the eye tracking device 130 is optionally a device separate from the handheld device or the CGR chamber. In some embodiments, the eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a head-mounted display generation component or a non-head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device and is optionally used in conjunction with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device and is optionally part of a non-head-mounted display generation component.

In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or semi-transparent display that allows the user to view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, whereby an individual using the system can observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

As shown in FIG. 5, in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, may be aimed at a "hot" mirror disposed between the user's eye and a display panel that reflects the IR or NIR light from the eye to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye tracking camera and lighting source.

In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device parameters for the particular operating environment 100, e.g., 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR/VR equipment to an end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. The user-specific calibration process may include estimation of the particular user's eye parameters, e.g., pupil location, central vision location, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters have been determined for the eye tracking device 130, images captured by the eye tracking camera may be processed using glint-assisted methods to determine the user's current visual axis and viewpoint relative to the display.

As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, projector, etc.) and may be directed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown in the upper part of FIG. 5), or may be directed at the eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown in the lower part of FIG. 5).

In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses eye-tracking input 542 from the eye-tracking camera 540 for various purposes, e.g., in processing the frames 562 for display. The controller 110 optionally estimates the user's viewpoint on the display 510 based on the eye-tracking input 542 obtained from the eye-tracking camera 540 using a glint-assisted method or other suitable method. The viewpoint estimated from the eye-tracking input 542 is optionally used to determine the direction in which the user is currently looking.

Below, some possible use cases of the user's current gaze direction are described, which are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display certain virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on objects or surfaces in the environment that the user is currently viewing on the display 510. As another exemplary use case, the eyepiece 520 may be a focusable lens, and the eye tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. The controller 110 can leverage the eye tracking information to orient and focus the eyepiece 520 so that a nearby object the user is looking at appears at the correct distance.

In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR LED or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, so it does not introduce noise into the eye tracking system. Note that the locations and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality (e.g., including virtual reality and/or mixed reality) applications to provide a user with a computer-generated reality (e.g., including virtual reality, augmented reality, and/or augmented virtual) experience.

Figure 6 illustrates a glint-assisted eye tracking pipeline, according to some embodiments. In some embodiments, the eye tracking pipeline is realized by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

As shown in FIG. 6, an eye tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

At 610, if the tracking status is yes for the current captured image, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eye.

At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints, based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

FIG. 6 is intended to serve as an example of an eye-tracking technique that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking techniques, either currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technique described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

In the present disclosure, various input methods are described with respect to interaction with a computer system. If one example is provided using one input device or input method and another example is provided using another input device or input method, it should be understood that each example may be compatible with and optionally utilize the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. If one example is provided using one output device or output method and another example is provided using another output device or output method, it should be understood that each example may be compatible with and optionally utilize the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. If one example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with and optionally utilize the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.
User Interface and Related Processes

Attention is now directed to embodiments of user interfaces ("UIs") and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, with a display generation component, one or more input devices, and (optionally) one or more cameras.

7A-7P illustrate a three-dimensional environment displayed via a display generating component (e.g., display generating component 7100, display generating component 7200, display generating component 120, etc.) and interactions occurring in the three-dimensional environment caused by user input directed to the three-dimensional environment and/or input received from other computer systems and/or sensors. In some embodiments, the input is directed to a virtual object in three dimensions by a user's gaze detected at the position of the virtual object in the three-dimensional environment, by a hand gesture performed at a location in the physical environment corresponding to the position of the virtual object, by a hand gesture performed at a location in the physical environment unrelated to the position of the virtual object while the virtual object has input focus (e.g., selected by a simultaneously and/or previously detected gaze input, selected by a simultaneously or previously detected pointer input, selected by a simultaneously and/or previously detected gesture input, etc.), by an input device that has placed a focus selector object (e.g., a pointer object, a selector object, etc.) at the position of the virtual object, etc. In some embodiments, input is directed to a physical object or a representation of a virtual object corresponding to a physical object by a user's hand movements (e.g., moving the entire hand, moving the entire hand in an individual pose, moving one part of the hand relative to another part of the hand, relative movement between two hands, etc.) and/or manipulations on the physical object (e.g., touching, swiping, tapping, opening, moving towards, moving relatively, etc.). In some embodiments, the computer system alters the display of the three-dimensional environment (e.g., displaying additional virtual content or ceasing to display existing virtual content, transitioning between different immersion levels displaying visual content, etc.) according to input from sensors (e.g., image sensors, temperature sensors, biometric sensors, motion sensors, proximity sensors, etc.) and contextual conditions (e.g., location, time, presence of different objects in the environment, etc.). In some embodiments, the computer system modifies the display of the three-dimensional environment (e.g., displaying additional virtual content or ceasing to display existing virtual content, transitioning between different immersion levels displaying visual content, etc.) according to input from other computers used by other users sharing the computer-generated environment with the user of the computer system (e.g., in a shared computer-generated experience, a shared virtual environment, a shared virtual or augmented reality environment of a communication session, etc.).

In some embodiments, the three-dimensional environment displayed via the display generation component is a virtual three-dimensional environment that includes virtual objects and content at various virtual positions within the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at various virtual positions within the three-dimensional environment that are constrained by one or more physical aspects of the physical environment (e.g., positions and orientations of walls, floors, surfaces, direction of gravity, time of day, etc.). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. The representation of the physical environment includes respective representations of physical objects and surfaces at different positions within the three-dimensional environment such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. When the virtual objects are positioned relative to the positions of the representations of the physical objects and surfaces in the three-dimensional environment, the virtual objects appear to have corresponding spatial relationships with the physical objects and surfaces in the physical environment. In some embodiments, the computer system transitions between displaying different types of environments based on user input and/or contextual conditions (e.g., transitioning between presenting computer-generated environments or experiences with different levels of immersion, adjusting the relative prominence of audio/visual sensory inputs from the virtual content and from the representation of the physical environment, etc.).

In some embodiments, the display generation component includes a pass-through portion in which a representation of the physical environment is displayed. In some embodiments, the pass-through portion is a transparent or semi-transparent (e.g., see-through) portion of the display generation component that surrounds the user's field of view and reveals at least a portion of the physical environment in the user's field of view. For example, the pass-through portion is a portion of a head-mounted display that is semi-transparent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent, allowing the user to view the real world surrounding the user through the pass-through portion without removing the head-mounted display or moving away from the head-up display. In some embodiments, the pass-through portion gradually transitions from semi-transparent or transparent to fully opaque when displaying a virtual or mixed reality environment. In some embodiments, the pass-through portion of the display generation component displays a live feed of images or video of at least a portion of the physical environment captured by one or more cameras (e.g., rear-facing camera(s) of a mobile device or associated with the head-mounted display, or other cameras that provide image data to an electronic device). In some embodiments, the one or more cameras are aimed at a portion of the physical environment that is directly in front of the user (e.g., behind the display generation component). In some embodiments, the one or more cameras are aimed at a portion of the physical environment that is not directly in front of the user (e.g., in a different physical environment, or to the side or behind the user).

In some embodiments, when displaying virtual objects in positions corresponding to the location of one or more physical objects in a physical environment (e.g., a virtual reality environment, a mixed reality environment, an augmented reality environment, etc.), at least some of the virtual objects are displayed in place of (e.g., replace) a portion of the camera's live view (e.g., a portion of the physical environment captured in the live view). In some embodiments, at least some of the virtual objects and content are projected onto physical surfaces or open space in the physical environment and are visible through a pass-through portion of the display generation component (e.g., as part of the camera view of the physical environment or visible through a transparent or semi-transparent portion of the display generation component). In some embodiments, at least some of the virtual objects and content are displayed over a portion of the display and block a view of at least a portion of the physical environment that is visible through a transparent or semi-transparent portion of the display generation component.

In some embodiments, the display generating component displays different views of the three-dimensional environment according to user input or movement that changes the virtual position of the viewpoint of the currently displayed view of the three-dimensional environment relative to the three-dimensional environment. In some embodiments, when the three-dimensional environment is a virtual environment, the viewpoint moves according to a navigation or movement request (e.g., a hand gesture in the air, a gesture made by moving one part of a hand relative to another part of a hand, etc.) without requiring movement of the user's head, torso, and/or the display generating component in the physical environment. In some embodiments, movement of the user's head and/or torso relative to the physical environment (e.g., by the user holding the display generating component or wearing an HMD, etc.) and/or movement of the display generating component or other location sensing element of the computer system, etc., causes a corresponding movement of the viewpoint relative to the three-dimensional environment (e.g., with a corresponding change in direction of movement, distance of movement, speed of movement, and/or orientation, etc.), resulting in a corresponding change in the currently displayed view of the three-dimensional environment. In some embodiments, when a virtual object has a pre-established spatial relationship to the viewpoint, movement of the viewpoint relative to the three-dimensional environment causes the virtual object to move relative to the three-dimensional environment while the position of the virtual object in the field of view is maintained (e.g., the virtual object is said to be head-gated). In some embodiments, the virtual object is body-gated to the user and moves relative to the three-dimensional environment when the user moves as a whole in the physical environment (e.g., carrying or wearing the display generating component and/or other location sensing component of the computer system), but does not move within the three-dimensional environment in response to movement of the user's head (e.g., the display generating component and/or other location sensing component of the computer system rotates around the user's fixed location in the physical environment).

In some embodiments, the view of the three-dimensional environment shown in FIGS. 7A-7P includes a representation(s) of the user's hand(s), arm(s), and/or wrist(s). In some embodiments, the representation(s) is part of the representation of the physical environment provided via the display generation component. In some embodiments, the representation is not part of the representation of the physical environment and is displayed within the three-dimensional environment independent of the view of the three-dimensional environment (e.g., by one or more cameras pointed at the user's hand(s), arm(s), and wrist(s)). In some embodiments, the representation(s) includes a stylized version of the arm(s), wrist(s), and/or hand(s) based on camera images captured by one or more cameras of the computer system(s) or information captured by various sensors. In some embodiments, the representation(s) replaces the display of, overlaps with, or blocks the view of, a portion of the representation of the physical environment. In some embodiments, when the display generation component does not provide a view of the physical environment, but rather provides an entirely virtual environment (e.g., no camera view or transparent pass-through portions), a real-time visual representation (e.g., a stylized representation or segmented camera image) of one or both of the user's arms, wrists, and/or hands may still be displayed within the virtual environment.

7A-7C are block diagrams illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments.

In some embodiments, the computer system allows multiple users (e.g., a first user 7102, a second user 7002, another user, etc.) to have the right to access a first user interface object (e.g., a first user interface object 7016, another user interface object, a control panel, a virtual menu, a media object, etc.) displayed in a three-dimensional environment (e.g., a three-dimensional environment 7015, another virtual or augmented reality environment, etc.), but prevents a user (e.g., the first user 7102, or another user different from the first user 7102, etc.) from accessing the first user interface object while another user (e.g., the second user 7002, another user different from the second user 7002, etc.) is interacting with the first user interface object. When displaying a view of a three-dimensional environment including a first user interface object via a first display generation component (e.g., display generation component 7200, a different type of display generation component such as an HMD, etc.) used by a first user (e.g., first user 7102), the computer system detects a first user input (e.g., gaze input, hand movement, a combination of gaze input and user hand movement, etc.) directed at the first user interface object. In response to detecting the first user input, the computer system either performs a first action on the first user interface object corresponding to the first user input (e.g., moving the first user interface object or a representation thereof towards the representation 7202' of the first user's hand 7202, performing a function associated with the first user interface object that changes the three-dimensional environment (e.g., causing virtual content to be displayed or released within the three-dimensional environment, changing other virtual content within the three-dimensional environment, etc.), depending on whether the first user interface object is currently available for interaction with the first user, or displays a visual indication that the first user interface object is not available for interaction with the first user and cancels performance of the first action. The computer system provides a visual indication and aborts execution of the first action pursuant to a determination that another user (e.g., second user 7002) currently controls the first user interface object (e.g., another user is interacting with the first user interface object, interacting with the first user interface object in a manner that precludes simultaneous interaction of the first user, and/or locks the first user interface object for the type of action the first user is attempting to perform, etc.). In some embodiments, displaying the visual indication includes moving the first user interface object in a view of the three-dimensional environment shown to the first user to maintain a preset distance between the first user interface object and a close representation of the first user's hand. In some embodiments, displaying the visual indication includes changing a visual appearance of the first user interface object in a view of the three-dimensional environment shown to the first user (e.g., a view from the user 7102's side as shown in FIG. 7C ). In some embodiments, when the first user interface object is released by the controlling user to the first user (e.g., by a throw gesture, a toss gesture, etc.), the computer system rotates the first user interface object such that the first user interface object is displayed in a preset orientation (e.g., content or control side toward the first user 7102) relative to the viewpoint of the currently displayed view of the three-dimensional environment shown to the first user. In some embodiments, the computer system controls the first user's access to the first user interface object by displaying a representation of the first user interface object at or near a representation of a portion of the first user (e.g., a representation of the first user's 7102 hand 7202, within arm's reach of a virtual position of the user's face, etc.).

In the example shown in Figures 7A-7C, according to some embodiments, the three-dimensional environment 7015 is shared between the first user 7102 and the second user 7002 in response to a request initiated by one of the users 7102 and 7002 using a computer system controlled by the one user and accepted by the other of the users 7102 and 7002 using a computer system controlled by the other user. In some embodiments, both users receive and accept a request to share the three-dimensional environment from a computer system used by a third user using their respective computer systems. In some embodiments, both users send a request to share the three-dimensional environment to a server using their respective computer systems, and the user's request is accepted by the server. When sharing a computer-generated three-dimensional environment, the location and orientation of the users, as well as the location and orientation of their respective heads, eyes, hands, arms, and/or wrists, are captured in real time or periodically by sensors (e.g., cameras, motion sensors, etc.), and the location and orientation data is provided to one or both of the computer systems controlled by the users and/or to a server in communication with the computer systems. The location data is used by the computer system and/or server to determine the respective positions and orientations of the users within the computer-generated three-dimensional environment, as well as the respective positions and orientations of the users' respective heads, eyes, hands, arms, and/or wrists, and correspondingly, the respective positions of the representations of the users including the users' respective heads, arms, hands, and/or wrists within views of the three-dimensional environment provided via the different display generation components associated with the users, as well as the fields of view and viewpoints of the views of the three-dimensional environment provided via the different display generation components associated with the users.

In some embodiments, when two or more users share a computer-generated environment (e.g., a virtual conference call, a chat session, a multiplayer game, a shared computer-generated experience (e.g., a group meditation, exercise, game, collaboration, etc.), they may wish to control and/or manipulate the same user interface objects (e.g., a virtual ball, a virtual control panel, a document or media content, a virtual menu, a user interface, etc.) present in the computer-generated environment. This may make it difficult for the computer system to consistently prioritize the actions of different users on the user interface objects, and the resulting changes in the three-dimensional environment may be confusing to the users. As disclosed herein, the computer system provides visual feedback in response to the first user's 7102 attempt to interact with a first user interface object 7016 already under the control of the second user 7002 in the environment by modifying a set of appearance characteristics of the first user interface object in the view 7015-1 of the environment presented to the first user 7102, thereby reducing conflicts between the users' actions and reducing user confusion when the users interact with the first user interface object 7016. In some embodiments, the first user interface object 7016 presented in the view 7015-2 of the three-dimensional environment shown to the second user 7002 controlling the first user interface object is not altered as a result of the first user's attempt to interact with the first user interface object and does not cause confusion to the second user 7002 when the second user 7002 interacts with the first user interface object 7016.

7A illustrates an exemplary physical environment (e.g., scene 105, another indoor or outdoor physical environment, etc.). In some embodiments, as illustrated in FIG. 7A, two or more users (e.g., user 7102, user 7002, etc.) are present in the same physical environment. A first user 7102 views a first view 7015-1 of a three-dimensional environment 7015 (e.g., an augmented reality environment, a virtual environment, etc.) through a first display generating component (e.g., display generating component 7200, another type of display generating component such as an HMD used by the first user, etc.). A second user 7002 views a second view 7015-2 of the same three-dimensional environment 7015 through a second display generating component (e.g., display generating component 7100, another type of display generating component such as an HMD used by the second user, etc.). In some embodiments, the three-dimensional environment 7015 (e.g., labeled 7015-1 when presented via the first display generating component 7200 and labeled 7015-2 when presented via the second display generating component 7100) is an environment for a shared computer-generated experience, a communication session, an application environment, a game, a movie, etc.

In some embodiments, the first user 7102 and the second user 7002 are not necessarily located in the same physical environment at the same time, but may be located separately in two different physical environments. In some embodiments, the three-dimensional environment 7015 includes a representation of the physical environment of the first user but not the second user, and the first user and the second user have a shared experience in the three-dimensional environment based on the physical environment of the first user. In some embodiments, the three-dimensional environment 7015 includes a representation of the physical environment of the second user but not the first user, and the first user and the second user have a shared experience in the three-dimensional environment based on the physical environment of the second user. In some embodiments, the three-dimensional environment 7015 includes a representation of a third physical environment that is not the physical environment of the first user or the physical environment of the second user, and the first user and the second user have a shared experience in the three-dimensional environment based on the third physical environment (e.g., the physical environment of a third user participating in the shared experience, another physical environment not associated with the user or associated with a user not participating in the shared experience, etc.). In some embodiments, the three-dimensional environment 7015 includes a virtual three-dimensional environment, and the first user and the second user have a shared experience within the virtual three-dimensional environment. In some embodiments, the positions and movements of the first user and the second user within their respective physical environments (e.g., the same physical environment, different physical environments, etc.) are mapped (e.g., using the same mapping relationship, different mapping relationships, etc.) to positions and movements within the same three-dimensional environment, but the appearance of the three-dimensional environment can be adjusted (e.g., with different wallpaper, color schemes, different virtual furniture, etc.) to accommodate the individual users within the view of the three-dimensional environment shown to the individual users.

In some embodiments, the computer system determines that the three-dimensional environment is at least partially shared between the first user 7102 and the second user 7002 according to a determination that at least a spatial portion of the environment 7015 is shared (e.g., a spatial portion of the environment corresponding to the living room but not the kitchen, a spatial portion of the environment corresponding to a portion of the physical space in front of the first user but not a portion of the physical space behind the first user, etc.). In some embodiments, the computer system determines that the three-dimensional environment is at least partially shared between the first user and the second user according to a determination that at least a spatial portion of the environment 7015 is shared during at least a period of time (e.g., during a communication session between the first user and the second user, during the morning, during work hours, when both users are online, etc.). In some embodiments, the computer system determines that the three-dimensional environment 7015 is at least partially shared between the first user and the second user according to a determination that an object in the environment 7105 is fully or partially shared (e.g., visible and accessible at the same time, visible but not accessible at the same time, visible but not accessible when the other person has control (e.g., the other person can see the object or cannot see it, etc.)). In some embodiments, the computer system determines that the three-dimensional environment 7015 is at least partially shared between the first user and the second user according to a determination that at least a portion of the three-dimensional environment 7015 (e.g., the portion shown in the first view 7015-1 of the three-dimensional environment, another portion of the three-dimensional environment 7015, etc.) is displayed for viewing by both the first user and the second user simultaneously. In some embodiments, the computer system determines that the three-dimensional environment 7015 is at least partially shared between the first user and the second user in accordance with a determination that some or all of the virtual objects in the three-dimensional environment are simultaneously displayed to both the first user and the second user in the three-dimensional environment.

7B and 7C, the computer system displays a first view 7015-1 of the three-dimensional environment 7015 that is at least partially shared between the first user 7102 and the second user 7002 via the first display generation component 7200, and substantially simultaneously (e.g., adjusted for network delays, processing time delays, etc.), the computer system or another computer system in communication with the computer system displays a second view 7015-2 of the three-dimensional environment 7105 via the second display generation component 7100. According to some embodiments, both the first view 7015-1 and the second view 7015-2 include at least a first portion of the three-dimensional environment (e.g., separate portions corresponding to the same portion of the physical environment represented in the three-dimensional environment, separate portions corresponding to the same portion of the virtual environment of the three-dimensional environment, etc.). In some embodiments, the first portion of the three-dimensional environment is optionally shown from different viewing angles in the first view 7015-1 and the second view 7015-2 of the three-dimensional environment 7105 (e.g., based on the respective spatial relationships between the first user and its physical environment, and/or the respective spatial relationships between the first user and its physical environment, etc.).

In some embodiments, the first view 7015-1 has a first viewpoint having a position corresponding to a current location of the first user 7102 in the first user's 7102 physical environment, which position moves within the three-dimensional environment 7015 according to the first user's 7102 movement within the first user's 7102 physical environment (e.g., scene 105, another physical environment, etc.). In some embodiments, the second view 7015-2 has a second viewpoint having a position in the three-dimensional environment 7015 corresponding to a current location in the second user's 7002 physical environment, which position moves within the three-dimensional environment 7015 according to the second user's 7002 movement within the second user's physical environment (e.g., scene 105, another physical environment, etc.). In some embodiments, a viewpoint of a currently displayed view of the three-dimensional environment 7015 shown via an individual display generating component (e.g., first display generating component 7200, second display generating component 7100, etc.) has a position in the three-dimensional environment 7015 that corresponds to a current location of the individual display generating component, and that position moves within the three-dimensional environment 7015 according to movement of the individual display generating component within the individual display generating component's physical environment (e.g., scene 105, another physical environment, etc.). In some embodiments, a viewpoint of a currently displayed view of the three-dimensional environment 7015 shown via an individual display generating component (e.g., first display generating component 7200, second display generating component 7100, etc.) has a position in the three-dimensional environment that corresponds to a current location of one or more cameras associated with the individual display generating component, and that position moves within the three-dimensional environment 7015 according to movement of the one or more cameras associated with the individual display generating component within the individual display generating component's physical environment (e.g., scene 105, another physical environment, etc.). In the examples shown in FIGS. 7A-7C, the first view 7015-1 and the second view 7015-2 appear to have the same viewpoint, however, it should be understood that the respective views and their corresponding viewpoints shown via the first display generation component 7200 and the second display generation component 7100 are determined separately and independently based on the spatial relationships and movements present in the respective physical environments of the first display generation component (and first user) and the second display generation component (and second user), and need not be exactly the same at a given time.

In Figures 7B and 7C, the first view 7015-1 and the second view 7015-2 of the three-dimensional environment 7015 include one or more user interface objects (e.g., first user interface object 7016, second user interface object 7018, other user interface objects, virtual three-dimensional objects, etc.), and optionally, one or more surfaces (e.g., a representation 7004' or 7004" of wall 7004, a representation 7006' or 7006" of wall 7006, a representation 7008' or 7008" of floor 7008, a virtual surface such as a virtual wall, a virtual screen, a virtual window, a virtual landscape, etc.), and/or a representation of one or more physical objects (e.g., a representation 7014' or 7014" of physical object 7014 in physical environment 7015, a representation of another physical object in another physical environment represented in the three-dimensional environment 7014, etc.). In some embodiments, the first view 7015-1 and the second view 7015-2 do not include a representation of a physical environment, but rather include a virtual three-dimensional environment (e.g., a virtual conference room, a gaming environment, a virtual experience, a virtual stadium, etc.).

In some embodiments, the first user interface object 7016 is a representation of an application, and interaction with the first user interface object that meets pre-set criteria causes the computer system to launch the application in the three-dimensional environment or execute an application function of the application. In some embodiments, the first user interface object 7016 is a user interface that includes multiple user interface objects (e.g., selectable avatars, selectable menu items, selectable device controls, selectable content items, slider controls, buttons, etc.). In some embodiments, the first user interface object 7016 is a virtual three-dimensional object that can be manipulated (e.g., transformed, separated into parts, rotated, moved, etc.) in the three-dimensional environment according to the movement of the user's hand in the physical environment. In some embodiments, the first user interface object 7016 is a single control or control panel that includes multiple controls corresponding to different functions or operations. In some embodiments, the first user interface object 7016 is an information item, a notification, an alert, etc. In some embodiments, the first user interface object 7016 is a media item, a document, etc.

In some embodiments, as shown in FIGS. 7B and 7C, the first view 7015-1 includes a representation 7202' of the hand 7202 of the first user 7102 and a representation 7028' of the hand 7028 of the second user 7002, and the second view 7015-2 includes a representation 7202'' of the hand 7202 of the first user 7102 and a representation 7028'' of the hand 7028 of the second user 7002. In the scenario shown in FIGS. 7B and 7C, the second user 7002 controls the first user interface object 7016, excluding any simultaneous interaction between the first user 7102 and the first user interface object 7016. For example, in some embodiments, when the first user interface object 7016 is under the control of the first user 7002, the first user interface object 7016 is displayed at a position in the three dimensional environment 7015 that corresponds to the location of the hand 7028 of the second user 7002 in the physical environment of the second user 7002. In some embodiments, when the first user interface object 7016 is under the control of the second user 7002, a representation of the first user interface object 7016 is displayed at a position in the three dimensional environment 7015 that corresponds to the location of the hand 7028 of the second user 7002 in the physical environment of the second user 7002, and the first user interface object 7016 is displayed at another position away from the position of the representation of the first user interface object 7016. In this example, the second user 7002 is controlling a first user interface object 7016, and the first user interface object 7016 is displayed at a position within the three dimensional environment 7015 that corresponds to the location of the second user's hand 7028. In some embodiments, when the first user interface object 7016 is under the control of the second user 7002, the first user interface object 7016 is oriented within the three dimensional environment 7015 such that a pre-defined surface of the first user interface object 7016 (e.g., front surface A, a content presentation surface, an interactive surface, etc.) faces a viewpoint that corresponds to a currently displayed second view 7015-2 of the three dimensional environment 7105 (e.g., a view shown to the second user 7002 controlling the first user interface object 7016, a view displayed by the second display generation component 7100, etc.). In some embodiments, the first user interface object 7016 can be reoriented within the three-dimensional environment by the second user 7002 controlling the first user interface object 7016 such that a preset surface of the first user interface object 7016 faces toward a viewpoint corresponding to a currently displayed first view 7015-1 of the three-dimensional environment (e.g., the view shown to a first user 7102 who is not currently controlling the first user interface object 7016, the view displayed by the first display generation component 7200, etc.). In some embodiments, if the first user interface object 7016 is under the control of the second user 7002 and is not shared with the first user 7102 (e.g., even if the content-presenting side of the first user interface object 7016 is in the first view 7015-1 of the three-dimensional environment presented to the first user 7102 by the first display generating component 7200), at least a portion of the content on the first user interface object 7016 is shown only in the second view 7015-2 of the three-dimensional environment and not in the first view 7015-1 of the three-dimensional environment. In some embodiments, the second user 7002 can make the hidden content of the first user interface object 7016 visible to the first user 7102 by reorienting the first user interface object 7016 such that the content-presenting side of the first user interface object 7016 faces away from the viewpoint of the second view 7015-2.

In Figures 7B and 7C, the first view 7015-1 and the second view 7015-2 both include a respective representation of the first user's hand 7202 (e.g., representation 7202' or 7202") and a respective representation of the second user's hand 7028 (e.g., representation 7028' or 7028"). In some embodiments, the computer system displays the representation of the hand based on a camera view of the user's hand. In some embodiments, the computer system provides a view of the representation of the hand through a transparent portion of the display generation component(s). In some embodiments, the computer system generates a style representation of the user's hand based on sensor information received from one or more sensors located in the first user's and the second user's physical environment(s). In some embodiments, the position and configuration of the representation of the user's hand(s) varies according to the location(s) and configuration(s) of the user's hand(s) in the user's physical environment(s). In some embodiments, the computer system displays hand representations based on camera views of the users' hands. In some embodiments, the computer system displays hand representations of one user but not the other user at a given time. For example, if the second user 7002 is controlling the first user interface object 7016, a representation of the second user's hand 7028 is optionally displayed only in the second view 7015-2 shown to the second user 7002 and not in the first view 7015-1 shown to the first user 7102. In another example, the user's hand representation may move in and out of the field of view provided via the respective display generation components due to movement of the first user and/or the second user (and/or the respective display generation components or cameras, etc.) within their respective physical environments.

In Figures 7B and 7C, within a second view 7015-2 of the three-dimensional environment displayed via a second display generating component 7100 used by a second user 7002 to control a first user interface object 7016, the first user interface object 7016 is displayed with a first set of appearance characteristics (e.g., a normal appearance of the first user interface object as displayed to the second user by the second display generating component (e.g., a first shape, a first size, a first color, a first opacity, a first saturation, a first brightness, etc.)). The first user interface object 7016 maintains the first set of appearance characteristics under the control of the second user 7002 regardless of whether the first user 7102 seeks to access the first user interface object 7016 with a separate movement or input directed at the first user interface object. The first user interface object 7016 can change its appearance in a distinct manner according to an interaction between the second user 7002 and the first user interface object 7016 via a computer system used by the second user 7002. These changes in appearance caused by the interaction between the second user 7002 and the first user interface object 7016 are, optionally, shown in both the first view 7015-1 and the second view 7015-2 at any given time that the change occurs.

In FIG. 7B , while the first user interface object 7016 is under the control of the second user 7002, if the first user 7002 is not attempting to access or gain control of the first user interface object 7016 (e.g., via movement of a part of the user, such as the user's hand, via gaze input, via an air gesture, via a gesture involving movement of a part of the hand relative to another part of the hand, via input provided via a control object, etc.), the first user interface object 7016 is displayed in the first view 7015-1 with the same first set of appearance characteristics as in the second view 7015-2 of the three-dimensional environment (optionally from a different perspective and/or with editing of hidden content, etc.). If both the first view and the second view capture a portion of the three-dimensional environment that corresponds to a location in physical space that includes the first user's hand 7202, the movement of the first user's hand 7202 within the first user's 7102 physical environment may be represented in both the first view 7015-1 and the second view 7015-2.

In contrast, in FIG. 7C , the computer system detects a first user input provided by the first user 7102 directed to the first user interface object 7016. For example, in some embodiments, the computer system detects a movement of a part of the first user 7102 (e.g., the user's hand 7202, another hand of the first user, etc.) to a location in the first user's 7102 physical environment that corresponds to the position of the first user interface object 7016 in the three-dimensional environment 7015. In some embodiments, the computer system detects a gaze input directed to the first user interface object 7016 and a control input (e.g., a finger movement gesture, an air gesture, an input provided by a controller, etc.) that is detected together with the gaze input. In the example shown in FIG. 7C , the first user input is a movement of the first user's hand 7202 to a location that corresponds to the position of the first user interface object 7016, optionally accompanied by a movement or posture to grasp the first user interface object 7016 in the three-dimensional environment. In some embodiments, a representation of the movement, position, and/or posture of the hand 7202 of the first user 7102 is shown in both the first view 7015-1 and the second view 7015-2. In some embodiments, a representation of the movement, position, and/or posture of the hand 7202 of the first user 7102 is shown only in the first view 7015-1 and not in the second view 7015-2. In some embodiments, by not showing the movement, position, and/or posture of the hand 7202 of the first user 7102 in the second view 7015-2, the computer system used by the second user 7002 reduces confusion to the second user 7002 when the second user 7002 interacts with the first user interface object 7016.

7C, in response to detecting a first user input directed at the first user interface object 7016 and in accordance with a determination that the second user 7002 is currently interacting with the first user interface object (e.g., controlling the first user interface object 7016, controlling the first user interface object to the exclusion of a requested interaction by the first user 7102, etc.), the computer system displays a visual indication that the first user interface object 7016 is not available for interaction with the first user 7102. In some embodiments, displaying the visual indication includes modifying at least one of an appearance of the first user interface object 7016 or a position of the first user interface object 7016 within the first view 7015-1 of the three-dimensional environment 7015.

In some embodiments, the computer system determines that the second user 7002 is currently interacting with the first user interface object 7016 according to a determination that the first user interface object 7016 has a pre-established spatial relationship with respect to the virtual position of the second user 7002 in the three-dimensional environment (e.g., the first user interface object 7016 is within a representation of the palm or hand 7028 of the second user, the first user interface object 7016 is within a private space of the second user within the first view 7015-1 of the three-dimensional environment, etc.). In some embodiments, the computer system determines that the second user 7002 is currently interacting with the first user interface object 7016 according to a determination that the second user 7002 is controlling, selecting, moving, modifying, and/or otherwise interacting with the first user interface object 7016 through the computer system displaying the second view 7015-2 of the three-dimensional environment via the second display generating component 7100.

In some embodiments, to display a visual indication in the first view 7015-1 of the three-dimensional environment 7015 that the first user interface object 7016 is unavailable for interaction with the first user 7102, the computer system displays the first user interface object 7016 with a second set of appearance characteristics (e.g., a second shape, a second size, a second color, a second opacity, a second saturation, a second brightness, etc.) that are different from the first set of appearance characteristics (e.g., the second set of appearance characteristics provide a visual indication that the first user interface object is now controlling the second user and is unavailable for interaction with the first user). For example, the first user interface object 7016 shown in the first view 7015-1 of FIG. 7C is semi-transparent compared to that shown in the second view 7015-2 of FIG. 7C. In some embodiments, to display a visual indication in the first view 7015-1 of the three-dimensional environment 7015 to indicate that the first user interface object 7016 is unavailable for interaction with the first user 7102, the computer system moves the first user interface object 7016 out of the way when the first user 7102 attempts to grasp it. In some embodiments, the first user interface object 7016 maintains its appearance and/or position in the second view 7015-2 displayed to the second user 7002, as the visual indication only needs to be displayed to the first user 7102. In some embodiments, if the first user input provided by the first user 7102 corresponds to a request to perform a first action with respect to the first user interface object 7016, the computer system does not perform the first action with respect to the first user interface object 7016 pursuant to a determination that the second user 7002 is currently interacting with the first user interface object 7016 (e.g., controlling the first user interface object 7016, controlling the first user interface object 7016 excluding the interaction requested by the first user 7102, etc.). For example, in some embodiments, the computer system does not show the first user interface object 7106 being grasped by the representation 7202' of the first user's hand 7202. In some embodiments, the computer system does not show a ghost image or another representation of the first user interface object 7016 moving into the representation 7202' of the first user's hand 7202.

In some embodiments, in response to detecting a first user input directed at the first user interface object 7106 and in accordance with a determination that the second user 7002 is not currently interacting with the first user interface object 7016, the computer system performs a first operation on the first user interface object in accordance with the first user input. In some embodiments, performing the first operation includes indicating that the first user interface object 7016 is being grasped or moved by the first user 7102 in accordance with the first user input (e.g., moved toward a virtual position of the first user 7102 in the three-dimensional environment, moved in accordance with the movement of the first user input, etc.). In some embodiments, performing the first operation includes indicating a ghost image or other representation of the first user interface object 7016 being grasped and/or moved within the representation 7202' of the first user's hand 7202. In some embodiments, pursuant to a determination that the second user 7002 was not interacting with the first user interface object 7106 when the first user input from the first user 7102 was detected, the first user interface object 7016 continues to be displayed with the first set of appearance characteristics (e.g., in its original location, or with a representation of the first user's hand, etc.).

In some embodiments, when the first user 7102 attempts to grasp or otherwise interact with the first user interface object 7016 while the second user 7002 is interacting with the first user interface object, the computer system changes the appearance of the first user interface object, such as fading out the first user interface object in the first view 7015-1 displayed to the first user 7102 as the first user 7102 attempts to grasp the first user interface object 7016. For example, the computer system may modify at least one of the first set of appearance characteristics of the first user interface object 7016 (e.g., increase the transparency level, decrease the saturation, decrease the opacity, blur, darken, decrease the resolution, reduce the size, etc. of the first user interface object) to, optionally, reduce the visual prominence of the first user interface object 7016 in the first view 7015-1 of the three-dimensional environment while maintaining the appearance of the surrounding environment of the first user interface object 7016 (e.g., not changing the appearance and/or visual prominence of the surrounding environment). In some embodiments, in response to detecting that the first user 7102 has stopped attempting to interact with the first user interface object 7016, the computer system restores at least one (e.g., some, all, etc.) of the first set of appearance characteristics of the first user interface object that were changed in response to the first user's attempt to grasp or interact with the first user interface object (e.g., to a level that existed immediately prior to detecting the first user input or before the change was made in response to detecting the first user input, etc.) to restore the visual prominence of the first user interface object.

In some embodiments, when the first user interface object 7016 is moved away from a position corresponding to the location of the first user's hand 7202 (e.g., moved away from the representation 7202' of the hand 7202 in the three-dimensional environment 7015 due to an action of the second user 7002 and/or in accordance with other events occurring in the three-dimensional environment (e.g., events unrelated to an interaction attempt by the first user 7102), etc.), the computer system restores at least one (e.g., some, all, etc.) of the first set of appearance characteristics of the first user interface object that was modified in response to the first user's attempt to grasp or otherwise interact with the first user interface object (e.g., to a level that existed immediately prior to detecting the first user input or before the modification was made in response to detecting the first user input) to restore the visual prominence of the first user interface object.

In some embodiments, after the visual indication that the first user interface object 7016 is not available for interaction with the first user 7102 is displayed in the first view 7015-1, the computer system continues to display the visual indication until the computer system detects that the second user 7002 is no longer interacting with the first user interface object and/or has relinquished control of the first user interface object so that the first user interface object becomes available for interaction with the first user 7102. In some embodiments, after the visual indication that the first user interface object 7016 is not available for interaction with the first user 7102 is displayed in the first view 7015-1, the computer system continues to display the visual indication for a preset period of time (e.g., 10 seconds, 5 seconds, etc.) after the first user stops attempting to interact with the first user interface object 7106 via the first user input or another input.

In some embodiments, the first user interface object 7016 can be sent to a position corresponding to the location of the first user (e.g., a position corresponding to the hand 7202 of the first user 7102, a position corresponding to a private space surrounding the first user 7102, etc.) according to a gesture input (e.g., a toss gesture, a throw gesture, a push gesture, etc.) provided by the second user 7002 controlling the first user interface object 7016. In some embodiments, the first user interface object 7016 rotates (e.g., reorients, changes its facing direction, etc.) while moving from a first position to a second position within the three-dimensional environment 7015 as a result of the gesture input provided by the second user 7002. In some embodiments, the first user interface object 7016 can also be sent to a position corresponding to the location of the second user 7002 according to a gesture input (e.g., a toss gesture, a throw gesture, a push gesture, etc.) provided by the first user 7102 after the first user 7102 gains control of the first user interface object 7016. In some embodiments, the first user interface object 7016 rotates (e.g., changes orientation, changes facing direction, etc.) while moving from the second position to the third position in the three-dimensional environment 7015 as a result of the gesture input provided by the first user 7102. In some embodiments, the first user interface object 7106 rotates such that its content presenting or interactive side faces the side receiving the first user interface object.

In some embodiments, the first user interface object 7016 can be sent to a position in the three-dimensional environment where the first user interface object can be better viewed by both the first user and the second user (e.g., displayed at the center of the three-dimensional environment 7015, displayed at a position corresponding to a wall of the physical environment 105, displayed on a virtual surface in the three-dimensional environment 7015, etc.) in response to a gesture input (e.g., a toss gesture, a throw gesture, a push gesture, etc.) provided by a user controlling the first user interface object. In some embodiments, the first user interface object has an orientation that allows both the first user and the second user to see its content and/or interactive side when it reaches the position in the three-dimensional environment, and/or rotates while moving to a position in the three-dimensional environment (e.g., re-facing, changing the facing direction, etc.) such that it has a pre-defined spatial relationship (e.g., overlapping, parallel, diagonally, vertical, upright, etc.) with respect to a surface (e.g., a representation of a wall surface, a table surface, a virtual surface, a virtual screen, a virtual table top, etc.) at the position of the three-dimensional environment.

In some embodiments, the computer system alters a position of the first user interface object 7016 within the first view 7015-1 of the three-dimensional environment as a visual indication that the first user interface object 7016 is unavailable for interaction with the first user 7102. In some embodiments, altering the position of the first user interface object within the first view 7015-1 of the three-dimensional environment includes moving the first user interface object 7016 from an original position of the first user interface object to maintain at least a preset distance between the first user interface object and the representation 7202' of the hand 7202 of the first user 7102 that provided the first user input (e.g., the first user interface object appears to move in one or more directions to avoid the representation 7202' of the hand 7202 of the first user 7102 attempting to grasp the first user interface object). In some embodiments, the movement of the first user interface object 7016 is accompanied by changes made to the appearance of the first user interface object (e.g., the first user interface object appears to fade or darken while being moved to avoid the representation 7202' of the first user's 7102's hand getting too close to itself).

In some embodiments, if the first user interface object 7016 is not under the control of the second user 7002 and is available for interaction with the first user 7102, the computer system moves the first user interface object 7016 towards the representation 7202' of the first user's hand 7202 in the first view 7015-1 of the three-dimensional environment 7015, and optionally also into the second view 7015-2 of the three-dimensional environment.

In some embodiments, the first user input provided by the first user 7102 includes a predefined selection gesture (e.g., is, includes, starts with, ends with, etc.) (e.g., the selection gesture is a pinch gesture involving touching down of the index finger on the thumb of the same hand (optionally followed by lifting the index finger from the thumb, or flicking the wrist connected to the hand, or translating the entire hand, etc.), a gesture involving pulling the index finger and thumb of the same hand away from each other from a touching position, a pinch gesture, a pinch and drag gesture, a pinch and flick gesture, etc.). In some embodiments, in response to detecting a first user input while the second user 7002 is not interacting with the first user interface object 7016, the computer system selects the first user interface object 7016 as a target for a subsequent input received from the first user 7102 (e.g., a drag gesture while a pinch gesture is maintained, a flick gesture while a pinch gesture is maintained, a drag gesture after a predefined selection gesture is completed, etc.). In some embodiments, in conjunction with selecting the first user interface object 7016 as a target for subsequent input received from the first user 7102, the computer system maintains the first user interface object 7016 in a first position within the first view 7015-1 of the three-dimensional environment while displaying a representation of the first user interface object 7106 (e.g., a replica of the first user interface object, a ghost image of the first user interface object, etc.) in a position corresponding to the location of the hand 7202 of the first user 7102 (e.g., the first user interface object remains in its original location but can be "remotely" controlled by the first user 7102 according to interactions between the first user 7102 and the representation of the first user interface object). In some embodiments, the representation of the first user interface object is displayed near the representation 7202' of the first user's hand 7102, but does not advance to the position corresponding to the location of the first user's hand until the computer system detects another selection input provided by the first user 7202. In some embodiments, the computer system changes a shape of the representation of the first user interface object in accordance with a determination that the first user 7102 is providing an input consistent with the requirements for the selection input, the change in shape of the representation of the first user interface object optionally providing visual guidance regarding the requirements for completing the selection input. In some embodiments, a user interaction with the representation of the first user interface object is translated into an interaction with the first user interface object, causing the computer system to perform an action on the first user interface object in accordance with the interaction between the first user 7102 and the representation of the first user interface object. In some embodiments, the representation of the first user interface object remains displayed at the position of the representation 7202' of the first user's hand 7202, optionally indicating that the first user 7102 controls the first user interface object, to the exclusion of interactions of other users sharing the three-dimensional environment with the first user.

In some embodiments, some or all of the features described above with respect to the behavior of the computer system, first display generating component 7200, and second display generating component 7100 of FIGS. 7A-7C are equally applicable to other scenarios in which the roles of the first user 7102 and the second user 7002 with respect to the first user interface object 7016 are reversed. In such other scenarios, the operation of the computer system and display generating components used by the first user and the second user may be reversed as appropriate for the particular scenario. The features described above remain valid and therefore will not be repeated herein for the sake of brevity.

7D-7F are block diagrams illustrating a method of displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, according to some embodiments, where the viewpoint moves according to a user's movements in a first physical environment, the representations of the physical objects move according to movements of the physical objects in a second physical environment different from the first physical environment, and a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion.

In some embodiments, the computer system displays a view of the three-dimensional environment 7204 including representations of physical objects (e.g., the second user 7102, animals, a moving drone, etc.) located in a different physical environment (e.g., scene 105-b, or another indoor or outdoor physical environment, etc.) than the physical environment (e.g., scene 105-a, or another indoor or outdoor physical environment, etc.) of the first user (and the first display generating component 7100 used by the first user 7002 to view the three-dimensional environment 7304). The computer system optionally moves a viewpoint corresponding to the currently displayed view of the three-dimensional environment 7304 in accordance with the movement of the first user 7002 within the physical environment (e.g., scene 105-a, or another physical environment, etc.) of the first user 7002 (and/or the first display generating component 7100). The computer system determines positions and paths of movement of the representations of the physical objects (e.g., representation 7102'-a of the second user 7102, representation of another physical object, etc.) in the three-dimensional environment 7204 based on the locations and paths of movement of the physical objects in the physical environment (e.g., scene 105-b, or another physical environment, etc.). The computer system utilizes a first type of correspondence (e.g., mapping and transformation relationships, optionally different mapping and transformation relationships between viewpoints, physical objects, and the first user, etc.) between positions in the three-dimensional environment 7304 and locations in the respective physical environments (e.g., physical environment 105-a of the first user 7002 and the first display generating component 7100, the physical environment of the physical objects (e.g., physical environment 105-b of the second user 7102, another physical environment of the physical objects, etc.). Under some conditions (e.g., due to movement of the first user 7002 and/or movement of a physical object (e.g., the physical object represented by the second user 7102 in this example), etc.), the position(s) of the representation of the physical object is within a threshold distance (e.g., an arm's length, three feet, a user-specified distance, etc.) of the position of the viewpoint of the currently displayed view (e.g., view 7304-a, 7304-a', etc.) of the three-dimensional environment 7304 shown via the first display generation component 7100, when the position is determined using a first type of correspondence between positions in the three-dimensional environment 7304 and locations in the physical environment (e.g., scenes 105-a, 105-b, etc.). Under such conditions, the computer system displays the representation of the physical object (e.g., representation 7102'-a in this example) at an adjusted position that is offset from the position determined based on the first type of correspondence (e.g., as shown in FIG. 7F ). In some embodiments, the adjusted position is determined based on a second type of correspondence different from the first type of correspondence to ensure that the adjusted position remains greater than a threshold distance from the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generating component (e.g., view 7304-a'', a subsequent view shown via the first display generating component 7100, etc.). The computer system continues to use the second type of correspondence to determine the adjusted position of the representation of the physical object (e.g., representation 7102'-a in this example) until the unadjusted position calculated based on the first type of correspondence is greater than a threshold distance away from the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generating component (e.g., view 7304-a'', a subsequent view shown via the first display generating component 7100, etc.).

In some embodiments, when the computer system provides a view of the three-dimensional environment 7304 to the first user 7002 and the viewpoint position corresponding to the currently displayed view of the three-dimensional environment 7304 is based on the location of the first user's head, body, or eyes within the first user's 7002 physical environment, the computer system sometimes displays representations of other physical objects (e.g., a physical object represented by the second user 7102 in this example, but which may be an inanimate object or an animate object that does not share the computer-generated environment 7304 with the first user 7002, etc.) at positions corresponding to the location of the physical objects in their respective physical environments. In some circumstances, even when there is no risk or possibility of an actual physical collision or uncomfortable spatial proximity between the first user 7002 and other physical objects in the real world, the position of the representation of the physical object may clash with or be too close (e.g., if not specifically adjusted, otherwise addressed, etc.) to the position of the viewpoint corresponding to the view shown to the first user, making the first user's visual experience in the three-dimensional environment uncomfortable or sometimes frustrating for the first user.

As disclosed herein, the computer system determines a position of a representation of a physical object located in a physical environment other than the first user based on a first type of correspondence or mapping relationship between a position in the three-dimensional environment and a corresponding location in the physical environment of the physical object when the position of the representation of the physical object determined based on the first type of correspondence is not within a threshold range of a viewpoint corresponding to a currently displayed view of the three-dimensional environment shown to the first user. This means that if the representation of the physical object is at a certain distance from the virtual position of the viewpoint, the movement of the representation of the physical object in the three-dimensional environment can correspond to the movement of the physical object in a way that mimics the movement and spatial relationship in the real world, and the representation of the physical object does not violate the first user's sense of personal space. However, if the representation of the physical object is very close to the virtual position of the viewpoint, the movement of the representation of the physical object that corresponds to the movement of the physical object in the same way (e.g., according to the first type of correspondence or mapping relationship) causes the representation of the physical object to be displayed at an unreasonable size, to overlap with the viewpoint, and/or to violate the first user's sense of personal space. Thus, pursuant to a determination based on the first type of correspondence or mapping relationship that the representation of the physical object is within a threshold distance from the viewpoint, the computer system uses the second type of correspondence or mapping relationship between the position in the three-dimensional environment and the corresponding location in the physical environment of the physical object to calculate an adjusted position of the representation of the physical object, such that the representation of the physical object may be displayed in the adjusted position and/or moved to avoid being displayed at an unreasonable size, overlapping with the viewpoint, and/or violating the first user's sense of personal space.

7D illustrates a scenario in which two users, e.g., a first user 7002 and a second user 7102, are sharing a computer-generated three-dimensional environment 7304, according to some embodiments. In some embodiments, the first user 7002 is located in a first physical environment 105-a and the second user 7102 is located in a second physical environment 105-b. In some embodiments, the first physical environment and the second physical environment are part of the same physical environment that may overlap with each other. In some embodiments, the first physical environment and the second physical environment are separate physical environments that do not overlap with each other. In some embodiments, the first physical environment and the second physical environment are, optionally, an indoor environment, an outdoor environment, one indoor environment and one outdoor environment, a mix of indoor and outdoor environments, etc. In this example, the first physical environment includes physical surfaces (e.g., walls 7004-a and 7006-a, floor 7008-a, etc.) and physical objects (e.g., physical object 7010, other physical objects, etc.). The second physical environment includes physical surfaces (e.g., walls 7004-b and 7006-b, floor 7008-b, etc.) and physical objects (e.g., physical object 7014, other physical objects, etc.). A first user 7002 is a user of a first view generation component 7100 and is provided with a first view 7304-a (and subsequently updated first views 7304-a', 7304-a", etc.) of a shared three-dimensional environment 7304 via the first view generation component 7100. A second user 7102 is a user of a second view generation component 7200 and is provided with a second view 7304-b (and subsequently updated first views 7304-b', 7304-b", etc.) of a shared three-dimensional environment 7304 via the second view generation component 7200. For purposes of illustration, a first user 7002 moves forward along a straight line 7300 in a first physical environment 105-a, a second user 7102 moves forward along a straight line 7302 in a second physical environment 105-b, a representation 7300' of the straight line 7300 in a second view 7304-b of a three-dimensional environment 7304 passes through the viewpoint of the second view 7304-b, and a representation 7302' of the straight line 7302 in a first view 7304-a of a three-dimensional environment 7304 passes through the viewpoint of the first view 7304-a. In some embodiments, the paths of travel of the first user 7002 and the second user 7102 need not be straight lines, and the paths may be of any shape and/or have any suitable spatial extent within their physical environments. In some embodiments, it is not necessary for both the first user and the second user to move within separate physical environments. In some embodiments, the viewpoint of the currently displayed view of the three-dimensional environment provided via the individual display generating components may not be stationary and/or may move according to the movement of the individual display generating components and/or the movement of the individual users of the individual display generating components. In some embodiments, there is no requirement that the three-dimensional environment is a shared environment between the first user and the second user. For example, in some embodiments, from the perspective of the first display generating component 7100, the second user 7102 in this example is simply a representation of a physical object in the second physical environment (e.g., an animal, a drone, a person not using or providing input to the three-dimensional environment, etc.). Similarly, in some embodiments, from the perspective of the second display generating component, the first user 7002 in this example is simply a representation of a physical object in the first physical environment (e.g., an animal, a drone, a person not using or providing input to the three-dimensional environment, etc.). In some embodiments, only one of the display generation components (e.g., the first display generation component, the second display generation component, etc.) is used, and the other display generation components are not present or do not participate in the processes described herein.

In the example shown in FIG. 7D, according to some embodiments, a three-dimensional environment 7304 is shared between user 7002 and user 7102 in response to a request initiated by one of users 7002 and 7102 using a computer system controlled by one user and accepted by another of users 7002 and 7102 using a computer system controlled by the other of users 7002 and 7102. In some embodiments, both users receive and accept a request to share the three-dimensional environment from a computer system used by a third user using their respective computer systems. In some embodiments, both users send a request to share the three-dimensional environment to a server using their respective computer systems, and the user's request is accepted by the server. When sharing a computer-generated three-dimensional environment, the location and orientation of the users, as well as the location and orientation of their respective heads, eyes, hands, arms, and/or wrists, are captured in real time or periodically by sensors (e.g., cameras, motion sensors, etc.), and the location and orientation data is provided to one or both of the computer systems controlled by the users and/or to a server in communication with the computer systems. The location data is used by the computer system and/or server to determine the location and orientation of each of the users within the computer-generated three-dimensional environment, as well as the location and orientation of each of the users' heads, eyes, hands, arms, and/or wrists, and correspondingly, the position of each of the representations of the users including their respective heads, arms, hands, and/or wrists within the view of the three-dimensional environment provided via the different display generation components associated with the users, as well as the field of view and viewpoint of the view of the three-dimensional environment provided via the different display generation components associated with the users. In some embodiments, the computer-generated environment shared by the users is an environment such as a virtual conference call, a chat session, a multiplayer game, a shared computer-generated experience (e.g., a group meditation, exercise, game, collaboration, etc.). In some embodiments, the representation of the user is an avatar of the user. In some embodiments, the representation of the user is, optionally, not attached to or supported by a surface within the three-dimensional environment.

In part (A) of FIG. 7D, the computer system displays a first view 7304-a of a three-dimensional environment 7304 via the first display generation component 7100. In the first view 7304-a of the three-dimensional environment, a representation 7102'-a of the second user 7102 is displayed at a position corresponding to the current location of the second user 7102 in the second physical environment 105-b. In the first view 7304-a of the three-dimensional environment, there are other objects such as a virtual path 7306-a, a virtual object 7308-a, etc. The appearance and display positions of each of the representation 7102'-a of the second user 7102, the virtual object 7308-a, and the virtual path 7306-a in the first view 7304-a are based on their respective positions in the three-dimensional environment relative to the position of the viewpoint of the currently displayed first view 7304-a of the three-dimensional environment shown via the first display generation component 7100. In some embodiments, the representation 7002'-a of the first user 7002 is optionally visible in the first view 7304-a of the three-dimensional environment at a position corresponding to the virtual position of the first user 7002 and/or the viewpoint of the first view 7304-a currently displayed in the three-dimensional environment. In this example, as shown in Fig. 7D, part (A), the computer system displays the movement of the representation 7102'-a along the representation 7302' of the straight line 7302 toward the virtual position of the viewpoint of the first view 7304-a. At the moment shown in Fig. 7D, the representation 7102'-a is displayed at a position calculated according to a first type of correspondence between a position in the three-dimensional environment 7304 and a location in the second physical environment (e.g., scene 105-b, or another physical environment of the second user 7102, etc.). The representation 7102'-a of the second user 7102 is shown moving toward and approaching the viewpoint of the first view 7304-a as the second user 7102 moves forward along the line 7302 in the second physical environment.

In some embodiments, as shown in part (B) of FIG. 7D, the computer system, or another computer system in communication with the computer system, optionally displays a second view 7304-b of the three-dimensional environment 7304 via the display generation component 7200. In the second view 7304-b of the three-dimensional environment, the representation 7002'-b of the first user 7002 is displayed in a position corresponding to the current location of the first user 7002 in the first physical environment (e.g., scene 105-a, or another physical environment of the first user, etc.). In the second view 7304-b of the three-dimensional environment, there are other objects, such as a virtual path 7306-b (e.g., the same virtual path as virtual path 7306-a, but viewed from the perspective of the second view 7304-b), a virtual object 7308-b (e.g., the same virtual object as virtual object 7308-a, but viewed from the perspective of the second view 7304-b). The appearance and display position of each of the representation 7002'-b of the first user 7002, the virtual object 7308-b, and the virtual path 7306-b in the second view 7304-b is based on their respective positions in the three-dimensional environment relative to the position of the viewpoint of the currently displayed second view 7304-b of the three-dimensional environment shown via the second display generation component 7200. In some embodiments, the representation 7102'-b of the second user 7102 is visible in the second view 7304-b of the three-dimensional environment at a position corresponding to the virtual position of the second user 7102 and/or the viewpoint of the currently displayed second view 7304-b. In this example, as shown in part (B) of FIG. 7D, the computer system displays the movement of the representation 7002'-b along the representation 7300' of the straight line 7300 toward the virtual position of the viewpoint of the second view 7304-b. At the moment shown in FIG. 7D, the representation 7002'-b is displayed at a position calculated according to a first type of correspondence between positions in the three-dimensional environment 7304 and locations in the first physical environment (e.g., scene 105-a, or another physical environment of the first user). The representation 7002'-b of the first user 7002 is shown moving toward and approaching the virtual position of the viewpoint of the second view 7304-b as the first user 7002 moves forward along the line 7300 in the first physical environment.

7E illustrates a time when either or both of the first user 7002 and the second user 7102 have moved within their respective physical environments such that the respective positions of the first user and the second user within the three-dimensional environment 7304 calculated according to a first type of correspondence (e.g., a first type of correspondence between a position within the three-dimensional environment 7304 and a location within the first physical environment, a first type of correspondence between a position within the three-dimensional environment 7304 and a location within the second physical environment, etc.) are at a respective pre-set threshold distance from each other within the three-dimensional environment 7304. In some embodiments, at this point, as shown in part (A) of FIG. 7E, the respective positions of the representation 7102'-a of the second user 7102 and the position of the viewpoint of the updated first view 7304-a' calculated according to the first type of correspondence between the position within the three-dimensional environment 7304 and the location within the second physical environment are at a first threshold distance from each other within the three-dimensional environment 7304. In some embodiments, optionally, as shown in part (B) of FIG. 7E, the individual positions of the representation 7002'-b of the first user 7002 and the position of the viewpoint of the updated second view 7304-b', calculated according to a first type of correspondence between positions in the three-dimensional environment 7304 and locations in the first physical environment, are at a second preset threshold distance from each other in the three-dimensional environment (e.g., at the same preset threshold distance, a different preset threshold distance, etc. than the first preset threshold distance). In some embodiments, the first threshold distance is different from the second threshold distance depending on the personal settings and other characteristics (e.g., size, shape, posture, activity, etc.) of the first user and the second user, respectively.

In part (A) of FIG. 7E, in response to detecting movement of the first user 7002 and the second user 7102 within his/her physical environment, the computer system displays an updated first view 7304-a' of the three-dimensional environment having a viewpoint that is moved according to the movement of the first user 7002 within the first physical environment. In some embodiments, the viewpoint of the updated first view 7304-a' is stationary within the three-dimensional environment if the first user 7002 and/or the first display generation component 7100 have not moved within the first physical environment. In some embodiments, in response to a determination that a respective position of the representation 7102'-a of the second user 7102 in the three dimensional environment, calculated based on the second user's current location in the second physical environment according to the first type of correspondence, is greater than or equal to a first preset threshold distance from a respective position in the three dimensional environment corresponding to a viewpoint associated with the updated first view 7304-a' of the three dimensional environment, the computer system displays the representation 7102'-a at a first display position within the updated first view 7304-a' of the three dimensional environment, the first display position being the respective position of the representation 7102'-a in the three dimensional environment.

In some embodiments, the first pre-set threshold distance is an arm's length, a pre-set radius of a personal space for the first user 7002 in the three-dimensional environment 7304, and is defined by a pre-set bounding surface surrounding the virtual position of the first user 7002 in the three-dimensional environment (e.g., a virtual surface of a representation of the first user 7002, or a bounding box surrounding the virtual position of the first user 7002).

Optionally, in some embodiments, as shown in part (B) of FIG. 7E, in response to detecting movement of the second user 7102 within the second user's 7102 physical environment, the computer system of the second user 7102 displays an updated second view 7304-b' of the three-dimensional environment 7304 having a viewpoint that is moved according to the movement of the second user 7102 within the second physical environment. In some embodiments, the viewpoint of the updated second view 7304-b' is stationary within the three-dimensional environment if the second user 7102 and/or the second display generating component 7200 have not moved within the second physical environment. In some embodiments, in response to a determination that the individual position of the representation 7002'-b of the first user 7002 in the three-dimensional environment, calculated based on the current location of the first user 7002 in the first physical environment according to the first type of correspondence, is greater than or equal to a second preset threshold distance from the individual position in the three-dimensional environment corresponding to the viewpoint associated with the updated second view 7304-b' of the three-dimensional environment, the computer system of the second user 7102 displays the representation 7002'-b at a second display position within the updated second view 7304-b' of the three-dimensional environment, the second display position being the individual position of the representation 7002'-b in the three-dimensional environment.

In some embodiments, the second pre-set threshold distance is an arm's length, a pre-set radius of the second user's 7102 personal space in the three-dimensional environment, and is defined by a pre-set bounding surface surrounding the second user's 7102 virtual position in the three-dimensional environment (e.g., a virtual surface of a representation of the second user 7102, a bounding box surrounding the second user's 7102 virtual position, etc.).

7F, the next moment after that shown in FIG. 7E, the movement of either or both of the first user 7002 and the second user 7102 continues in their respective physical environments such that the respective positions of the first user and the second user in the three-dimensional environment as calculated according to a first type of correspondence (e.g., a first type of correspondence between a position in the three-dimensional environment 7304 and a location in the first physical environment, a first type of correspondence between a position in the three-dimensional environment 7304 and a location in the second physical environment, etc.) are within a respective preset threshold distance of each other in the three-dimensional environment. In some embodiments, at this point, the respective positions of the representation of the second user 7102 and the position of the viewpoint of the further updated first view 7304-a'' are less than a first preset threshold distance of each other in the three-dimensional environment, calculated according to the first type of correspondence between a position in the three-dimensional environment 7304 and a location in the second physical environment.

In portion (A) of FIG. 7F, in response to detecting further movement of the first user 7002 and/or the second user 7102 within their respective physical environments, the computer system displays a further updated first view 7304-a'' of the three-dimensional environment having a viewpoint that is moved in accordance with the further movement of the first user 7002 within the first physical environment. In some embodiments, the viewpoint of the further updated first view 7304-a'' is stationary within the three-dimensional environment if the first user 7002 and/or the first display generation component 7100 have not moved within the first physical environment. In some embodiments, following a determination that an individual position of the representation 7102'-a of the second user 7102 in the three dimensional environment calculated based on the current location of the second user 7102 in the second physical environment in accordance with the first type of correspondence is less than a first preset threshold distance from an individual position in the three dimensional environment corresponding to a viewpoint associated with the further updated first view 7304-a'' of the three dimensional environment, the computer system displays the representation 7102'-a at an adjusted display position in the further updated first view 7304-a'' of the three dimensional environment, the adjusted display position being offset from the individual position of the representation 7102'-a in the three dimensional environment at this point in time. For example, in portion (A) of Figure 7F, instead of displaying representation 7002'-a in a position directly in front of representation 7102'-a, or overlapping with representation 7002'-a in the further updated first view 7304-a", the adjusted display position of representation 7002'-a is offset to the side (e.g., to the right, or to another side or direction, etc.) of representation 7002'-a of first user 7002. In general, instead of displaying representation 7102'-a in a position that is within a first preset threshold distance of the viewpoint of the currently displayed first view 7304-a", the computer system displays representation 7102'-a in an adjusted display position that is offset from the unadjusted position calculated according to the first type of correspondence. In some embodiments, the computer system continues to apply adjustments to the display position of the representation 7102'-a as the first user 7002 and/or the second user 7102 move until the distance between the position of the representation 7102'-a and the position of the viewpoint of the currently displayed first view 7304-a'' is no longer within the first preset threshold distance of each other.

In some embodiments, optionally, as shown in part (B) of FIG. 7F, the individual positions of the representation 7002'-b of the first user 7002 and the position of the viewpoint of the further updated second view 7304-b'' calculated according to a first type of correspondence between positions in the three-dimensional environment 7304 and locations in the first physical environment are less than a second preset threshold distance from each other in the three-dimensional environment (e.g., the same as the first preset threshold distance, a different preset threshold distance, etc.).

In portion (B) of FIG. 7F, in response to detecting further movement of the first user 7002 and/or the second user 7102 within their respective physical environments, the computer system of the second user 7102 displays a further updated second view 7304-b'' of the three-dimensional environment having a viewpoint that is moved in accordance with the further movement of the second user 7102 within the second physical environment. In some embodiments, the viewpoint of the further updated second view 7304-a'' is stationary within the three-dimensional environment if the second user 7102 and/or the second display generation component 7200 have not moved within the second physical environment. In some embodiments, in response to a determination that an individual position of the representation 7002'-b of the first user 7002 in the three dimensional environment calculated based on the current location of the first user 7002 in the first physical environment in accordance with the first type of correspondence is less than a second preset threshold distance from an individual position in the three dimensional environment corresponding to a viewpoint associated with the further updated second view 7304-b'' of the three dimensional environment, the computer system of the second user 7102 displays the representation 7002'-b at an adjusted display position in the further updated second view 7304-b'' of the three dimensional environment, the adjusted display position being offset from the individual position of the representation 7002'-b in the three dimensional environment at this time. For example, in portion (B) of Figure 7F, instead of displaying representation 7002'-b in a position directly in front of representation 7102'-b, or even overlapping with representation 7102'-b in the updated second view 7304-b", the adjusted display position of representation 7002'-b is offset to the side (e.g., to the right, to another side or direction, etc.) of representation 7102'-b of the second user 7102. In general, instead of displaying representation 7002'-b in a position that is within a second preset threshold distance of the viewpoint of the currently displayed second view 7304-b", the computer system of the second user 7102 displays representation 7002'-b in an adjusted display position that is offset from the unadjusted position calculated according to the first type of correspondence. In some embodiments, the computer system of the second user 7102 continues to apply the adjustments during the movement of the first user and/or the second user until the distance between the position of the representation 7002'-b and the position of the viewpoint of the currently displayed second view 7304-b'' is no longer within the preset second threshold distance of each other.

In some embodiments, in the above example, the first user 7002 is moving and the second user 7102 is stationary. As a result, unless adjusted as described above, the viewpoints of the currently displayed views 7304-a, 7304-a', and 7304-a" have different positions in the three-dimensional environment, and the representations 7002'-b of the first user 7002 have different positions in the three-dimensional environment (e.g., in the currently displayed first views 7304-a, 7304-a', and 7304-a" and the currently displayed second views 7304-b, 7304-b', 7304-b" in Figures 7D-7F). Unless adjusted as described above, the viewpoints of the currently displayed views 7304-b, 7304-b', and 7304-b'' have the same position in the three-dimensional environment, and the representation 7102'-a of the second user 7102 has the same position in the three-dimensional environment (e.g., in the currently displayed first views 7304-a, 7304-a', and 7304-a'' and the currently displayed second views 7304-b, 7304-b', 7304-b'' in Figures 7D-7F).

In some embodiments, in the above example, the first user 7002 is stationary and the second user 7102 is moving within the second physical environment. As a result, unless adjusted as described above, the viewpoints of the currently displayed views 7304-b, 7304-b', and 7304-b" have different positions in the three-dimensional environment and the representation 7102'-a of the second user 7102 has different positions in the three-dimensional environment (e.g., in the currently displayed first views 7304-a, 7304-a', and 7304-a" and the currently displayed second views 7304-b, 7304-b', 7304-b" in Figures 7D-7F). Unless adjusted as described above, the viewpoints of the currently displayed views 7304-a, 7304-a', and 7304-a'' have the same position in the three-dimensional environment, and the representation 7002'-b of the first user 7002 has the same position in the three-dimensional environment (e.g., in the currently displayed first views 7304-a, 7304-a', and 7304-a'' and the currently displayed second views 7304-b, 7304-b', and 7304-b'' in Figures 7D-7F).

In some embodiments, in the example above, both the first user 7002 and the second user 7102 are moving within their respective physical environments. As a result, the viewpoints of the currently displayed first views 7304-b, 7304-b', and 7304-b'', and the viewpoints of the currently displayed second views 7304-a, 7304-a', 7304-a'', all have different positions in the three-dimensional environment. The representation 7102'-a of the second user 7102 has different positions in the three-dimensional environment in the currently displayed first views 7304-a, 7304-a', and 7304-a'' and the currently displayed second views 7304-b, 7304-b', and 7304-b'' of Figures 7D-7F, and the representation 7002'-b of the first user 7002 has different positions in the three-dimensional environment in the currently displayed first views 7304-a, 7304-a', and 7304-a'' and the currently displayed second views 7304-b, 7304-b', and 7304-b'' of Figures 7D-7F.

In some embodiments, the representation 7002'-b of the first user 7002 and/or the representation 7102'-a of the second user 7102 are floating in space within the first view and the second view. For example, in some embodiments, the representation 7002'-b of the first user 7002 is a floating avatar of the first user 7002, floating in the second views 7034-b, 7034-b', and 7034-b'', etc. of the three-dimensional environment, and automatically moves out of the way when the viewpoint of the second view 7034-b'' comes within a second preset threshold distance of the representation 7002'-b due to the movement of the first user and/or the movement of the second user. Similarly, in some embodiments, the representation 7102'-a of the second user 7102 is a floating avatar of the second user 7102 floating in the first view 7034-a, 7034-a', and 7034-a", etc. of the three dimensional environment, and automatically moves out of the way when the viewpoint of the first view 7034-a" comes within a first preset threshold distance of the representation 7102'-a due to the first user's movement and/or the second user's movement. In some embodiments, the user's avatar in the three dimensional environment is a level of realism selected based on the realism level of the three dimensional environment (e.g., a photorealism level, a cartoon realism level, etc.). In some embodiments, following a determination that the three-dimensional environment 7304 is displayed at a first level of reality, the user's representation is displayed with a first set of display characteristics (e.g., a first resolution, a first number of dimensions, a first clarity, a first color palette, no lighting effects, etc.) corresponding to the first level of reality, and following a determination that the three-dimensional environment is displayed at a second level of reality different (e.g., higher, lower, etc.) from the first level of reality, the user's representation is displayed with a second set of display characteristics (e.g., a second resolution, a second number of dimensions, a second clarity, a second color palette, with lighting effects, etc.) corresponding to the second level of reality, where the second set of display characteristics is different (e.g., higher, lower, adding, subtracting, etc.) from the first set of display characteristics.

In some embodiments, when the display position of the individual user's representation is adjusted, the individual user's representation moves with a movement component that does not correspond to the movement of the individual user in the physical environment in a normal manner (e.g., according to a first type of correspondence, without adjustment, etc.). In some embodiments, the amount of offset applied to the adjusted position of the individual user's individual representation is variable based on the spatial relationship between the individual representation and the virtual position of the viewpoint in the three-dimensional environment. In some embodiments, the adjustment to the display position of the representation 7102'-a is optionally applied to the first view 7304-a'' displayed to the first user 7002 and not to the second view 7304-b'' displayed to the second user 7102. In some embodiments, the adjustment to the display position of the representation 7002'-b is optionally applied to the second view 7304-b'' displayed to the second user 7102 rather than to the first view 7304-a'' displayed to the first user 7002.

In some embodiments, the three-dimensional environment 7304 includes a virtual three-dimensional environment or an augmented reality environment, and the first user and the second user have a shared experience in the virtual three-dimensional environment. In some embodiments, the positions and movements of the first user and the second user in their respective physical environments (e.g., the same physical environment, different physical environments, etc.) are mapped (e.g., using the same mapping relationship, or different mapping relationships, etc.) to positions and movements in the same three-dimensional environment, but the appearance of the three-dimensional environment can be adjusted (e.g., with different wallpaper, color schemes, different virtual furniture, etc.) to accommodate the individual users within the view of the three-dimensional environment shown to the individual users.

7G-7J are block diagrams illustrating changes in immersion levels displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments.

In some embodiments, the computer system alters the immersion level at which it presents a computer-generated experience (e.g., a visual experience, an audiovisual experience, a virtual reality experience, an augmented reality experience, etc.) to a user according to biometric data (e.g., biometric data represented by bar 7312, other biometric data, etc.) corresponding to a user (e.g., user 7002). For example, after a computer-generated experience has begun, the computer system can detect changes in biometric data (e.g., heart rate, blood pressure, respiratory rate, etc.) corresponding to the user as the user adjusts his or her physical and emotional state, e.g., actively and/or under the influence of the computer-generated content. In response to changes in the biometric data relative to respective sets of pre-established criteria (e.g., the thresholds represented by indicators 7326 or other types of thresholds or criteria) associated with various levels of immersion, the computer system increases or decreases the level of immersion provided to the user in the computer-generated experience by altering the visual prominence (including, e.g., spatial extent, visual depth, saturation, visual contrast, etc.) of the virtual content relative to the visual prominence of the representation of the physical environment (e.g., by enhancing the complexity, spatial extent, and/or visual characteristics of the virtual content and/or reducing the visual clarity, blur radius, opacity, saturation, etc. of the representation of the physical environment).

In the examples shown in Figures 7G-7J, a computer system initially displays a view 7316 of a three-dimensional environment via a display generating component (e.g., display generating component 7100 or another type of display generating component such as an HMD). In some embodiments, the view 7316 of the three-dimensional environment is a pass-through view of the user's 7002 physical environment and includes no virtual content or a minimal amount of virtual content (e.g., system controls, indicators, etc.) within a peripheral portion of the field of view provided by the display generating component. The view 7316 corresponds to a low level of immersion that provides a computer-generated experience to the user, for example, due to a minimal amount of virtual content displayed relative to a representation of the user's physical environment. In this example, the view 7316 of the three-dimensional environment includes representations of physical surfaces (e.g., representations 7004' and 7006' of two adjacent walls 7004 and 7006 in the user 7002's physical environment 105, a representation 7008' of a floor 7008, etc.) and representations of physical objects (e.g., a representation 7010' of a physical object 7010 in the user 7002's physical environment 105, as well as representations of other physical objects).

FIG. 7G also illustrates that while the computer system is displaying a view 7316 of the three-dimensional environment at a low immersion level (e.g., displaying a pass-through view of the physical environment or displaying a representation of the physical environment with a minimal amount of virtual content), the computer system receives biometric data corresponding to the user 7002. Pursuant to a determination that the biometric data of the user 7002 does not meet the preset criteria corresponding to the next higher immersion level, the computer system maintains the display of the first view 7316 of the three-dimensional environment without reducing the visual prominence of the representation of the physical environment in the currently displayed view of the three-dimensional environment. For example, as illustrated in FIG. 7G, the biometric data has a value or set of values indicated by the length of the bar 7312 relative to the full range of values of the biometric data, and a threshold value corresponding to the preset criteria for transitioning to a different higher immersion level is indicated by the position of the indicator 7326 relative to the full range of values of the biometric data.

In some embodiments, the biometric data corresponding to the user 7002 includes one or more of the user's 7002 heart rate, respiration rate, body temperature, serum concentrations of certain chemicals, drugs, and/or hormones, blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose level, and the like. In some embodiments, the biometric data corresponding to the user 7002 includes one or more types of biometric data (e.g., respiration rate, blood pressure, concentration, blood glucose level, and the like) that may change over time during the user's engagement with the computer-generated experience. In some embodiments, the biometric data corresponding to the user includes one or more types of biometric data that may vary through the user's physical actions (e.g., meditation, changes in breathing patterns, exercise, and the like, as opposed to direct interaction with user interface elements or controls provided by the computer system during the user's engagement with the computer-generated experience). In some embodiments, the biometric data corresponding to the user includes one or more types of composite metrics of multiple types of biometric data corresponding to the user's mood, happiness, and/or stress level, and the like. In some embodiments, the biometric data includes real-time data corresponding to the user's physiological state within a time or a preset amount of time prior to the display of the current view of the three-dimensional environment via the display generation component. In some embodiments, the biometric data is continuously and/or periodically collected through one or more biometric sensors (e.g., various suitable medical devices, vibration sensors, cameras, thermal sensors, chemical sensors, etc.) connected to or directed at the user and continuously and/or periodically transmitted to the computer system. In some embodiments, the biometric data does not include non-primary human characteristics (e.g., fingerprints, iris patterns and colors, facial features, voice prints, etc.) that typically do not change over the period that an average user is engaged with a computer-generated experience.

In some embodiments, the computer system determines that the biometric data does not meet the preset criteria for transitioning to displaying the computer-generated experience at a preset higher immersion level according to a determination that the heart rate is greater than a first threshold heart rate, the blood pressure is greater than a first threshold blood pressure, the user's movement is greater than a first threshold movement amount for a threshold amount of time, the user's body temperature is greater than a first threshold body temperature, the stress level metric is greater than a first threshold stress level, a metric corresponding to the user's mood indicates that the user is upset and dissatisfied, etc. In some embodiments, the computer system switches to displaying the three-dimensional environment at a preset higher immersion level (e.g., as shown in FIG. 7J) when the preset criteria are met, without going through a gradual transition based on changes in the biometric data before the preset criteria are met. In some embodiments, optionally, the computer-generated experience includes visual and/or audio guidance (e.g., music, scenery, motivational messages, medication record guidance, visual, audio, or verbal breathing instructions, etc.) to assist the user in entering a state in which the corresponding biometric data received from the user would meet the preset criteria.

7H-7I illustrate that in some embodiments, the computer system gradually adjusts the level of immersion provided to the user in the computer-generated experience according to the trend and/or magnitude of the change in the biometric data corresponding to the user. For example, in some embodiments, when the biometric data exhibits a change approaching satisfaction of a preset criterion for switching to a preset higher immersion level (e.g., an augmented reality view, an augmented virtuality view, a virtual reality view, etc.), the computer system increases the visual prominence and/or amount of the virtual content corresponding to the computer-generated experience and reduces the visual prominence and/or amount of the representation of the physical environment in the currently displayed view of the three-dimensional environment. In some embodiments, the computer system changes the visual balance between the virtual content corresponding to the computer-generated experience and the representation of the physical environment by an amount corresponding to the amount and/or nature of the change in the biometric data corresponding to the user. Similarly, in some embodiments, when the biometric data exhibits a change away from meeting the pre-set criteria for switching to a pre-set higher level of immersion, the computer system decreases the visual prominence and/or amount of the virtual content corresponding to the computer-generated experience and increases the visual prominence and/or amount of the representation of the physical environment within the currently displayed view of the three-dimensional environment.

In some embodiments, the computer system changes the visual balance between the virtual content and the representation of the physical environment by an amount corresponding to the amount and/or nature of the change in the biometric data corresponding to the user. As shown in FIG. 7H, when the value of the biometric data changes to meet a pre-set criterion (e.g., as indicated by the increased length of bar 7312 approaching the position of indicator 7326), the amount of virtual content displayed in the view of the three-dimensional environment (e.g., view 7318 of FIG. 7H) increases compared to a previous state (e.g., view 7316 of FIG. 7G) and the visual prominence of the representation of the physical environment decreases. More specifically, in FIG. 7H , representations 7004' and 7006' of walls 7004 and 7006 are replaced or hidden by the display of virtual content 7320 and 7322 (e.g., a visual effect, a virtual surface, a virtual object, a virtual scenery, etc. that visually obscures a portion of the representation of the physical environment to which the visual effect is applied), and at least a portion of the surface of representation 7010' is also replaced or hidden by the display of virtual content 7324 (e.g., a visual effect, a virtual surface, a virtual object, a virtual scenery, etc. that visually obscures a portion of the representation of the physical environment to which the visual effect is applied). As shown in Figure 7I, which follows Figure 7H, when the value of the biometric data changes away from meeting the pre-set criteria (e.g., as indicated by the reduced length of bar 7312 receding from the position of indicator 7326), the amount of virtual content displayed in the view of the three-dimensional environment (e.g., view 7328 in Figure 7I) is reduced compared to the previous state (e.g., view 7318 in Figure 7H), and the visual prominence of the representation of the physical environment is increased again (e.g., optionally still lower than the state shown in Figure 7G). More specifically, in Figure 7I, the representation 7006' of wall 7006 is redisplayed after the virtual content 7332 is removed, and the representation 7004' of wall 7004 is partially redisplayed when the visual prominence of the virtual content 7320 is reduced (e.g., the magnitude of visual effects that visually obscure the portion of the representation of the physical environment to which the visual effect is applied is reduced, virtual surfaces and objects are reduced, removed, reduced in number or made more translucent, etc.). The visual prominence of the portions of the surface of the representation 7010' that are replaced or obscured by the display of the virtual content 7324 is similarly increased by the changes made to the virtual content 7324 (e.g., made more translucent, less opaque, includes a lesser amount of distortion to the representation 7010', etc.). In some embodiments, before a preset criterion for transitioning to a preset higher immersion level is met (e.g., before a threshold indicated by the indicator 7326 is met by the biometric data corresponding to the user, or before other criteria are met by the biometric data, etc.), the computer system continuously or periodically adjusts the visual balance between the virtual content and the representation of the physical environment in the currently displayed view of the three-dimensional environment according to the biometric data as the biometric data is updated based on the user's current state (e.g., increases the visual prominence of the virtual content relative to the representation of the physical environment, decreases the visual prominence of the virtual content relative to the representation of the physical environment, etc.).

In FIG. 7J, the computer system detects that updated biometric data corresponding to the user meets preset criteria for transitioning to a preset higher immersion level (e.g., an augmented reality environment, an augmented virtual reality environment, a virtual reality environment, etc.) having a higher immersion level compared to what was displayed before the preset criteria was met by the biometric data (e.g., views 7316, 7318, 7328, etc. of FIGs. 7G-7I), and the computer system transitions to a display of a three-dimensional environment at the preset higher immersion level (e.g., a display such as view 7334 of FIG. 7J, or another view of the three-dimensional environment at the preset higher immersion level). In this example, as shown in Figure 7J, the computer system, through the visual characteristics of the virtual content (e.g., virtual content 7322, 7320, 7330, and 7324, which visually obscure representations 7006', 7004', 7008', and 7010' of walls 7006, 7004, floor 7008, and physical object 7010 in view 7334 of Figure 7J), increases the visual prominence of the virtual content and further decreases the visual prominence of the representations of the physical environment such that only hints of the physical environment (e.g., structural relationships between walls and floors, presence of physical objects, etc.) are still visible within the three-dimensional environment. In some embodiments, the computer system further displays virtual objects at various positions within the three-dimensional environment. For example, virtual object 7332 may be displayed in a position corresponding to the location of physical object 7010 in the physical environment, virtual object 7326 may be displayed in a position corresponding to a location on floor 7008, and other virtual objects may be displayed in positions corresponding to free space in the physical environment or independent of the state of the physical environment. In some embodiments, after the preset criteria for transitioning to a preset higher immersion level are met, the computer system abruptly increases the amount of virtual content in the currently displayed view of the three-dimensional environment or displays an entirely new environment (e.g., a new virtual world, a new scene, etc.) corresponding to the computer-generated experience. In some embodiments, after the preset criteria are met and the computer system displays the three-dimensional environment at the preset higher immersion level, pursuant to a determination that the preset criteria are no longer met by the updated biometric data, the computer system gradually adjusts the immersion level at which the three-dimensional environment is displayed based on the change in the biometric data, as shown in Figures 7H and 7I. In some embodiments, after the preset criteria are met and the computer system displays the three-dimensional environment at a preset higher level of immersion, pursuant to a determination that the preset criteria are no longer met by the updated biometric data, the computer system abruptly switches back to displaying the three-dimensional environment at a lower level of immersion (e.g., as shown in FIG. 7G).

In some embodiments, the pre-set criteria are met according to a determination that the heart rate is below a first threshold heart rate, the respiration rate is below a first threshold respiration rate, the blood pressure is below a first threshold blood pressure, the user's movement is less than a first threshold movement amount during a threshold time, the user's body temperature is below a first threshold body temperature, a stress level metric is below a first threshold stress level, and a metric corresponding to the user's mood indicates that the user is relaxed and happy.

In some embodiments, a view of the three-dimensional environment shown at a low immersion level (e.g., as shown in FIG. 7G, or another view of the three-dimensional environment, etc.) is displayed when the display generating component of the computer system is first turned on or placed in front of the user's head or eyes, and no virtual elements or a minimal amount of virtual elements are displayed in the three-dimensional environment. This allows the user to start with a view of the three-dimensional environment that closely resembles a direct view of the real world, without the display generating component blocking the user's eyes. In some embodiments, a view of the three-dimensional environment corresponding to a low immersion level is a view of the user interface or environment (e.g., a two-dimensional environment, a three-dimensional environment, etc.) of the application or computer-generated experience that is displayed in a two-dimensional window or is limited within a display area displayed in a position relative to a representation of the physical environment. In some embodiments, a view of the three-dimensional environment shown at a low immersion level (e.g., as shown in FIG. 7G, or another view of the three-dimensional environment, etc.) is displayed when the application or computer-generated experience is first launched or started by the user, and the full spatial extent of the application or experience has not yet been displayed in the three-dimensional environment. This allows the user to start with a less immersive view of the three-dimensional environment that is seen in the context of the real-world view.

In some embodiments, the virtual content (e.g., virtual wallpaper, virtual objects, virtual surfaces, virtual landscapes, virtual three-dimensional environments, etc.) displayed by the computer system at least partially blocks or obscures the view of the physical environment. In some embodiments, when displaying a view of the three-dimensional environment at a higher than preset immersion level, the computer system replaces or blocks the view of a first class of physical objects or surfaces (e.g., the front wall, the front wall, and the ceiling, etc.) with a newly displayed virtual element or a newly displayed portion of an existing virtual element. In some embodiments, an animated transition is displayed in which the virtual element gradually expands or becomes more opaque and fills up, thereby obscuring or blocking the view of the first class of physical objects or surfaces. In some embodiments, when displaying a view of the three-dimensional environment at a higher than preset immersion level, the computer system adds the virtual element to the three-dimensional environment without replacing any entire class of physical elements. In some embodiments, the virtual elements added optionally include user interface objects such as menus (e.g., application menus, documents, etc.), controls (e.g., display brightness controls, display focus controls, etc.), or other objects that can be manipulated by user input or that provide information or feedback to the three-dimensional environment (e.g., virtual assistants, documents, media items, etc.). In some embodiments, the virtual elements added optionally include non-interactive objects or surfaces that cannot be manipulated by user input and that function to provide a look and feel of the three-dimensional environment that replaces the look and feel of the physical environment. In some embodiments, the virtual content displayed by the computer system includes visual effects that at least partially block or obscure the view of the physical environment (e.g., fading out, blurring, dimming the representation of the physical environment, etc.).

In some embodiments, the biometric data is updated, and pursuant to a determination that the updated biometric data meets the preset criteria for transitioning to displaying the three-dimensional environment at a higher level of immersion, the computer system increases the visual prominence of the virtual content corresponding to the computer-generated experience and reduces the visual cues from the physical environment to another level corresponding to the higher level of immersion. For example, in some embodiments, the computer system replaces, obscures, or blocks the additional class of physical objects or surfaces (e.g., floors) with newly displayed virtual elements or newly displayed portions of existing virtual elements. In some embodiments, an animated transition is displayed in which the virtual elements gradually expand or become more opaque and full, thereby obscuring or blocking the v-view of the additional class of physical objects and surfaces.

In some embodiments, the three-dimensional environment is an environment for a computer-generated mediated experience, and when the biometric data indicates that the user has achieved a level of focus, relaxation, concentration, etc. required to enter a state of a deeper meditative experience, the computer system transforms the currently displayed view of the environment into a more immersive environment, e.g., with an expanded spatial extent (e.g., width, depth, angle, etc.) and visual prominence of the virtual content corresponding to the meditative experience, and a reduced spatial extent and visual prominence of the representation of the physical environment.

In some embodiments, along with increasing the level of immersion in displaying the visual content of the computer-generated experience, the computer system also increases the level of sound suppression in the physical environment perceivable by the user through actions of an audio output device of the computer system and/or increases the level of immersion in the audio content of the computer-generated experience output by the audio output device (e.g., increasing the volume, changing from a stereo audio output mode or a surround sound output mode to a spatial audio output mode, or changing from a stereo audio output mode to a surround sound output mode, etc.).

In some embodiments, the computing system is configured to display the visual components of the CGR content via the display generation component at two or more immersion levels. In some embodiments, the computer system displays the visual components of the CGR content at least a first immersion level, a second immersion level, and a third immersion level. In some embodiments, the computer system displays the visual components of the CGR content at least two immersion levels, providing less immersive and more immersive visual experiences relative to each other, respectively. In some embodiments, the computing system transitions the visual content displayed via the display generation component between different immersion levels in response to biometric data corresponding to the user satisfying different sets of criteria. In some embodiments, the first, second, and third immersion levels correspond to increasing the amount of virtual content corresponding to the CGR experience present in the CGR environment and/or decreasing the amount of a representation of the surrounding physical environment present in the CGR environment. In some embodiments, the first, second, and third immersion levels correspond to different modes of content presentation having increasing image fidelity (e.g., increasing pixel resolution, increasing color resolution, increasing color saturation, increasing luminance, increasing opacity, increasing image detail, etc.) and/or spatial extent (e.g., angular extent, spatial depth, etc.) of the computer-generated content and/or decreasing image fidelity and/or spatial extent of the representation of the surrounding physical environment. In some embodiments, the first immersion level is a pass-through mode, where the physical environment is fully visible to the user through the display generation components (e.g., as a camera view of the physical environment or through transparent or semi-transparent portions of the display generation components). In some embodiments, the visual CGR content presented in the pass-through mode includes a pass-through view of the physical environment with a minimal amount of virtual elements simultaneously visible as the view of the physical environment, or with only virtual elements peripheral to the user's view of the physical environment (e.g., indicators and controls displayed in a peripheral area of the display). For example, the view of the physical environment occupies a central and majority area of the field of view provided by the display generation component, and only a smaller number of controls (e.g., movie title, progress bar, playback controls (e.g., play button), etc.) are displayed in a peripheral area of the field of view provided by the display generation component. In some embodiments, the first level of immersion is a pass-through mode, where the physical environment is fully visible to the first user through the display generation component (e.g., as a camera view of the physical environment or through a transparent portion of the display generation component), and the visual CGR content is displayed in a virtual window or frame, such as overlaid on a representation of the physical environment, replacing a portion of the representation of the physical environment, or blocking the view of a portion of the representation of the physical environment. In some embodiments, the second level of immersion is a mixed reality mode, where the pass-through view of the physical environment is augmented with virtual elements generated by the computer system, where the virtual elements occupy a central and/or majority area of the user's field of view (e.g., the virtual content is integrated with the physical environment in the view of the computer-generated environment). In some embodiments, the second immersion level is a mixed reality mode in which a pass-through view of the physical environment is augmented with a virtual window, viewport, or frame that is overlaid on, replaces the display of, or blocks the view of a portion of the representation of the physical environment, and has additional depth or spatial extent that is revealed when the display generating component is moved relative to the physical environment. In some embodiments, the third immersion level is an augmented reality mode in which virtual content is displayed in the three-dimensional environment along with the representation of the physical environment, and virtual objects are distributed throughout the three-dimensional environment in positions that correspond to different locations of the physical environment. In some embodiments, the third immersion level is a virtual reality mode in which virtual content is displayed in the three-dimensional environment without a representation of the physical environment. In some embodiments, the different immersion levels described above represent increasing levels of immersion relative to each other.

In some embodiments, the computer system selects an audio output mode for outputting audio content of a computer-generated experience (e.g., an application, a communication session, a movie, a video, a game, etc.) according to an immersion level at which visual content of the computer-generated experience is being displayed by the display generation component. In some embodiments, as the immersion level at which the visual content is displayed increases (e.g., from a first immersion level to a second immersion level, from the first immersion level to a third immersion level, or from the second immersion level to the third immersion level, etc.), the computer system switches the audio output mode from an output mode with a lower immersion level to an output mode with a higher immersion level (e.g., the first audio output mode, the second audio output mode, and the third audio output mode corresponding to audio output with increasing immersion levels, such as from the first audio output mode to the second audio output mode, or from the first audio output mode to the third audio output mode, or from the second audio output mode to the third audio output mode, etc.). As described herein, the spatial audio output mode corresponds to a higher level of immersion than the stereo audio output mode and the mono audio output mode. The spatial audio output mode corresponds to a higher level of immersion than the surround sound output mode. The surround sound output mode corresponds to a higher level of immersion than the stereo audio output mode and the mono audio output mode. The stereo audio output mode corresponds to a higher level of immersion than the mono audio output mode. In some embodiments, the computer system selects an audio output mode from a plurality of available audio output modes, e.g., a mono audio output mode, a stereo audio output mode, a surround sound output mode, a spatial audio output mode, etc., based on a level of immersion at which the computer-generated experience's visual content is provided via the display generation component.

Figures 7K-7M are block diagrams illustrating the aggregation of effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

In some embodiments, the computer system provides multiple types of sensory modulation features that enhance the user's ability to perceive various aspects of the physical environment that may not be readily perceived without the assistance of special equipment or the computer system. Instead of only allowing the user to use a single type of sensory modulation feature when viewing a portion of the physical environment at a time, the computer system aggregates the effects of two or more types of sensory enhancement features on the representation of a portion of the physical environment such that features and characteristics present in a portion of the physical environment that were previously hidden within the view of the physical environment provided by the computer system may be revealed.

In some embodiments, when a computer system displays a three-dimensional environment including a representation of a physical environment via a display generation component (e.g., display generation component 7100 or another type of display generation component such as an HMD), the computer system optionally uses sensor input or information corresponding to a currently displayed portion of the physical environment to enhance and adjust the representation of the physical environment, thereby allowing a user to perceive a portion of the physical environment using sensory information that is not available to the user when the user views the portion of the physical environment without the assistance of the display generation component.

In FIG. 7K, the computer system displays a view 7340 of a three-dimensional environment including a first representation of a first portion of the physical environment. In the view 7340, the first representation of the first portion of the physical environment corresponds to an appearance of the first portion of the physical environment without sensory adjustments made by the computer system. In some embodiments, the first representation of the first portion of the physical environment corresponds to a view of the first portion of the physical environment captured by a color camera having a first level of imaging sensitivity corresponding to average color and intensity detection within the range of normal human perception. In some embodiments, the first representation of the first portion of the physical environment corresponds to a view of the first portion of the physical environment through a transparent portion of the display generation component and is not augmented or adjusted by the computer system.

In some embodiments, the computer system provides a plurality of affordances (e.g., hardware controls 7354, 7356, and 7358, user interface elements displayed in the three-dimensional environment, etc.) for activating each of the plurality of sensory adjustment functions provided by the computer system. In some embodiments, the computer system activates each of the plurality of sensory adjustment functions in sequence or in combination according to a user activation input (e.g., a button press input, a tap input, a gesture input, a touch input, a gaze input, a selection input, combinations thereof, etc.) directed at an affordance corresponding to each of the plurality of sensory adjustment functions. In some embodiments, each one of the plurality of sensory adjustment functions is optionally activated by a preset input (e.g., a gesture input, a touch input, a voice command, etc.) without requiring the presence of a corresponding hardware affordance associated with the computer system or a corresponding user interface control in the three-dimensional environment.

In this example, as shown in FIG. 7K, the first representation of the first portion of the physical environment includes a view from inside the room toward a window on a wall of the room. This example is non-limiting, and according to various embodiments, the first portion of the physical environment can be any indoor or outdoor environment. In this example, the first representation of the first portion of the physical environment includes a representation 7344' of a wall, a representation 7346' of a window, a representation 7348' of a hill outside the window at a first distance from the window, and a representation 7350' of a tree near the top of the hill a second distance away from the window. Because the hills and trees are at a large distance from the view generation component and the hill representation 7348' and the tree representation 7350' are far away from the viewpoint corresponding to the currently displayed view of the three-dimensional environment (e.g., the respective distances between the viewpoint and the representations 7348' and 7350' correspond to the respective distances from the user's eyes (or the view generation component) to the hills and trees), the hill representation 7348' and the tree representation 7350' occupy a small portion of the field of view provided by the view generation component 7100.

In FIG. 7L, the computer system detects a user input that activates a first sensory adjustment feature of a plurality of sensory adjustment features provided by the computer system. For example, the computer system detects that a hardware affordance 7354 has been activated by a user's input, that a user interface object corresponding to the first sensory adjustment feature has been activated or selected by a user's input, that a gesture input, a voice command, and/or a touch input, etc., that meets criteria for activating the first sensory adjustment feature has been provided by the user, etc. In response, the computer system displays a second view 7361 of the three-dimensional environment including a second representation of a second portion of the physical environment, the second portion of the physical environment being contained within the first portion of the physical environment (e.g., all or a portion of the first portion of the physical environment shown in FIG. 7K, or the portion of the physical environment shown prior to detection of the input that activated the first sensory adjustment feature). In the second view 7361 of the three-dimensional environment, as shown in the example of Figure 7L, the display characteristics of the representation 7350" of the tree are adjusted relative to the representation 7350' of the tree shown in the first view 7340 of the three-dimensional environment in accordance with the operation of the first sensory adjustment function. For example, if the first sensory adjustment function is a simulated telescopic view that reduces the focal length of the object so that the object appears closer to the user, as shown in Figure 7L, the representation 7350" of the tree will appear to be located much closer to the viewpoint than at the second distance, as shown in Figure 7K (e.g., the adjusted distance is one-fifth of the second distance, the adjusted distance is one-tenth of the second distance, the adjusted distance is a distance selected based on a preset percentage of the second distance and/or the maximum magnification of the simulated telescopic view, etc.). Similarly, the representation 7348'' of the hill also appears to be located much closer to the user than the first distance, as shown in FIG. 7K (e.g., the adjusted distance is one-fifth the first distance, the adjusted distance is one-tenth the first distance, the adjusted distance is a distance selected based on a preset percentage of the first distance and/or the maximum magnification of the simulated telescope function, etc.). In this example, the user's viewpoint or virtual position in view 7361 is moved to the position of the window in view 7340, according to some embodiments. In this example, the user's viewpoint or virtual position in view 7361 is still based on the actual location of the user and/or the display generation component in the physical environment, according to some embodiments.

In some embodiments, when applying the first sensory adjustment function, the computer selects a target portion of the physical environment based on the location of the user's gaze directed at the current view of the three-dimensional environment. For example, as shown in FIG. 7K, the computer system detects that the user's gaze 7352 is directed at a representation 7350' of a tree in the first view 7340 of the three-dimensional environment and selects a portion of the physical environment that includes the tree from the first portion of the physical environment as the second portion of the physical environment to which the first sensory adjustment function is applied.

In some embodiments, the simulated telescopic vision is an illustrative example of a first type of sensory accommodation function provided by the computer system, and may be replaced by another type of sensory accommodation function provided by the computer system and selected by user input.

In FIG. 7M, while the computer system is displaying a second view 7361 of the three-dimensional environment including a second representation of the physical environment adjusted according to the operation of the first sensory adjustment feature activated by the user's input, the computer system detects a second user input that activates a second sensory adjustment feature of the plurality of sensory adjustment features that is different from the first sensory adjustment feature. For example, the computer system detects that a hardware affordance 7356 has been activated by the user's input, that a user interface object corresponding to the second sensory adjustment feature has been activated or selected by the user's input, that a gesture input, a voice command, and/or a touch input, etc., that meets the criteria for activating the second sensory adjustment feature has been provided by the user, etc. In response, the computer system displays a third view 7364 of the three-dimensional environment including a third representation of a third portion of the physical environment, the third portion of the physical environment being included within the second portion of the physical environment (e.g., all or a portion of the second portion of the physical environment shown in FIG. 7L, or the portion of the physical environment shown prior to the detection of the input that activated the second sensory adjustment feature, etc.). In a third view 7364 of the three dimensional environment, as shown in the example of FIG. 7M , the display characteristics of the tree representation 7350'' are further adjusted relative to the tree representation 7350'' shown in the second view 7361 of the three dimensional environment in accordance with operation of the second sensory adjustment function. For example, if the second sensory adjustment function is a simulated heat vision that presents variations in color and/or intensity according to variations in temperature and/or thermal radiation as shown in FIG. 7M , the tree representation 7350''' will appear to have a different color and/or intensity relative to the background environment in the third view 7364, and the display characteristics of portions 7366''' and 7368''' of representation 7350''' will be further adjusted based on the temperature of those portions of the tree relative to other portions of the tree in the physical environment (e.g., as detected by a thermal imaging sensor or other sensor in communication with the computer system, as indicated by thermal data transmitted to the computer system or retrieved by the computer system from another computer system, etc.). For example, the higher temperatures of these portions represented by portions 7366''' and 7368''' are more likely to reveal small animals or objects that radiate more heat or have a higher temperature than the tree itself. Portions 7366''' and 7368''' in representation 7350''' have display characteristics that are generated based on the operation of both the first sensory modulation function and the second sensory modulation function on the original first representation 7350' of the tree, as shown in FIG. 7K.

In some embodiments, when applying the second sensory adjustment function, the computer system selects a target portion of the physical environment based on the location of the user's gaze directed toward the currently displayed view of the three-dimensional environment. For example, as shown in FIG. 7L, the computer system detects that the user's gaze 7360 is directed toward a representation 7350'' of a tree in the second view 7361 of the three-dimensional environment and selects a portion of the physical environment that includes the tree from the second portion of the physical environment as a third portion of the physical environment to which both the first sensory adjustment function and the second sensory adjustment function are applied.

In some embodiments, simulated heat vision is an illustrative example of a second type of sensory modulation functionality provided by the computer system, and may be replaced by another type of sensory modulation functionality provided by the computer system and selected by user input.

In some embodiments, a first display characteristic (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) is adjusted relative to a baseline representation of the individual portion of the physical environment (e.g., tree representation 7350' of FIG. 7K corresponding to portions 7366''', 7368''' of FIG. 7M, another portion of the physical environment, etc.) according to a first type of computer-generated sensory adjustment (e.g., binocular vision, telescopic vision, microscopic vision, night vision, thermal vision, etc.) to generate a first adjusted representation of the individual portion of the physical environment (e.g., portions 7366''', 7368''', of tree representation 7350' of FIG. 7M, another portion of the physical environment, etc.). 7L corresponding to portions 7366''', 7368''', or a further adjusted representation of another portion of the physical environment, and second display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) are adjusted for the second representation of the physical environment according to the second type of computer-generated sensory adjustment to obtain a third representation of the respective portion of the physical environment (e.g., portions 7366''', 7368''', or a further adjusted representation of another portion of the physical environment, etc.). In some embodiments, the second display characteristics have the same value in the first representation and the second representation in some combinations of the first and second types of sensory adjustment functions, and the second display characteristics have different values in the first representation and the second representation in some combinations of the first and second types of sensory adjustment functions.

In some embodiments, the computer system enables the representation of the physical environment to be further adjusted based on a third sensory adjustment feature (e.g., an affordance 7358 corresponding to the third sensory adjustment feature, a sensory adjustment feature that can be activated by interaction with a user interface object, a gestural input, a voice command, etc.). In some embodiments, while displaying a third view 7364 of the three-dimensional environment including the third representation of the physical environment, the computer system detects a third user input corresponding to a request to activate a third type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.) that is distinct from the first and second types of sensory adjustment features. In response, the computer system displays a fourth view of the three-dimensional environment including a fourth representation of a fourth portion of the physical environment (e.g., all or a portion of a third portion of the physical environment), the fourth representation of the physical environment having first display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the first representation of the fourth portion of the physical environment according to a first type of sensory adjustment function, second display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the second representation of the fourth portion of the physical environment according to a second type of sensory adjustment function, and third display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the third representation of the physical environment of the fourth portion of the physical environment according to a third type of sensory adjustment function.

In some embodiments, the first sensory adjustment function includes simulated telescopic viewing (e.g., binocular, monocular, telescopic, etc.) for viewing distant physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user), and the second sensory adjustment function includes simulated microscopic viewing for magnifying nearby physical objects.

In some embodiments, the first sensory adjustment function includes simulated telescopic vision for viewing remote physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user) and the second sensory adjustment function includes simulated night vision for viewing physical objects under low light conditions (e.g., increased sensitivity in low light conditions, the brightness of an object is visually enhanced, small variations in brightness are magnified, etc.).

In some embodiments, the first sensory adjustment function includes simulated telescoping for viewing a remote physical object (e.g., reducing the focal length of the object so that the object appears closer to the user), and the second sensory adjustment function includes modifying the view of the physical object with filters (e.g., color filters, light frequency filters, intensity filters, motion filters, etc.).

In some embodiments, the first sensory adjustment function includes simulated telescopic viewing for viewing distant physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user), and the second sensory adjustment function includes selective sound enhancement (e.g., boosting the volume, selectively enhancing/suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object that is centered in the current field of view, etc.).

In some embodiments, simultaneously with displaying the third representation of the physical environment, the computer system outputs sounds corresponding to the portion of the physical environment that is visible in the third representation of the physical environment, and the sounds are selectively enhanced (e.g., increasing the volume, modifying the amplitude of some selected frequencies, etc.) relative to sounds from sources outside the portion of the physical environment.

In some embodiments, concurrently with displaying the third representation of the physical environment, the computer system displays text output corresponding to speech emanating from a portion of the physical environment that is visible in both the second and third representations of the physical environment, the speech being selectively enhanced relative to sounds from a sound source outside the portion of the physical environment.

In some embodiments, the first sensory modulation function includes simulated microscopic vision to magnify nearby physical objects, and the second sensory modulation function includes simulated thermal vision to view physical objects having different thermal emission profiles (e.g., high sensitivity to temperature variations, presenting variations in color and/or intensity according to temperature and/or thermal emission variations, etc.).

In some embodiments, the first sensory adjustment function includes simulated night vision for viewing physical objects in low light conditions (e.g., high sensitivity in low light conditions, the brightness of objects is visually enhanced, small variations in brightness are magnified, etc.), and the second sensory adjustment function includes simulated telescopic vision for viewing remote physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user).

In some embodiments, the first sensory adjustment function includes simulated night vision for viewing physical objects in low light conditions (e.g., high sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.), and the second sensory adjustment function includes simulated microscopic vision for magnifying nearby physical objects.

In some embodiments, the first sensory adjustment function includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, the brightness of the object is visually enhanced, small variations in brightness are magnified, etc.), and the second sensory adjustment function includes simulated heat vision for viewing physical objects having different thermal radiation profiles (e.g., high sensitivity to variations in temperature, presenting variations in color and/or intensity in response to variations in temperature and/or thermal radiation, etc.).

In some embodiments, the first sensory modulation function includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, luminance of objects is visually enhanced, small variations in luminance are magnified, etc.), and the second sensory modulation function and the second type of computer-generated sensory modulation includes selective sound enhancement (e.g., boosting volume, selectively enhancing/suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object that is centered in the current field of view, etc.).

In some embodiments, the first sensory adjustment function includes simulated thermal vision for viewing physical objects having different thermal radiation profiles (e.g., high sensitivity to temperature variations, presenting color and/or intensity variations according to temperature and/or thermal radiation variations, etc.), and the second sensory adjustment function includes simulated telescopic vision for viewing remote physical objects (e.g., reducing the focal length of the object so that it appears closer to the user).

In some embodiments, the first sensory modulation function includes simulated thermal vision for viewing physical objects having various thermal radiation profiles (e.g., high sensitivity to temperature variations, presenting color and/or intensity variations according to temperature and/or thermal radiation variations, etc.), and the second sensory modulation function includes simulated microscopic vision for magnifying nearby physical objects.

In some embodiments, the first sensory adjustment operation includes simulated heat vision for viewing physical objects having various thermal radiation profiles (e.g., increased sensitivity to temperature variations, presenting variations in color and/or intensity according to temperature and/or thermal radiation variations, etc.), and the second sensory adjustment operation includes simulated night vision for viewing physical objects under low light conditions (e.g., increased sensitivity in low light conditions, luminance of objects is visually enhanced, small variations in luminance are magnified, etc.).

In some embodiments, the first sensory modulation function includes simulated thermal vision for viewing physical objects having various thermal emission profiles (e.g., high sensitivity to temperature variations, presenting variations in color and/or intensity according to temperature and/or thermal emission variations, etc.), and the second sensory modulation action includes selective sound enhancement (e.g., enhancing the volume, selectively enhancing/suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object that is centered in the current field of view, etc.).

In some embodiments, the order in which multiple selected sensory adjustment features selected by the user are applied to the baseline representation of a portion of the physical environment is adjusted by the computer system based on one or more pre-set constraints, and optionally differs from the order in which these sensory adjustment features are activated by the user. For example, in some embodiments, adjustments corresponding to simulated telescopic viewing are performed before adjustments corresponding to other types of sensory adjustments in order to reduce the portions of the physical environment in which other types of sensory adjustments need to be performed in order to present the final result to the user. In some embodiments, the computer system observes the order in which different types of sensory adjustment features are activated by the user and presents intermediate results obtained in response to each additional sensory adjustment activated by the user.

7N-7P are block diagrams illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment pursuant to a determination that a portion of the physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise, according to some embodiments.

In some embodiments, the computer system displays virtual content (e.g., virtual open water 7406, virtual hiking trail 7412, etc.) (e.g., virtual scenery, visual and functional enhancements to exercise equipment, user interfaces, health and scoreboards, etc.) corresponding to a particular type of exercise (e.g., rowing, hiking, etc.) pursuant to a determination that a physical location (e.g., location of physical object 7404, location of physical object 7402, etc.) represented in a view of the three-dimensional environment (e.g., view 7408, view 7410, etc.) is associated with a particular type of exercise (e.g., rowing, hiking, etc.). For example, as the user and display generating component (e.g., user 7002 and display generating component 7100, or another user having another type of display generating component such as an HMD, etc.) move between locations in the real world (e.g., within scene 105, or within another physical environment, etc.), the virtual content shown in the view of the three-dimensional environment is adjusted to correspond to the type of exercise associated with the current location of the user and display generating component. In some embodiments, if a location is associated with multiple types of exercises, the computer system selects a type of exercise from the multiple types of exercises associated with the location based on other contextual information (e.g., the user's movements, the user's engagement with objects at the location, etc.) and displays visual content corresponding to the selected type of exercise.

Part (A) of FIG. 7N shows that a user 7002 is located in a physical environment (e.g., scene 105, or another physical environment, etc.). The user 7002 may be located in different physical environments, which may be an outdoor environment or an indoor environment, or may move between an indoor environment and an outdoor environment. The user 7002 views the physical environment through a field of view provided via a first display generating component (e.g., display generating component 7100, another type of display generating component such as an HMD, etc.). The physical environment includes physical surfaces (e.g., walls 7004 and 7006, floor 7008, other physical surfaces, etc.) and one or more physical objects (e.g., exercise equipment 7402, 7404, other physical objects, etc.). In some embodiments, the physical environment is a building that includes multiple rooms or sections that are separate from one another that cannot be viewed by the user simultaneously. In some embodiments, the physical environment includes multiple areas that are separate from one another, such as rooms within individual buildings, different parks, different geographic areas, etc. In some embodiments, the physical environment is an outdoor environment including outdoor physical objects and surfaces such as roads, trees, sky, open water, rocks, mountains, vehicles, animals, people, etc. In some embodiments, the computer system stores information and/or implements rules and artificial intelligence to determine one or more types of exercises (e.g., indoor exercises, indoor sports, outdoor exercises, outdoor sports, physical activities that promote health and physical performance, restorative training and therapy, etc.) associated with individual locations within the user's physical environment (e.g., within the user's field of view through the display generation component, within a threshold vicinity of the user (e.g., within 5 meters, within a few steps, etc.), etc.). In some embodiments, the computer system determines the type of exercise associated with the individual locations based on the type of physical objects present at the individual locations. In some embodiments, the computer system determines the type of exercise associated with the individual locations based on the type of environment or setting present at the individual locations. In some embodiments, the computer system determines the type of exercise associated with the individual locations based on other types of markers and signals, or a combination of information present at the individual locations.

In part (B) of FIG. 7N, the computer system displays a first view 7405 of a three-dimensional environment including a representation of the physical environment of the user 7002. In some embodiments, the first view 7405 of the three-dimensional environment is a real view without or with minimal virtual elements, as shown in FIG. 7N(B). In this example, the first view 7405 includes representations of physical surfaces (e.g., representations 7004' and 7006' of walls 7004 and 7006, representation 7008 of floor 7008, etc.) and representations of physical objects (e.g., representation 7402' of physical object 7402, representation 7404' of physical object 7404, etc.) without virtual content. In some embodiments, the first view of the three-dimensional environment is a real view (e.g., view 7405 shown in FIG. 7N(B)) with user interface objects for controlling basic functions of the computer system (e.g., application icons for launching various computer-generated experiences, display settings, audio controls, etc.). In some embodiments, the first view of the three-dimensional environment is an augmented reality view that is displayed at a low level of immersion (e.g., displays user interface objects (e.g., application launch pad, welcome user interface, settings user interface) that are not part of a particular application experience (e.g., wellness application, meditation application, workout application, gaming application, etc.) and that as a whole occupy only a small percentage (e.g., less than 10%, less than 20%) of the user's field of view or are displayed in a limited floating window, etc.). In some embodiments, the representation of the physical environment included in the first view of the three-dimensional environment 7405 is a camera view of a portion of the physical environment. In some embodiments, the portion of the physical environment shown in the first view of the three-dimensional environment 7405 changes as the user moves around the physical environment (e.g., when the user is wearing a display generating component on his or her head or holding a display generating component in his or her hand, etc.). In some embodiments, the portion of the physical environment shown in the first view of the three-dimensional environment 7405 changes as the display generating component moves around the physical environment. In some embodiments, the representation of the physical environment included in the first view 7405 of the three-dimensional environment is a view of the physical environment through a transparent portion of the display generation component. In this example, the physical object 7402 is located at a first location within the first portion of the physical environment shown in the first view 7405 of the three-dimensional environment, and the physical object 7404 is located at a second location within the first portion of the physical environment shown in the first view 7405 of the three-dimensional environment. In some embodiments, the first location and the second location are not necessarily in the same view of the three-dimensional environment, but may be located in two separate locations within the same physical environment, or in different physical environments completely separated from each other. In this example, the physical object 7402 corresponds to equipment or a setting corresponding to a first type of exercise (e.g., running, walking, etc.), and the physical object 7402 corresponds to equipment or a setting corresponding to a second type of exercise (e.g., rowing, boating, water skiing, etc.).

In Figures 7O and 7P, the computer system detects movement of a user 7002 within the physical environment while displaying a first view 7450 of the three-dimensional environment. In some embodiments, the portions of the physical environment that are visible in the first view of the three-dimensional environment change as the user moves around in the physical environment. Figure 7O illustrates a first scenario in which the user 7002 has moved to a first location that includes a physical object 7404 or setting that corresponds to a first type of exercise. Figure 7O illustrates a second scenario in which the user 7002 has moved to a second location that includes a physical object 7402 or setting that corresponds to a second type of exercise.

In some embodiments, the user's movement includes movement of the entire user to a separate location (e.g., a first location including the first physical object 7404, a second location including the second physical object 7402, etc.) (e.g., while the user is holding or wearing the display generating component, while the spatial relationship between the display generating component and the user is maintained such that the user can continue to view the physical environment through the display generating component, etc.). In some embodiments, the user's movement includes movement of the user to orient the display generating component or a camera associated with the display generating component to capture a view of the separate location (e.g., a first location including the first physical object 7404, a second location including the second physical object 7402, etc.) (e.g., while the user is holding or wearing the display generating component, while the spatial relationship between the display generating component and the user is maintained such that the user can continue to view the physical environment through the display generating component, etc.). In some embodiments, the user's movements further include movements corresponding to the manipulation of a physical object(s) at a discrete location (e.g., turning on the exercise equipment at a discrete location, picking up the exercise equipment at a discrete location, beginning to use the exercise equipment at a discrete location, etc.).

7O, the user has moved to a first location in a physical environment that includes a physical object 7404 corresponding to a first type of exercise. In this example, the user has also moved to a position relative to the physical object 7404 that allows the user to begin using the physical object 7404 for the first type of exercise (e.g., sitting on the equipment, standing on the equipment, holding one or more portions of the equipment, etc.). In some embodiments, the computer system optionally detects that the user has begun one or more repetitions of a movement of an exercise corresponding to the first type of exercise (e.g., rowing an oar, pulling a shift lever, assuming a starting position, etc.). In response to detecting movement of the first user to a first location including a physical object 7404 corresponding to the first type of exercise, and optionally in accordance with a determination that the first location corresponds to the first type of exercise, and a determination that the user's movement satisfies a first set of criteria (e.g., criteria corresponding to the first location, criteria corresponding to the first type of exercise, etc.), the computer system displays a second view 7408 of the three-dimensional environment, the second view 7408 including a first virtual content corresponding to the first type of exercise, the view of the first virtual content replacing at least a portion of the view of the physical environment including the first location (e.g., a location including the physical object 7404 but not the physical object 7402, a location not corresponding to the second type of exercise, etc.). In some embodiments, the first virtual content completely replaces the view of the physical environment in the second view 7408 of the three-dimensional environment. In some embodiments, the virtual content is displayed overlaying, occluding, or replacing the display of the representation of the physical environment in the second view 7408 of the three-dimensional environment.

In some embodiments, the computer system determines that the first location corresponds to a first type of exercise according to a determination that the first location has a first type of exercise equipment (e.g., a rowing machine, a boat, etc.) corresponding to the first type of exercise. In some embodiments, the computer system determines that the first location corresponds to a first type of exercise according to a determination that the first location is a location designed (e.g., has a suitable floor surface, structure, etc.) for the first type of exercise (e.g., rowing, meditation, etc.).

As shown in part (B) of FIG. 7O, the computer system displays a second view 7408 of the three-dimensional environment when the user 7002 moves to a first location corresponding to the first type of exercise. In some embodiments, the second view 7408 is an augmented reality view having more virtual elements corresponding to the first location and the first computer-generated experience corresponding to the first type of exercise. In some embodiments, the second view 7408 is an augmented reality view showing a preview or start of the first computer-generated experience corresponding to the first location and the first type of exercise. In some embodiments, the second view 7408 is an augmented reality view displayed at a higher level of immersion (e.g., displays user interface objects that are part of the first particular application experience corresponding to the first type of exercise (e.g., virtual hiking trails, virtual scenery, leaderboards, exercise statistics, controls to change exercise parameters, etc.) and occupies a significant percentage (e.g., more than 60%, more than 90%, etc.) of the user's field of view as a whole or is displayed in the three-dimensional virtual or augmented reality environment. In this example, the virtual content displayed in the second view of the three-dimensional environment includes a virtual open body of water 7406 that replaces the view of representations 7004', 7006', and/or 7008' of various portions of the physical environment potentially within the field of view provided by the display generation component of the first location. In some embodiments, all portions of the physical environment within the potential field of view provided by the display generation component are replaced or blocked by the display of the virtual content. In some embodiments, portions of the physical environment, such as parts of the user's body, at least a portion of the exercise equipment, etc., remain visible in the second view 7408 of the three-dimensional environment.

7P, the user has moved to a second location in the physical environment that includes a physical object 7402 corresponding to a second type of exercise. In this example, the user has also moved to a position relative to the physical object 7402 that allows the user to begin using the physical object 7402 for the second type of exercise (e.g., sitting on the equipment, standing on the equipment, holding one or more portions of the equipment, etc.). In some embodiments, the computer system optionally detects that the user has begun one or more repetitions of a movement corresponding to the second type of exercise (e.g., taking a step, starting to pedal, starting to walk, etc.). In response to detecting movement of the user to a second location including a physical object 7402 corresponding to the second type of exercise, and in accordance with a determination that the second location corresponds to the second type of exercise, and optionally a determination that the user's movement satisfies a second set of criteria (e.g., criteria corresponding to the second location, criteria corresponding to the second type of exercise, etc.), the computer system displays a third view 7412 of the three-dimensional environment, the third view 7412 including a second virtual content corresponding to the second type of exercise, the view of the second virtual content replacing at least a portion of the view of the physical environment including the second location (e.g., a location corresponding to the second type of exercise but not the first type of exercise, a location not including the physical object 7404, etc.). In some embodiments, the first virtual content completely replaces the view of the physical environment in the third view 7410 of the three-dimensional environment. In some embodiments, the virtual content is displayed overlaying, occluding, or replacing the display of at least a portion of the representation of the physical environment.

In some embodiments, the computer system determines that the second location corresponds to a second type of exercise according to a determination that the second location has a second type of exercise equipment (e.g., stairs, stepper, treadmill, etc.) corresponding to the second type of exercise. In some embodiments, the computer system determines that the second location corresponds to a second type of exercise according to a determination that the second location is a location designed (e.g., has a suitable floor surface, structure, etc.) for the second type of exercise (e.g., hiking, running, etc.).

As shown in part (B) of FIG. 7P, the computer system displays a third view 7410 of the three-dimensional environment when the user 7002 moves to a second location corresponding to the second type of exercise. In some embodiments, the third view 7410 is an augmented reality view with more virtual elements corresponding to the second location and the second computer-generated experience corresponding to the second type of exercise. In some embodiments, the third view 7410 is an augmented reality view showing a preview or start of the second computer-generated experience corresponding to the second location and the second type of exercise. In some embodiments, the third view 7410 is an augmented reality view displayed at a higher level of immersion (e.g., displaying user interface objects that are part of a second specific application experience (e.g., virtual hiking trails, virtual scenery, leaderboards, exercise statistics, controls to change exercise parameters, etc.) corresponding to the second type of exercise and that occupy a significant percentage (e.g., more than 60%, more than 90%, etc.) of the user's field of view as a whole or that are displayed in the three-dimensional virtual or augmented reality environment). In this example, the virtual content displayed in the third view 7410 of the three-dimensional environment includes a virtual hiking trail 7412 that replaces the views of the representations 7004', 7006', and/or 7008' of various portions of the physical environment potentially within the field of view provided by the display generation component at the second location. In some embodiments, all portions of the physical environment within the potential field of view provided by the display generation component are replaced or blocked by the display of the virtual content. In some embodiments, portions of the physical environment, such as parts of the user's body, at least a portion of the exercise equipment, etc., remain visible in the third view 7410 of the three-dimensional environment.

In some embodiments, the computer system determines that the current location corresponds to a particular type of exercise according to detecting a particular type of exercise equipment corresponding to the particular type of exercise at the current location. In some embodiments, detecting the particular type of exercise equipment is based on detecting an RFID signal corresponding to the particular type of exercise equipment, detecting an image of the particular type of exercise equipment in a camera feed capturing the current location, detecting that the current location matches a registered location of the particular type of exercise equipment, etc.

In some embodiments, pursuant to a determination that the user's current location corresponds to a location associated with the individual type of exercise, the computer system displays a view of the three-dimensional environment corresponding to the individual type of exercise, which includes gradually reducing the visual prominence of a representation of the physical environment in the currently displayed view of the three-dimensional environment while increasing the visual prominence of virtual content corresponding to the individual type of exercise associated with the current location in the view of the three-dimensional environment. In some embodiments, reducing the visual prominence of the representation of the physical environment includes ceasing to display more and more portions of the representation of the physical environment, fading out the representation of the physical environment, etc. In some embodiments, gradually increasing the visual prominence of the virtual content corresponding to the individual type of exercise includes beginning to display the virtual content in the region of the view of the three-dimensional environment where the representation of the physical environment is gradually reduced, increasing the visibility of the virtual content, increasing a percentage of the user's field of view occupied by the virtual content, increasing the opacity or brightness of the virtual content, etc.

In some embodiments, the individual locations may correspond to multiple types of exercises, and the computer system requires the user to make some movement corresponding to a respective one of the multiple types of exercises to clarify which type of exercise the user wants to perform, and selects corresponding virtual content for display in the view of the three-dimensional environment at the individual location. For example, in some embodiments, the computer system detects a movement corresponding to a respective one of the multiple types of exercises associated with the individual location (e.g., the start of a characteristic motion (e.g., starting to walk on a treadmill, stepping on a stair stepper, moving legs back and forth on an elliptical, or starting rowing on a rowing machine, etc.), stepping/sitting on exercise equipment corresponding to the individual type of exercise (e.g., sitting on a rowing machine or weight training machine, etc.), a preparation position corresponding to the individual type of exercise (e.g., standing up in a preparation position to hit a virtual tennis ball, sitting on the floor to start meditation or yoga, etc.), and the computer system displays a view of the three-dimensional environment including virtual content corresponding to the individual type of exercise.

In some embodiments, the computer system gradually changes the virtual content displayed in the view of the three-dimensional environment according to the progress of the individual type of exercise performed by the user at the individual location. For example, in some embodiments, the view of the real world gradually disappears and/or is stopped from being displayed and gradually replaced by the virtual content corresponding to the individual type of exercise. In some embodiments, the computer system gradually increases the amount of virtual content displayed in the first user's field of view until the individual virtual environment corresponding to the individual type of exercise is fully displayed via the first display generation component (e.g., the second view of the three-dimensional environment includes the virtual environment corresponding to the first type of exercise, the third view of the three-dimensional environment includes the virtual environment corresponding to the second type of exercise, etc.). For example, in some embodiments, when the open gym is a location associated with both yoga and dance, after the first user arrives at the open gym, if the first user sits in a namaste pose, the computer system displays a virtual ocean view with ocean sounds for the user to practice yoga on a virtual beach, and if the first user stands in a dancer's pose, the computer system displays a virtual stage with dance music for the user to practice dance.

In some embodiments, when the computer system detects that the user has left the individual location, the computer system stops displaying the virtual content corresponding to the type of exercise associated with the individual location. For example, in FIG. 7O, if the computer system detects that the user 7002 has left the first location including the physical object 7404, after the view 7408 is displayed, the computer system stops displaying the view 7408 corresponding to the first type of exercise. In some embodiments, the computer system redisplays the view 7405, which does not include the virtual content corresponding to either the first type of exercise or the second type of exercise. In some embodiments, when the computer system detects that the user has moved from the first location to the second location, the computer system displays the virtual content 7410 corresponding to the second type of exercise.

In some embodiments, when the computer system displays a view of the three-dimensional environment including virtual content corresponding to the particular type of exercise, it displays status information corresponding to the particular type of exercise (e.g., progress during the current session, duration, speed, force, height, pace, stride length, performance level, score, number of repetitions completed, etc., historical statistics, average statistics for the first user and/or across multiple users, status of other users performing the same type of exercise, etc.).

In some embodiments, the computer system displays corresponding health information to the user (e.g., real-time biometric data (e.g., heart rate, blood pressure, respiratory rate, body temperature, blood glucose, etc.), weight, BMI, etc.) when displaying a view of a three-dimensional environment including virtual content corresponding to a particular type of exercise.

In some embodiments, the computer system visually presents progress information (e.g., real-time score, laps completed, laps remaining, duration, steps, distance traveled, completed poses, etc.) of a particular type of exercise performed by the user when displaying a view of a three-dimensional environment including virtual content corresponding to the particular type of exercise.

In some embodiments, the three-dimensional environment including the virtual content corresponding to the particular type of exercise is an immersive environment and includes a spatial extent that is greater than the spatial extent included in the currently displayed view of the three-dimensional environment. For example, as the user turns his/her head or otherwise changes the viewpoint that corresponds to the currently displayed view of the three-dimensional environment, different portions of the virtual content are displayed in the currently displayed view of the three-dimensional environment.

In some embodiments, the second and/or third views of the three-dimensional environment include a virtual representation of the user shown competing with the user to perform a particular type of exercise (e.g., based on the first user's previous best performance, based on the first user's pre-set configuration for the first type of exercise, etc.).

In some embodiments, the second and/or third views of the three-dimensional environment include a virtual representation of at least another user, different from the user, who is shown competing with the user to perform a particular type of exercise.

As disclosed herein, in some embodiments, the three-dimensional environment displayed via the display generation component is a virtual three-dimensional environment that includes virtual objects and content at various virtual positions within the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at various virtual positions within the three-dimensional environment that are constrained by one or more physical aspects of the physical environment (e.g., positions and orientations of walls, floors, surfaces, direction of gravity, time of day, etc.). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. The representation of the physical environment includes respective representations of physical objects and surfaces at different positions within the three-dimensional environment such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. When the virtual objects are positioned relative to the positions of the representations of the physical objects and surfaces in the three-dimensional environment, the virtual objects appear to have corresponding spatial relationships with the physical objects and surfaces in the physical environment. In some embodiments, the display generation component includes a pass-through portion in which the representation of the physical environment is displayed. In some embodiments, the pass-through portion is a transparent or semi-transparent (e.g., see-through) portion of the display generation component that surrounds the user's field of view and reveals at least a portion of the physical environment in the user's field of view. For example, the pass-through portion is a portion of a head-mounted display that is semi-transparent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent, allowing the user to view the real world surrounding the user through the pass-through portion without removing the head-mounted display or moving away from the head-up display. In some embodiments, the pass-through portion gradually transitions from semi-transparent or transparent to fully opaque when displaying a virtual or mixed reality environment. In some embodiments, the pass-through portion of the display generation component displays a live feed of images or video of at least a portion of the physical environment captured by one or more cameras (e.g., rear-facing camera(s) of a mobile device or associated with the head-mounted display, or other cameras that provide image data to an electronic device). In some embodiments, the one or more cameras are aimed at a portion of the physical environment that is directly in front of the user (e.g., behind the display generation component). In some embodiments, one or more cameras are directed at a portion of the physical environment that is not directly in front of the user (e.g., in a different physical environment, or to the side or behind the user). In some embodiments, when displaying virtual objects or content in positions that correspond to the location of one or more physical objects in the physical environment, at least some of the virtual objects are displayed in place of (e.g., replace the display of) a portion of the live view of the camera (e.g., a portion of the physical environment captured in the live view). In some embodiments, at least some of the virtual objects and content are projected onto physical surfaces or open space in the physical environment and are visible through a pass-through portion of the display generation component (e.g., as part of a camera view of the physical environment, or visible through a transparent or semi-transparent portion of the display generation component, etc.). In some embodiments, at least some of the virtual objects and content are displayed to overlay a portion of the display, blocking the view of at least a portion, but not all, of the physical environment that is visible through the transparent or semi-transparent portion of the display generation component. In some embodiments, at least some of the virtual objects are projected directly onto the user's retina in a position relative to an image of a representation of the physical environment (e.g., as viewed through a camera view of the physical environment or through a transparent portion of a display generation component).

In some embodiments, the input gestures used in the various examples and embodiments described herein (e.g., with respect to FIGS. 7A-7P and 8-12) optionally include discrete, small movement gestures performed by moving a user's finger(s) relative to another finger(s) or a portion(s) of the user's hand, optionally without requiring the user to significantly move the user's entire hand or arm away from their natural location(s) and posture(s) to perform an action immediately before or during the gesture, according to some embodiments, to interact with a virtual or mixed reality environment.

In some embodiments, the input gesture is detected by analyzing data and signals captured by a sensor system (e.g., sensor 190 of FIG. 1, image sensor 314 of FIG. 3). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, a depth camera, etc.). For example, the one or more imaging sensors are components of or provide data to a computer system (e.g., computer system 101 of FIG. 1 (e.g., portable electronic device 7100 or HMD)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a touch screen display that functions as a display and a touch-sensitive surface, a stereoscopic display, a display with a pass-through portion, etc.)). In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on a side of the device opposite the display of the device. In some embodiments, the input gesture is detected by a sensor system of a head-mounted system (e.g., a VR headset including a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras that are components of the head-mounted system are mounted on the front and/or underside of the head-mounted system. In some embodiments, one or more imaging sensors are positioned in the space in which the head-mounted system is used (e.g., arrayed around the head-mounted system at various locations in a room) such that the imaging sensors capture images of the head-mounted system and/or a user of the head-mounted system. In some embodiments, the input gesture is detected by a sensor system of a head-up device (e.g., a head-up display, an automobile windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, the one or more imaging sensors are mounted on an interior surface of an automobile. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array). For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and/or one or more acoustic-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and/or sound emitters (e.g., ultrasonic emitters). For example, while light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7200), an image of the hand under the illumination of the light is captured by one or more cameras, and the captured image is analyzed to determine the position and/or configuration of the hand. By determining input gestures using signals from an image sensor directed at the hand, as opposed to using signals from a touch-sensitive surface or other direct contact or proximity-based mechanism, a user can freely choose to perform large movements or remain relatively still when providing input gestures with their hand without experiencing constraints imposed by a particular input device or input area.

In some embodiments, the tap input optionally indicates a thumb tap input on the index finger of the user's hand (e.g., on the side of the index finger adjacent to the thumb). In some embodiments, the tap input is detected without having to lift the thumb from the side of the index finger. In some embodiments, the tap input is detected according to a determination that a downward movement of the thumb is followed by an upward movement of the thumb and the thumb is in contact with the side of the index finger for less than a threshold time. In some embodiments, the tap hold input is detected according to a determination that the thumb moves from an up position to a touch down position and remains in the touch down position for at least a first threshold time (e.g., a tap time threshold or another time threshold longer than the tap time threshold). In some embodiments, the computer system requires that the entire hand remains substantially stationary at a location for at least a first threshold time to detect a thumb tap hold input on the index finger. In some embodiments, the touch hold input is detected without requiring the hand to remain substantially stationary (e.g., the entire hand can move while the thumb is resting on the side of the index finger). In some embodiments, a taphole drag input is detected when the thumb touches the side of the index finger and the entire hand moves while the thumb remains stationary on the side of the index finger.

In some embodiments, the flick gesture optionally indicates a push or flick input with the thumb moving across the index finger (e.g., from the palm side of the index finger to the back side). In some embodiments, the extension movement of the thumb is accompanied by an upward movement away from the side of the index finger, e.g., an upward flick input with the thumb. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves forward and upward. In some embodiments, a reverse flick input is performed by the thumb moving from an extended position to a retracted position. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves backward and downward.

In some embodiments, the swipe gesture is a swipe input, optionally by movement of the thumb along the index finger (e.g., along the side of the index finger adjacent to the thumb or along the side of the palm). In some embodiments, the index finger is optionally in an extended state (e.g., substantially straight) or bent state. In some embodiments, the index finger moves between an extended state and a bent state during the movement of the thumb in the swipe input gesture.

In some embodiments, different phalanges of various fingers correspond to different inputs. Thumb tap inputs across the various phalanges of various fingers (e.g., index, middle, ring, and optionally pinky) are optionally mapped to different actions. Similarly, in some embodiments, different push or click inputs can be performed by the thumb across different fingers and/or different portions of the fingers to trigger different actions in distinct user interface contacts. Similarly, in some embodiments, different swipe inputs performed by the thumb along different fingers and/or in different directions (e.g., toward the distal or proximal ends of the fingers) trigger different actions in distinct user interface contexts.

In some embodiments, the computer system processes tap, flick, and swipe inputs as different types of inputs based on the type of thumb movement. In some embodiments, the computer system processes inputs having different finger locations tapped, touched, or swiped by the thumb as different sub-input types (e.g., proximal, intermediate, distal subtypes, or index, middle, ring, or pinky subtypes) of a given input type (e.g., tap input type, flick input type, swipe input type, etc.). In some embodiments, the amount of movement performed by the moving finger (e.g., thumb) and/or other movement measures associated with the finger movement (e.g., velocity, initial velocity, ending velocity, duration, direction, movement pattern, etc.) are used to quantitatively affect the action triggered by the finger input.

In some embodiments, the computer system recognizes combination input types that combine a series of thumb movements, such as a tap-swipe input (e.g., thumb swipes along the side of a finger after touching down on another finger), a tap-flick input (e.g., thumb flicks across the finger from the side of the palm to the back of the finger after touching down on another finger), and a double-tap input (e.g., two consecutive taps on the side of a finger in approximately the same location).

In some embodiments, the gesture input is performed by the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move towards each other to perform a pinch gesture). In some embodiments, a wrist movement (e.g., a wrist flick in a horizontal or vertical direction) is performed immediately before, after (e.g., within a threshold time), or simultaneously with the finger movement input to trigger an additional, different, or modified action in the current user interface context compared to a finger movement input without a modified wrist movement input. In some embodiments, a finger input gesture performed with the palm of the user facing the user's face is treated as a different type of gesture than a finger input gesture performed with the palm of the user facing away from the user's face. For example, a tap gesture performed with the palm of the user facing the user performs an action with added (or reduced) privacy protection compared to an action (e.g., the same action) performed in response to a tap gesture performed with the palm of the user facing away from the user's face.

While one type of finger input may be used to trigger an action type in the examples provided in this disclosure, in other embodiments, other types of finger input are optionally used to trigger the same type of action.

Additional description of Figures 7A-7P is provided below with reference to methods 8000, 9000, 10000, 11000, and 12000 described with respect to Figures 8-12 below.

Figure 8 is a flowchart of a method for supporting interaction with user interface objects in a computer-generated three-dimensional environment shared among two or more users, according to some embodiments.

In some embodiments, the method 8000 is performed on a computer system (e.g., the first computer system 101 of FIG. 1) including a display generating component (e.g., the display generating component 120 of FIGS. 1, 3, and 4) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera (e.g., a color sensor, an infrared sensor, and other depth-sensing camera)) facing forward from the user's hand or the user's head. In some embodiments, the method 8000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., the control unit 110 of FIG. 1A). Some operations of the method 8000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, the method 8000 is executed in a computer system (e.g., the first computer system 101 of FIG. 1 ) in communication with a display generation component (e.g., display generation component 120, display generation component 7100, etc., of FIGS. 1, 3, and 4) (e.g., a heads-up display, HMD, display, touch screen, projector, etc.) and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, buttons, etc.). In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as the display generation component and at least some of the one or more input devices. In some embodiments, the computer system includes a computing component including one or more processors and memory that are separate from the display generation component and/or the one or more input devices. In some embodiments, the display generation component and the one or more input devices are integrated within and enclosed in the same housing.

In method 8000, the computer system displays (8002) a first user interface object (e.g., user interface object 7016 of FIG. 7B, another user interface object, etc.) (e.g., a representation of an application, a user interface including a plurality of user interface objects (e.g., a selectable avatar, a selectable menu item, a selectable device control, a selectable content item, a slider control, a button, etc.), a virtual three-dimensional object, a control, a control panel including a plurality of controls corresponding to different functions or actions, an information item, a media item, etc.) in a first view (e.g., first view 7015-1 of FIG. 7B, another first view, etc.) of a three-dimensional environment, and the three-dimensional environment is at least partially shared (e.g., at least a spatial portion of the environment is shared, the environment is shared at least for a period of time, objects in the environment are fully or partially shared (e.g., can be viewed simultaneously) between a first user (e.g., user 7102 of FIG. 7A-7C) and a second user (e.g., user 7002 of FIG. 7A-7C). and accessible, simultaneously viewable but not simultaneously accessible, viewable but not accessible when another has control (e.g., the other can view or cannot view the object), etc.), (e.g., when at least a portion of the three dimensional environment (e.g., a portion shown in a first view of the three dimensional environment, another portion of the three dimensional environment, etc.) is displayed for viewing by both the first user and the second user, and/or when a virtual object within the three dimensional environment (e.g., a first user interface object, another user interface object, etc.) is displayed for viewing by both the first user and the second user. ) is displayed in a three dimensional environment shown to both the first user and the second user simultaneously), the first user interface object is displayed in a first position in a first view of the three dimensional environment (e.g., first view 7015-1 of FIG. 7B ) with a first set of appearance characteristics (e.g., the normal appearance of the first user interface object as displayed to the second user by the second display generation component (e.g., first shape, first size, first color, first opacity, first saturation, first brightness, etc.)) (e.g., as shown in FIG. 7B ). While displaying a first user interface object having a first set of appearance characteristics at a first position within the first view of the three-dimensional environment, the computer system detects (8004) a first user input directed to the first user interface object, the first user input being provided by a first user (e.g., detecting the user input includes detecting movement of a portion of the first user to a first location within the physical environment, the first location within the physical environment corresponding to a discrete position of the first user interface object within the first view of the three-dimensional environment, and detecting the user input includes detecting a gaze input directed at the first user interface object and a control input (e.g., a finger movement gesture, an air gesture, an input provided by a controller, etc.) detected along with the gaze input). In response to detecting 8006 a first user input directed at the first user interface object, and in accordance with a determination that a second user (e.g., user 7002 of FIGS. 7A-7C ) is not currently interacting with the first user interface object (e.g., user interface object 7016 of FIGS. 7A-7C ) (e.g., the first user interface object does not have a pre-defined spatial relationship with a virtual position of the second user within the first view of the three dimensional environment (e.g., the first user interface object is not within a representation of the second user's palm or hand, the first user interface object is outside of a private space of the second user that is visible within the first view of the three dimensional environment, etc.), the second user is In accordance with a determination that the second user was not interacting with the first user interface object when the first user input was detected (e.g., not controlling, selecting, moving, modifying, and/or interacting with the first user interface object via the second computer system displaying the second view of the three-dimensional environment within the partially shared three-dimensional environment), the computer system performs (8008) a first action on the first user interface object in accordance with the first user input (e.g., indicating that the first user interface object is being grasped or moved by the first user in accordance with the first user input (e.g., moved towards the user, moved in accordance with the movement of the first user input, etc.), showing a ghost image of the first user interface object being grasped or moved in the hand representation of the first user, etc.). In some embodiments, in accordance with a determination that the second user was not interacting with the first user interface object when the first user input was detected, the first user interface object continues to be displayed with the first set of appearance characteristics (e.g., in its original location or with the hand representation of the first user, etc.). In response to detecting 8006 a first user input directed at a first user interface object (e.g., user interface object 7016 of FIGS. 7A-7C ) and in accordance with a determination that a second user is currently interacting with the first user interface object (e.g., the first user interface object has a pre-defined spatial relationship with a virtual position of the second user within the first view of the three-dimensional environment) (e.g., the first user interface object is within a representation of the second user's palm or hand, the first user interface object is within a private space of the second user visible within the first view of the three-dimensional environment, etc.), the second user is prompted to view the second view of the three-dimensional environment from a shared third-party application. 7C , the first user 7102 controls, selects, moves, modifies, and/or otherwise interacts with the first user interface object through a second computer system that displays the first user interface object in the three-dimensional environment, and the computer system displays a visual indication that the first user interface object is unavailable for interaction with the first user (8010), where displaying the visual indication includes changing at least one of an appearance of the first user interface object or a position of the first user interface object within the first view of the three-dimensional environment (e.g., in FIG. 7C , the appearance of user interface object 7016 is changed within first view 7015-1 shown to the first user 7102). In some embodiments, the computer system displays the first user interface object having a second set of appearance characteristics (e.g., a second shape, a second size, a second color, a second opacity, a second saturation, a second brightness, etc.) different from the first set of appearance characteristics (e.g., the second set of appearance characteristics provide a visual indication that the first user interface object is now controlling the second user and is not available for interaction with the first user) and/or moves the first user interface object out of the way when the first user attempts to grasp the first user interface object. In some embodiments, the first user interface object maintains its appearance and/or position within the view of the at least partially shared three-dimensional environment displayed to the second user, since the visual indication need only be displayed to the first user. In some embodiments, the visual indication is displayed while the second user is interacting with the first user interface object in the at least partially shared three-dimensional environment. The computer system ceases performing the first action on the first user interface object in accordance with the first user input (8014). In some embodiments, the computer system does not show the first user interface object being grasped by the representation of the first user or does not show the first user interface object moving in accordance with the movement of the first user input (e.g., the object does not move to avoid being grasped by the hand of the first user, the object does not shrink or change shape to avoid being grasped by the representation of the first user, etc.). In some embodiments, the computer system does not show a ghost image of the first user interface object moving within the representation of the hand of the first user.

These features are shown, for example, in Figures 7A-7C, where a first user 7102 and a second user 7002 share a three-dimensional environment shown via display generation components 7200 and 7100, respectively. When the second user 7002 controls the first user interface object 7016 (e.g., interacts with the first user interface object 7016, holds the first user interface object 7016 or a representation thereof via a representation 7028'' of the hand 7028 in the second view 7015-2 (as well as the representation 7028' in the first view 7015-1 shown to the first user 7102), etc.), if the first user 7102 attempts to interact with the first user interface object 7016 by moving the first user's hand 7102, the computer system of the first user 7102 changes the appearance of the first user interface object 7016 in the first view 7015-1 shown via the first display generation component 7200 and does not perform an action corresponding to the first user interface object 7016. In contrast, if the second user 7002 is not interacting with the first user interface object 7016, the computer system executes the first action according to the movement of the first user's hand 7202. This is indirectly illustrated by the interaction between the second user 7002 and the first user interface object 7016 in FIG. 7B, where the first user 7102 does not control or interact with the first user interface object 7016 (e.g., consider a role reversal of the first and second users in that scenario).

In some embodiments, the computer system alters the appearance of the first user interface object as a visual indication that the first user interface object is unavailable for interaction with the first user, where altering the appearance of the first user interface object includes altering at least one of a first set of appearance characteristics of the first user interface object (e.g., increasing transparency, decreasing saturation, decreasing opacity, blurring, darkening, decreasing resolution, decreasing size, etc. of the first user interface object, optionally reducing the visual prominence of the first user interface object while preserving the appearance of the first user interface object's surrounding environment (e.g., without changing the visual prominence of the surrounding environment)) (e.g., in FIG. 7C , the appearance of user interface object 7016 is altered in first view 7015-1 shown to first user 7102). In some embodiments, the computing system changes the appearance of the first user interface object while maintaining a position of the first user interface object in the first view of the three-dimensional environment that is determined independent of the first user input (e.g., the first position, another position that is determined in response to an interaction between the first user interface object and the second user, another position that is determined according to pre-programmed autonomous movement of the first user interface object (e.g., the first user interface object has a pre-set movement pattern or a pre-set animation effect, etc.), another position that is determined according to other events in the computer system, etc.). In some embodiments, in response to detecting a first user input directed at the first user interface object, in accordance with a determination that the second user is not currently interacting with the first user interface object, the computer system does not alter the appearance of the first user interface object to reduce the visual prominence of the first user interface object, and the computer system performs a first action on the first user interface object in accordance with the first user input (e.g., the appearance and visual prominence of the first user interface object is maintained, or the appearance may be altered as a result of performing the first action, but not for the purpose of reducing the visual prominence of the first user interface object, etc.).

Modifying the appearance of the first user interface object, including modifying at least one of the first set of appearance characteristics of the first user interface object to reduce visual prominence for the first user interface object as a visual indication that the first user interface object is unavailable for interaction with the first user, provides improved visual feedback to the user (e.g., improved visual feedback that the first user interface object is unavailable for interaction with the first user). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the computer system detects an end of a first user input directed to a first user interface object (e.g., detects movement of a portion of the first user away from a first location in the physical environment corresponding to a distinct position of the first user interface object in a first view of the three-dimensional environment, detects that a gaze input directed to the first user interface object moves away from the first user interface object, detects that a hand of the first user that provided the first user input moves out of a posture required to maintain the first user input, etc.). In response to detecting an end of the first user input directed to the first user interface object, the computer system restores at least one of the first set of appearance characteristics of the first user interface object that was modified in response to the first user input (e.g., to a level that existed immediately prior to detecting the first user input, or before the modification was made in response to detecting the first user input, etc.) to restore visual prominence of the first user interface object. In some embodiments, the computer system optionally restores an increased transparency, restores a decreased saturation, restores a decreased opacity, stops blurring and/or darkening, restores a decreased resolution, restores a decreased size, etc. of the first user interface object, while preserving the appearance of the first user interface object's surroundings (e.g., without changing the visual prominence of the surroundings). For example, in some embodiments, when a first user extends his/her hand towards a location corresponding to a virtual object with which a second user is currently interacting, the virtual object appears to fade out or become darker when the first user's hand is in a location in the physical environment that corresponds to the position of the virtual object in the three-dimensional environment. Then, when the first user moves their hand away from that location, the appearance of the virtual object is restored (e.g., no longer faded out or appearing dark). This is shown in FIG. 7B (continued in FIG. 7C), where when the first user 7102 stops attempting to interact with the first user interface object 7016, the appearance of the first user interface object 7016 is no longer changed in the first view 7015-1 shown to the first user 7102.

Restoring at least one of the first set of appearance characteristics of the first user interface object that were altered in response to the first user input in response to detecting an end of the first user input directed to the first user interface object to restore the visual prominence of the first user interface object provides improved visual feedback to the user (e.g., improved visual feedback that the first user interface object is available for interaction). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the computer system detects movement of the first user interface object away from the first position in the first view of the three-dimensional environment while continuing to detect the first user input (e.g., detecting a portion of the first user remaining in a first location in the physical environment that corresponds to the distinct position of the first user interface object in the first view of the three-dimensional environment when the first user input was initially detected, detecting gaze input directed at the first user interface object remaining in the same position in the three-dimensional environment, detecting the hands of the first user who provided the first user input remaining in a posture required to maintain the first user input, etc.), independently of detection of the first user input (e.g., in accordance with user input provided by a second user, in accordance with a unique movement pattern of the first user interface object, in response to other events in the computer system independent of the first user input, etc.). In response to detecting movement of the first user interface object away from the first position within the first view of the three-dimensional environment independent of detection of the first user input, the computer system restores at least one of the first set of appearance characteristics of the first user interface object that was modified in response to the first user input (e.g., to a level that existed immediately prior to detecting the first user input, or to a level that existed before the modification was made in response to detection of the first user input, etc.) to restore visual prominence of the first user interface object. In some embodiments, the computer system restores an increased transparency, restores a decreased saturation, restores a decreased opacity, stops blurring and/or darkening, restores a decreased resolution, restores a decreased size, etc. of the first user interface object, optionally while preserving the appearance of the first user interface object's surroundings (e.g., without changing the visual prominence of the surroundings). For example, in some embodiments, when a first user extends his/her hand towards a location corresponding to a virtual object with which a second user is currently interacting, the virtual object appears to fade out or become darker when the first user's hand is in a location in the physical environment that corresponds to the position of the virtual object in the three-dimensional environment. Then, when the first user interface object is moved away from its current position and away from the position corresponding to the current location of the first user's hand (e.g., moved by the second user according to its own movement pattern, according to other system-generated events not related to the first user input, etc.), the appearance of the virtual object is restored (e.g., no longer appears faded out or darkened).

In response to detecting movement of the first user interface object away from the first position in the first view of the three-dimensional environment independent of detection of the first user input, restoring at least one of the first set of appearance characteristics of the first user interface object that was modified in response to the first user input to restore visual prominence of the first user interface object provides improved visual feedback to the user (e.g., improved visual feedback that the first user interface object has moved away from the first position). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, displaying a visual indication that the first user interface object is unavailable for interaction with the first user includes maintaining the changes to the appearance of the first user interface object made in response to the first user input until the second user stops interacting with the first user interface object. For example, the modified appearance of the user interface object 7016 is maintained until the second user 7002 no longer controls the first user interface object 7016 to the exclusion of the first user 7102, even after the first user 7102 has stopped attempting to interact with the first user interface object 7016. For example, in some embodiments, once a visual indication is displayed in response to detection of a first user input and in accordance with a determination that the second user was interacting with the first user interface object at the time the first user input was initially detected, the computer system continues to display the visual indication (e.g., a changed appearance, a changed position, etc. of the first user interface object) in accordance with a determination that the second user is still interacting with the first user interface object (e.g., the second user continues to maintain the virtual object in a position corresponding to the location of the second user's palm or hand and/or continues to select, modify, or otherwise interact with the virtual object through operation of the second user's computer system, etc.).

In some embodiments, the visual indication continues to be displayed even when the computer system detects that the first user input has ended and that the first user is no longer providing input to attempt to interact with the first user interface object. In some embodiments, the visual indication is maintained for a period of time regardless of whether the first user input is maintained or whether the first user continues to attempt to interact with the first user interface object, but not necessarily until the second user ceases interacting with the first user interface object. In some embodiments, the computer system of the first user determines that the second user is no longer interacting with the first user interface object, and in response to detecting that the second user is no longer interacting with the first user interface object, the computer system stops displaying the visual indication that the first user interface object is not available for interaction with the first user (e.g., the computer system stops displaying the first user interface object in a dimmed or darkened state and restores the original appearance characteristics of the first user interface object that were altered in response to detecting the first user input). In some embodiments, a persistent visual indication that the first user interface object is still within the second user's range of control and/or is unavailable for interaction with the first user helps the first user know when the device is ready to respond to another attempt to interact with the first user interface object and avoids repeated failed attempts to interact with the first user interface object.

Maintaining the changes made to the appearance of the first user interface object in response to the first user input until the second user stops interacting with the first user interface object provides the user with improved visual feedback (e.g., improved visual feedback that the second user is interacting with the first user interface object, improved visual feedback that the first user interface object is unavailable for interaction with the first user, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to detecting a first user input directed at a first user interface object (e.g., user interface object 7016 in FIGS. 7B-7C ) and a second user (e.g., user 7002 in FIGS. 7B-7C ) is not currently interacting with the first user interface object (the first user interface object does not have a pre-defined spatial relationship with the second user's virtual position in the first view of the three-dimensional environment) (e.g., the first user interface object is not within a representation of the second user's palm or hand, the first user interface object is not within a representation of the second user's palm or hand, the first user interface object is not within a representation of the third ... In accordance with a determination that the first user interface object is located within the first view of the environment (e.g., outside the second user's private space visible within the first view of the environment), the second user is not controlling, selecting, moving, modifying, and/or otherwise interacting with the first user interface object via a second computer system displaying a second view of the three-dimensional environment within the at least partially shared three-dimensional environment, and the computer system displays the first user interface object at a second position within the first view of the three-dimensional environment, the second position being selected according to a current location of the hand of the first user (e.g., user 7102 of FIGS. 7B-7C or another user). In some embodiments, the second position is selected to correspond to a current location of the first user's hand (e.g., hand 7202 in FIGS. 7B-7C, or another hand, etc.) providing the first user input (e.g., overlaying a representation of the first user's hand, replacing the display of that representation, changing the appearance of that representation, etc.), or the second position is selected to correspond to a position within the three-dimensional environment or a location within the physical environment that is pointed to or indicated by the first user input, etc.

In some embodiments, the computer system presents a first user interface object being held by a representation of the first user's hand in a first view of the three-dimensional environment, presents the first user interface object moving in a direction corresponding to a direction of movement of a first user input provided by the first user's hand (e.g., an upward movement of the user's hand causes the virtual object to lift off of the representation of the tabletop, an upward movement of the user's hand causes the virtual object to jump off of the representation of the tabletop and land on the representation of the user's hand, etc.), and moves the first user interface object (e.g., user interface object 7016 of Figures 7B-7C, or another user interface object, etc.) to a position indicated by the first user's hand (e.g., a flick and point gesture by the first user's hand causes the virtual object to move from its original position and move to a position indicated by the index finger of the first user's hand, etc.). In response to detecting a first user input directed at the first user interface object, while displaying the first user interface object at a second position selected according to a current location of the first user's hand (e.g., while the first user interface object is displayed at a position corresponding to the first user's hand representation (e.g., appearing to be held by the first user's hand), while the first user interface object is displayed hovering at the position selected by the first user input, etc.), the computer system detects a movement of the first user's (e.g., user 7102, or another user, etc.) hand corresponding to a throwing gesture of the first user's hand. In some embodiments, the computer system detects that the first user's hand is moving from a location close to the first user to another location further away from the first user, optionally toward a location corresponding to a position of the representation of the second user. In some embodiments, the computer system detects that the first user lifts their index finger away from the first user and optionally flicks it toward a location corresponding to a position of the representation of the second user, and/or detects that the first user is performing a tossing or throwing motion using their hand. In some embodiments, detecting the throwing gesture includes detecting a sudden acceleration of the hand followed by a sudden deceleration. In some embodiments, detecting the throwing gesture includes detecting an opening of a closed hand (and optionally in conjunction with detecting a sudden deceleration). In response to detecting a first user input directed at the first user interface object, and in response to detecting a movement of the first user's hand corresponding to a throwing gesture of the first user's hand, the computer system moves the first user interface object in the first view of the three-dimensional environment in a first direction corresponding to a direction of the movement of the first user's hand, and rotates the first user interface object during the movement of the first user interface object. In some embodiments, as the first user interface object moves in a direction corresponding to the direction of the throwing gesture, the first user interface object also rotates about a virtual center of gravity of the first user interface object (e.g., a geometric center, another point inside or outside the first user interface object depending on the object simulated by the first user interface object, etc.) (e.g., to simulate conservation of angular momentum during linear motion of the first user interface object, to simulate a physical effect of the throwing gesture on the first user interface object, to present a predetermined user-facing side of the first user interface object at the destination end of the throwing gesture towards a second user, to land on a representation of a physical surface or on a virtual surface having a predetermined upright orientation, etc.).

In some embodiments, when a second user interacts with the first user interface object and subsequently performs a throwing gesture to throw the first user interface object into the three-dimensional environment, the computer system indicates that the first user interface object moves in a second direction in the first view of the three-dimensional environment corresponding to the direction of the second user's hand movement and rotates the first user interface object during the movement of the first user interface object (e.g., as the first user interface object moves in the direction of the throwing gesture, the first user interface object also rotates about a virtual center of gravity of the first user interface object (e.g., to simulate conservation of angular momentum during linear motion of the first user interface object, to simulate a physical effect of a throwing gesture on the first user interface object, to indicate a predetermined user-facing side of the first user interface object toward the first user at the destination end of the throwing gesture, to land on a representation of a physical surface or on a virtual surface having a predetermined upright orientation, etc.)).

In response to detecting a movement of the first user's hand corresponding to a throwing gesture of the first user's hand, moving a first user interface object in a first direction corresponding to the direction of the movement of the first user's hand in a first view of the three-dimensional environment and rotating the first user interface object during the movement of the first user interface object provides improved visual feedback to the user (e.g., improved visual feedback that the computer system has detected a throwing gesture of the first user's hand, improved visual feedback that the first user interface object is being moved, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, rotating the first user interface object (e.g., user interface object 7016, or another user interface object, etc.) during the movement of the first user interface object includes rotating the first user interface object in a first manner (e.g., rotating the first user interface object by a first amount in a first rotational direction, a second amount in a second rotational direction, and/or a third amount in a third rotational direction) such that the first user interface object has a first preset orientation in the three-dimensional environment when it reaches a destination position (e.g., a position of the representation of the second user's hand, a representation of a surface associated with the second user, etc.) in the three-dimensional environment selected in accordance with the movement of the first user's hand in the physical environment in accordance with a determination that the direction of the movement of the first user's hand is toward the representation of the second user in the first view of the three-dimensional environment. In some embodiments, the first preset orientation is different from the orientation that the first user interface object had when the first user interface object began to move in response to the throwing gesture of the first user. In some embodiments, the first preset orientation is an orientation in which a preset front side of the first user interface object faces towards a representation of a second user. In some embodiments, the first preset orientation is an orientation in which the first user interface object stands upright when hooked and/or placed on a hand representation of a second user, etc. In some embodiments, when the second user interacts with the first user interface object and subsequently performs a throwing gesture in a direction corresponding to the direction of the representation of the first user, the computer system determines that the first user interface object has a first preset orientation (e.g., the first preset orientation is an orientation in which the first user interface object stands upright when hooked and/or placed on a hand representation of a second user, etc.) in accordance with a determination that the direction of the hand movement of the second user is towards a viewpoint of the first view of the three-dimensional environment when the selected destination position in the three-dimensional environment is reached according to the hand movement of the second user in the physical environment. The first user interface object is rotated in a separate manner (e.g., by a first amount in a first rotational direction, a second amount in a second rotational direction, and/or a third amount in a third rotational direction, etc.) such that the first preset orientation is different from the orientation the first user interface object had when it began moving in response to the mouse, the first preset orientation being an orientation in which the preset forward-facing side of the first user interface object faces the representation of the first user, or the first preset orientation is an orientation in which the first user interface object will stand upright when caught by and/or placed on the representation of the hand of the first user, etc.).

Rotating the first user interface object in a first manner such that the first user interface object has a first preset orientation in the three-dimensional environment when a destination position in the three-dimensional environment selected according to the movement of the first user's hand in the physical environment is reached reduces the number of inputs to display the first user interface object in a desired orientation at the destination position (e.g., the user does not need to perform an additional gesture to rotate the first user interface object in a desired orientation (e.g., for viewing) after the first user interface object is rotated during the movement of the first user interface object). Reducing the number of inputs required to perform an operation improves the operability of the device, as well as reducing power usage and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, rotating the first user interface object during movement of the first user interface object includes rotating the first user interface object (e.g., a first user 7102 in FIGS. 7A-7C, another user, etc.) in accordance with a determination that a direction of hand movement of a first user (e.g., a first view 7015-1 in FIGS. 7B-7C, another view, etc.) points toward a representation of a first surface (e.g., a physical surface (e.g., a wall, a table top, a sofa seat, etc.), a virtual surface (e.g., a virtual shelf, a virtual wall, a virtual sofa, etc.), etc.) in a first view of the three-dimensional environment (e.g., a first view 7015-1 in FIGS. 7B-7C, another view, etc.). , the first user interface object 7016 of FIGS. 7B-7C, or another user interface object) in a second manner (e.g., rotating the first user interface object a first amount in a first rotational direction, a second amount in a second rotational direction, and/or a third amount in a third rotational direction) such that the first user interface object has a second preset orientation with respect to the representation of the first surface in the three-dimensional environment when the first user interface object arrives at a destination position on the representation of the first surface selected in accordance with the movement of the first user's hand in the physical environment. In some embodiments, the second preset orientation is different from the orientation that the first user interface object had when the first user interface object began moving in response to the throwing gesture of the first user. In some embodiments, the second preset orientation is an orientation in which a preset front side of the first user interface object faces the representation of the first user. In some embodiments, the second preset orientation is an orientation in which the first user interface object is upright when it lands on the representation, such as the first surface.

In some embodiments, when a second user interacts with the first user interface object and subsequently performs a throwing gesture in a direction corresponding to the direction of the representation of the first surface, the computer system, in accordance with a determination that the direction of the second user's hand movement is toward the representation of the first surface in the three-dimensional environment, rotates the first user interface object in a separate manner such that the first user interface object has a second pre-set orientation relative to the representation of the first surface in the three-dimensional environment (e.g., by a first amount in a first rotational direction, a second amount in a second rotational direction, and/or a third amount in a third rotational direction, etc.) when the first user interface object reaches a destination position on the representation of the first surface selected in accordance with the movement of the second user's hand in the physical environment. In some embodiments, regardless of which user performed the throwing gesture in the direction of the representation of the first surface, the first user interface object is rotated in a separate manner during movement toward the representation of the first surface such that the first user interface object lands on the representation of the first surface in a pre-set spatial relationship (e.g., orientation, location, etc.) relative to the representation of the first surface. For example, in some embodiments, when the first user interface object is a virtual picture frame, the virtual picture frame rotates while being thrown toward a representation of a table and lands on the table with an upright orientation toward the user who performed the throwing gesture. In some embodiments, when the virtual picture frame is thrown toward a representation of a wall, the virtual picture frame rotates while traveling toward the representation of the wall and lands on the representation of the wall with its back surface parallel to the representation of the wall. In some embodiments, when the virtual picture frame is thrown by a first user toward a second user, the virtual picture frame rotates to have its front surface facing toward the representation of the second user when the virtual picture frame lands on a representation of the second user's palm.

Rotating the first user interface object in a second manner such that the first user interface object has a second pre-set orientation relative to the representation of the first surface in the three-dimensional environment when the first user interface object reaches a destination position on the representation of the first surface selected according to the movement of the first user's hand in the physical environment reduces the number of inputs required to display the first user interface object in a desired orientation on the first surface (e.g., the user does not need to perform an additional gesture to rotate the first user interface object in the desired orientation (e.g., for viewing) after the first user interface object is rotated during the movement of the first user interface object). Reducing the number of inputs required to perform an operation improves the usability of the device, as well as reducing power usage and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the computer system alters a position of a first user interface object (e.g., user interface object 7016 of FIGS. 7B-7C or another user interface object) in a first view of the three-dimensional environment (e.g., 7015-1 of FIG. 7C or another view) as a visual indication that the first user interface object is unavailable for interaction with the first user, and altering the position of the first user interface object in the first view of the three-dimensional environment includes moving the first user interface object from a first position to maintain at least a predetermined distance between the first user interface object and a representation of the first user's hand that provided the first user input (e.g., the first user interface object appears to move in one or more directions to avoid the representation of the first user's hand attempting to grasp the first user interface object). In some embodiments, the movement of the first user interface object is accompanied by changes made to the appearance of the first user interface object (e.g., the first user interface object appears to fade or darken while moving to avoid the first user's hand representation getting too close to itself). In some embodiments, in response to detecting a first user input directed at the first user interface object, following a determination that a second user is not currently interacting with the first user interface object, the computer system does not move the first user interface object from the first position until the first user's hand representation reaches the first position, and then the computer system moves the first user interface object away from the first position or manipulates the first user interface object in accordance with subsequent movement of the hand from the first position (e.g., subsequent movement of the first user interface object is to move with or follow the hand rather than to avoid the hand). In some embodiments, in response to detecting a first user input directed at the first user interface object and following a determination that the second user is not currently interacting with the first user interface object, the computer system moves the first user interface object from a first position toward the representation of the hand of the first user until the position of the first user interface object and the representation of the hand overlap, and then the computer system moves or manipulates the first user interface object in accordance with subsequent movements of the hand (e.g., the first user interface object moves with or follows the representation of the hand).

Moving the first user interface object from the first position to maintain at least a preset distance between the first user interface object and the representation of the hand of the first user that provided the first user input as a visual indication that the first user interface object is unavailable for interaction with the first user provides the user with improved visual feedback (e.g., improved visual feedback that the first user interface object is unavailable for interaction with the first user). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, performing a first operation on a first user interface object (e.g., user interface object 7016 of FIGS. 7B-7C, or another user interface object, etc.) in accordance with the first user input includes moving the first user interface object toward a representation of a hand of the first user (e.g., hand 7202 of FIGS. 7B-7C, or another hand, etc.) (e.g., the hand providing the first user input, a hand of the first user different from the hand that provided the first user input, any hand of the first user, etc.). In some embodiments, the computer system shows the first user interface object moving toward the representation of the hand as the hand moves toward the first user interface object to indicate that the first user interface object is available for grasping by the hand of the first user. In some embodiments, the movement of the first user interface object ceases when a position of the first user interface object overlaps a position of the representation of the hand, and the computer system then moves or manipulates the first user interface object in accordance with subsequent movements, if any, of the hand. In some embodiments, the movement of the first user interface object is only a limited distance from the first position (e.g., the first user interface object moves a small distance toward the first user's hand representation when the first user's hand representation approaches the first user interface object to within a threshold distance of the first position), and this movement provides visual feedback to the first user that the first user interface object is available for interaction with the first user (e.g., in response to the first user's hand representation moving close to the first user interface object to grasp the first user interface object when the first user provides a correct selection gesture, etc.).

Moving the first user interface object toward the hand representation of the first user pursuant to a determination that the second user is not currently interacting with the first user interface object provides the user with improved visual feedback (e.g., improved visual feedback that the user interface object is available for interaction with the first user, improved visual feedback that the computer system is performing an action on the first user interface object, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, performing a first operation on a first user interface object (e.g., user interface object 7016, or another user interface object, etc.) in accordance with a first user input includes selecting the first user interface object as a target for subsequent input (e.g., a drag gesture while the pinch gesture is maintained, a flick gesture while the pinch gesture is maintained, a drag gesture after the predetermined selection gesture is terminated, etc.) received from a first user (e.g., user 7102 of FIGS. 7A-7C, or another user, etc.) in accordance with a determination that the first user input includes (e.g., includes, begins, ends, etc.) a predetermined selection gesture. In some embodiments, the selection gesture is a pinch gesture involving a touch down of an index finger onto a thumb of the same hand (optionally followed by lifting the index finger from the thumb, or a flick of the wrist connected to the hand, or translation of the entire hand, etc.), a gesture involving an index finger and thumb of the same hand pulling away from each other from a touch pose, a pinch gesture, a pinch-and-drag gesture, a pinch-and-flick gesture, etc. In some embodiments, when the hand of the first user approaches a location corresponding to a nearby position of the first user interface object and performs a predefined selection gesture, the computer system generates visual feedback indicating that the first user interface object is not available for interaction with the first user in accordance with a determination that a second user is currently interacting with the first user interface object, and the computer system does not select the first user interface object for subsequent interaction with the first user. If the computer system determines that the second user is not currently interacting with the first user interface object, the computer system does not display visual feedback and selects the first user interface object for subsequent interaction with the first user.

Selecting the first user interface object as a target for subsequent input received from the first user pursuant to a determination that the first user input includes a predefined selection gesture provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting the first user interface object). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, in conjunction with selecting a first user interface object (e.g., user interface object 7016 of FIGS. 7B-7C, or another user interface object, etc.) as a target for subsequent input received from a first user (e.g., in response to detecting at least a portion of a first user input, in response to detecting a predefined selection gesture provided by the first user while the second user is not currently interacting with the first user interface object, etc.), the computer system may select the first user interface object (e.g., in response to detecting at least a portion of a first user input, in response to detecting a predefined selection gesture provided by the first user while the second user is not currently interacting with the first user interface object, etc.) while maintaining the first user interface object in a first position within the first view of the three-dimensional environment (e.g., in response to detecting a predefined selection gesture provided by the first user while the second user is not currently interacting with the first user interface object). the first user interface object remains in its original location but can be "remotely" controlled by the first user according to interactions between the first user and the representation of the first user interface object), displaying a representation of the first user interface object in a position corresponding to the location of the first user's hand (e.g., the hand that performed the predefined selection gesture, or a hand of the first user different from the hand that performed the first predefined selection gesture, any hand of the first user, etc.) (e.g., a representation or ghost image of the first user interface object is displayed in the palm portion of the representation of the first user's hand near or at a position of the representation of the first user's hand). In some embodiments, the representation of the first user interface object is a replica of the first user interface object, an image of the first user interface object that is more semi-transparent than the first user interface object, a scaled-down version of the first user interface object having a subset of the characteristics (e.g., a simplified internal structure, outline, etc.) and/or functionality of the first user interface object (e.g., scaled-down user interface elements, removed text content, etc.), etc. In some embodiments, when a first user interface object is selected in response to a selection gesture performed by the first user, a ghost image of the first user interface object is displayed in a position corresponding to the location of the first user's hand (e.g., floating above a representation of the first user's hand, over a representation of the user's palm, etc.).

Selecting the first user interface object as a target for subsequent input received from the first user in conjunction with displaying a representation of the first user interface object in a position corresponding to the location of the first user's hand while maintaining the first user interface object in a first position within the first view of the three-dimensional environment provides the user with improved visual feedback (e.g., improved visual feedback that the first user interface object is now a target for subsequent input received from the first user). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, after a first action is performed on a first user interface object in accordance with a first user input (e.g., a representation of the first user interface object is displayed near the first position with the first user interface object), a representation of the first user interface object (e.g., user interface object 7016 in FIGS. 7B-7C, or another user interface object, etc.) moves near a position of a representation of the first user's hand (e.g., representation 7202' of first view 7015-1 in FIGS. 7B-7C, or another hand representation 7202, etc.), and the first user interface object moves toward the position of the first user. and indicating that the first user interface object is ready for interaction with the first user), the computer system detects a second user input directed at the first user interface object (e.g., detect a predefined pinch gesture directed at a position of the representation of the first user interface object, detect a predefined pinch gesture directed at a position of the first user interface object, detect a gaze input directed at a position of the representation of the first user interface object in conjunction with the detection of the predefined selection input, detect a gaze input directed at a position of the first user interface object in conjunction with the detection of the predefined selection input, etc.). In response to detecting the second user input and in accordance with a determination that the second user input includes (e.g., includes, begins, ends, etc.) the predefined selection gesture, the computer system selects the first user interface object as a target for a subsequent input received from the first user (e.g., a drag gesture while the pinch gesture is maintained, a flick gesture while the pinch gesture is maintained, a drag gesture after the predefined selection gesture is ended, etc.).

In some embodiments, the selection gesture is a pinch gesture including a touchdown of an index finger on a thumb of the same hand (optionally followed by moving the index finger away from the thumb, or by flicking the wrist connected to the hand, or by translating the entire hand, etc.), a gesture including an index finger and thumb of the same hand pulling away from each other from a touch pose, a pinch gesture, a pinch-and-drag gesture, a pinch-and-flick gesture, etc. In some embodiments, performing a first operation on a first user interface object (e.g., user interface object 7106 of FIGS. 7B-7C, or another user interface object, etc.) in accordance with a first user input includes displaying a representation of the first user interface object (e.g., in a second position different from the first position at which the first user interface object is displayed and/or different from the position of a representation of the hand of the first user that provided the first user input) while maintaining the first user interface object in a first position within a first view of the three-dimensional environment. In some embodiments, the representation of the first user interface object is a duplicate of the first user interface object, an image of the first user interface object that is more semi-transparent than the first user interface object, a scaled-down version of the first user interface object having a subset of the characteristics (e.g., simplified internal structure, outlines, etc.) and/or functionality of the first user interface object (e.g., scaled-down user interface elements, removal of text content, etc.), etc. In some embodiments, the representation of the first user interface object is displayed slightly offset from the first user interface object (e.g., floating above, in front of, etc., of the first user interface object in the first view of the three-dimensional environment, rather than in a position corresponding to the location of the first user's hand, etc.). In some embodiments, the representation of the first user interface object moves toward but remains outside of the representation of the first user's hand.

After the first action is performed, selecting the first user interface object as a target for subsequent input received from the first user pursuant to a determination that the second user input includes a predetermined gesture provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting the first user interface object). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, performing a first operation on a first user interface object (e.g., user interface object 7106 of FIGS. 7B-7C, or another user interface object) in accordance with the first user input includes displaying a representation of the first user interface object (e.g., a replica of the first user interface object, an image of the first user interface object that is more semi-transparent than the first user interface object, a reduced version of the first user interface object having a subset of the characteristics (e.g., a simplified internal structure or contours) and/or functionality (e.g., reduced user interface elements or reduced text content) of the first user interface object) (e.g., in a second position different from the first position at which the first user interface object is displayed and/or different from the position of a representation of the hand of the first user that provided the first user input) while maintaining the first user interface object in a first position within the first view of the three-dimensional environment. In some embodiments, the representation of the first user interface object is displayed slightly offset from the first user interface object (e.g., floating above, in front of, etc., the first user interface object in the first view of the three-dimensional environment, rather than in a position corresponding to the location of the first user's hand). In some embodiments, the representation of the first user interface object moves toward the representation of the first user's hand, but remains outside the representation of the first user's hand. Displaying the representation of the first user interface object while maintaining the first user interface object in the first position in the first view of the three-dimensional environment provides the user with improved visual feedback (e.g., improved visual feedback that the computer system is performing a first action on the first user interface object in accordance with the first user input). Providing improved feedback improves usability of the device, as well as reducing power usage and improving battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a representation of a first user interface object (e.g., user interface object 7106 of FIGS. 7B-7C, or another user interface object, etc.) is initially displayed in a position away from a representation of a first user's hand (e.g., hand 7202 of FIGS. 7B-7C, or another hand, etc.) (e.g., the hand that provided the first user input, the first user's other hand, etc.), and pursuant to a determination that the first user interface object has been selected by a subsequent user input provided by the first user, the computer system moves the representation of the first user interface object from a position away from the representation of the first user's hand (e.g., the hand that provided the first user input, the first user's other hand, etc.) to a position of the representation of the first user's hand (e.g., moving the representation of the first user interface object from its initial display position to a position of the representation of the first user's hand, over the representation of the user's palm, etc.). In some embodiments, the computer system moves the representation of the first user interface object in response to detecting a second user input directed to the first user interface object that satisfies a selection criterion or includes a selection gesture (e.g., detecting a predefined pinch gesture directed at the representation of the first user interface object, detecting a predefined pinch gesture directed at the first user interface object, detecting a gaze input directed at the representation of the first user interface object along with detecting the predefined selection input, or detecting a gaze input directed at the first user interface object along with detecting the predefined selection input). Moving the representation of the first user interface object from a position away from the representation of the first user's hand to a position of the representation of the first user's hand pursuant to a determination that the first user interface object has been selected by a subsequent user input provided by the first user provides improved visual feedback to the first user (e.g., improved visual feedback that the first user interface object has been selected by a subsequent user input provided by the first user). Providing improved feedback improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, while displaying a representation of the first user interface object (e.g., the representation of the first user interface object is being displayed in response to detection of a first user input (e.g., detection of the first user's hand moving near a location corresponding to the position of the first user interface object in the three-dimensional environment)), the computer system detects movement of the first user's hand (e.g., hand 7202, or another hand of user 7102, etc.) (e.g., the hand that provided the first user input, a hand of the first user different from the hand that provided the first user input, etc.). In response to detecting movement of the first user's hand (e.g., the hand that provided the first user input, a first user's hand different from the hand that provided the first user input, etc.), and in accordance with a determination that the movement of the first user's hand meets preset criteria for identifying an initial portion of a preset selection gesture, the computer system modifies an appearance (e.g., changes a shape, size, color, opacity, level of detail, etc.) of the representation of the first user interface object (e.g., user interface object 7016 of FIGS. 7B-7C , or another user interface object, etc.), optionally causing the representation of the first user interface object to more closely resemble an appearance of the first user interface object, etc. In some embodiments, the appearance of the representation of the first user interface object continues to change as the movement of the first user's hand continues to conform to the required progression of the selection gesture. In some embodiments, in response to detecting that the first user's hand movement does not meet the preset criteria for identifying an initial portion of the preset selection gesture, the computer system does not change or does not change the appearance of the representation of the first user interface object in a different manner to indicate to the first user that the preset selection gesture has not been detected. In some embodiments, in response to detecting that the first user's hand movement no longer conforms to the required progression of the preset selection gesture, the computer system stops changing the appearance of the representation of the first user interface object, restores the appearance of the first user interface object, or stops displaying the representation of the first user interface object.

Altering the appearance of the representation of the first user interface object pursuant to a determination that the first user's hand movement meets preset criteria for identifying an initial portion of a preset selection gesture provides improved visual feedback to the first user (e.g., improved visual feedback that the computer system has detected the first user's hand movement that meets the preset criteria). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, while displaying a representation of a first user interface object (e.g., user interface object 7106 of FIGS. 7B-7C, or another user interface object, etc.) in a position of a representation of a first user's hand (e.g., hand 7202 of FIGS. 7B-7C, or another hand, etc.) (e.g., on a representation of the user's palm after showing the representation of the first user interface object flying from its initial display position to a position of the representation of the first user's hand), pursuant to a determination that the first user interface object has been selected by the first user, the computer system detects a third user input interacting with the representation of the first user interface object (e.g., the third input being an input provided by the hand that provided the first user input, the hand that selected the representation of the first user interface object, a hand of the first user different from the hand that provided the first user input, or a hand of the first user different from the hand that selected the representation of the first user interface object, etc. In response to detecting the third user input, the computer system controls the movement of the representation of the first user interface object and the display of the first user interface object. and displaying visual feedback for the third user input through at least one of changing an appearance of the representation of the first user interface object, and the computer system performs a second operation on the first user interface object in accordance with the third user input. In some embodiments, the visual feedback corresponds to the second operation performed on the first user interface object. In some embodiments, the visual feedback corresponds to directly manipulating the representation of the first user interface object in accordance with a movement of the first user's hand at a location corresponding to a position of the representation of the first user interface object. In some embodiments, performing the second operation includes actuating a control on the first user interface object, moving the first user interface object in three-dimensional space, playing a media item represented by the first user interface object, launching an application represented by the first user interface object, and/or initiating a communication session with another user represented by the first user interface object, etc.

In response to detecting a third user input interacting with the representation of the first user interface object, displaying visual feedback for the third user input through at least one of moving the representation of the first user interface object and changing the appearance of the representation of the first user interface object, and performing a second operation on the first user interface object in accordance with the third user input provides improved visual feedback to the user (e.g., improved visual feedback that the computer system has detected the third user input and/or improved visual feedback that the computer system is performing the second operation on the first user interface object). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the computer system updates the position of the representation of the first user interface object (e.g., user interface object 7106 of FIGS. 7B-7C, or another user interface object) according to the movement of the first user's hand such that the representation of the first user interface object maintains (e.g., remains fixed, follows, etc.) an existing spatial relationship with the updated position of the representation of the first user's hand (e.g., while the first user interface object remains in the first position within the first view of the three-dimensional environment). In some embodiments, the representation of the first user interface object remains displayed at or near the representation of the first user's hand, regardless of whether the first user actually interacts with the representation of the first user interface object. In some embodiments, the representation of the first user interface object is displayed at a position corresponding to the location of the palm of the first user's hand, and ceases to display when the representation of the hand is not within the currently displayed view of the three-dimensional environment or when the first user's hand is closed. In some embodiments, the representation of the first user interface object reappears when the representation of the first user's hand becomes visible again in the currently displayed view of the three-dimensional environment or when the representation of the palm of the hand becomes visible again in the currently displayed view of the three-dimensional environment. The first user can interact with the representation of the first user interface object (e.g., by providing a throwing gesture toward the second user or toward a shared portion of the three-dimensional environment (e.g., a representation of a wall or tabletop, etc.)) unless the first user has specifically relinquished control of the first user interface object. In some embodiments, the first user relinquishes control of the first user interface object when the first user closes his or her hand such that the representation of the first user interface object is no longer displayed in the currently displayed view of the three-dimensional environment (e.g., the first user must regain control by starting over again with another selection input while the second user is not interacting with the first user interface object).

Updating the position of the representation of the first user interface object in accordance with the movement of the first user's hand such that the representation of the first user interface object maintains an existing spatial relationship with the updated position of the representation of the first user's hand provides the user with improved visual feedback (e.g., improved visual feedback that the first user interface object is selected by subsequent user input provided by the first user). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

It should be understood that the particular order described for the operations in FIG. 8 is merely an example, and that the order described is not intended to indicate the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 9000, 10000, 11000, and 12000) are also applicable in a similar manner to method 8000 described above in connection with FIG. 8. For example, the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described above with reference to method 8000 may optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described herein with reference to other methods described herein (e.g., methods 9000, 10000, 11000, and 12000), the details of which are not repeated here for the sake of brevity.

9A-9B are flowcharts of a method, according to some embodiments, for displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where the viewpoint moves according to a user's movements in a first physical environment, the representations of the physical objects move according to movements of the physical objects in a second physical environment different from the first physical environment, and a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a preset criterion.

In some embodiments, the method 9000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera (e.g., a color sensor, an infrared sensor, and other depth-sensing camera)) facing forward from a user's hand or a user's head. In some embodiments, the method 9000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 9000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, the method 9000 is executed in a computer system (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component (e.g., display generation component 120, display generation component 7100, etc., of FIGS. 1, 3, and 4) (e.g., a heads-up display, HMD, display, touch screen, projector, etc.) and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, buttons, etc.). In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as the display generation component and at least some of the one or more input devices. In some embodiments, the computer system includes a computing component including one or more processors and memory that are separate from the display generation component and/or the one or more input devices. In some embodiments, the display generation component and the one or more input devices are integrated within and enclosed in the same housing.

In method 9000, a first user (e.g., user 7002 of FIGS. 7D-7F, or another user) is at a first location in a first physical environment (e.g., scene 105-a of FIGS. 7D-7F, or another physical environment, etc.), the computer system generates a three-dimensional environment (e.g., scene 730 of FIG. 7D) corresponding to a first viewpoint associated with the first location in the first physical environment (e.g., the first viewpoint is a virtual position corresponding to the location of the user's eyes or face when the first user is at the first location in the physical environment, the first viewpoint corresponding to a first perspective into the three-dimensional environment, etc.). 4-a) displays (9002) a first view of a three-dimensional environment (e.g., a virtual three-dimensional environment, an augmented reality environment, a three-dimensional mixed reality environment, etc.), where the first view of the three-dimensional environment includes a first object (e.g., a virtual three-dimensional environment, an augmented reality environment, a three-dimensional mixed reality environment, etc.) in a second physical environment (e.g., scene 105-b of FIGS. 7D-7F, or another physical environment, etc.) that is different from the first physical environment (e.g., the first physical environment and the second physical environment are in two different rooms, two different geographic locations, etc., and the first user and the second user are physically out of reach of each other (e.g., there is no risk of physical collision) when located in the first physical environment and the second physical environment, respectively). For example, the representation 7102′-a of the second user 7102 in FIGS. 7D-7F , or another representation of another physical object, may include a first user interface object (e.g., an avatar of the second user different from the first user, a virtual representation of a moving physical object (e.g., an animal, a flying drone, a multiplayer game opponent, etc.)) that represents a second user, a moving physical object (e.g., an animal, a vehicle, a flying drone, a multiplayer game opponent, etc.), and a respective position of the first user interface object within the three-dimensional environment is represented in a first manner within the second physical environment. corresponding to individual locations of the first object (e.g., a moving distance and a moving direction of the first object (e.g., a second user, an animal, a flying drone, etc.) in the second physical environment are respectively mapped to a moving distance and a moving direction of the first user interface object in the three-dimensional environment according to a first pre-defined object mapping relationship (e.g., a linear mapping relationship, and/or the same type of coordinate system having the same base direction, etc.), and a moving distance and a moving direction of an individual position corresponding to a viewpoint of a currently displayed view of the three-dimensional environment (e.g., the viewpoint of the first user, the virtual position of the first user in the three-dimensional environment, etc.) are respectively mapped to a moving distance and a moving direction of the first user in the first physical environment according to a first pre-defined user mapping relationship (e.g., a linear mapping relationship, and/or the same type of coordinate system having the same base direction, etc.). In some embodiments, the correspondence in the first manner is a default correspondence between motions and locations in the real world and motions and locations in the three-dimensional environment (e.g., a reality-mimicking correspondence, a correspondence that also applies to a movement of one part of the first user to another part of the first user in front of the first user, etc.). In some embodiments, the three-dimensional environment is a virtual or augmented reality environment that is at least partially shared between the first user and the second user, and the first user and the second user may view the same part of the three-dimensional environment from two different perspectives relative to the three-dimensional environment.

In some embodiments, the three-dimensional environment is not a shared environment. In some embodiments, the three-dimensional environment is a virtual three-dimensional environment. In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of a first physical environment (e.g., a camera view or a transparent pass-through view of the first physical environment, etc.) and, optionally, a representation of a physical object (e.g., a second user, an animal, a flying drone, etc.) in the second physical environment (e.g., without including a representation of the second physical environment). In some embodiments, the three-dimensional environment is an augmented reality view that includes a representation of a second physical environment (e.g., a camera view or a transparent pass-through view of the second physical environment, a video recording captured in the second physical environment, etc.) in which the camera view or recorded image of the first object has been removed and replaced by the first user interface object (e.g., such that the position of the first user interface object may be computationally or digitally corrected with respect to the representation of the second physical environment in the augmented reality environment shown via the first display generation component). In some embodiments, movement of the first user as a whole within the first physical environment (e.g., walking, running, riding a bike, jumping on, riding an elevator, etc. within the first physical environment instead of simply moving the first user's head or arm without moving the entire person within the first physical environment) causes a corresponding change in perspective of a currently displayed view of the three-dimensional environment (e.g., a translation of the perspective relative to the three-dimensional environment in a corresponding direction and/or a corresponding distance, etc.). Movement of a first object (e.g., the second user as a whole, an animal, a flying drone, etc.) within the second physical environment (e.g., walking, running, riding a bike, jumping on, riding an elevator, flying, etc.) causes a movement of a first user interface object in a corresponding direction and/or a corresponding distance, etc. within the three-dimensional environment.

In some embodiments, when the movement and/or position of the first user interface object is determined based on the movement and/or location of the first object in the second physical environment in a first manner (e.g., regardless of the first user's current location and/or movement history in the first physical environment), the first user interface object may arrive at a position in the three-dimensional environment that is within a threshold distance of a viewpoint of a currently displayed view of the three-dimensional environment shown via the first display generation component (e.g., a viewpoint determined according to the first user's current location and movement history in the first physical environment). In some embodiments, causing the first user interface object to be displayed within the threshold distance of a virtual position of the viewpoint of the currently displayed view of the three-dimensional environment causes the first user interface object to appear too large, unnatural, and/or intrusive relative to the personal space of a person (e.g., the first user) viewing the three-dimensional environment. In some embodiments, the currently displayed view of the three-dimensional environment shown via the first display generating component does not visually include a virtual representation of the first user's body (e.g., the first user's virtual position relative to the three-dimensional environment is reflected by the currently displayed view itself and the corresponding viewpoint). In some embodiments, the currently displayed view of the three-dimensional environment shown via the first display generating component visually includes a virtual representation of a part of the first user's body (e.g., the view includes an outline of the first user at the bottom of the view, a representation of the first user's hands or feet in front of the first user's eyes, and the view includes an avatar of the first user whose position within the three-dimensional environment is determined in a first manner based on the first user's movement and/or current location within the first physical environment and remains stationary relative to the display (e.g., has a fixed spatial relationship with the viewpoint of the currently displayed view of the three-dimensional environment).

In method 9000, the computer system records movement (e.g., movement from a first location to a second location in a first direction (e.g., forward, backward, left, right, up, down, 2 o'clock, 10 o'clock, etc., in the first physical environment) at a first distance and/or a first velocity) of a first user (e.g., first user 7002 of FIGS. 7D-7F) in a first physical environment and a first object (e.g., second object 7003 of FIGS. 7D-7F) in a second physical environment. and (9004) detecting at least one of (e.g., only one, only the first user, only the first object (e.g., a second user, an animal, a flying drone, etc.), both, etc.) movement of a second user 7102 or another physical object (e.g., movement in a second direction (e.g., forward, backward, left, right, up, down, 3 o'clock, 8 o'clock, etc.) from a third location to a fourth location at a second distance and/or a second speed). In some embodiments, detecting the movement of the first user is performed directly using sensors disposed with the first user and the first display generating component, and optionally detecting the movement of the first object in the second physical environment is performed indirectly (e.g., via a sensor disposed with the first object in the second physical environment, via receiving an updated position of the first user interface object transmitted from a computer system used in the second physical environment (e.g., used by the second user or another user different from the first and second users), which computer system updates the position of the first user interface object according to a first manner (e.g., according to a first preset reality-mimicking mapping relationship).

In method 9000, in response to detecting at least one of a movement of a first user within the first physical environment and a movement of a first object within the second physical environment (9006) (e.g., in response to detecting only the movement of the first user, in response to detecting only the movement of the first object, in response to detecting one of either the user or the object movement, in response to detecting both the user movement and the object movement, etc.), the computer system displays (9008) a second view of the three-dimensional environment corresponding to a second perspective (e.g., view 7304-a' of FIG. 7E, view 7304-a'' of FIG. 7F, etc.) (e.g., if the first user has not moved within the first physical environment, the second perspective is the same as the first perspective and the first user has moved within the first physical environment). When the first user moves within the three-dimensional environment, the second viewpoint differs from the first viewpoint (e.g., the first user's movement within the physical environment is mapped to the first user's movement within the three-dimensional environment, causing a perspective shift of the first user's view of the three-dimensional environment displayed to the first user). The computer system displays (9010) a first user interface object (e.g., representation 7102'-a of FIGS. 7E and 7F ) in a second view of the three-dimensional environment (e.g., view 7304-a' of FIG. 7E , view 7304-a'' of FIG. 7F , etc.). Displaying the first user interface object includes mapping the individual location of the first object within the second physical environment in a first manner (e.g., a first preset mapping relationship determined according to a reality-mimetic manner, etc.). a distinct position of the first user interface object in the three dimensional environment corresponding to a second viewpoint associated with the second view of the three dimensional environment (e.g., the same previous position of the first user interface object in the three dimensional environment if the first user interface object was substantially stationary in the three dimensional environment, a different position in the three dimensional environment if the first user interface object was moving in the three dimensional environment) is more than a threshold distance (e.g., greater than an arm's length, greater than a pre-set radius of the first user's personal space in the three dimensional environment ... (e.g., a virtual surface of a representation of the first user, a bounding box surrounding the virtual position of the first user, etc.)) from a distinct position of the three dimensional environment corresponding to a second viewpoint associated with the second view of the three dimensional environment. 7A , where the first user interface object is a virtual position of the first user in the three dimensional environment (e.g., a visible position or an invisible position in a currently displayed view of the three dimensional environment), the individual position is a virtual position of the first user's head or a virtual representation of him/her in the three dimensional environment, the individual position is one of one or more positions of a boundary surface surrounding the virtual position of the first user or his/her virtual representation in the three dimensional environment), where the first display position is an individual position of the first user interface object in the three dimensional environment. This is, for example, the scenario shown in FIG. 7E , where the display position of representation 7102′-a of the second user is the same as the individual position of representation 7102′-a calculated according to the first scheme. For example, in some embodiments, the first display position is determined in a first manner (e.g., according to a first pre-defined mapping relationship, independent of the first user's current location and/or movement history within the first physical environment) based on a current location and/or movement history of a first object (e.g., a second user, an animal, a flying drone, etc.) within the second physical environment. The first display position is a position of the first user interface object when the first user interface object is not too close to a viewpoint of a currently displayed view of the three-dimensional environment (or is not too close to a virtual position of the first user and/or is not too close to a position of a virtual representation of the first user within the three-dimensional environment) and does not appear too large, unnatural, and/or invasive relative to the first user's personal space.

Displaying the first user interface object further includes displaying the first user interface object at a second display position within the second view of the three dimensional environment in accordance with a determination that a respective position of the first user interface object within the three dimensional environment that corresponds to a respective location of the first object within the second physical environment in a first manner is less than a threshold distance from a respective position within the three dimensional environment that corresponds to a second perspective associated with the second view of the three dimensional environment (9014), wherein the second display position is offset from the respective position (e.g., shifted laterally, shifted in a respective direction (e.g., sideways, upward, downward, etc.) by more than a threshold distance, shifted laterally (e.g., left, right, etc.) by more than a threshold distance relative to the currently displayed view of the three dimensional environment, shifted upward by more than a threshold distance relative to the currently displayed view of the three dimensional environment of the first user interface object within the three dimensional environment, etc.). For example, the second display position is determined based on the current location and/or movement history of the first user within the first physical environment as well as the current location and/or movement history of the first object (e.g., the second user, an animal, a flying drone, etc.) within the second physical environment, the second display position is determined according to a second scheme different from the first scheme, and the second display position is shifted from the default position of the first user interface object if the first user interface object moves too close to the viewpoint of the currently displayed view of the three-dimensional environment such that the first user interface object does not appear too large, unnatural, and/or intrusive with respect to the personal space of the first user. This is, for example, the scenario shown in FIG. 7F, where the display position of the representation 7102'-a of the second user is offset from the individual position of the representation 7102'-a calculated according to the first scheme and calculated according to the second scheme.

In some embodiments, to determine the position of the first user interface object according to a second manner different from the first manner, the movement distance and movement direction of the first object (e.g., a second user, an animal, a flying drone, etc.) in the second physical environment is mapped to the movement distance and movement direction of the first user interface object in the three dimensional environment, respectively, according to a second pre-defined object mapping relationship (e.g., a non-linear mapping relationship, a linear mapping relationship with an additional linear or non-linear offset amount based on the current position, movement distance, and/or movement direction of an individual position corresponding to a viewpoint of a currently displayed view of the three dimensional environment (e.g., a viewpoint of the first user, a virtual position of the first user in the three dimensional environment, etc.) and/or the current position, movement distance, and/or movement direction of the first user interface object in the three dimensional environment). In some embodiments, the correspondence in the second manner includes a modification to a default correspondence (e.g., a reality-mimetic correspondence) between movements and locations in the real world and in the three-dimensional environment in order to avoid the first user interface object appearing too close in the first user's view of the three-dimensional environment (e.g., visual collision avoidance correspondence). In some embodiments, the direction and amount by which the second display position of the first user interface object is shifted or offset from the default display position (e.g., a position determined according to the reality-mimetic correspondence, according to a first pre-set mapping relationship, etc.) is determined according to a size and/or shape of the first object, and/or a size and/or shape of the first user, a size and/or shape of a bounding box associated with the first user, a size and/or shape associated with a bounding box of the first object, and/or a size and/or shape of a virtual representation of the first user in the three-dimensional environment, etc. In a first example, in some embodiments, the first object is a second user, and when the first user and/or the second user walk within their respective physical environments such that a respective position of the virtual representation of the second user within the currently displayed view of the three-dimensional environment (calculated according to the first reality-mimicking mapping relationship) exceeds a threshold distance of a viewpoint corresponding to the currently displayed view, the virtual representation of the second user is displayed at a respective position calculated according to the first reality-mimicking mapping relationship within the currently displayed view of the three-dimensional environment (e.g., based on the second user's current location and movement history within the second physical environment without taking into account the first user's current location and movement history within the first physical environment). However, as the first user and/or the second user walk within their respective physical environments, the display position of the virtual representation of the second user is shifted from its respective position (e.g., a position calculated based on the second user's current location and movement history in the second physical environment without taking into account the first user's current location and movement history in the first physical environment) so that the virtual representation of the second user does not appear to collide with the first user and/or does not overwhelm the first user's view from the first user's perspective and/or does not overlap with the first user's virtual representation in the three dimensional environment (e.g., a virtual representation that is visible or a virtual representation that is invisible in the currently displayed view of the three dimensional environment) such that the respective position of the second user's virtual representation in the currently displayed view of the three dimensional environment calculated according to the first realism-mimetic mapping relationship falls within a threshold distance of a viewpoint corresponding to the currently displayed view. In some embodiments, even if the display position of the representation of the second user is shifted in the view of the three-dimensional environment, the individual position of the representation of the second user in the three-dimensional environment is not shifted (it is just not visually reflected in the display view of the three-dimensional environment shown to the first user). The above features according to some embodiments are illustrated, for example, in Figures 7D-7F. In Figure 7E, for example, in the view 7304-a' shown to the first user 7102, the display position of the representation 7102'-a of the second user 7002 is the same as the individual position of the representation 7102'-a calculated according to the first type of correspondence between the position in the three-dimensional environment and the location in the second physical environment of the second user 7102. This is a normal scenario, where the representation 7102'-a of the second user 7102 is not within a threshold distance of the viewpoint of the first view 7304-a shown to the first user 7002 (left view of Figure 7E). In FIG. 7F, for example, in the view 7034-a'' shown to the first user 7002, the display position of the representation 7102'-a is offset from the individual position of the representation 7102'-a calculated according to the first type of correspondence. This is because the individual position places the second user's representation 7102'-a too close to the viewpoint of the first view 7304-a'' shown to the first user 7002. Thus, in the scenario shown in FIG. 7F, the representation 7102'-a is shown to the right of the representation 7002'-a of the first user 7002, even though the individual position of the second user's representation 7102'-a is actually directly in front of the representation 7002'-a in the first view 7304-a''.

In some embodiments, detecting at least one of a movement of a first user (e.g., user 7002 of FIGS. 7D-7F, or another user, etc.) within a first physical environment (e.g., scene 105-a of FIGS. 7D-7F, or another scene, etc.) and a movement of a first object (e.g., user 7102 of FIGS. 7D-7F, or another person or object, etc.) within a second physical environment (e.g., scene 105-b of FIGS. 7D-7F, or another scene, etc.) includes detecting a first movement of the first user (e.g., user 7002 of FIGS. 7D-7F, or another user, etc.) within the first physical environment while the first object (e.g., user 7102 of FIGS. 7D-7F, or another person or object, etc.) is stationary in the second physical environment. During a first movement of the first user within the first physical environment (e.g., during a movement of the entire first user that causes a movement of the viewpoint towards a representation of a stationary second user or virtual object within the three dimensional environment), a distinct position of the first user interface object within the three dimensional environment that corresponds to a distinct location of the first object within the second physical environment in a first manner (e.g., a stationary position within the three dimensional environment that corresponds to a stationary location of the first object within the second physical environment during the first movement of the first user) is greater than a threshold distance from a distinct position within the three dimensional environment that corresponds to a stationary location of the first object within the second physical environment during the first movement of the first user (e.g., a stationary position within the three dimensional environment that corresponds to a stationary location of the first object within the second physical environment during the first movement of the first user) that corresponds to a stationary location of the first object within the three dimensional environment that corresponds to a stationary location of the first object within the second physical environment during the first movement of the first user, the stationary location of the first object within the three dimensional environment that corresponds to a stationary location of the first object within the second physical environment during the first movement of the first user) that corresponds to a stationary location of the first object within the three dimensional environment ... In accordance with a determination that the representation of the first user or virtual object is outside a threshold range of the first user's viewpoint or virtual position within the three dimensional environment, the first user interface object is displayed in a first manner at a respective position of the user interface object within the three dimensional environment that corresponds to the respective location of the first object in the second physical environment (e.g., the appearance and location of the first user interface object may visually change to the first user due to the change in viewpoint (e.g., appearing to be a different size or in a different portion of the first user's field of view), but the respective three dimensional position of the first user interface object within the three dimensional environment does not change). pursuant to a determination that during a first movement of the first user within the first physical environment (e.g., during a movement of the entire first user that causes a movement of the viewpoint towards a representation of a stationary second user or virtual object within the three dimensional environment), a respective position of the first user interface object within the three dimensional environment that corresponds to a respective location of the first object within the second physical environment in a first manner (e.g., a resting position within the three dimensional environment that corresponds to a resting location of the first object within the second physical environment during the first movement of the first user) is less than or equal to a threshold distance (e.g., equal to, less than, etc.) from a respective position within the three dimensional environment that corresponds to a viewpoint associated with a currently displayed view of the three dimensional environment. (e.g., when the representation of the second user or virtual object is within a threshold range of the first user's viewpoint or virtual position within the three dimensional environment and is not otherwise shifted), the first user interface object is displayed at an adjusted position within the three dimensional environment (e.g., a position offset from a distinct position determined based on the reality-mimicking mapping relationship, a position that is outside a threshold distance from the viewpoint of the first user's virtual representation or currently displayed view of the three dimensional environment) where the first object is stationary within the second physical environment, the adjusted position within the three dimensional environment corresponding to the distinct location of the first object within the second physical environment in a second manner that is different from the first manner. For example, in some embodiments, the first user interface object appears to move autonomously within the currently displayed view of the three dimensional environment in response to movement of the first user (e.g., movement within the first physical environment corresponding to movement of the first user's viewpoint or virtual representation towards the distinct position of the first user interface object within the three dimensional environment) even though the first object remains stationary in the second physical environment. In a more specific example, when a first user makes a movement in a first physical environment corresponding to a movement of the virtual representation of the first user towards a representation of a second user or virtual object in the three-dimensional environment, if the virtual representation of the first user arrives within a threshold distance of the representation of the second user or virtual object, the representation of the second user or virtual object automatically moves out of the way from the virtual representation of the first user to avoid colliding with the virtual representation of the first user, even if the second user or virtual object remains stationary in the second physical environment. The representations of the second user and virtual object are restored to their default positions calculated using the default reality-mimicking mapping relationship after the conditions for triggering the collision avoidance offset are no longer met. In some embodiments, the first user interface object is an avatar representing a second user having a virtual conference call with the first user. When the first user walks in a direction corresponding to a movement in the three-dimensional environment towards the avatar of the second user, the avatar of the second user moves out of the way as the avatar gets too close to the virtual position of the first user. This occurs even if the second user is not moving around in their environment.

Detecting a first movement of a first user within the first physical environment while the first object is stationary within the second physical environment, displaying a first user interface object in a first manner at an individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in accordance with a determination that an individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment corresponding to a viewpoint associated with a currently displayed view of the three dimensional environment is more than a threshold distance from an individual position in the three dimensional environment corresponding to a viewpoint associated with a currently displayed view of the three dimensional environment, and displaying a first user interface object in a first manner at an individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment corresponding to a viewpoint associated with a currently displayed view of the three dimensional ... threshold distance from Displaying the first user interface object in the three dimensional environment in a second manner different from the first manner at an adjusted position in the three dimensional environment corresponding to the individual location of the first object in the second physical environment according to a determination that the individual position of the first user interface object corresponding to the individual location of the first object in the environment is less than or equal to a threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment displays the first user interface object in an appropriate position when a set of conditions are met without requiring further user input (e.g., further user input adjusting the position of the first user interface object). Performing an action when a set of conditions are met without requiring further user input enhances operability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, detecting at least one of a movement of a first user (e.g., user 7002 of Figures 7D-7F, or another user) in a first physical environment (e.g., scene 105-a of Figures 7D-7F, or another scene) and a movement of a first object (e.g., user 7102 of Figures 7D-7F, or another person or object) in a second physical environment (e.g., scene 105-b, or another scene) includes detecting a second movement of the first object in the second physical environment while the first user is stationary in the first physical environment. During a second movement of the first object within the second physical environment (e.g., during a movement of the first object (e.g., a second user, an animal, a flying drone, etc.) as a whole that moves the first user interface object in the three dimensional environment), pursuant to a determination that a respective position of the first user interface object within the three dimensional environment that corresponds to a respective location of the first object within the second physical environment in a first manner (e.g., a new position within the three dimensional environment that corresponds to a new location of the first object within the second physical environment during the second movement of the first object) is more than a threshold distance from a respective position within the three dimensional environment that corresponds to a viewpoint associated with a currently displayed view of the three dimensional environment (e.g., a representation of the second user or virtual object is outside a threshold range of the first user's viewpoint or virtual position within the three dimensional environment). Then, the first user interface object is displayed (e.g., the first user interface object is shown moving to) at an updated position within the three dimensional environment (e.g., a position that is an individual position determined based on a reality-mimicking relationship, a position that is outside a threshold distance from the virtual representation of the first user or the viewpoint of the currently displayed view of the three dimensional environment, etc.) (e.g., while the viewpoint of the currently displayed view of the three dimensional environment is stationary), where the updated position within the three dimensional environment corresponds to an updated location of the first object in the second physical environment as a result of the second movement in the first manner (e.g., the appearance and display position of the first user interface object changes due to the change in the individual three dimensional position of the first user interface object within the three dimensional environment while the viewpoint of the currently displayed view is maintained). During a second movement of the first object within the second physical environment (e.g., during a movement of the first object (e.g., a second user, an animal, a flying drone, etc.) that collectively causes a movement of the first user interface object within the three dimensional environment), individual positions of the first user interface object within the three dimensional environment that correspond to individual locations of the first object within the second physical environment in a first manner (e.g., new positions within the three dimensional environment that correspond to new locations of the first object within the second physical environment during the second movement of the first object) are less than or equal to (e.g., equal to, less than, etc.) a threshold distance from individual positions within the three dimensional environment that correspond to a viewpoint associated with a currently displayed view of the three dimensional environment (e.g., a viewpoint associated with a viewpoint of the second user or In accordance with a determination that the representation of the virtual object is within a threshold range of the first user's viewpoint or virtual position in the three dimensional environment, unless shifted away, the first user interface object is displayed (e.g., the first user interface object is shown moving to) at an adjusted updated position in the three dimensional environment (e.g., a position offset from the individual position determined based on the reality-mimicking mapping relationship, a position that is outside a threshold distance from the viewpoint of the first user's virtual representation or currently displayed view of the three dimensional environment, etc.) while the first object is stationary in the second physical environment, and the adjusted updated position in the three dimensional environment corresponds to the updated location of the first object in the second physical environment in a second manner that is different from the first manner. For example, in some embodiments, when the second user or virtual object moves in a direction in the second physical environment corresponding to movement towards the currently displayed view's viewpoint or the first user's virtual position in the three-dimensional environment, the first user interface object appears to be under the influence of simulated physical forces, such as magnetic or electrostatic repulsion, that push it away from the first user's viewpoint or virtual position and/or prevent it from getting too close to the first user's viewpoint or virtual position when the representation of the second user or virtual object is about to exceed a threshold distance of the first user's viewpoint or virtual position in the three-dimensional environment. The representations of the second user and virtual object are restored to their original movement paths, calculated using the default reality-mimicking mapping relationship, after the conditions for triggering the visual collision avoidance offset are no longer met. In some embodiments, the first user interface object is a floating conversational avatar representing the second user in a virtual conference call with the first user. When the second user walks in a direction corresponding to the movement in the three-dimensional environment towards the virtual position of the first user, the floating conversational avatar of the second user moves around the virtual position of the first user and does not get too close to the virtual position of the first user. This is a modification performed in addition to the normal movement calculated based on the default mapping relationship between the movement of the second user in the second physical environment and the movement of the second user's avatar in the three-dimensional environment.

Detecting a second movement of the first object in the second physical environment while the first user is stationary in the first physical environment, and displaying the first user interface object at a respective position of the first user interface object in the three dimensional environment that corresponds to the respective location of the first object in the second physical environment in the first manner in accordance with a determination that a respective position of the first user interface object in the three dimensional environment that corresponds to the respective location of the first object in the second physical environment in the first manner exceeds a threshold distance from a respective position in the three dimensional environment that corresponds to a viewpoint associated with a currently displayed view of the three dimensional environment, and displaying the first user interface object at a respective position of the first user interface object in the three dimensional environment that corresponds to the respective location of the first object in the second physical environment in the first manner in accordance with a determination that a respective position of the first user interface object in the three dimensional environment that exceeds a threshold distance from a respective position in the three dimensional environment that corresponds to a viewpoint associated with a currently displayed view of the three dimensional environment in the first manner while the first user is stationary in the second physical environment. Displaying the first user interface object at an adjusted position in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in a second manner different from the first manner in accordance with a determination that the individual position of the first user interface object in the three dimensional environment is less than or equal to a threshold distance from the individual position in the three dimensional environment corresponding to the viewpoint associated with the currently displayed view of the three dimensional environment displays the first user interface object in an appropriate position when a set of conditions is met without requiring further user input (e.g., further user input adjusting the position of the first user interface object). Performing an action when a set of conditions is met without requiring further user input enhances operability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, detecting at least one of a movement of a first user (e.g., user 7002 of FIGS. 7D-7F or another user) in a first physical environment (e.g., scene 105 of FIGS. 7D-7F or another scene) and a movement of a first object (e.g., user 7102 of FIGS. 7D-7F or another person or object) in a second physical environment (e.g., scene 105-b or another scene) includes simultaneously detecting a third movement of the first user in the first physical environment (e.g., movement of the first user 7002 along path 7300 of FIGS. 7D-7F or movement along another path) and a fourth movement of the first object in the second physical environment (e.g., movement of the second user 7102 or an object represented by 7102 along path 7302 of FIGS. 7D-7F or movement along another path). In some embodiments, detecting a third movement of the first user within the first physical environment is performed using a sensor positioned in-line with the first user, and detecting a fourth movement of the first object is, optionally, performed using a sensor positioned in-line with the first object and/or according to an unadjusted updated position of the first user interface object received from another computer system. During at least one of the third movement of the first user within the first physical environment and the fourth movement of the first object within the second physical environment (e.g., during a movement of the first object (e.g., a second user, an animal, a flying drone, etc.) that collectively causes a movement of the first user interface object within the three dimensional environment and a movement of the first user that collectively causes a movement of a viewpoint of the currently displayed view of the three dimensional environment and/or a movement of a virtual position of the first user within the three dimensional environment), a respective position of the first user interface object within the three dimensional environment that corresponds to a location of the first object within the second physical environment in a first manner (e.g., during the fourth movement of the first object, a new position within the three dimensional environment that corresponds to a new location of the first object within the second physical environment) is more than a threshold distance away from a respective position within the three dimensional environment that corresponds to a viewpoint associated with the currently displayed view of the three dimensional environment (e.g., updated due to the third movement of the first user within the first physical environment in accordance with a reality-mimicking relationship). In accordance with a determination that the representation of the virtual object is outside a threshold range of the first user's viewpoint or virtual position within the three dimensional environment being updated due to the third movement of the first user, the first user interface object is displayed (e.g., shown to move to) at an updated position within the three dimensional environment (e.g., a position that is a discrete position determined based on the reality-mimicking relationship, a position that is outside a threshold distance from the viewpoint of the virtual representation of the first user or the currently displayed view of the three dimensional environment, etc.) (e.g., while the viewpoint of the currently displayed view of the three dimensional environment is updated according to the third movement of the first user), where the updated position within the three dimensional environment corresponds to an updated location of the first object in the second physical environment as a result of the fourth movement in the first manner (e.g., an appearance and location of the first user interface object visually changes to the first user due to a change in the first user's discrete three dimensional position within the three dimensional environment while the viewpoint of the currently displayed view is updated according to the third movement of the first user). During at least one of a third movement of the first user within the first physical environment and a fourth movement of the first object within the second physical environment (e.g., during a movement of the first object (e.g., a second user, an animal, a flying drone, etc.) that collectively causes a movement of the first user interface object within the three dimensional environment and a movement of the first user that collectively causes a movement of a viewpoint of the currently displayed view of the three dimensional environment and/or a movement of a virtual position of the first user within the three dimensional environment), a respective position of the first user interface object within the three dimensional environment that corresponds to a location of the first object within the second physical environment in the first manner (e.g., a new position within the three dimensional environment that corresponds to a new location of the first object within the second physical environment during the second movement of the first object) is updated (e.g., updated in accordance with the third movement of the first user within the first physical environment in accordance with a reality-mimicking relationship) in the currently displayed view of the three dimensional environment. pursuant to a determination that the first user interface object is not more than (e.g., equal to, less than, etc.) a threshold distance from a respective position in the three dimensional environment corresponding to a viewpoint associated with the first user (e.g., within a threshold range of the first user's viewpoint or virtual position in the three dimensional environment if the representation of the second user or virtual object has not been shifted away), the first user interface object is displayed (e.g., shown to move to) at an adjusted updated position in the three dimensional environment (e.g., a position that is offset from the respective position determined based on the reality-mimicking relationship, a position that is outside the threshold distance from the viewpoint of the virtual representation of the first user or the currently displayed view of the three dimensional environment, etc.), where the updated position in the three dimensional environment corresponds to the location of the first object in the second physical environment in a third manner (e.g., taking into account both the fourth movement of the first object and the third movement of the first user) different from the first manner (and, optionally, the second manner). For example, in some embodiments, when a second user or virtual object moves in a direction in the second physical environment corresponding to movement towards the currently displayed view's viewpoint or the first user's virtual position in the three-dimensional environment and/or the first user moves in a direction in the first physical environment corresponding to movement towards a representation of the second user or virtual object in the three-dimensional environment, the first user interface object appears to be under the influence of another force that pushes it away from the first user's viewpoint or virtual position and/or prevents it from getting too close to the first user's viewpoint or virtual position when the representation of the second user or virtual object is about to cross a threshold distance of the first user's changing viewpoint or virtual position in the three-dimensional environment. The representations of the second user and virtual object are restored to their original movement paths calculated using the default reality-mimetic mapping relationships after the conditions for triggering the collision avoidance offsets are no longer met. In some embodiments, the virtual position of the first user or the viewpoint of the currently displayed view of the three-dimensional environment corresponds to the location of the first user in the first physical environment in a first manner (e.g., a reality-mimicking manner, a first preset mapping relationship, etc.), regardless of the movement and current location of the second user or virtual object in the second physical environment. In some embodiments, the first user interface object is a floating conversation avatar representing the second user in a virtual conference call with the first user, and when the first user and the second user walk in their physical environment, respectively, the virtual positions of the first user and the second user in the three-dimensional environment are determined based on the movements of the first user and the second user according to some preset default mapping relationship applied to the first user and the second user, respectively. For example, the virtual positions of the first user and the second user are confined to the same common virtual space even if the first user or the second user performs a significant movement distance in their respective physical environments. If the default mapping results in a visual collision of the first and second user's virtual positions within the three-dimensional environment, the second user's floating conversation avatar (as viewed by the first user via the first display generation component) moves out of the way as the floating conversation avatar gets too close to the first user's virtual position. At the same time, the first user's floating conversation avatar (as viewed by the second user via the second display generation component) moves out of the way as the floating conversation avatar gets too close to the second user's virtual position.

Concurrently detecting a third movement of the first user within the first physical environment and a fourth movement of the first object within the second physical environment, and displaying the first user interface object in the first manner at a respective position of the first user interface object in the three dimensional environment corresponding to the respective location of the first object in the second physical environment in accordance with a determination that the respective position of the first user interface object in the three dimensional environment corresponding to the respective location of the first object in the second physical environment in accordance with a determination that the respective position of the first user interface object in the three dimensional environment corresponding to the respective position ... Displaying the first user interface object in a third manner different from the first manner at an adjusted position in the three-dimensional environment that corresponds in the first manner to a respective location of the first object in the second physical environment in accordance with a determination that the respective position of the first user interface object in the three-dimensional environment corresponding to the respective position in the three-dimensional environment corresponding to the viewpoint associated with the currently displayed view of the three-dimensional environment is not more than a threshold distance from the respective position in the three-dimensional environment corresponding to the viewpoint associated with the currently displayed view of the three-dimensional environment, displays the first user interface object in an appropriate position when a set of conditions is met without requiring further user input (e.g., further user input adjusting the position of the first user interface object). Performing an action when a set of conditions is met without requiring further user input improves operability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first object is a second user (e.g., second user 7102 of FIGS. 7D-7F, or another user, etc.) located in a second physical environment (e.g., scene 105-b of FIGS. 7D-7F), and the first user (e.g., first user 7002) and the second user (e.g., second user 7102) at least partially share (e.g., both users can simultaneously view, access, and/or interact with at least a portion of) the three-dimensional environment (e.g., the environment as viewed via display generation components 7100 and 7200 of FIGS. 7D-7F). For example, the second user is provided with a view of the three-dimensional environment via a second display generation component in communication with the second computer system. The location of the second user in the second physical environment is detected by one or more sensors in communication with the second computer system. In some embodiments, the second computer system transmits data representing the second physical environment, the first computer system receives the data representing the second physical environment, and both the first computer system and the second computer system display a representation of the second physical environment (e.g., a camera view or a pass-through view of the second physical environment). In some embodiments, the first computer system transmits data representing the first physical environment, the second computer system receives the data representing the first physical environment, and both the first computer system and the second computer system display a representation of the first physical environment (e.g., a camera view or a pass-through view of the first physical environment). In some embodiments, both the first computer system and the second computer system display a representation of a virtual environment (e.g., a virtual conference room, a game world, etc.) or a representation of a third physical environment that is different from the first physical environment and the second physical environment (e.g., a physical environment of a third user that is different from the first user and the second user and that shares a three-dimensional environment with the first user and the second user, etc.). In some embodiments, when a second user's viewing position is modified with a visual collision avoidance shift/offset (e.g., in a second manner, with a second collision avoidance mapping relationship, etc.) within a view of the three-dimensional environment provided to a first user via a first display generation component, a view of the three-dimensional environment provided to a second user via a second display generation component instead indicates that the first user's viewing position is modified with a visual collision avoidance shift/offset, but that a viewpoint of the three-dimensional environment based on the second user's location has no such shift or offset.

displaying the first user interface object at the individual position of the first user interface object in the three dimensional environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment, which corresponds to the individual location of the first object in the second physical environment in a first manner, is within a threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment; and In accordance with the determination that the first user interface object exceeds a threshold distance from the respective position in the three-dimensional environment, displaying the first user interface object at a second display position in a second view of the three-dimensional environment that is offset from the respective position of the first user interface object in the three-dimensional environment, the first object being a second user located in the second physical environment, and the first user and the second user at least partially sharing the three-dimensional environment displaying the first user interface object in an appropriate position when a set of conditions is met without requiring further user input (e.g., further user input adjusting the position of the first user interface object). Performing an action when a set of conditions is met without requiring further user input improves usability of the device, as well as reducing power usage and improving battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, in a third view (e.g., view 7304-b shown in FIG. 7F(B)) of the three-dimensional environment displayed to a second user (e.g., user 7102, or another user, etc.) via a second display generation component (e.g., display generation component 7200, or another display generation component, etc.), individual positions of a second user interface object (e.g., representation 7002'-b of FIG. 7F, another representation, etc.) in the three-dimensional environment (e.g., an avatar of the first user, a talking head representing the first user, etc.) that corresponds to an individual location of the first user (e.g., user 7002 of FIG. 7D-FIG. 7F, another user, etc.) in the first physical environment (e.g., scene 105-a of FIG. 7D-FIG. 7F) in a first manner (e.g., a first predetermined mapping relationship, determined, for example, according to a reality-mimetic manner) are displayed in the three-dimensional environment corresponding to a third viewpoint associated with the third view of the three-dimensional environment ( For example, in accordance with the objection of not exceeding a threshold distance from the virtual position of the second user in the three-dimensional environment), the computer system displays the second user interface object (e.g., the representation 7002'-b of the first user 7002) at a modified position in the third view of the three-dimensional environment, the modified position being offset from the respective position of the second user interface object in the three-dimensional environment that corresponds to the respective location of the first user in the first physical environment in a first manner (e.g., a first pre-set mapping relationship determined according to a reality-mimicking manner, etc.) (e.g., as shown in FIG. 7F(B), the representation 7002'-b of the first user 7002 is displayed to the right of the representation 7102'-b in the view 7304-b'' shown to the second user 7102, even though the position of the representation 7102'-b of the first user 7002 should be directly in front of the representation 7102'-b in the view 7304-b'').

Displaying the second user interface object at a modified position in the third view of the three dimensional environment offset from the individual position of the second user interface object in the three dimensional environment corresponding to the individual location of the first user in the first physical environment in a first manner in accordance with a determination that the individual position of the second user interface object in the three dimensional environment corresponding to the third viewpoint associated with the third view of the three dimensional environment in a first manner does not exceed a threshold distance from the individual position in the three dimensional environment corresponding to the third viewpoint associated with the third view of the three dimensional environment displays the first user interface object at an appropriate position when a set of conditions is met without requiring further user input (e.g., further user input to adjust the position of the first user interface object). Performing an action when a set of conditions is met without requiring further user input improves operability of the device, as well as reducing power usage and improving battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first object is a physical object located in a second physical environment (e.g., user 7102 of FIGS. 7D-7F may represent a physical object that is not another user using display generation component 7200 in some embodiments). In some embodiments, the three-dimensional environment includes representations of other physical objects in a different physical environment than the person viewing the three-dimensional environment. For example, in some embodiments, the three-dimensional environment is optionally part of a simulated natural walk on a hiking trail for the first user and includes a real-time camera feed or recorded video of the physical environment on the hiking trail in a different physical location than the first user's physical environment. When a first user is walking in a first physical environment and experiencing a simulated natural walk with a view of the physical environment from a hiking trail, if a physical object, such as a wild bird, a flying drone, or another hiker, is also moving in the physical environment surrounding the hiking trail such that the individual position of the physical object overlaps or is within a threshold distance of the virtual position of the first user (e.g., a position corresponding to the viewpoint of the view of the hiking trail currently shown to the first user, or the position of the virtual representation of the first user in the view, etc.) without visual collision avoidance adjustment, the first user's experience is disturbed by a visual collision (e.g., the first user feels that the physical object has run through his/her head or body and/or is afraid that such a sensation or experience will occur, etc.). By automatically adjusting the display position of the representation of the physical object in such a scenario so that visual collision is avoided, the first user can experience the simulated natural walk more comfortably and smoothly. In another example, the first object is a stationary object, such as a large rock on a hiking trail, or a person sitting in the middle of the hiking trail. In some embodiments, the first user interface object is automatically moved out of the way (e.g., through digital image editing means) when the first user's virtual position approaches a large rock or a seated person's position in the view of the hiking trail. In some embodiments, following a determination that the first object is a stationary physical object (e.g., throughout the experience), the first computer system optionally stops displaying the first user interface object for a short period of time when the first user interface object is too close to the first user's virtual position in the currently displayed view of the three-dimensional environment.

In accordance with a determination that an individual position of a first user interface object in the three dimensional environment, which corresponds to an individual location of a first object in the second physical environment in a first manner, exceeds a threshold distance from an individual position in the three dimensional environment corresponding to a viewpoint associated with a currently displayed view of the three dimensional environment, displaying the first user interface object in the three dimensional environment at an individual position of the first user interface object in the three dimensional environment; in accordance with a determination that an individual position of the first user interface object in the three dimensional environment, which corresponds to an individual location of the first object in the second physical environment in a first manner, is less than or equal to a threshold distance from an individual position in the three dimensional environment corresponding to a viewpoint associated with a currently displayed view of the three dimensional environment, displaying the first user interface object in a second display position in a second view of the three dimensional environment offset from the individual position of the first user interface object in the three dimensional environment, the first object being a physical object located in the second physical environment; and displaying the first user interface object in an appropriate position when a set of conditions is met without requiring further user input (e.g., further user input to adjust the position of the first user interface object). Executing an action when a set of conditions is met without requiring further user input improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first user interface object (e.g., the representation 7102'-a of the second user 7102 in Figures 7D-7F) is an object (e.g., a floating conversational avatar of the second user, a flying bird representation, a flying drone representation, etc.) that floats in the three-dimensional environment (e.g., not attached to and/or in contact with another object or surface). In some embodiments, the representation of an inanimate object in the currently displayed view of the three-dimensional environment also has a corresponding visual collision avoidance behavior in the same manner. When the first user moves to bring a virtual position corresponding to the viewpoint of the currently displayed view of the three-dimensional environment closer to the floating virtual lantern or virtual representation of the digital assistant, the virtual representation of the floating virtual lantern or digital assistant automatically moves out of the way to the side in the currently displayed view of the three-dimensional environment. displaying the first user interface object at the first user interface object's individual position in the three dimensional environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in a first manner exceeds a threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment; displaying the first user interface object at a second display position in the second view of the three dimensional environment offset from the individual position of the first user interface object in the three dimensional environment in accordance with a determination that the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in a first manner is not more than a threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment, wherein the first user interface object is an object floating in the three dimensional environment, displaying the first user interface object in the appropriate position when a set of conditions is satisfied without requiring further user input (e.g., further user input to adjust the position of the first user interface object). Executing an action when a set of conditions is met without requiring further user input improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in accordance with a determination that the three-dimensional environment is displayed at a first level of reality, a first user interface object (e.g., representation 7102'-a of second user 7102 in FIGS. 7D-7F) is displayed with a first set of display characteristics corresponding to the first level of reality (e.g., a first resolution, a first number of dimensions, a first sharpness, a first color palette, no lighting effects, etc.), and in accordance with a determination that the three-dimensional environment is displayed at a second level of reality different (e.g., higher, lower, etc.) from the first level of reality, the first user interface object is displayed with a second set of display characteristics corresponding to the second level of reality (e.g., a second resolution, a second number of dimensions, a second sharpness, a second color palette, with lighting effects, etc.), and the second set of display characteristics differ from the first set of display characteristics (e.g., higher, lower, adding, subtracting, etc.). For example, in some embodiments, the first user interface object is a three-dimensional virtual model of the second user with fixed conversational animation when the three-dimensional environment is a virtual three-dimensional environment, and the first user interface object is a more realistic three-dimensional model of the second user with facial expressions and conversational animation generated according to a real-time video image of the second user when the three-dimensional environment is an augmented reality environment. In some embodiments, the computer system switches the display of the three-dimensional environment from a first reality level to a second reality level in response to a request of a user (e.g., the first user, the second user, etc.) switching from a virtual mode to an augmented reality mode of the shared experience. In some embodiments, the first user interface object is a two-dimensional image of the second user floating within the three-dimensional environment when the three-dimensional environment has simple, flat, opaque surfaces, and the first user interface object becomes a three-dimensional model of the second user with more elaborate facial features and lighting effects when the three-dimensional environment is switched to a more realistic virtual rendering of the physical environment. In some embodiments, automatically matching the level of realism of the first user interface object and/or how the first user interface object is rendered in the three-dimensional environment to the level of realism of the three-dimensional environment and/or how the three-dimensional environment is rendered makes the first user interface object appear a natural and unobtrusive part of the three-dimensional environment, thereby improving the viewing experience of the first user.

Displaying the first user interface object with a first set of display characteristics corresponding to the first level of reality in accordance with a determination that the three-dimensional environment is displayed at a first level of reality, and displaying the first user interface object with a second set of display characteristics different from the first set of display characteristics corresponding to the second level of reality in accordance with a determination that the three-dimensional environment is displayed at a second level of reality different from the first level of reality provides improved visual feedback to the user (e.g., improved visual feedback regarding at which level of reality the three-dimensional environment is displayed). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the second display position within the second view of the three-dimensional environment (e.g., the display position of the representation 7102'-a of the second user 7102 within view 7304-a'' shown to the first user 7002) is displaced in a first manner by a first displacement amount from a respective position of the first user interface object (e.g., the representation 7102'-a of the second user 7102) within the three-dimensional environment corresponding to a respective location of the first object (e.g., the second user 7102 or another object in FIG. 7F) within the second physical environment (e.g., scene 105-b or another scene), and the first displacement amount (e.g., having a first direction and/or a first magnitude, etc.) does not correspond to a movement in a first manner (e.g., movement along the path 7302 of FIGS. 7D-7F or movement along another path) of the first object in the second physical environment. In some embodiments, the first amount of displacement does not correspond to a movement of the first object in the second physical environment at all, or the first amount of displacement corresponds to a movement of the first object in the second physical environment in a separate manner that is different from the first manner.

In accordance with a determination that an individual position of the first user interface object in the three dimensional environment corresponding to an individual location of the first object in the second physical environment in a first manner is less than a threshold distance from an individual position in the three dimensional environment corresponding to a second viewpoint associated with the second view of the three dimensional environment, displaying the first user interface object at a second display position in the second view of the three dimensional environment that is displaced from the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner by a first displacement amount that does not correspond to a movement of the first object in the second physical environment in the first manner provides improved visual feedback to the user (e.g., improved visual feedback that the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner is less than a threshold distance from an individual position in the three dimensional environment corresponding to the second viewpoint associated with the second view of the three dimensional environment). Providing improved feedback improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first displacement amount has a direction that is determined in a first manner according to a spatial relationship (e.g., distance and relative position, etc.) between a viewpoint of a currently displayed view of the three-dimensional environment (e.g., the viewpoint of view 7304-a'' in FIG. 7F) and a respective position of a first user interface object (e.g., representation 7102'-a of second user 7102 in FIG. 7F, or another representation, etc.) in the three-dimensional environment that corresponds to a respective location of the first object (e.g., second user 7102 in FIG. 7D-FIG. 7F, or another user or object, etc.) in the second physical environment (e.g., scene 105-b, or another scene, etc.).

Displaying the first user interface object at a second display position in the second view of the three-dimensional environment, where the second display position in the second view of the three-dimensional environment is displaced from a respective position of the first user interface object in the three-dimensional environment that corresponds to a respective location of the first object in the second physical environment in a first manner by a first displacement amount having a direction determined in a first manner according to a spatial relationship between a viewpoint of the currently displayed view of the three-dimensional environment and the respective position of the first user interface object in the three-dimensional environment that corresponds to the respective location of the first object in the second physical environment, provides the user with improved visual feedback (e.g., improved visual feedback regarding the spatial relationship between the viewpoint of the currently displayed view and the respective position of the first user interface object). Providing the improved feedback improves usability of the device, as well as reducing power usage and improving battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the second display position in a second view of the three-dimensional environment (e.g., view 7304-a'' in FIG. 7F(A), or another view, etc.) is calculated in a first manner from a distinct position of a first user interface object (e.g., representation 7102'-a of a second user, another representation, etc.) in the three-dimensional environment that corresponds to a distinct location of a first object (e.g., second user 7102, another user or object, etc.) in a second physical environment (e.g., scene 105-b, another scene, etc.). The representation 7102'-a is displaced by a second displacement amount (e.g., the same as the first displacement amount, different from the first displacement amount, etc.), the second displacement amount having a direction different from the forward direction (e.g., vertical or orthogonal, pointing left, right, up, etc.) toward a respective position in the three-dimensional environment corresponding to a second viewpoint associated with the second view of the three-dimensional environment (e.g., as shown in FIG. 7F(A), the representation 7102'-a is shifted toward the viewpoint even though its position should be directly in front of the viewpoint in the view 7304-a'' shown to the first user 7002). For example, in some embodiments, regardless of from which direction the first user interface object is approaching or reaching the individual position corresponding to the viewpoint associated with the currently displayed view of the three dimensional environment (e.g., the second view, or another view that is subsequently displayed, etc.), the first user interface object is biased to the side or above the individual position corresponding to the viewpoint associated with the currently displayed view, such that the displayed position of the first user interface object does not enter into the bounded space surrounding the individual position corresponding to the viewpoint, even if the individual position of the first user interface object calculated based on the movement of the first user and/or first object within the respective physical environment according to a default, unadjusted manner (e.g., the first manner, the reality-mimicking manner, the first pre-set mapping relationship, etc.) approaches further toward or passes through the individual viewpoint associated with the currently displayed view of the three dimensional environment. In some embodiments, the first user interface object is slid sideways or upwards along the invisible glass wall as if the invisible glass wall were at a threshold distance in front of the respective position corresponding to the viewpoint of the currently displayed view of the three-dimensional environment until the respective position of the first user interface object calculated in a first manner based on the current location and movement history of the first user and the first object is no longer within the threshold distance of the respective position corresponding to the viewpoint of the currently displayed view of the three-dimensional environment. This implementation helps to avoid the first user interface object crossing the viewpoint of the first user or the virtual representation of the first user's face.

Displaying the first user interface object at a second display position in the second view of the three-dimensional environment, where the second display position in the second view of the three-dimensional environment is displaced from a respective position of the first user interface object in the three-dimensional environment that corresponds to a respective location of the first object in the second physical environment in the first manner by a second displacement amount having a direction different from a forward direction toward the respective position in the three-dimensional environment that corresponds to a second viewpoint associated with the second view of the three-dimensional environment, provides the user with improved visual feedback (e.g., improved visual feedback that the respective position of the first user interface object in the three-dimensional environment that corresponds to a respective location of the first object in the second physical environment in the first manner is less than a threshold distance from the respective position in the three-dimensional environment that corresponds to a second viewpoint associated with the second view of the three-dimensional environment). Providing the improved feedback improves operability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the second display position in the second view of the three-dimensional environment (e.g., view 7304-a'' in FIG. 7F) is displaced in a first manner by a third amount of displacement (e.g., the same as the first amount of displacement, from a distinct position of the first user interface object (e.g., representation 7102'-a of the second user, another representation, etc.) in the three-dimensional environment that corresponds to a distinct location of the first object (e.g., user 7102, another user or object, etc., in FIG. 7F) in the second physical environment (e.g., scene 105-b, or another scene, etc.). the first user interface object 7102'-a is displaced by a distance (e.g., different from the first displacement amount) and the third displacement amount has a direction (e.g., perpendicular or orthogonal, pointing left, right, upward, etc.) different from the approach direction (e.g., pointing to a common center) between the first user interface object and the respective position in the three dimensional environment corresponding to a second viewpoint associated with the second view of the three dimensional environment (e.g., as shown in FIG. 7F(A), representation 7102'-a is shifted toward the viewpoint even though it should move linearly through the viewpoint in view 7304-a'' shown to the first user 7002). For example, in some embodiments, depending on from which direction the first user interface object is approaching and reaching the individual position corresponding to the viewpoint associated with the currently displayed view of the three dimensional environment (e.g., the second view, or another view that is subsequently displayed, etc.), the third displacement amount causes the first user interface object to deviate from its approach direction to the side or above the individual position corresponding to the viewpoint associated with the currently displayed view, such that the displayed position of the first user interface object does not enter a bounded space surrounding the individual position corresponding to the viewpoint even though the individual position of the first user interface object calculated based on the movements of the first user and/or first object within the respective physical environments approaches further or passes through the individual viewpoint associated with the currently displayed view of the three dimensional environment. In some embodiments, the first user interface object is slid to the side or above along the invisible glass dome as if there were an invisible glass dome surrounding the individual position corresponding to the viewpoint of the currently displayed view of the three dimensional environment until the individual position of the first user interface object calculated based on the current location and movement history of the first user and the first object is no longer within a threshold distance of the individual position corresponding to the viewpoint of the currently displayed view of the three dimensional environment. This implementation helps to prevent the first user interface object from crossing the first user's viewpoint or the first user's virtual representation from any direction (e.g., from the front, from the side, etc.).

Displaying the first user interface object at a second display position within the second view of the three-dimensional environment, where the second display position within the second view of the three-dimensional environment is displaced from a respective position of the first user interface object within the three-dimensional environment that corresponds to a respective location of the first object within the second physical environment in the first manner by a third displacement amount having a direction different from an approach direction between the first user interface object and the respective position within the three-dimensional environment that corresponds to a second viewpoint associated with the second view of the three-dimensional environment, provides the user with improved visual feedback (e.g., improved visual feedback that the respective position of the first user interface object within the three-dimensional environment that corresponds to a respective location of the first object within the second physical environment in the first manner is less than a threshold distance from the respective position within the three-dimensional environment that corresponds to a second viewpoint associated with the second view of the three-dimensional environment). Providing the improved feedback improves operability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, at least one of the magnitude and direction of the displacement between the second display position in the second view of the three-dimensional environment (e.g., view 7304-a'' in FIG. 7F, or another view) and the respective position of the first user interface object in the three-dimensional environment (e.g., representation 7102'-a in FIG. 7F, or another representation) that corresponds in the first manner to the respective location of the first object in the second physical environment (e.g., user 7102 in FIG. 7F, or another user or object) is based on a spatial relationship (e.g., relative direction and/or distance, etc.) between the respective position of the first user interface object in the three-dimensional environment that corresponds in the first manner to the respective location of the first object in the second physical environment and the respective position in the three-dimensional environment that corresponds to the second viewpoint associated with the second view of the three-dimensional environment (e.g., the displacement is dynamically adjusted in direction and/or magnitude according to the magnitude and/or direction of the change in magnitude and/or direction, or according to the absolute value of the magnitude and/or direction).

Displaying the first user interface object at a second display position within the second view of the three-dimensional environment, where at least one of the magnitude and direction of displacement between the second display position within the second view of the three-dimensional environment and the individual position of the first user interface object within the three-dimensional environment corresponding to the individual location of the first object within the second physical environment in a first manner is based on a spatial relationship between the individual position of the first user interface object within the three-dimensional environment corresponding to the individual location of the first object within the second physical environment in a first manner and the individual position within the three-dimensional environment corresponding to a second viewpoint associated with the second view of the three-dimensional environment, provides the user with improved visual feedback (e.g., improved visual feedback regarding the spatial relationship between the individual position of the first user interface object and the individual position within the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

It should be understood that the particular order described for the operations in Figures 9A-9B is merely an example, and that the order described is not intended to indicate the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 10000, 11000, and 12000) are also applicable in a similar manner to method 9000 described above in connection with Figures 9A-9B. For example, the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described above with reference to method 9000 may optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described herein with reference to other methods described herein (e.g., methods 8000, 10000, 11000, and 12000), the details of which are not repeated here for the sake of brevity.

Figure 10 is a flowchart of a method for varying the immersion level of displaying an environment of a computer-generated experience according to changes in biometric data of a user received by a computer system, according to some embodiments.

In some embodiments, the method 10000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera (e.g., a color sensor, an infrared sensor, and other depth-sensing camera)) facing forward from a user's hand or a user's head. In some embodiments, the method 10000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 10000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, the method 10000 is executed in a computer system (e.g., computer system 101 of FIG. 1) in communication with a display generation component (e.g., display generation component 120, display generation component 7100, etc., of FIGS. 1, 3, and 4) (e.g., a heads-up display, HMD, display, touch screen, projector, etc.) and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, buttons, etc.). In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as the display generation component and at least some of the one or more input devices. In some embodiments, the computer system includes a computing component including one or more processors and memory that are separate from the display generation component and/or the one or more input devices. In some embodiments, the display generation component and the one or more input devices are integrated within and enclosed in the same housing.

In method 10000, the computer system displays (10002) a first computer-generated experience (e.g., an application user interface, a virtual experience, an augmented reality experience, a mixed reality experience, etc.) at a first level of immersion (e.g., displaying a two-dimensional application user interface, displaying a two-dimensional view of a three-dimensional environment, displaying a window or viewpoint into the three-dimensional environment that occupies a small first portion of the user's field of view, displaying a computer-generated experience having non-spatial audio, etc.) (e.g., as shown in FIG. 7G, a minimal amount of the virtual content of the computer-generated experience is displayed and a representation of the physical environment dominates view 7316). While displaying the first computer-generated experience at the first immersion level, the computer system receives (10004) biometric data (e.g., through one or more biometric sensors (e.g., various suitable medical devices, vibration sensors, cameras, thermal sensors, chemical sensors, etc.) connected to or directed at the first user (e.g., in real time, etc.) corresponding to a first user (e.g., user 7002 of FIGS. 7G-7J) (e.g., corresponding to the first user's physiological state at a first time or time period). In some embodiments, the biometric data does not include non-primary human characteristics (e.g., fingerprints, iris patterns and colors, facial features, voice prints, etc.) that typically do not change over the period that an average user is engaged with the computer-generated experience. In some embodiments, the biometric data includes heart rate, respiration rate, body temperature, serum concentrations of certain chemicals, drugs, hormones, etc., blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose levels, one or more types of biometric data that may change over time while a user is engaged in the computer-generated experience, one or more types of biometric data that may change through the user's own actions while the user is engaged in the computer-generated experience (e.g., meditating, changing breathing patterns, exercising, etc., as opposed to direct interaction with user interface elements or controls provided by a computer system), one or more types of composite metrics of multiple types of biometric data corresponding to a user's mood, happiness, and/or stress level, etc. In response to receiving 10006 biometric data (e.g., periodically received data, continuously received data, etc.) corresponding to the first user (e.g., corresponding to a physiological state of the first user at a first time or time period) and in accordance with a determination that the biometric data (e.g., most recently received biometric data, biometric data received over a most recent period of a pre-set time period, etc.) corresponding to the first user (e.g., corresponding to a physiological state of the first user at the first time or time period) satisfies a first criterion, the computer system displays 10008 the first computer-generated experience at a second immersive level (e.g., , the second immersion level provides a more immersive experience than the first immersion level, the second immersion level provides a less immersive experience than the first immersion level, etc.), the first computer-generated experience displayed at the second immersion level occupies a larger portion of the first user's field of view (e.g., a wider angular range horizontally, a wider angular range vertically, a larger field of view size, etc.) than the first computer-generated experience displayed at the first immersion level (e.g., the first computer-generated experience occupying a larger portion of the first user's field of view, optionally providing the first user with a more immersive experience than if the first computer-generated experience occupies a smaller portion of the first user's field of view). In some embodiments, the first criterion includes a preset criterion for indicating that the first user is mentally and emotionally ready to enter a more immersive experience, a preset criterion for indicating that the first user is ready to exit to a less immersive experience, etc. This is shown in Figure 7J, where view 7334 includes virtual content of the computer-generated experience that occupies a larger spatial extent than view 7316 shown in Figure 7G, for example. In some embodiments, the computer system determines that the biometric data meets the pre-set criteria according to a determination that the heart rate is lower than a first threshold heart rate, the respiration rate is lower than a first threshold respiration rate, the blood pressure is lower than a first threshold blood pressure, the user's movement is less than a first threshold movement amount for a threshold amount of time, the user's body temperature is lower than a first threshold body temperature, a stress level metric is lower than a first threshold stress level, a metric corresponding to the user's mood indicates that the user is relaxed and happy, etc. In some embodiments, the first and second immersion levels optionally differ in the amount of virtual elements present in the user's view of the computer-generated experience, the number of physical surfaces that remain visible in the computer-generated experience, the audio output mode used to play sound effects of the computer-generated experience, the level of realism depicted by the computer-generated experience, the number of dimensions depicted by the computer-generated experience, and/or the number of features and interactions made available in the computer-generated experience, etc. In method 10000, in response to receiving biometric data corresponding to the first user, and following a determination that the biometric data corresponding to the first user does not meet a first criterion (e.g., a heart rate greater than a first threshold heart rate, a blood pressure greater than a first threshold blood pressure, user movement greater than a first threshold movement amount for a threshold amount of time, a user body temperature greater than a first threshold body temperature, a stress level metric above a first threshold stress level, a metric corresponding to the user's mood indicating the user is upset and frustrated, etc.), the computer system continues to display (10010) the first computer-generated experience at the first immersion level. This is illustrated in FIG. 7G, where the biometric data may fluctuate below threshold indicator 7326 and view 7316 is maintained. In some embodiments, optionally, the first computer-generated experience includes visual and audio guidance (e.g., music, scenery, motivational messages, medication record guidance, visual, audio, or verbal breathing instructions, etc.) to assist the first user in entering a state where corresponding biometric data received from the first user would meet the first criteria. These features are illustrated, for example, in FIGS. 7G-7J, where the visual balance between the virtual content and the representation of the physical environment changes gradually and/or abruptly in accordance with changes in the biometric data corresponding to the user 7002. The visual balance and relative visual prominence between the virtual content and the representation of the physical environment represents a level of immersion that the computer-generated experience provides to the user.

In some embodiments, a computer system, displaying the first computer-generated experience at the second immersive level, receives first updated biometric data (e.g., in real time, through one or more biometric sensors (e.g., various suitable medical devices, vibration sensors, cameras, thermal sensors, chemical sensors, etc.) connected to or directed at the first user) corresponding to the first user (e.g., corresponding to the physiological state of the first user at a second time or time period after the computer system transitions to displaying the first computer-generated experience at the second immersive level). In some embodiments, the first updated biometric data includes first updated values of heart rate, respiration rate, body temperature, serum concentrations of certain chemicals, drugs, hormones, etc., blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose level, one or more types of biometric data that may change over time while the user is engaged in the computer-generated experience, one or more types of biometric data that may change through the user's own actions while the user is engaged in the computer-generated experience (e.g., meditating, changing breathing patterns, exercising, etc., as opposed to direct interaction with user interface elements or controls provided by the computer system), one or more types of composite metrics of multiple types of biometric data corresponding to the user's mood, joy, and/or stress level received after a period of time, and the like. In method 10000, the computer system, in response to receiving first updated biometric data corresponding to the first user (e.g., corresponding to a physiological state of the first user at a second time point or time period after the first time point or time period) and in accordance with a determination that the first updated biometric data corresponding to the first user (e.g., corresponding to a physiological state of the first user at a second time point or time period after the first time point or time period) satisfies a second criterion different (e.g., more restrictive, more difficult to meet) than the first criterion, updates the first computer-generated experience to a third immersive level. a third immersion level that provides a more immersive experience than the second immersion level, the third immersion level that provides a less immersive experience than the second immersion level, etc.), and the first computer-generated experience displayed at the third immersion level occupies a larger portion of the first user's field of view than the first computer-generated experience displayed at the second immersion level (e.g., the first computer-generated experience occupying a larger portion of the first user's field of view, optionally providing the first user with a more immersive experience than if the first computer-generated experience occupies a smaller portion of the first user's field of view). In some embodiments, the first immersion level, the second immersion level, and the third immersion level optionally differ in the amount of virtual elements present in the user's view of the computer-generated experience, the number of physical surfaces that remain visible in the computer-generated experience, the audio output mode used to reproduce sound effects in the computer-generated experience, the level of realism portrayed by the computer-generated experience, the number of dimensions represented by the computer-generated experience, and/or the number of features and interactions made available in the computer-generated experience, etc. In method 10000, in response to receiving first updated biometric data corresponding to the first user and in accordance with a determination that the first updated biometric data corresponding to the first user meets a first criterion and does not meet a second criterion (e.g., a heart rate less than a first threshold heart rate but greater than a second threshold heart rate, a blood pressure less than a first threshold blood pressure but greater than a second threshold blood pressure, a user's movement during a threshold time less than a first threshold movement amount but greater than a second threshold movement amount, a user's body temperature less than a first threshold body temperature but greater than a second threshold temperature, a stress level metric less than a threshold stress level but greater than a second threshold stress level, a metric corresponding to the user's mood indicating that the user is relaxed and happy, but not yet focused and restful, etc.), the computer system continues to display the first computer-generated experience at a second immersion level. In some embodiments, optionally, the first computer-generated experience includes visual and audio guidance (e.g., music, scenery, motivational messages, medication record guidance, visual, audio, or verbal breathing instructions, etc.) that assist the first user in entering a state in which corresponding biometric data received from the first user would satisfy the second criterion. In some embodiments, the first, second, and third immersion levels correspond to increasing amounts of virtual content present in the computer-generated environment and/or decreasing amounts of representations of the surrounding physical environment present in the computer-generated environment. In some embodiments, the first, second, and third immersion levels correspond to different modes of content display having increased image fidelity and/or spatial extent (e.g., angular extent, spatial depth, etc.) of the computer-generated content and decreased image fidelity and/or spatial extent of the representations of the surrounding physical environment. In some embodiments, the first level of immersion is a pass-through mode in which the physical environment is fully visible to the user through the first display generation component (e.g., as a camera view of the physical environment or through a transparent portion of the first display generation component) and the computer-generated environment includes a pass-through view of the physical environment that has a minimal amount of virtual elements simultaneously visible as the view of the physical environment or includes virtual elements on the periphery of the user's view of the physical environment (e.g., indicators and controls displayed in a peripheral area of the display). In some embodiments, the second level of immersion is a mixed reality mode in which the pass-through view of the physical environment is augmented with virtual elements generated by the computing system and has positions in the computer-generated environment that correspond to a central portion of the user's view of the physical environment and/or has positions in the computer-generated environment that correspond to locations and objects in the physical environment (e.g., virtual content is integrated with the physical environment in the view of the computer-generated environment). In some embodiments, a third level of immersion in the virtual reality mode in which the user's view of the physical environment is completely replaced or occluded by a view of the virtual content provided by the first display generation component. In some embodiments, there are four different immersion levels, the first immersion level corresponds to a pass-through mode of the first display generating component, the second immersion level includes two sub-levels A and B corresponding to two distinct sub-modes of the first display generating component (e.g., second level-A, where the user interface or user interface objects are displayed in a main part of the user's field of view while a pass-through view of the physical environment is displayed in the background of the user interface or user interface objects, and second level-B, where virtual elements are integrated with representations of physical objects in the physical environment in the augmented reality view of the physical environment), and the third immersion level corresponds to a virtual reality mode of the first display generating component.

Displaying the first computer-generated experience at a third immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the second immersion level in accordance with a determination that the first updated biometric data corresponding to the first user satisfies a second criterion different from the first criterion, and continuing to display the first computer-generated experience at the second immersion level in accordance with a determination that the first updated biometric data corresponding to the first user satisfies the first criterion and does not satisfy the second criterion, displaying the first computer-generated experience at the third immersion level when a set of conditions is met without requiring further user input (e.g., further user input to change the immersion level). Performing an action when a set of conditions is met without requiring further user input enhances operability of the device, as well as reducing power usage and improving battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, a computer system is displaying a first computer-generated experience at a discrete immersion level (e.g., a second immersion level, a third immersion level, etc.), receives second updated biometric data corresponding to a first user (e.g., the computer system receives second updated biometric data corresponding to a physiological state of the first user at the first time or time period and/or a third time or time period later than the second time or time period after transitioning from a lower immersion level to displaying the first computer-generated experience at the discrete immersion level), and the first computer-generated experience displayed at the discrete immersion level occupies a larger portion of the first user's field of view than at the first immersion level (e.g., the discrete immersion level is the second immersion level or the third immersion level). In some embodiments, the second updated biometric data includes second updated values for one or more types of composite metrics of multiple types of biometric data corresponding to heart rate, respiration rate, body temperature, serum concentrations of certain chemicals, drugs, hormones, etc., blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose levels, one or more types of biometric data that may change over time while the user is engaged in the computer-generated experience, one or more types of biometric data that may change through the user's own actions while the user is engaged in the computer-generated experience (e.g., meditating, changing breathing patterns, exercising, etc., as opposed to direct interaction with user interface elements or controls provided by the computer system), the user's mood, happiness, and/or stress level received after a period of time, and the like. In response to receiving second updated biometric data corresponding to the first user (e.g., corresponding to a physiological state of the first user at the first time or time period and/or a third time point or time period later than the second time point or time period), and in accordance with a determination that the second updated biometric data corresponding to the first user (e.g., corresponding to a physiological state of the first user at the second time point or time period later than the first time point or time period) does not satisfy the individual criteria (e.g., the first criterion, the second criterion, etc.) met for transitioning to displaying a first computer-generated experience having an individual immersion level (e.g., the second immersion level, the third immersion level, etc.), the computer system displays a first computer-generated experience having a lower immersion level (e.g., the first immersion level, the second immersion level, etc.) to be used before displaying the first computer-generated experience having the individual immersion level (e.g., the second immersion level, the third immersion level, etc.). In some embodiments, changing the immersion level of the computer-generated environment displayed via the first display generation component includes switching from displaying the computer-generated environment having a third immersion level (e.g., virtual reality mode) to displaying the computer-generated environment having a second immersion level (e.g., mixed reality mode, or temporary pass-through mode, optionally with simultaneous display of virtual reality content) in accordance with a determination that the currently received biometric data no longer meets the second criterion but still meets the first criterion. In some embodiments, when the computer-generated environment is currently displayed at the second immersion level and the computing system detects that the current biometric data no longer meets the first criterion and does not meet the second criterion, the computing system switches from displaying the computer-generated environment at the first immersion level to displaying the computer-generated environment at the second immersion level (e.g., switching from a full pass-through mode to a mixed reality mode (e.g., sub-mode A of the mixed reality mode), or causing display of a graphical user interface (e.g., a home screen, an application launching user interface) or user interface objects (e.g., application launching icons, representations of content items and experiences, etc.) to be displayed in a major portion of the user's field of view). For example, in FIG. 7J, according to some embodiments, after the computer-generated experience is displayed at a high level of immersion in response to the biometric data 7312 meeting a preset threshold as indicated by indicator 7326, when the biometric data no longer meets the preset threshold, the computer system optionally reverts to any of the states shown in FIGS. 7G-7I.

Displaying the first computer-generated experience at a lower immersion level than used prior to displaying the first computer-generated experience at the individual immersion level pursuant to a determination that the second updated biometric data corresponding to the first user does not satisfy the individual criteria satisfied for transitioning to displaying the first computer-generated experience at the individual immersion level displays the first computer-generated experience at an appropriate immersion level when a set of conditions are satisfied without requiring further user input (e.g., further user input to select the immersion level). Performing an action when a set of conditions are satisfied without requiring further user input enhances usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the biometric data (e.g., biometric data 7312 of FIGS. 7G-7J) includes a respiration rate of the first user, and the first criterion includes a criterion that is met when the respiration rate of the first user is less than a first threshold respiration rate for the first criterion to be met. In some embodiments, the biometric data, such as the respiration rate, is used as an indication of whether the first user is ready to enter a deeper immersive experience provided by the computer system and receive less stimulation from the physical environment surrounding the first user. In some embodiments, the lesser respiration rate, optionally in combination with other types of biometric data, is used to indicate that the user is ready to move to the next stage of the guided meditation provided by the computer-generated experience. In some embodiments, the biometric data includes other types of physiological data, and the first criterion includes respective threshold values for each of the other types of physiological data. In some embodiments, the respective threshold values for at least a threshold number of biometric data types must be met for the first criterion to be met. In some embodiments, the second criterion includes a criterion that is met when the respiration rate of the first user falls below a second threshold respiration rate that is lower than the first respiration rate for the second criterion to be met. In some embodiments, different (not necessarily lower) values for different types of biometric data are used for the threshold of the second criterion.

Displaying the first computer-generated experience at a second immersion level in accordance with a determination that the biometric data corresponding to the first user, including the first user's respiration rate, does not satisfy a first criterion requiring the first user's respiration rate to be less than a first threshold respiration rate, and continuing to display the first computer-generated experience at the first immersion level in accordance with a determination that the biometric data corresponding to the first user, including the first user's respiration rate, does not satisfy a first criterion requiring the first user's respiration rate to be less than a first threshold respiration rate, displays the first computer-generated experience at an appropriate immersion level when a set of conditions is met without requiring further user input (e.g., further user input to select an immersion level). Performing an action when a set of conditions is met without requiring further user input enhances operability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first criterion includes a requirement that the biometric data (e.g., biometric data 7312 of FIGS. 7G-7J, other biometric data, etc.) meet one or more pre-set thresholds (e.g., the threshold indicated by indicator 7326, other thresholds, etc.) for at least a threshold amount of time in order for the first criterion to be met. For example, in some embodiments, the biometric data includes the first user's respiration rate and/or the first user's heart rate, and the first criterion is met when the first user's average respiration rate remains below 15 breaths/minute for at least three minutes and/or the first user's average heart rate remains below 65 beats/minute for at least five minutes. In another example, the biometric data includes the first user's blood pressure and/or the first user's oxygen level, and the first criterion is met when the first user's average blood pressure remains within a first range of 120/80 (e.g., +/- 10) for at least 10 minutes and/or the first user's oxygen level remains above 99.9% for at least 3 minutes.

Displaying the first computer-generated experience at a second immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion requiring at least a threshold amount of time, and continuing to display the first computer-generated experience at the first immersion level in accordance with a determination that the biometric data corresponding to the first user, including the first user's respiration rate, does not satisfy a first criterion requiring one or more pre-set thresholds to be satisfied for at least the threshold amount of time, displays the first computer-generated experience at an appropriate immersion level when a set of conditions is met without requiring further user input (e.g., further user input to select an immersion level). Performing an action when a set of conditions is met without requiring further user input enhances operability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, displaying the first computer-generated experience at the first immersion level includes displaying virtual content (e.g., the virtual content of the first computer-generated experience, optionally changing over time) at respective first positions corresponding to the locations of one or more first portions of the physical environment (e.g., the virtual content overlays, replaces the display of, or blocks the view of a representation of the first portion of the physical environment (e.g., a single contiguous portion, or multiple separate disjoint portions, etc.) that would have been within the user's field of view if the virtual content had not been displayed) while maintaining the display (e.g., at respective second positions) of a representation of one or more second portions (different from the first portion) of the physical environment (e.g., the portions of the physical environment remain visible to the user via the display generation component (e.g., adjacent to the virtual content, as an ambient background of the virtual content, etc.)), displaying an augmented reality view of the physical environment, or displaying a complete pass-through view of the physical environment with some user interface objects). In some embodiments, displaying the first computer-generated experience at the first immersive level includes displaying virtual content in a virtual window or screen that is overlaid on, replaces the display of, or blocks a view of the representation of the physical environment (e.g., a camera view, a pass-through view through a transparent display, etc.). In some embodiments, displaying the first computer-generated experience at the first immersive level includes displaying the virtual content in a position that corresponds to the location of a first physical surface (e.g., a real window, wall, tabletop, etc.) or a first number (e.g., less than all) of physical surfaces (e.g., all walls but the ceiling and floor, all walls but the furniture, the ceiling and floor, the tabletop but the walls, etc.) in the physical environment. Displaying the first computer-generated experience at the second immersion level includes displaying virtual content (e.g., the virtual content of the first computer-generated experience, optionally changing over time) at respective first positions corresponding to locations of one or more first portions of the physical environment (e.g., portions near the center of the user's field of view) and at respective second positions corresponding to at least a portion of one or more second portions of the physical environment (e.g., portions further from the center of the user's field of view) (e.g., at the second immersion level, fewer portions of the physical environment remain visible to the user through the display generation component). In some embodiments, displaying the first computer-generated experience at the second immersion level includes displaying the virtual content within the three-dimensional environment with virtual objects overlaid on, replacing, or blocking a view of more or a larger portion of the representation of the physical environment (e.g., a camera view, a pass-through view through a transparent display, etc.). In some embodiments, displaying the first computer-generated experience at the second immersion level includes displaying virtual content in positions that correspond to the locations of more physical surfaces and/or more types of physical surfaces (e.g., real windows, walls, tabletops, furniture, etc.) in the physical environment. In some embodiments, displaying the first computer-generated experience at the third immersion level includes displaying a virtual environment (e.g., displaying a virtual reality environment) without displaying a representation of any portion of the physical environment. In some embodiments, the virtual environment still corresponds to the physical environment, e.g., the locations and spatial relationships of virtual objects and surfaces in the virtual environment still correspond to the locations and spatial relationships of at least some physical objects and surfaces in the physical environment. In some embodiments, the virtual environment does not correspond to the physical environment, except to a minimal extent (e.g., the direction of gravity and the orientation of the floor, etc.). In some embodiments, the first computer-generated experience displayed at the first immersion level is an augmented reality experience and the first computer-generated experience displayed at the second immersion level is a virtual experience.

Displaying the first computer-generated experience at a first immersion level, including displaying virtual content at respective first positions corresponding to locations of the one or more first portions of the physical environment while maintaining display of a representation of the one or more second portions of the physical environment, and displaying the first computer-generated experience at a second immersion level, including displaying virtual content at respective first positions corresponding to locations of the one or more first portions of the physical environment and respective second positions corresponding to at least some of the one or more second portions of the physical environment, provide the user with improved visual feedback (e.g., improved visual feedback regarding a current immersion level). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to receiving biometric data (e.g., biometric data 7312 of FIGS. 7G-7J, or other biometric data) corresponding to a first user (e.g., corresponding to a physiological state of the first user at a first time or time period) (e.g., periodically received data, continuously received data, etc.), and determining that a change in the biometric data (e.g., most recently received biometric data, biometric data received over a most recent period of a pre-set duration, etc.) corresponding to the first user (e.g., corresponding to a physiological state of the first user at a first time or time period) is progressing to meet a first criterion (e.g., heart rate slowing to approach a first threshold heart rate, respiration rate slowing to approach a first threshold respiration rate, body temperature decreasing to approach a first threshold body temperature, serum concentrations of certain chemicals, drugs, hormones, etc., blood pressure, brain waves, concentration level, pupil size, metabolic rate, blood glucose level, and one or more types of biometric data that may vary over time during a user's engagement in the computer-generated experience, computer-generated body In accordance with a determination that one or more types of biometric data that may vary through the user's own actions during the user's engagement with the experience (e.g., meditating, changing breathing patterns, exercising, etc., as opposed to direct interaction with user interface elements or controls provided by the computer system), one or more types of composite metrics of the multiple types of biometric data corresponding to the user's mood, happiness, and/or stress level, etc., are changing with a trend toward the first criterion being met upon continuation), the computer system gradually reduces (e.g., blurring, darkening, blocking, replacing, overlaying, etc.) a visual emphasis of at least a portion of the representation of the physical environment that was visible through the first display generation component while the first computer-generated experience was displayed at the first immersion level, and displaying the first computer-generated experience at the second immersion level includes displaying virtual content of the first computer-generated experience in a position corresponding to the portion of the representation of the physical environment such that the portion of the representation of the physical environment ceases to be visible through the first display generation component. For example, in some embodiments, when the first computer-generated experience is displayed at the first immersion level, a representation of a physical wall facing the first user (e.g., a pass-through view or a camera view of the wall) is occluded, replaced, or overlaid by a virtual wall (e.g., having a virtual wallpaper), a virtual window (e.g., having a virtual view), a virtual scenery (e.g., an open ocean view, an open scenery, etc.), a virtual desktop, a virtual movie screen, etc., while other physical walls, ceilings, floors, furniture in the room are still visible to the user through the display generation components. When the biometric data received from the first user meets a first criterion, the computer system gradually blurs and/or darkens portions of the representation of the physical environment that are still visible and replaces them with virtual content (e.g., augmenting existing virtual content, adding new virtual content, etc.). In some embodiments, the computer system displays virtual content, such as virtual wallpaper, virtual room decor, virtual scenery, a virtual movie screen, a virtual desktop, etc., that gradually replaces the blurred and/or darkened portions of the representation of the physical environment (e.g., fading in from behind portions of the representation of the physical environment, creeping in from surrounding areas of portions of the representation of the physical environment, etc.). When the transition is complete, the user's field of view of the first computer-generated experience has expanded and less of the physical environment is visible through the display generation component.

Gradually reducing visual emphasis of at least a portion of the representation of the physical environment that was viewable via the first display generation component while the first computer-generated experience was displayed at the first immersion level and displaying the first computer-generated experience at the second immersion level includes displaying virtual content of the first computer-generated experience at a position corresponding to the portion of the representation of the physical environment such that the portion of the representation of the physical environment ceases to be viewable via the first display generation component in accordance with a determination that a change in the biometric data corresponding to the first user is progressing toward satisfying the first criterion, providing improved visual feedback to the user (e.g., improved visual feedback that the biometric data corresponding to the first user is progressing toward satisfying the first criterion, visual feedback regarding the relative progress of the biometric data corresponding to the first user toward satisfying the first criterion, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to receiving biometric data (e.g., biometric data 7312 of FIGS. 7G-7J, or other biometric data) corresponding to a first user (e.g., corresponding to a physiological state of the first user at a first time or time period) (e.g., periodically received data, continuously received data, etc.), and determining that a change in the biometric data (e.g., most recently received biometric data, biometric data received over a most recent period of a pre-set duration, etc.) corresponding to the first user (e.g., corresponding to a physiological state of the first user at a first time or time period) is progressing to meet a first criterion (e.g., heart rate slowing to approach a first threshold heart rate, respiration rate slowing to approach a first threshold respiration rate, body temperature decreasing to approach a first threshold body temperature, serum concentrations of certain chemicals, drugs, hormones, etc., blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose level, and computer-generated body temperature). In accordance with a determination that one or more types of biometric data that may vary over time during the user's engagement with the experience, one or more types of biometric data that may vary through the user's own actions during the user's engagement with the computer-generated experience (e.g., meditating, changing breathing patterns, exercising, etc., as opposed to direct interaction with user interface elements or controls provided by the computer system), one or more types of composite metrics of the multiple types of biometric data corresponding to the user's mood, happiness, and/or stress level, etc., are changing with a trend that, over time, a first criterion will be met, the computer system alters (e.g., blurs, darkens, blocks, replaces, overlays, etc.) visual characteristics of at least a portion of a representation of the physical environment that was visible via the first display generation component while the first computer-generated experience was displayed at the first immersion level in an amount corresponding to the change in the biometric data corresponding to the first user. For example, in some embodiments, when the first computer-generated experience is displayed at the first immersion level, a representation of a physical wall facing the first user (e.g., a pass-through view or a camera view of the wall) is occluded, replaced, or overlaid by a virtual wall (e.g., with a virtual wallpaper), a virtual window (e.g., with a virtual view), a virtual scenery (e.g., an open ocean view, an open scenery, etc.), a virtual desktop, a virtual movie screen, etc., while other physical walls, ceilings, floors, furniture in the room remain visible to the user through the display generation component. When the biometric data received from the first user changes with a trend toward meeting the first criterion, the computer system gradually increases the amount of blurring and/or darkening applied to areas of the user's field of view that are not yet covered by the virtual content. Optionally, when the biometric data changes with an opposite trend, the amount of blurring and/or darkening is gradually reduced, and the clarity of the view of the physical environment in those areas improves again.

Pursuant to a determination that the change in the biometric data corresponding to the first user is progressing toward satisfying the first criterion, altering visual characteristics of at least a portion of the representation of the physical environment that was viewable through the first display generation component while the first computer-generated experience was displayed at the first immersion level by an amount corresponding to the change in the biometric data corresponding to the first user provides the user with improved visual feedback (e.g., improved visual feedback that the biometric data corresponding to the first user is progressing toward satisfying the first criterion, improved visual feedback regarding the relative progress of the biometric data corresponding to the first user toward satisfying the first criterion, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to receiving biometric data (e.g., biometric data 7312 of FIGS. 7G-7J, or other biometric data) corresponding to a first user (e.g., corresponding to a physiological state of the first user at a first time or time period) (e.g., periodically received data, continuously received data, etc.), and determining that a change in the biometric data (e.g., most recently received biometric data, biometric data received over a most recent period of a preset duration, etc.) corresponding to the first user (e.g., corresponding to a physiological state of the first user at a first time or time period) is progressing to meet a first criterion (e.g., heart rate slowing to approach a first threshold heart rate, respiration rate slowing to approach a first threshold respiration rate, body temperature decreasing to approach a first threshold body temperature, serum concentrations of certain chemicals, drugs, hormones, etc., blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose level, and computer In accordance with a determination that one or more types of biometric data that may vary over time during the user's engagement with the generated experience, one or more types of biometric data that may vary through the user's own actions during the user's engagement with the computer-generated experience (e.g., meditating, changing breathing patterns, exercising, etc., as opposed to direct interaction with user interface elements or controls provided by the computer system), one or more types of composite metrics of the multiple types of biometric data corresponding to the user's mood, happiness, and/or stress level, etc., are changing with a trend toward a first criterion being met over time, the computer system extends (e.g., occludes, replaces, overlays, etc.) the display of the virtual content over at least a portion of the representation of the physical environment that was visible through the first display generation component while the first computer-generated experience was displayed at the first immersion level by an amount of the change in the biometric data corresponding to the first user. For example, in some embodiments, when the first computer-generated experience is displayed at the first immersion level, a representation of a physical wall facing the first user (e.g., a pass-through view of the wall or a camera view) is occluded, replaced, or overlaid by a virtual wall (e.g., with a virtual wallpaper), a virtual window (e.g., with a virtual view), a virtual landscape (e.g., an open ocean view, an open landscape, etc.), a virtual desktop, a virtual movie screen, etc., while representations of other physical walls, ceilings, floors, furniture in the room are still displayed to the user via the display generation component. When the biometric data received from the first user changes with a trend toward meeting the first criterion, the computer system gradually expands the area of the user's field of view that is covered by the virtual content, occluding more of the view of the surrounding physical environment. Optionally, when the biometric data changes with an opposite trend, the previously occluded/covered areas are gradually restored, again revealing a view of the physical environment in those areas.

In accordance with a determination that the change in the biometric data corresponding to the first user is progressing toward satisfying the first criterion, expanding the display of the virtual content onto at least a portion of the representation of the physical environment that was visible through the first display generation component while the first computer-generated experience was displayed at the first immersion level by an amount corresponding to the change in the biometric data corresponding to the first user provides the user with improved visual feedback (e.g., improved visual feedback that the biometric data corresponding to the first user is progressing toward satisfying the first criterion, improved visual feedback regarding the relative progress of the biometric data corresponding to the first user toward satisfying the first criterion, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first criterion includes a criterion that the first user (e.g., user 7002 of FIGS. 7G-7J, or another user, etc.) makes less than a threshold amount of movement of a first type (e.g., less than a preset threshold amount of movement of the first type for a threshold amount of time, less than a threshold cumulative amount of movement of the first type, less than an absolute amount of movement of the first type, etc.) when the biometric data (e.g., biometric data 7312 of FIGS. 7G-7J, or other biometric data, etc.) is being received for the first criterion to be met (e.g., the first type of movement includes head movement, center of body movement, limb movement, and/or eye movement, etc.). For example, in some embodiments, the first user is required to remain substantially stationary when the biometric data is received and evaluated to ensure that the biometric data received is valid and/or to ensure that the first user settles into a higher immersion level of the computer-generated experience. In some embodiments, if movement of the first user that exceeds a threshold amount of movement is detected for a threshold amount of time, the computer system may (e.g., cancel, stop, reverse, etc.) the display of the first computer-generated experience having the second immersion level (or another subsequent immersion level) even if the biometric data meets requirements specified for the biometric data in the first criteria (e.g., thresholds for breathing rate, heart rate, etc.).

Displaying the first computer-generated experience at a second immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion requiring the first user to move less than a threshold amount of movement of a first type when the biometric data is received, and continuing to display the first computer-generated experience at the first immersion level in accordance with a determination that the biometric data corresponding to the first user, including the first user's respiration rate, does not satisfy the first criterion requiring the first user to move less than a threshold amount of movement of a first type when the biometric data is received, displays the first computer-generated experience at an appropriate immersion level when a set of conditions is met without requiring further user input (e.g., further user input to select an immersion level). Performing an action when a set of conditions is met without requiring further user input enhances operability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, while displaying the first computer-generated experience at the second immersion level, the computer system detects a first type of movement (e.g., head movement, body center movement, limb movement, eye movement, etc.) being performed by a first user (e.g., user 7002 of FIGS. 7G-7J, another user, etc.). In response to detecting the first type of movement being performed by the first user, and in accordance with a determination that the first type of movement exceeds a preset threshold amount of movement (e.g., exceeding a preset threshold amount of movement during a threshold time, exceeding an accumulated amount of movement, exceeding an absolute amount of movement, etc.), the computer system re-displays the first computer-generated experience at the first immersion level. For example, in some embodiments, after the biometric data received from the first user meets a first criterion, if the first user moves in one or more pre-defined ways (e.g., the first user stands, moves his/her head, stretches his/her arms, moves his/her gaze, etc.) more than a threshold amount while the first computer-generated experience is displayed at the second immersive level, the computer system interprets the first user's movement as an intent to exit the more immersive level experience and return to the less immersive level computer-generated experience previously displayed. This feature is useful when the first user is using the computer-generated experience for meditation or sleep, and the first user's movement allows the user to return to a normal state. In some embodiments, to ensure that the biometric data received is valid and/or to ensure that the first user wants to settle in to enter a more immersive level of the computer-generated experience, the first user is required to remain substantially still when the biometric data is received and evaluated. In some embodiments, if movement of the first user that exceeds a threshold amount of movement is detected during the threshold time, the computer system stops or reverses the display of the first computer-generated experience having the second immersion level (or whatever the next immersion level is), regardless of whether the biometric data still meets the requirements specified for the biometric data in the first criteria (e.g., thresholds for respiration rate, heart rate, etc.).

In response to detecting a first type of movement being performed by a first user and in accordance with a determination that the first type of movement exceeds a preset threshold amount of movement, redisplaying the first computer-generated experience at the first immersion level when a set of conditions are met without requiring further user input (e.g., further user input to select the first immersion level). Performing an action when a set of conditions are met without requiring further user input enhances usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, while displaying the first computer-generated experience at the second immersion level (e.g., as shown in FIG. 7J), the computer system detects a first type of movement (e.g., head movement, body center movement, limb movement, eye movement, etc.) being performed by a first user (e.g., user 7002 of FIG. 7J). In response to detecting the first type of movement being performed by the first user, and in accordance with a determination that the first type of movement exceeds a preset threshold amount of movement (e.g., exceeds a preset threshold amount of movement for a threshold amount of time, exceeds an accumulated amount of movement, exceeds an absolute amount of movement, etc.), the computer system switches from displaying the first computer-generated experience at the second immersion level at a first perspective to displaying the first computer-generated experience at the second immersion level at a second perspective different from the first perspective (e.g., a change in perspective of the first computer-generated experience at the second immersion level corresponds to the first type of movement performed by the first user). For example, when a more immersive experience is triggered by a change in biometric data, the first user may move around in the physical environment, turn their head, or look in different directions to change the perspective from which the view of the virtual environment is displayed.

Switching from displaying the first computer-generated experience at a second immersion level in a first perspective to displaying the first computer-generated experience at a second immersion level in a second perspective different from the first perspective in response to detecting a first type of movement being performed by a first user and in accordance with a determination that the first type of movement exceeds a preset threshold amount of movement switches the displayed perspective when a set of conditions are met without requiring further user input (e.g., further user input to change from the first perspective to the second perspective). Performing an action when a set of conditions are met without requiring further user input improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the transition from displaying a first computer-generated experience (e.g., a computer-generated experience shown via the first display generation component 7100 of FIG. 7G-7J, another computer-generated experience, etc.) at a first immersion level to displaying the first computer-generated experience at a second immersion level is a discontinuous transition (e.g., an abrupt change that simultaneously replaces, blocks view of, and/or is superimposed on a large portion of the representation of the physical environment with the virtual content without gradually blurring or fading in, or incrementally changing along one or more directions across a position corresponding to the physical surface) that occurs at a time corresponding to the time the first criterion is met. For example, in some embodiments, the first computer-generated experience is displayed at the first immersion level for an extended period of time before the first criterion is met, and once the first criterion is met by the biometric data, a clear and sudden visual change occurs that is shown as the first computer-generated experience displayed at the second immersion level replaces the first computer-generated experience displayed at the first immersion level.

The transition from displaying the first computer-generated experience at the first immersion level to displaying the first computer-generated experience at the second immersion level provides the user with improved visual feedback (e.g., improved visual feedback that the computer system has transitioned from the first immersion level to the second immersion level, improved visual feedback that the first criterion has been met, etc.) with a discontinuous transition that occurs at a time corresponding to the time the first criterion is met. Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a first computer-generated experience displayed at a first immersion level depicts a first virtual environment and a first computer-generated experience displayed at a second immersion level depicts a second virtual environment having a greater virtual depth than the first virtual environment (e.g., the first virtual environment has virtual content on a flat two-dimensional surface and the second virtual environment has virtual content at a different depth than the first user's viewpoint). Displaying the first computer-generated experience at a first immersion level depicting the first virtual environment and displaying the first computer-generated experience at a second immersion level depicting the second virtual environment having a greater virtual depth than the first virtual environment provides improved visual feedback to the user (e.g., improved visual feedback regarding whether the computer system is displaying the first or second immersion level). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, displaying the first computer-generated experience at the first immersion level includes displaying the first computer-generated experience with at least a first visual characteristic (e.g., a movement of a first virtual object, a change in lighting, etc.) that changes according to changes in the biometric data received while displaying the first computer-generated experience at the first immersion level, and displaying the first computer-generated experience at the second immersion level includes displaying the first computer-generated experience with at least a second visual characteristic (e.g., a movement of a first virtual object, a change in lighting, etc.) that changes according to changes in the biometric data received while displaying the first computer-generated experience at the second immersion level. For example, in some embodiments, the first computer-generated experience displayed at the first immersion level shows a viewport into a virtual forest nighttime scene, with the virtual trees dimly lit by a moon and stars on a dark virtual sky. In accordance with changes in the biometric data received from the first user, such as a decrease in breathing rate and/or an increase in oxygen concentration, the lighting level shown in the virtual forest increases accordingly and the virtual dark sky gradually becomes brighter and redder, simulating the onset of dawn. When the first criterion is met by the biometric data, the first computer-generated experience displayed at the second immersion level shows an expanded area in the user's field of view occupied by the virtual forest (e.g., the virtual forest expands around the user to encompass a viewpoint corresponding to the currently displayed view of the three-dimensional environment) and night falls in the virtual scene with the edge of the sun visible on the virtual horizon. In accordance with further changes in the biometric data received from the first user, such as a continued decrease in breathing rate (e.g., dropping to a threshold level) and/or a continued increase in oxygen concentration (e.g., rising to a threshold level), the lighting level shown in the virtual forest continues to increase accordingly and the virtual sky gradually becomes brighter, simulating the onset of daylight. In another example, the first computer-generated experience displayed at the first immersion level shows a virtual ocean view with crashing waves at a position in the augmented reality environment corresponding to the location of a first physical wall in front of the first user. In accordance with changes in the biometric data received from the first user, such as a decrease in breathing rate and/or a decrease in heart rate, the frequency and/or size of the ocean waves decreases accordingly. When the first criterion is met by the biometric data, the first computer-generated experience displayed at the second immersion level shows an expanded area in the user's field of view being occupied by the ocean scene (e.g., the virtual ocean view also expands to positions corresponding to the locations of the two side walls). In accordance with further changes in the biometric data received from the first user, such as a continued decrease in breathing rate (e.g., down to a threshold level) and/or a continued decrease in heart rate (e.g., down to a threshold level), the frequency and/or size of the virtual ocean waves continues to decrease accordingly.

Displaying the first computer-generated experience at the first immersion level includes displaying the first computer-generated experience with at least a first visual characteristic that changes according to changes in the biometric data received while displaying the first computer-generated experience at the first immersion level, and displaying the first computer-generated experience at the second immersion level includes displaying the first computer-generated experience with at least a second visual characteristic that changes according to changes in the biometric data received while displaying the first computer-generated experience at the second immersion level, providing improved visual feedback to the user (e.g., improved visual feedback regarding whether the computer system is displaying the first or second immersion level, improved visual feedback regarding changes in the biometric data corresponding to the first user, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to receiving biometric data (e.g., biometric data 7312 of FIGS. 7G-7J, other biometric data, etc.) corresponding to a first user (e.g., corresponding to a physiological state of a first user (e.g., user 7002 of FIGS. 7G-7J, another user, etc.) at a first time point or time period) (e.g., periodically received data, continuously received data, etc.), biometric data (e.g., most recently received biometric data, pre-set biometric data, etc.) corresponding to a first user (e.g., corresponding to a physiological state of a first user (e.g., user 7002 of FIGS. 7G-7J, another user, etc.) at a first time point or time period) corresponding to the first user (e.g., corresponding to a physiological state of a first user at a first time point or time period) (e.g., recently received biometric data, pre-set biometric data, etc.) corresponding to a ... periodically received data, continuously received data, etc.) Pursuant to a determination that the biometric data received over a period immediately preceding the period of time that was received satisfies a first criterion (e.g., the biometric data meets a threshold indicated by indicator 7326 as shown in FIG. 7J), the computer system changes the audio output mode from the first audio output mode to a second audio output mode (e.g., from stereo sound to surround sound, from headlocked audio to spatial audio, etc.), the first audio output mode having fewer computational control variables (e.g., volume of each audio source, phase of each audio source, number of audio sources, activation sequence of available audio sources, etc.) than the second audio output mode.

Changing the audio output mode from the first audio output mode to a second audio output mode having more computational control variables than the first audio output mode in accordance with a determination that the biometric data corresponding to the first user meets the first criterion provides improved audio feedback to the user (e.g., improved audio feedback that the computer system has transitioned from the first immersion level to the second immersion level, improved audio feedback that the biometric data corresponding to the first user met the first criterion, etc.). Providing improved feedback improves operability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

It should be understood that the particular order described for the operations in FIG. 10 is merely an example, and that the described order is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 9000, 11000, and 12000) are also applicable in a similar manner to method 10000 described above in connection with FIG. 10. For example, the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described above with reference to method 10000 may optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described herein with reference to other methods described herein (e.g., methods 8000, 9000, 11000, and 12000), the details of which are not repeated here for the sake of brevity.

Figure 11 is a flowchart of a method for aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

In some embodiments, the method 11000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera (e.g., a color sensor, an infrared sensor, and other depth-sensing camera)) facing forward from a user's hand or a user's head. In some embodiments, the method 11000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 11000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, the method 11000 is executed in a computer system (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component (e.g., display generation component 120, display generation component 7100, etc., of FIGS. 1, 3, and 4) (e.g., a heads-up display, HMD, display, touch screen, projector, etc.) and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, buttons, etc.). In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as the display generation component and at least some of the one or more input devices. In some embodiments, the computer system includes a computing component including one or more processors and memory that are separate from the display generation component and/or the one or more input devices. In some embodiments, the display generation component and the one or more input devices are integrated within and enclosed in the same housing.

The computer system displays (11002) a first view of the physical environment (e.g., view 7340 of FIG. 7K, or another view), where the first view of the physical environment includes a first representation (e.g., representation 7350', 7348' of FIG. 7K, etc.) of a first portion of the physical environment (e.g., the first representation is a normal color or B/W camera view of the first portion of the physical environment, a view of the physical environment through a pass-through portion of the display generation component, etc.) (e.g., the first representation is a baseline representation displayed without one or more types of computer-generated sensory augmentation). While displaying the first view of the physical environment, the computer system detects a first user input (e.g., selecting a first user interface control, activating a first hardware button in a first manner, performing a first predefined gesture input, speaking a first preset voice command, etc.) corresponding to a request to activate a first type of computer-generated sensory accommodation (e.g., binocular vision, heat vision, microscopic vision, night vision, super hearing, etc.) of two or more types of computer-generated sensory accommodation (e.g., binocular vision, heat vision, microscopic vision, night vision, super hearing, etc.) (11004). In response to detecting the first user input, the computer system displays (11006) a second view of the physical environment (e.g., second view 7361 shown in FIG. 7L , or another view, etc.), where the second view of the physical environment includes a second representation of the first portion of the physical environment (e.g., representations 7350", 7348", etc. of FIG. 7L ), where the second representation of the first portion of the physical environment has first display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) that are adjusted for the first representation of the first portion of the physical environment according to a first type of computer-generated sensory accommodation (e.g., binocular vision, microscopic vision, thermal vision, night vision, etc.). In some embodiments, the representation of the first portion of the physical environment is altered relative to the baseline representation with respect to size, resolution, focal length, magnification, and color and light intensity distribution of subject matter captured within the representation due to enhancement and/or suppression of different portions of the light and/or color spectrum due to the applied sensory accommodation. In the example shown in Figures 7K-7L, representations 7350" and 7348" are enlarged and/or moved closer to the viewpoint of view 7361 as compared to representations 7350' and 7348' in view 7340. While displaying a second view of the physical environment (e.g., view 7361 of Figure 7L, or another view, etc.), the computer system detects (11008) a second user input (e.g., selecting a second user interface control, activating a first hardware button in a second manner, activating a second hardware button in the first manner, performing a second predefined gesture input, speaking a first preset voice command, etc.) corresponding to a request to activate a second type of computer-generated sensory adjustment of the two or more types of computer-generated sensory adjustment, where the second type of computer-generated sensory adjustment is different from the first type of computer-generated sensory adjustment. In response to detecting the second user input, the computer system displays (11010) a third view of the physical environment (e.g., view 7364, or another view, etc.), where the third view of the physical environment includes a third representation of the first portion of the physical environment (e.g., representation 7350''', 7348''', etc.), where the third representation of the first portion of the physical environment has first display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focus, etc.) adjusted for the first representation of the first portion of the physical environment according to a first type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, etc.). and second display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the second representation of the physical environment according to a second type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.) (e.g., the second display characteristics have the same values in the first and second representations in some combinations of the first and second types of sensory enhancements, and the second display characteristics have different values in the first and second representations in some combinations of the first and second types of sensory enhancements). In the example shown in Figures 7K-7M, representations 7350" and 7348" are enlarged and/or moved closer to the viewpoint of view 7340 compared to representations 7350' and 7348' in view 7361, and representations 7350"" and 7348"" are altered in color and intensity compared to representations 7350" and 7348".

In some embodiments, while displaying a third view of the physical environment (e.g., view 7364 of FIG. 7M, or another view, etc.), the computer system detects a third user input corresponding to a request to activate a third type of computer-generated sensory adjustment (e.g., an adjustment function corresponding to affordance 7358 of FIG. 7M, another adjustment function not yet activated, etc.) of the two or more types of computer-generated sensory adjustments that is different from the first type of computer-generated sensory adjustments and the second type of computer-generated sensory adjustments (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.), and in response to detecting the third user input, the computer system displays a fourth view of the physical environment, the fourth view of the physical environment including a fourth representation of the first portion of the physical environment, A first display characteristic (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) that is adjusted for a first representation of a first portion of a physical environment according to a first type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.), a second display characteristic (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) that is adjusted for a second representation of the physical environment according to a second type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.), and a third display characteristic (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) that is adjusted for a third representation of the physical environment according to a third type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.).

In response to detecting a third user input corresponding to a request to activate a third type of computer-generated sensory adjustments of the two or more types of computer-generated sensory adjustments, displaying a fourth view of the physical environment including a fourth representation of the first portion of the physical environment having first display characteristics adjusted relative to the first representation of the first portion of the physical environment according to the first type of computer-generated sensory adjustments provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting and/or activating the first, second, or third types of computer-generated sensory adjustments). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensory adjustment includes simulated telescopic viewing (e.g., binocular viewing, monocular viewing, telescopic viewing, etc.) to view a distant physical object (e.g., as shown in Figures 7K-7L) (e.g., reducing the focal length of the object so that it appears closer to the user), and the second type of computer-generated sensory adjustment includes simulated microscopic viewing to magnify a nearby physical object. In some embodiments, displaying the first representation of the physical environment includes displaying a representation of the distant physical object at a first virtual position corresponding to the location of the distant physical object in the physical environment (e.g., at a size and display resolution corresponding to that virtual position in the three-dimensional environment displayed via the first display generation component). For example, the first representation of the remote physical object also appears distant in the first representation of the physical environment as the remote physical object appears in the physical environment. Displaying the second representation of the physical environment includes displaying a representation of the remote physical object at a second virtual position (e.g., at a size and display resolution corresponding to the second virtual position within the three-dimensional environment displayed via the first display generation component) that is closer to the user's viewpoint or virtual position than the first virtual position. For example, the second representation of the remote physical object appears less far away in the second representation of the physical environment and occupies a larger portion of the user's field of view of the second representation of the physical environment. Displaying the third representation of the physical environment includes displaying a representation of the remote physical object at a third virtual position, optionally closer to the user's viewpoint or virtual position than the second virtual position, and at a positive magnification factor (e.g., 100x, 20x, etc.) relative to the size of the second representation of the remote physical object (e.g., the remote physical object appears enlarged in the second or third virtual position). In an exemplary usage scenario, the display generation component first displays a camera view of a tree at a first distance (e.g., 30 meters, 50 meters, etc.) away, then with the telescope view activated, the display generation component displays the telescope view of the tree at a virtual position at a second distance (e.g., 5 meters, 10 meters, etc.) away from a viewpoint corresponding to the currently displayed representation of the physical environment, and with both the telescope view and the microscopic view activated, the display generation component displays a magnified view (e.g., 30x, 100x, etc.) of at least a portion of the tree at the current virtual position (e.g., 5 meters, 10 meters, etc.) away. In some embodiments, the camera view, telescope view, and microscopic view of the same portion of a physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing distant physical objects, and in response to detecting a second user input, displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing distant physical objects and second display characteristics adjusted for the second representation of the physical environment according to a simulated microscopic view for magnifying nearby physical objects, provides additional displayed control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between a simulated telescopic view for viewing distant objects and a simulated microscopic view for magnifying nearby physical objects). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensory adjustment includes simulated telescopic vision (e.g., as shown in 7K-7L) for viewing the remote physical object (e.g., reducing the focal length of the object so that the object appears closer to the user), and the second type of computer-generated sensory adjustment includes simulated night vision for viewing the physical object under low light conditions (e.g., high sensitivity in low light conditions, the brightness of the object is visually enhanced, small variations in brightness are magnified, etc.). In some embodiments, displaying the first representation of the physical environment includes displaying the representation of the remote physical object under low light conditions at a first virtual position (e.g., at a size and display resolution corresponding to that virtual position in the three-dimensional environment displayed via the first display generation component) corresponding to the location of the remote physical object in the physical environment. For example, the first representation of the remote physical object also appears far away in the first representation of the physical environment as the remote physical object appears in the physical environment, the first representation of the physical environment appears dark, and the object cannot be clearly identified due to the low light conditions of the physical environment. Displaying the second representation of the physical environment includes displaying representations of the remote physical objects in a second virtual position (e.g., at a size and display resolution corresponding to the second virtual position within the three-dimensional environment displayed via the first display generation component) that is closer to the user's viewpoint or virtual position than the first virtual position, but still under low light conditions. For example, the second representation of the remote physical objects may appear less distant in the second representation of the physical environment and occupy a larger portion of the user's field of view of the second representation of the physical environment, but the second representation of the physical environment still appears dark and the objects are not clearly discernible due to the low light conditions of the physical environment. Displaying the third representation of the physical environment includes displaying representations of the remote physical objects in the second virtual position with enhanced brightness and/or contrast (e.g., enhanced with images from a low-light camera or digitally enhanced, such as by combining multiple photographs and/or using machine learning). In an exemplary usage scenario, the display generation component first displays a camera view of a tree at night at a first displacement (e.g., 30 meters, 50 meters, etc.), then with the telescope view activated, the display generation component displays the telescope view of the tree at a virtual position at a second distance (e.g., 5 meters, 10 meters, etc.) away from the viewpoint corresponding to the currently displayed representation of the physical environment, but the entire scene is still dark due to low light conditions at night, and with both the telescope view and night vision activated, the display generation component displays a bright, high contrast image of the tree at the current virtual position (e.g., 5 meters, 10 meters, etc.) away. In some embodiments, the camera view, telescope view, and night vision view of the same portion of the physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing remote physical objects, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing remote physical objects and second display characteristics adjusted for the second representation of the physical environment according to a simulated night vision for viewing physical objects under low light conditions provides additional displayed control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between the simulated telescopic view for viewing remote objects and the simulated night vision for viewing physical objects under low light conditions). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated telescopic vision (e.g., as shown in 7K-7L) for viewing remote physical objects (e.g., reducing the focal length of the object so that the object appears closer to the user), and a second type of computer-generated sensory modulation includes simulated thermal vision (e.g., as shown in 7L-7M) for viewing physical objects having various thermal emission profiles (e.g., high sensitivity to temperature variations, presenting variations in color and/or intensity according to variations in temperature and/or thermal emission, etc.). In some embodiments, displaying the first representation of the physical environment includes displaying a representation of the remote physical object at a first virtual position corresponding to the location of the remote physical object in the physical environment (e.g., at a size and display resolution corresponding to that virtual position in the three-dimensional environment displayed via the first display generation component). For example, the first representation of the remote physical object also appears distant within the first representation of the physical environment as the remote physical object appears in the physical environment. Displaying the second representation of the physical environment includes displaying a representation of the remote physical object at a second virtual position (e.g., at a size and display resolution corresponding to the second virtual position in the three-dimensional environment displayed via the first display generation component) that is closer to the user's viewpoint or virtual position than the first virtual position. For example, the second representation of the remote physical object appears less far away in the second representation of the physical environment and occupies a larger portion of the user's field of view of the second representation of the physical environment. Displaying the third representation of the physical environment includes displaying a representation of the remote physical object at the second virtual position along with its thermal radiation profile or temperature map. In an exemplary usage scenario, the display generation component first displays a camera view of a tree a first distance away (e.g., 30 meters, 50 meters, etc.), then with the telescope view activated, the display generation component displays the telescope view of the tree at a virtual position a second distance away (e.g., 5 meters, 10 meters, etc.) from the viewpoint corresponding to the currently displayed representation of the physical environment, and with both the telescope view and thermal vision activated, the display generation component displays a heat map of the tree showing the luminous profile of a squirrel hidden among the leaves at the current virtual position (e.g., 5 meters, 10 meters, etc. away). In some embodiments, the camera view, telescope view, and thermal vision views of the same portion of a physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing remote physical objects, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing remote physical objects and second display characteristics adjusted for the second representation of the physical environment according to a simulated thermal view for viewing physical objects having various thermal radiation profiles, provides additional displayed control options without cluttering the UI with additional displayed controls (e.g., for selecting or switching between the simulated telescopic view for viewing remote objects and the simulated thermal view for viewing physical objects having various thermal radiation profiles. Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device and, in addition, reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory adjustment includes simulated telescopic viewing (e.g., reducing the focal length of an object so that the object appears closer to the user) to view a remote physical object (e.g., as shown in 7K-7L), and a second type of computer-generated sensory adjustment includes modifying the view of the physical object with a filter (e.g., a color filter, a light frequency filter, an intensity filter, a motion filter, etc.). In some embodiments, displaying the first representation of the physical environment includes displaying a representation of the remote physical object at a first virtual position corresponding to the location of the remote physical object in the physical environment (e.g., at a size and display resolution corresponding to that virtual position in the three-dimensional environment displayed via the first display generation component). For example, the first representation of the remote physical object also appears distant in the first representation of the physical environment as the remote physical object appears in the physical environment. Displaying the second representation of the physical environment includes displaying a representation of the remote physical object (e.g., at a size and display resolution corresponding to the second virtual position in the three-dimensional environment displayed via the first display generation component) at a second virtual position that is closer to the user's viewpoint or virtual position than the first virtual position. For example, the second representation of the remote physical object appears less far away in the second representation of the physical environment and occupies a larger portion of the user's field of view of the second representation of the physical environment. Displaying the third representation of the physical environment includes displaying the representation of the remote physical object at the second virtual position with some filtering, such as color and/or intensity. In some embodiments, a motion filter is applied to filter portions of the second representation of the physical environment that do not have motion and highlight portions that have motion (e.g., moving leaves, animals, people, etc.). In an exemplary usage scenario, the display generation component initially displays a camera view of a tree at a first distance (e.g., 30 meters, 50 meters, etc.). When the telescope view is activated, the display generation component displays a telescope view of the tree at a virtual position a second distance (e.g., 5 meters, 10 meters, etc.) away from the viewpoint corresponding to the currently displayed representation of the physical environment, and when both the telescope view filter and the color/intensity/motion filter are activated, the display generation component displays a filtered image of the tree at the current virtual position (e.g., 5 meters, 10 meters, etc.). A blurred, desaturated image of the tree (with a color filter applied) at the current virtual position (e.g., 5 meters, 10 meters, etc.) or a bright orange hat and safety vest over the filtered image of the tree) showing visual highlighting of a camouflaged insect moving over the blurred, desaturated image of the tree. In some embodiments, the camera view and the telescope view of the same portion of the physical object are, optionally, captured by different cameras and/or sensors, or, optionally, enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing a remote physical object, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing a remote physical object and second display characteristics adjusted for the second representation of the physical environment according to a filter that modifies the view of the physical object, provides additional displayed control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between the simulated telescopic view for viewing a remote object and a filter that modifies the view of the physical object). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory adjustment includes simulated telescopic viewing (e.g., as shown in 7K-7L) to view a remote physical object (e.g., reducing the focal length of the object so that the object appears closer to the user), and a second type of computer-generated sensory adjustment includes selective sound enhancement (e.g., enhancing the volume, selectively enhancing/suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-generating physical objects, a physical object that is centered in the current field of view, etc.). In some embodiments, displaying the first representation of the physical environment includes displaying a representation of the remote physical object at a first virtual position corresponding to the location of the remote physical object in the physical environment (e.g., at a size and display resolution corresponding to that virtual position in the three-dimensional environment displayed via the first display generation component). For example, the first representation of the remote physical object also appears distant in the first representation of the physical environment as the remote physical object appears in the physical environment. Displaying the second representation of the physical environment includes displaying a representation of the remote physical object at a second virtual position (e.g., at a size and display resolution corresponding to the second virtual position in the three-dimensional environment displayed via the first display generation component) that is closer to the user's viewpoint or virtual position than the first virtual position. For example, the second representation of the remote physical object may appear less distant in the second representation of the physical environment and occupy a larger portion of the user's field of view of the second representation of the physical environment. Displaying the third representation of the physical environment includes displaying the representation of the remote physical object at the second virtual position along with a visual identification of a localized sound source in the physical environment on or near the representation of the remote physical object, where an enhanced audio output corresponding to sound from the localized sound source is output along with the display of the third representation of the physical environment. In an exemplary usage scenario, the display generation component first displays a camera view of a tree at night a first distance away (e.g., 30 meters, 50 meters, etc.), then with the telescope view activated, the display generation component displays the telescope view of the tree at a virtual position a second distance away (e.g., 5 meters, 10 meters, etc.) from the viewpoint corresponding to the currently displayed representation of the physical environment, and with both the telescope view and the enhanced hearing activated, the display generation component displays a circle superimposed on the image of the tree at the current virtual position (e.g., 5 meters away, 3 meters away, etc.) indicating the position of a singing bird in the tree. A localized chirp from the bird is played, optionally in a spatial audio output mode, along with the three views at the second distance away (e.g., 5 meters, 10 meters, etc.) from the viewpoint. In some embodiments, the camera view, telescope view, and localized sound of the same part of the physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated telescopic view for viewing remote physical objects, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to the simulated telescopic view for viewing remote physical objects and second display characteristics adjusted for the second representation of the physical environment according to selective audio adjustments for sounds corresponding to a subset of the physical objects in the physical environment provides additional displayed control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between the simulated telescopic view for viewing remote objects and selective audio adjustments for sounds corresponding to a subset of the physical objects in the physical environment). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, while displaying a third representation of the physical environment (e.g., the representation shown in FIG. 7M, or another representation, etc.), the computer system outputs sounds corresponding to the first portion of the physical environment that is visible in the third representation of the physical environment (e.g., portions 7366″ and 7368″ of FIG. 7M, or another portion, etc.), the sounds being selectively enhanced (e.g., enhanced in volume, with modifications to the amplitude of some selected frequencies, etc.) relative to sounds from sources outside the first portion of the physical environment. In an exemplary scenario, when two trees are visible in the first representation of the physical environment with audio output of sounds captured from the entire physical environment, and the first tree is seen in a telescope view and enhanced hearing is activated, the bird call in the first tree is enhanced (e.g., made louder) relative to the squirrel call in the second tree and played in spatial audio to have a virtual position corresponding to the virtual position of the first tree. Concurrently displaying a third representation of the physical environment and outputting a sound corresponding to a first portion of the physical environment that is visible in the second representation of the physical environment, the sound being selectively enhanced relative to sounds from a sound source outside the first portion of the physical environment, provides improved audio feedback to the user (e.g., improved audio feedback regarding the first portion of the physical environment). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, simultaneously with displaying the third representation of the physical environment, the computer system displays text output corresponding to speech emanating from a first portion of the physical environment that is visible in both the second and third representations of the physical environment (e.g., portions 7366'' and 7368'', or another portion, of FIG. 7M), where the speech is selectively emphasized relative to sounds from sources outside the first portion of the physical environment. In an exemplary scenario, when trees and a house are visible in the first representation of the physical environment along with audio output of sounds captured from the entire physical environment, the house is viewed in a telescope view, and enhanced hearing is activated, the sounds of the speech from the house are enhanced (e.g., made louder, clearer, etc.) relative to sounds of birds singing in the trees, and text output, such as subtitles, a transcript, a translation, etc., is displayed. In some embodiments, the sounds of the speech are replaced with a corresponding audio translation. Displaying a text output corresponding to speech emanating from a first portion of the physical environment that is visible in both the second and third representations of the physical environment, while simultaneously displaying the third representation of the physical environment, selectively highlights the speech relative to sounds from sources outside the first portion of the physical environment, providing improved visual feedback to the user (e.g., improved visual feedback regarding the speech emanating from the first portion of the physical environment). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated microscopic vision to magnify nearby physical objects, and a second type of computer-generated sensory modulation includes simulated thermal vision (e.g., as shown in Figs. 7L-7M) to view physical objects having various thermal radiation profiles (e.g., high sensitivity to temperature variations, presenting variations in color and/or intensity according to temperature and/or thermal radiation variations, etc.). An exemplary usage scenario is where the display generation component first displays a camera view of a microchip in a mobile phone, then with the microscopic view activated, the display generation component displays a magnified view of the microchip, and with both the microscopic view and thermal vision activated, the display generation component displays a heat map of the microchip at the current magnification level, showing hot versus cold areas on the microchip. In some embodiments, the camera view, microscopic view, and thermal vision views of the same portion of a physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques. In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated microscopic view to magnify nearby physical objects, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to a simulated microscopic view to magnify nearby physical objects and second display characteristics adjusted for the second representation of the physical environment according to a simulated thermal view to view physical objects having different thermal emission profiles, provides additional displayed control options without cluttering the UI with additional displayed controls (e.g., for selecting or switching between the simulated microscopic view to magnify nearby physical objects and the simulated thermal view to view physical objects having different thermal emission profiles). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory adjustment includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, the brightness of the object is visually enhanced, small variations in brightness are magnified, etc.), and a second type of computer-generated sensory adjustment includes simulated telescopic vision for viewing remote physical objects (e.g., as shown in Figures 7K-7L) (e.g., reducing the focal length of the object so that it appears closer to the user). In an exemplary usage scenario, the display generation component first displays a camera view of a tree at night at a first distance (e.g., 30 meters, 60 meters, etc.) away, with little detail of the tree visible due to low light conditions, then with night vision activated, the display generation component displays a bright, high-contrast image of the camera view showing the tree at the first distance (e.g., 30 meters, 60 meters, etc.) away, and with both night vision and telescopic view activated, the display generation component displays a bright, high-contrast image of the tree in a virtual position 5 meters away.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated night vision for viewing physical objects under low light conditions, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated night vision for viewing physical objects under low light conditions and second display characteristics adjusted for the second representation of the physical environment according to simulated telescopic vision for viewing remote physical objects, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated night vision for viewing physical objects under low light conditions and simulated telescopic vision for viewing remote physical objects). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensory adjustment includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, object brightness is visually enhanced, small variations in brightness are magnified, etc.), and the second type of computer-generated sensory adjustment includes simulated microscopic vision for magnifying nearby physical objects. In an exemplary usage scenario, the display generation component first displays a camera view of a tabletop in a dark room. Room details are barely visible due to the low light conditions, and when night vision is activated, the display generation component displays a bright, high-contrast image of the room showing several coins on the table, and when both night vision and the microscopic view are activated, the display generation component displays a bright, high-contrast image of the coins magnified, showing the coin details. In some embodiments, the microscopic image of the coins is optionally captured at a different time and/or from a camera view of the tabletop than that of the night vision image. Information from the different types of sensors and cameras is combined to generate a third representation of the physical environment. In some embodiments, information extracted from the first and/or second representations of the physical environment (e.g., size of the coin, characteristics of the image on the coin, etc.) is used as a basis (e.g., from online sources, image databases, etc.) to obtain additional information about details of the physical environment (e.g., type of coin, year of coin, material of coin, etc.) and generate a third representation of the physical environment (e.g., showing further details of the coin not captured by the first and second representations).

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated night vision for viewing physical objects under low light conditions, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated night vision for viewing physical objects under low light conditions and second display characteristics adjusted for the second representation of the physical environment according to simulated microscopic vision for magnifying nearby physical objects, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated night vision for viewing physical objects under low light conditions and simulated microscopic vision for magnifying nearby physical objects). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, the brightness of the object is visually enhanced, small variations in brightness are magnified, etc.), and a second type of computer-generated sensory modulation includes simulated heat vision (e.g., as shown in Figures 7L-7M) for viewing physical objects having various thermal radiation profiles (e.g., high sensitivity to variations in temperature, presenting variations in color and/or intensity according to variations in temperature and/or thermal radiation, etc.). In an exemplary usage scenario, the display generation component first displays a camera view of a tree at night. The details of the tree are barely visible due to the low light conditions. When night vision is activated, the display generation component displays a brightened, high-contrast image of the tree showing two nests, and when both night vision and heat vision are activated, the display generation component displays a brightened, high-contrast image of the nests, one nest having a different color and/or light intensity than the other nest, indicating that one nest has recently been inhabited by an animal and the other has not. In some embodiments, the camera view, night vision view, and thermal vision view of the same portion of a physical object are, optionally, captured by different cameras and/or sensors, or, optionally, enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated night vision for viewing physical objects under low light conditions in response to the second representation of the first portion of the physical environment, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated night vision for viewing physical objects under low light conditions and second display characteristics adjusted for the second representation of the physical environment according to simulated thermal vision for viewing physical objects having various thermal radiation profiles provide additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated night vision for viewing physical objects under low light conditions and simulated thermal vision for viewing physical objects having various thermal radiation profiles). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensory modulation includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, brightness of objects is visually enhanced, small variations in brightness are magnified, etc.), and the second type of computer-generated sensory modulation includes selective audio enhancement (e.g., enhancing volume, selectively enhancing/suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object that is centered in the current field of view, etc.). In some embodiments, displaying the first representation of the physical environment includes displaying a representation of the physical objects under low light conditions. Displaying the second representation of the physical environment includes displaying a bright, high-contrast image of a dark room with normal audio output of sounds captured from throughout the room. Displaying the third representation of the physical environment includes displaying the same bright, high-contrast image of a dark room with localized sound sources identified and visually enhanced in the image, and with enhanced audio output corresponding to sounds from the localized sound sources. In an exemplary usage scenario, the display generation component first displays a camera view of a dark room with no discernible sounds, then with night vision activated, the display generation component displays a high brightness and high contrast view of the dark room showing the furniture and appliances in the room, and with both night vision and high hearing activated, the display generation component displays a circle overlaid on a bright, high contrast image of the dark room showing the position of the refrigerator where a low frequency vibration sound is heard. The localized sound from the refrigerator is enhanced and played along with the night vision view of the room, and optionally with a spatial audio output mode, along with an enhancement of the frequencies in the vibration of the refrigerator. In some embodiments, the camera view, telescopic view, and localized sound of the same portion of the physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques. In some embodiments, a user input is detected that selects a sound source for which enhanced sound is desired (e.g., a tap on the refrigerator in the night vision view, another input selecting a different sound source, etc.).

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with simulated night vision for viewing physical objects under low light conditions, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with simulated night vision for viewing physical objects under low light conditions and second display characteristics adjusted for the second representation of the physical environment in accordance with selective sound adjustments for sounds corresponding to a subset of the physical objects in the physical environment provide additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated night vision for viewing physical objects under low light conditions and selective sound adjustments for sounds corresponding to a subset of the physical objects in the physical environment). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated heat vision (e.g., as shown in FIGS. 7L-7M) for viewing physical objects having various thermal radiation profiles (e.g., high sensitivity to temperature variations, presenting variations in color and/or intensity according to temperature and/or thermal radiation variations, etc.), and a second type of computer-generated sensory modulation includes simulated telescopic vision (e.g., as shown in FIGS. 7K-7L) for viewing remote physical objects (e.g., reducing the focal length of the object so that it appears closer to the user). In an exemplary usage scenario, the display generation component first displays a camera view of a forest, and then, when heat vision is activated, one area of the forest in the heat vision view appears to have a higher temperature than other areas of the forest in the heat vision view, with the area of the forest having the higher temperature being a first distance away in the heat vision view (e.g., 50 meters, 100 meters, etc.). When both the thermal and telescopic views are activated, the display generation component displays a telescopic view of the area having a higher temperature at a virtual position that is a second distance (e.g., 5 meters, 10 meters, etc.) away from the viewpoint that corresponds to the currently displayed representation of the physical environment. A heat map of the area having a higher temperature displayed at the virtual position that is a second distance (e.g., 5 meters, 10 meters, etc.) away shows a representation of a smoldering dead tree trunk. In some embodiments, the camera view, telescopic view, and thermal view of the same portion of the physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated thermal vision for viewing physical objects having varying thermal emission profiles, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated thermal vision for viewing physical objects having varying thermal emission profiles and second display characteristics adjusted for the second representation of the physical environment according to simulated telescopic vision for viewing remote physical objects, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated thermal vision for viewing physical objects having varying thermal emission profiles and simulated telescopic vision for viewing remote physical objects). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated thermal vision (e.g., as shown in Figs. 7L-7M) for viewing physical objects with various thermal radiation profiles (e.g., high sensitivity to temperature variations, presentation of color and/or intensity variations according to temperature and/or thermal radiation variations, etc.), and a second type of computer-generated sensory modulation includes simulated microscopic vision for magnifying nearby physical objects. In an exemplary usage scenario, the display generation component first displays a camera view of the internal structure of the mobile device. Then, when the thermal vision is activated, one area of the internal structure of the mobile device appears to have a higher temperature in the thermal vision view than other areas of the structure, and the area of the structure having the higher temperature is shown in a modified color. When both the thermal vision view and the microscopic view are activated, the display generation component displays a magnified view of the area having the higher temperature. In some embodiments, the camera view, the telescopic view, and the thermal vision view of the same portion of the physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated thermal vision for viewing physical objects having varying thermal emission profiles, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment according to simulated thermal vision for viewing physical objects having varying thermal emission profiles and second display characteristics adjusted for the second representation of the physical environment according to simulated microscopic vision for magnifying nearby physical objects, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated thermal vision for viewing physical objects having varying thermal emission profiles and simulated microscopic vision for magnifying nearby physical objects). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated heat vision (e.g., as shown in Figures 7L-7M) for viewing physical objects having various thermal radiation profiles (e.g., high sensitivity to variations in temperature, presenting variations in color and/or intensity according to variations in temperature and/or thermal radiation, etc.), and a second type of computer-generated sensory modulation includes simulated night vision for viewing physical objects under low light conditions (e.g., high sensitivity in low light conditions, luminance of objects is visually enhanced, small variations in brightness are magnified, etc.). In an exemplary usage scenario, the display generation component first displays a camera view of a house at night. Details of the house are barely visible due to the low light conditions. With thermal vision activated, the display generation component displays one area of the thermal view as having a higher temperature than other areas of the thermal view, but it is unclear which structural portion of the house hosts the hot area; with night vision and thermal vision activated, the display generation component displays a bright, high contrast image of the house, showing the hot area in the downspout down the front side of the house. In some embodiments, the camera view, night vision view, and thermal vision view of the same portion of a physical object are optionally captured by different cameras and/or sensors, or are optionally enhanced using computational techniques.

In response to detecting a first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with simulated thermal vision for viewing physical objects having varying thermal emission profiles, and displaying a third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with simulated thermal vision for viewing physical objects having varying thermal emission profiles and second display characteristics adjusted for the second representation of the physical environment in accordance with simulated night vision for viewing physical objects under low light conditions provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or switching between simulated thermal vision for viewing physical objects having varying thermal emission profiles and simulated night vision for viewing physical objects under low light conditions). It improves usability of the device by providing additional control options without cluttering the UI with additional displayed controls, and also reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, a first type of computer-generated sensory modulation includes simulated heat vision (e.g., as shown in Figures 7L-7M) for viewing physical objects having various thermal emission profiles (e.g., high sensitivity to variations in temperature, presenting color and/or intensity variations according to variations in temperature and/or thermal emission, etc.), and a second type of computer-generated sensory modulation includes selective sound enhancement (e.g., enhancing volume, selectively enhancing/suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-generating physical objects, a physical object that is centered in the current field of view, etc.). In an exemplary usage scenario, the display generation component initially displays a camera view of a house at night. Details of the house are barely visible due to low light conditions, then when thermal vision is activated the display generation component displays that one area of the thermal view appears to have a higher temperature than other parts of the thermal view, but it is unclear which structural part of the house hosts the hot area, when both night vision and thermal vision are activated the display generation component displays a bright, high contrast image of the house showing the hot area inside the downspout down the front side of the house, when both night vision and enhanced hearing are activated the display generation component displays a circle overlaid on the hot area indicating the position of a sound source (e.g., a nest of snoring rodents). Localized sounds from the highlighted sound source are enhanced and played along with the thermal view of the house, optionally with spatial audio output mode and enhancement of sounds from the sound source. In some embodiments the camera view, thermal view, and localized sounds of the same part of the physical object are optionally captured by different cameras and/or sensors, or optionally enhanced using computational techniques. In some embodiments, a user input is detected that selects the source for which enhanced audio is desired (e.g., a tap on a hotspot in the night vision view).

In response to detecting the first user input, displaying a second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having a first display characteristic that is adjusted for the first representation of the first portion of the physical environment according to simulated thermal vision for viewing physical objects having different thermal emission profiles; and a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having a first display characteristic that is adjusted for the first representation of the first portion of the physical environment according to simulated thermal vision for viewing physical objects having different thermal emission profiles. Displaying a third view of the physical environment including a third representation of the first portion of the physical environment having first display characteristics adjusted for the current view and second display characteristics adjusted for the second representation of the physical environment according to selective sound adjustments of sounds corresponding to a subset of the physical objects in the physical environment provides additional control options without cluttering the UI with additional display controls (e.g., additional display controls for selecting or switching between simulated heat vision to view physical objects having different thermal emission profiles and selective sound adjustments of sounds corresponding to a subset of the physical objects in the physical environment). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

It should be understood that the particular order described for the operations in FIG. 11 is merely an example, and that the described order is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, and 12000) are also applicable in a similar manner to method 11000 described above in connection with FIG. 11. For example, the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described above with reference to method 11000 may optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described herein with reference to other methods described herein (e.g., methods 8000, 9000, 10000, and 12000), the details of which are not repeated here for the sake of brevity.

FIG. 12 is a flowchart of a method for selectively displaying virtual content corresponding to a distinct type of exercise within a view of a three-dimensional environment pursuant to a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to a distinct type of exercise, according to some embodiments.

In some embodiments, the method 12000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera (e.g., a color sensor, an infrared sensor, and other depth-sensing camera)) facing forward from a user's hand or a user's head. In some embodiments, the method 12000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 12000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, the method 12000 is executed in a computer system (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component (e.g., display generation component 120, display generation component 7100, etc., of FIGS. 1, 3, and 4) (e.g., a heads-up display, HMD, display, touch screen, projector, etc.) and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, buttons, etc.). In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as the display generation component and at least some of the one or more input devices. In some embodiments, the computer system includes a computing component including one or more processors and memory that are separate from the display generation component and/or the one or more input devices. In some embodiments, the display generation component and the one or more input devices are integrated within and enclosed in the same housing.

The computer system displays a first view (e.g., view 7405 in FIG. 7N(B), another view, etc.) of a three-dimensional environment (e.g., scene 105 in FIG. 7N(A), another physical environment, etc.) (e.g., a reality view with no or minimal virtual elements, a reality view with user interface objects for controlling basic functions of the computer system (e.g., application icons for launching different computer-generated experiences, display settings, audio controls, etc.), an augmented reality view displayed at a low level of immersion (e.g., displaying user interface objects (e.g., application launch pad, welcome user interface, settings user interface), etc.) that occupy only a small percentage (e.g., less than 10%, less than 20%) of the user's field of view or that are displayed in a limited floating window, etc.) (12002), where the first view of the three-dimensional environment includes a first representation of a first portion of the physical environment (e.g., the first representation includes a normal cover of the first portion of the physical environment surrounding the user in a physical spatial relationship with the first display generation component to view the three-dimensional environment through the first display generation component). While displaying the first view of the three-dimensional environment including the first representation of the first portion of the physical environment, the computer system detects a movement of a first user (e.g., user 7002 of FIGS. 7N-7P, another user, etc.) from a first location (e.g., location of user 7002 of FIG. 7N(A), another location, etc.) in the physical environment to a second location (e.g., location of user 7002 of FIG. 7O(A), location of user 7002 of FIG. 7P(A), another location, etc.) (e.g., movement of the first user's entire body while wearing an HMD functioning as the first display generating component (e.g., walking, climbing, etc.), movement of the first user carrying a mobile device having a display or projector functioning as the first display generating component from a first location to a second location in the first physical environment (e.g., movement of the user's arm causing the mobile device and display to move, movement of the entire user carrying the mobile device having the display), etc.) (12004). In response to detecting movement of the first user from a first location to a second location (12006), and the movement to the second location satisfies a first criterion, the first criterion including a first requirement that, for the first criterion to be satisfied, the second location corresponds to a location associated with a first type of exercise (e.g., the location has a first type of exercise equipment (e.g., a rowing machine, stairs, a treadmill, a climbing wall, a stationary bike, a weight training machine, a punch bag, etc.)), and the location is suitable for the first type of exercise (e.g., having a suitable floor surface, mat, pool, walls, structure, etc.) In accordance with a determination that the current location is a location designed for a particular sport (e.g., walking, meditation, yoga, lifting, kicking, walking, running, dancing, climbing, tennis, basketball, gymnastics, etc.), the computer system displays (12008) a second view of the three-dimensional environment (e.g., view 7408 of FIG. 7O(B) or another view) (e.g., an augmented reality view with more virtual elements corresponding to a first particular computer-generated experience corresponding to the current location, an augmented reality view showing a preview or start of the first computer-generated experience corresponding to the current location, an augmented reality view displayed at a higher immersion level) (e.g., collectively covering a significant proportion of the user's field of view (e.g., 60%, more than 90%, etc.) or displays user interface objects that are part of a first particular application experience (e.g., virtual hiking trails, virtual scenery, scoreboards, exercise statistics, controls to change exercise parameters, etc.) displayed in the three-dimensional virtual or augmented reality environment), while a second view of the three-dimensional environment displays a first set of virtual content (e.g., virtual open water body 7406 of FIG. 7O(B) , other virtual content, etc.) corresponding to a first type of exercise (e.g., hiking trail scenery for a treadmill exercise program, a lake scene for a rowing machine exercise, an arena for kickboxing, a climbing wall for a climbing wall exercise, etc.) corresponding to a first type of exercise. a virtual cliff side, a virtual tennis court for a virtual tennis game, and/or user interface controls, scores, statistics, etc. for the first type of exercise, and the first set of virtual content replaces at least a portion of a second representation of a second portion of the physical environment (e.g., the location of the user 7002 in FIG. 7O(A), another location, etc.) (e.g., the virtual content corresponding to the first type of exercise is displayed overlaying a representation of the portion of the physical environment including the second location (e.g., a view of actual equipment for the first type of exercise or a room designed for the first type of exercise, etc.), blocking a view of that representation, replacing the view of that portion, etc.). In some embodiments, the above requirements are the only requirements for the first criterion to be met. In some embodiments, the above requirements are alternative requirements to one or more other requirements in the first criterion that do not all need to be met for the first criterion to be met. In some embodiments, the above requirements are in addition to one or more other requirements in the first criterion that all must be met for the first criterion to be met. In some embodiments, as an alternative (or additional) condition, the user must perform an action associated with the exercise, such as initiating a characteristic movement (e.g., starting to walk on a treadmill, walking on a stair stepper, moving legs back and forth on an elliptical, or starting rowing on a rowing machine, or standing/sitting on the exercise equipment) for the first criterion to be met. In method 12000, in response to detecting movement of the first user from the first location to a second location, the movement to the second location satisfies a second criterion different from the first criterion, the second criterion including a second requirement that for the second criterion to be satisfied, the second location corresponds to a location associated with a second type of exercise (e.g., the location has a second type of exercise equipment (e.g., a rowing machine, stairs, a treadmill, a climbing wall, a weight training machine, a punch bag, etc.), the location corresponds to a location associated with a second type of exercise (e.g., swimming, rowing, meditation, yoga, lifting, kicking, walking, running, dancing, etc.) (e.g., having a suitable floor surface, mat, pool, wall, structure, etc.) Pursuant to a determination that the location is a location designed for a particular sport (e.g., climbing, tennis, basketball, gymnastics, etc.) and the second type of exercise is different from the first type of exercise, the computer system displays (12010) a third view of the three dimensional environment (e.g., view 7410 of FIG. 7P(B) or another view) (e.g., an augmented reality view with more virtual elements corresponding to a second particular computer-generated experience corresponding to the current location, an augmented reality view showing a preview or start of the second computer-generated experience corresponding to the current location, an augmented reality view displayed at a higher immersion level) (e.g., views that collectively occupy a substantial proportion (e.g., more than 60%, more than 90%, etc.) of the user's field of view or that are displayed in a three dimensional virtual or augmented reality environment, etc.). a virtual tennis court for a virtual tennis game, a virtual tennis court for a virtual tennis match, a virtual tennis court for a virtual tennis match, and/or a virtual tennis court for a virtual tennis match; a virtual tennis court for a virtual tennis match, ... (e.g., a virtual hiking trail vs. a virtual lake scene; a virtual tennis court vs. a virtual boxing ring; a virtual grass field for meditation vs. a virtual stage for dancing, etc.), the second set of virtual content (e.g., the virtual hiking trail 7412 of FIG. 7P(B), other virtual content, etc.) replaces at least a portion of a third representation of a third portion of the physical environment (e.g., the virtual content corresponding to the second type of exercise is displayed overlaying a representation of a portion of the physical environment (e.g., a view of actual equipment for the first type of exercise or a room designed for the first type of exercise, etc.) that includes the second location (e.g., the location of the user 7002 of FIG. 7P(A), another location, etc.), blocking the view of that representation, replacing the display of that representation, etc.). In some embodiments, the above requirements are the only requirements for the first criterion to be met. In some embodiments, the above requirements are alternative requirements to one or more other requirements in the first criterion that do not all need to be met for the first criterion to be met. In some embodiments, the above requirements are in addition to one or more other requirements in the first criterion that must all be met for the first criterion to be met. In some embodiments, as an alternative condition (or additional condition), the user must perform an action associated with the exercise, such as starting a characteristic movement (e.g., starting to walk on a treadmill, walking on a stair stepper, moving legs back and forth on an elliptical, or starting rowing on a rowing machine, or standing/sitting on the exercise equipment) for the first criterion to be met. These characteristics are illustrated in Figures 7N-7P, where as the user 7002 moves between locations, depending on the current location of the user 7002, the computer system determines which type of exercise is associated with the current location of the user 7002. If the current location is associated with the first type of exercise (e.g., a location that includes the object 7404), the computer system displays virtual content 7408 (Figure 7O) corresponding to the first type of exercise (e.g., rowing, rowing, etc.). If the current location is associated with a second type of exercise (e.g., a location that includes object 7402), the computer system displays virtual content 7410 (FIG. 7P) that corresponds to the second type of exercise (e.g., hiking, walking, etc.).

In some embodiments, the computer system determines that the second location corresponds to a location associated with the first type of exercise (e.g., the first type of exercise equipment is different from the second type of exercise equipment and does not correspond to the second type of exercise) following detection of a first type of exercise equipment (e.g., object 7404 in FIG. 7O, other equipment, etc.) at the second location (e.g., detecting an RFID signal corresponding to the first type of exercise equipment at the second location, detecting an image of the first type of exercise equipment at the second location in a camera feed capturing the second location, detecting that the second location matches a registered location of the first type of exercise equipment, etc.). The computer system determines that the second location corresponds to a location associated with the second type of exercise equipment according to detecting a second type of exercise equipment (e.g., object 7402 in FIG. 7P , other equipment, etc.) at the second location (e.g., detecting an RFID signal corresponding to the second type of exercise equipment at the second location, detecting an image of the second type of exercise equipment at the second location in a camera feed capturing the second location, detecting that the second location is a registered location of the second type of exercise equipment, etc.), and the second type of exercise equipment is different from the first type of exercise equipment and does not correspond to the first type of exercise. For example, when the first user walks to a location in front of the treadmill, the HMD displays a virtual hiking trail that occludes, replaces, or overlays a representation of the treadmill in the user's field of view provided via the first display generation component, and when the first user walks to a location in front of the rowing machine, the HMD displays a virtual lake scene that occludes, replaces, or overlays a representation of the rowing machine in the user's field of view provided via the first display generation component. In some embodiments, as the first user moves from location to location (e.g., from the gym entrance to in front of the treadmill, from in front of the treadmill to in front of the rowing machine, etc.), the virtual content provided by the first display generation component is automatically changed (e.g., without user input to specifically select virtual content or programs using user interface elements or voice commands), between a real view, a different augmented reality view with a different virtual scene, and/or a different virtual environment, etc.

The computer system, in accordance with detection of the first type of exercise equipment at the second location, determines that the second location corresponds to a location associated with the first type of exercise; displaying a second view of the three-dimensional environment including a first set of virtual content corresponding to the first type of exercise; and, in accordance with detection of the second type of exercise equipment at the second location, the computer system, in accordance with a determination that the second location corresponds to a location associated with the second type of exercise; displaying a third view of the three-dimensional environment including a second set of virtual content corresponding to the second type of exercise, different from the first set of virtual content corresponding to the second type of exercise, in accordance with a determination that the movement to the second location satisfies a second criterion, different from the first criterion, requiring that the second location corresponds to a location associated with the second type of exercise, displaying an appropriate set of virtual content when the set of conditions is met without requiring further user input (e.g., further user input to select the set of virtual content corresponding to the first or second type of exercise). Executing an action when a set of conditions is met without requiring further user input improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to detecting movement of the first user from a first location to a second location in the physical environment, displaying a second view of the three-dimensional environment (e.g., view 7408 of FIG. 7O, or view 7410 of FIG. 7P, another view, etc.) includes gradually decreasing a second representation of a second portion of the physical environment (e.g., a portion of scene 105 including object 7404 of FIG. 7O and a portion of scene 105 including object 7402 of FIG. 7P, etc.) (e.g., ceasing to display more of the representation of the second portion of the physical environment, fading out the representation of the second portion of the physical environment, etc.) and gradually increasing the prominence of virtual content corresponding to the first type of exercise (e.g., starting to display the virtual content, increasing the visibility of the virtual content, increasing a percentage of the user's field of view occupied by the virtual content, increasing the opacity or brightness of the virtual content, etc.) in an area of the second view of the three-dimensional environment where the second representation of the second portion of the physical environment is gradually decreased. In some embodiments, in response to detecting movement of the first user from the first location to the second location within the physical environment, displaying the third view of the three-dimensional environment includes gradually reducing the representation of the third portion of the physical environment (e.g., ceasing to display more of the representation of the third portion of the physical environment, fading out the representation of the third portion of the physical environment, etc.) and gradually increasing virtual content corresponding to the second type of exercise (e.g., displaying the virtual content, increasing visibility of the virtual content, etc.) in areas of the third view of the three-dimensional environment where the representation of the third portion of the physical environment is gradually reduced. For example, in some embodiments, as the first user stands in front of and/or rides on the treadmill, the user's view of the physical environment (e.g., the hardware control panel of the treadmill, the wall in front of the user, other exercise machines in the same room, etc.) is gradually altered such that more and more of the representation of the physical environment gradually disappears and/or is replaced with virtual content corresponding to the treadmill exercise (e.g., scenery of a hiking trail, a virtual paved path around a virtual lake, etc.). Eventually, as the user begins to walk on the treadmill, the user's entire field of vision is filled with a virtual landscape of a mountain trail or lakefront. In some embodiments, the virtual content is a virtual three-dimensional environment, and the user can view different parts of the virtual three-dimensional environment by turning or tilting their head while walking on the treadmill.

Gradually reducing the second representation of the second portion of the physical environment and gradually increasing the prominence of the virtual content corresponding to the first type of exercise in an area of the second view of the three-dimensional environment in which the second representation of the second portion of the physical environment is gradually reduced provides the user with improved visual feedback (e.g., improved visual feedback that the computer system has detected the first user's movement from a first location to a second location in the physical environment). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first criterion includes a third requirement that, for the first criterion to be met, the movement of the first user from the first location to the second location must be followed by a first predetermined movement (e.g., sitting on object 7404 in FIG. 7O, stepping on object 7402 in FIG. 7P) corresponding to a first type of exercise (e.g., initiating a characteristic motion (e.g., starting to walk on a treadmill, stepping on a stair stepper, moving legs back and forth on an elliptical, or starting rowing on a rowing machine, etc.)), stepping/sitting on exercise equipment corresponding to the first type of exercise (e.g., sitting on a rowing machine or weight training machine), assuming a ready position corresponding to the first type of exercise (e.g., standing in a ready position to hit a virtual tennis ball, sitting on the floor to begin meditation or yoga, etc.). In some embodiments, the second criteria includes a fourth requirement that the movement of the first user from the first location to the second location is followed by a second predefined movement corresponding to the second type of exercise, the second predefined movement being different from the first type of movement. In some embodiments, the second predefined movement is the same as the first predefined movement. For example, the predefined movement for initiating a virtual environment for kickboxing is optionally the same as the predefined movement requirement for initiating a virtual environment for boxing. For example, the predefined movement for initiating a virtual environment for ballet is optionally different from the predefined movement requirement for initiating a virtual environment for modern dance. In some embodiments, the computer system does not start displaying the virtual content corresponding to the first type of exercise until the first predefined movement corresponding to the first type of exercise is detected, even if the first user is at the second location and the second location is a location corresponding to the first type of exercise. In some embodiments, when the first user is detected at a second location, the computer system displays a visual prompt for the first user to provide a first predetermined movement to trigger the display of virtual content associated with the first type of exercise when the first user is detected at a second location, the second location being a location associated with the first type of exercise.

Displaying a second view of the three-dimensional environment including a first set of virtual content corresponding to the first type of exercise pursuant to a determination that the movement to the second location satisfies the first criteria, the first criteria including a third requirement that the movement of the first user from the first location to the second location is followed by a first predetermined movement corresponding to the first type of exercise provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for displaying the first set of virtual content corresponding to the first type of exercise, additional displayed controls for ceasing to display the first set of virtual content corresponding to the first type of exercise, etc.). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, in response to detecting movement of the first user from a first location to a second location, and the movement to the second location satisfies a third criterion distinct from the first criterion and the second criterion, the third criterion including a requirement that, for the third criterion to be satisfied, the second location corresponds to a location associated with a third type of exercise distinct from the first type of exercise and the second type of exercise (e.g., the second location is optionally associated with both the first type of exercise and the third type of exercise), and After the first user movement, a third predefined movement corresponding to a third type of exercise (e.g., starting a characteristic motion (e.g., starting walking on a treadmill, stepping on a stair stepper, moving legs back and forth on an elliptical, or starting rowing on a rowing machine, etc.)), stepping/sitting on exercise equipment corresponding to the respective type of exercise (e.g., sitting on a rowing machine, sitting on a weight training machine, etc.), assuming a readiness position corresponding to the respective type of exercise (e.g., standing in a readiness position to hit a virtual tennis ball, sitting on the floor to begin meditation or yoga, etc.), pursuant to a determination that the third pre-defined movement is different from the first pre-defined movement, the computer system displays a fourth view of the three dimensional environment (e.g., an augmented reality view having more virtual elements corresponding to a third particular computer-generated experience corresponding to the current location, an augmented reality view showing a preview or start of the third computer-generated experience corresponding to the current location, an augmented reality view displayed at a higher immersion level (e.g., occupying a substantial percentage of the user's field of view (e.g., more than 60%, more than 90%, etc.) in total or displaying user interface objects that are part of a third particular application experience displayed in the three dimensional virtual or augmented reality environment, etc. (e.g., virtual hiking trails, virtual scenery, scoreboard, exercise statistics, controls to change exercise parameters, etc.)). The fourth view of the three dimensional environment includes a third set of virtual content corresponding to the third type of exercise (e.g., hiking trail scenery for a treadmill exercise program, a lake scene for a rowing machine exercise, an arena for a kickboxing program, a virtual cliff side for a climbing wall exercise, a virtual tennis court for a virtual tennis game, and/or user interface controls, scores, statistics for the third type of exercise, etc.). The third set of virtual content is distinct from the first set of virtual content and the second set of virtual content, and the third set of virtual content replaces at least a portion of the second representation of the second portion of the physical environment (e.g., the second location corresponds to both the first type of exercise and the third type of exercise, and whether the first set of virtual content or the third set of virtual content is displayed depends on whether the first predetermined movement or the third predetermined movement is detected while the first user is at the second location).

Displaying a fourth view of the three-dimensional environment including the first set of virtual content corresponding to the third type of exercise and the third set of virtual content corresponding to the third type of exercise, different from the second set of virtual content, according to a determination that the movement to the second location satisfies a third criterion different from the first criterion and the second criterion, the second location corresponds to a location associated with the first type of exercise and the third type of exercise, different from the second type of exercise, and the first user's movement from the first location to the second location requires a third predefined movement different from the first predefined movement corresponding to the third type of exercise, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for displaying the first set of virtual content corresponding to the first type of exercise, additional displayed controls for displaying the third set of virtual content corresponding to the third type of exercise, etc.). Providing additional control options without cluttering the UI with additional displayed controls improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the computer system gradually increases the amount of virtual content displayed in the first user's field of view (e.g., in the views shown in Figures 7O(B) and 7P(B), etc.) according to at least one of the progression or duration of a predetermined movement corresponding to a distinct type of exercise (e.g., a first type of exercise, a second type of exercise, a third type of exercise, etc.) associated with the second location. For example, in some embodiments, the view of the real world gradually disappears and/or ceases to be displayed and is gradually replaced by the virtual content corresponding to the distinct type of exercise. In some embodiments, the computer system gradually increases the amount of virtual content displayed in the first user's field of view until a distinct virtual environment corresponding to the distinct type of exercise is fully displayed via the first display generation component (e.g., the second view of the three-dimensional environment includes a virtual environment corresponding to the first type of exercise, the third view of the three-dimensional environment includes a virtual environment corresponding to the second type of exercise, etc.). For example, in some embodiments, when the open gym is a location associated with both yoga and dance, after the first user arrives at the open gym, if the first user sits in a namaste pose, the computer system displays a virtual ocean view with ocean sounds for the user to practice yoga on a virtual beach, and if the first user stands in a dancer's pose, the computer system displays a virtual stage with dance music for the first user to practice dance.

Gradually increasing the amount of virtual content displayed within the field of view of the first user according to at least one of the progress or duration of a predetermined movement corresponding to a distinct type of exercise associated with the second location provides the user with improved visual feedback (e.g., improved visual feedback regarding the progress or duration of a predetermined movement corresponding to a distinct type of exercise associated with the second location). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, while displaying a separate view (e.g., an augmented reality view, a virtual reality view, etc.) of the three-dimensional environment corresponding to a separate type of exercise (e.g., a first type of exercise, a second type of exercise, a third type of exercise, etc.) associated with the second location, the computer system detects a movement of the first user corresponding to a request to end the separate type of exercise associated with the second location (e.g., detecting the first user ceasing the separate type of exercise, standing up, stepping off the equipment, removing the HMD, and/or walking away from the second location, etc.). In response to detecting the first user's movement corresponding to the request to end the separate type of exercise associated with the second location, the computer system detects a fifth view of the three-dimensional environment including a representation of at least a fourth portion of the physical environment, the representation of at least the fourth portion of the physical environment occupying a portion of the first user's field of view in which the separate set of virtual content corresponding to the separate type of exercise was displayed while the first user was at the second location. For example, when a first user movement corresponding to a request to end the current exercise is detected, the virtual scene corresponding to the current exercise is stopped from being displayed (e.g., faded away, stopped quickly, etc.) and the representation of the physical environment is made visible again. In some embodiments, when the user 7002 moves away from the object 7404 and does not reach the object 7402 in the scene 105, neither the view 7408 nor the view 7410 of Figures 7O and 7P are displayed, and the representation of the physical environment as shown in Figure 7N is displayed.

Displaying a fifth view of the three-dimensional environment including a representation of at least a fourth portion of the physical environment, the representation of the fourth portion occupying a portion of the first user's field of view in which a respective set of virtual content corresponding to a respective type of exercise was displayed while the first user was at the second location, displays the fifth view of the three-dimensional environment when a set of conditions are met without requiring further user input (e.g., further user input to display the fifth view of the three-dimensional environment) in response to detecting a movement of the first user corresponding to a request to end the respective type of exercise associated with the second location. Performing an action when a set of conditions are met without requiring further user input improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the computer system displays status information corresponding to the first type of exercise (e.g., progress during the current session, duration, speed, force, height, pace, stride length, performance level, score, number of repetitions completed, etc., historical statistics, average statistics for the first user and/or across multiple users, status of other users performing the same type of exercise, etc.) when the second view of the three-dimensional environment is displayed (e.g., view 7408 of FIG. 7O, view 7410 of FIG. 7P, another view, etc.). In some embodiments, the status information corresponding to the first type of exercise is superimposed on a portion of the virtual scene corresponding to the first type of exercise. In some embodiments, the status information is displayed in response to a request of the first user that is detected while the virtual scene corresponding to the first type of exercise is displayed without the status information. In some embodiments, the second view of the three-dimensional environment evolves progressively throughout the first user's performance of the first type of exercise. In some embodiments, the status information is continuously updated (e.g., superimposed on a changing second view of the three-dimensional environment) throughout the first user's performance of the first type of exercise. In some embodiments, the status information is displayed in response to detecting that the value of one or more performance parameters meets a pre-defined threshold (e.g., a target speed or distance is achieved, a threshold score is reached, etc.).

Displaying the status information corresponding to the first type of exercise when the second view of the three-dimensional environment is displayed provides the user with improved visual feedback (e.g., improved visual feedback regarding the first type of exercise, improved visual feedback that the user's movement to the second position meets the first criterion, improved visual feedback that the computer system is displaying the second view of the three-dimensional environment, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently. In some embodiments, the computer system displays health information corresponding to the first user (e.g., real-time biometric data (e.g., heart rate, blood pressure, respiratory rate, body temperature, blood glucose level, etc.), weight, BMI, etc.) when the second view of the three-dimensional environment (e.g., view 7408 of FIG. 7O, view 7410 of FIG. 7P, another view, etc.) is displayed. In some embodiments, the health information corresponding to the first user is overlaid on a portion of the virtual scene corresponding to the first type of exercise. In some embodiments, the health information is displayed in response to a request of the first user that is detected while the virtual scene corresponding to the first type of exercise is displayed without the health information. In some embodiments, the second view of the three-dimensional environment evolves progressively throughout the first user's performance of the first type of exercise. In some embodiments, the health information is continuously updated (e.g., superimposed on the changing second view of the three-dimensional environment) throughout the first user's performance of the first type of exercise. In some embodiments, the health information is displayed in response to detecting that the value of one or more health parameters meets a preset threshold (e.g., a target heart rate is achieved, a threshold blood pressure is reached, etc.).

Displaying health information corresponding to the first user when the second view of the three-dimensional environment is displayed provides the user with improved visual feedback (e.g., improved visual feedback regarding the first type of exercise, improved visual feedback that the user's movement to the second position meets the first criterion, improved visual feedback that the computer system is displaying the second view of the three-dimensional environment, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the computer system visually presents progress information (e.g., real-time score, laps completed, laps remaining, duration, number of steps, distance traveled, completed poses, etc.) of the first type of exercise performed by the first user when a second view of the three-dimensional environment (e.g., view 7408 of FIG. 7O, view 7410 of FIG. 7P, another view, etc.) is displayed. In some embodiments, the progress information is visually represented by visual changes occurring in the virtual scene presented to the first user (e.g., virtual milestones on a virtual hiking trail, a number of virtual shooting targets shown in a descending position, a leaderboard as part of a virtual game arena, still water on a virtual lake representing a degree of deep meditation achieved, etc.). In some embodiments, the progress information corresponding to the performance of the first type of exercise by the first user is overlaid on a portion of the virtual scene corresponding to the first type of exercise. In some embodiments, the progress information is displayed in response to a request of the first user that is detected while the virtual scene corresponding to the first type of exercise is displayed without the progress information. In some embodiments, the second view of the three-dimensional environment evolves progressively throughout the first user's performance of the first type of exercise. In some embodiments, the progress information is continuously updated (e.g., superimposed on the changing second view of the three-dimensional environment) throughout the first user's performance of the first type of exercise. In some embodiments, the progress information is displayed in response to detecting that the value of one or more progress parameters meets a pre-set threshold (e.g., a target distance is achieved, a threshold score is reached, an exercise routine is completed, etc.).

Visually presenting progress information of a first type of exercise performed by a first user when a second view of the three-dimensional environment is displayed provides the user with improved visual feedback (e.g., improved visual feedback related to the progress information of the first type of exercise). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, in accordance with a determination that the first user is facing a first direction in the physical environment, the computer system displays a first subset of the first set of virtual content corresponding to the first type of exercise without displaying a second subset of the first set of virtual content, and in accordance with a determination that the first user is facing a second direction in the physical environment that is different from the first direction (e.g., a direction opposite to the first direction, a non-zero angle from the first direction, etc.), the computer system displays a second subset of the first set of virtual content corresponding to the first type of exercise without displaying the second subset of the first set of virtual content (e.g., a virtual open water body 7406 not shown in view 7408 of FIG. 7O, a virtual mountain trail 7412 not shown in view 7410 of FIG. 7P, etc.). For example, in some embodiments, the second view of the three-dimensional environment is an immersive view with virtual objects in a direction all around the first user (e.g., spanning an angle wider than the user's field of view so that different parts of the virtual environment are visible when the user turns his or her head). In some embodiments, the computer system outputs sound effects using an immersive audio output mode, such as a surround sound mode, or a spatial audio mode that provides localized sound to positions corresponding to sound-generating virtual objects in the virtual environment (e.g., a cheering crowd, ocean waves, a virtual coach, etc.).

Displaying a first subset of the first set of virtual content corresponding to a first type of exercise without displaying a second subset of the first set of virtual content in accordance with a determination that the first user is oriented in a first direction in the physical environment, and displaying a second subset of the first set of virtual content corresponding to the first type of exercise without displaying the second subset of the first set of virtual content in accordance with a determination that the first user is in a second direction in the physical environment different from the first direction displays an appropriate subset of the first set of virtual content corresponding to the first type of exercise when a set of conditions is met without requiring further user input (e.g., further user input to navigate the first set of virtual content corresponding to the first type of exercise). Performing an action when a set of conditions is met without requiring further user input improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.

In some embodiments, the first type of exercise is a rowing exercise, the second location is a location where a rowing exercise apparatus (e.g., object 7404, other rowing apparatus, etc.) is present, and the second view of the three-dimensional environment (e.g., view 7408 of FIG. 7O) includes a virtual scene having an open body of water (e.g., virtual open body of water 7406 of FIG. 7O). In some embodiments, the first criterion further includes a requirement that the first user be seated at the rowing exercise apparatus and place his/her hands on the oars of the rowing exercise apparatus (e.g., as shown in FIG. 7O(A)) for the first criterion to be met. In some embodiments, the second type of exercise is a walking exercise, the second location is a location involving a treadmill (e.g., object 7402, or other walking equipment, etc.), and the third view of the three-dimensional environment (e.g., view 7410 of FIG. 7P ) includes a virtual scene showing an outdoor walking path (e.g., virtual course 7412 of FIG. 7P ) (e.g., a hiking trail, a lakefront path, a city street, etc.). In some embodiments, the second criterion further includes a requirement that the first user gets on the treadmill and takes at least one step. Displaying the second view of the three-dimensional environment including a virtual scene having an open body of water, where the first type of exercise is a rowing exercise and the second location is a location where a rowing exercise equipment is present, provides the user with improved visual feedback (e.g., improved visual feedback that the first type of exercise is a rowing exercise, improved visual feedback that a rowing exercise equipment is present at the second location, etc.). Providing improved feedback improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in response to detecting movement of a first user (e.g., 7002 in Figures 7O-7P) from a first location to a second location, in accordance with a determination that the second location corresponds to a location associated with a fifth type of exercise and a sixth type of exercise, and in accordance with a determination that the movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a respective type of equipment associated with the fifth type of exercise, the computer system displays a sixth view of the three-dimensional environment, the sixth view of the three-dimensional environment including a fifth set of virtual content corresponding to the fifth type of exercise (e.g., volleyball, tennis, elliptical machine, etc.), the fifth set of virtual content being different from the first set of virtual content and the second set of virtual content, and the fifth set of virtual content replacing at least a portion of a fifth representation of a fifth portion of the physical environment. In some embodiments, in response to detecting movement of the first user from the first location to the second location and in accordance with a determination that the movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a particular type of equipment associated with a sixth type of exercise, the computer system displays a seventh view of the three-dimensional environment, the seventh view of the three-dimensional environment including a sixth set of virtual content corresponding to a sixth type of exercise (e.g., basketball, fencing, exercise bike, etc.), the sixth set of virtual content being distinct from the first set of virtual content, the second set of virtual content, and the fifth set of virtual content, and the sixth set of virtual content replacing at least a portion of the fifth representation of the fifth portion of the physical environment (e.g., the fifth portion of the physical environment is associated with both the fifth type of exercise and the sixth type of exercise). This is illustrated in FIGS. 7O and 7P, where, according to some embodiments, when a user 7002 moves from one location that includes an object 7404 corresponding to a first type of exercise (e.g., rowing, boating, etc.) to another location that includes an object 7402 corresponding to a second type of exercise (e.g., hiking, walking, etc.), the virtual content in view 7408 (e.g., a virtual open water body 7406, or other virtual content, etc.) is replaced with virtual content in view 7410 (e.g., a hiking trail 7412, or other virtual content, etc.).

A sixth view of the three-dimensional environment is displayed in accordance with a determination that movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a respective type of equipment associated with the fifth type of exercise, the sixth view of the three-dimensional environment including a fifth set of virtual content corresponding to the fifth type of exercise; and a seventh view of the three-dimensional environment is displayed in accordance with a determination that movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a respective type of equipment associated with the sixth type of exercise, the seventh view of the three-dimensional environment including a sixth set of virtual content corresponding to the sixth type of exercise, the seventh view of the three-dimensional environment being displayed in accordance with a determination that movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a respective type of equipment associated with the sixth type of exercise, the seventh view of the three-dimensional environment displaying an appropriate view of the three-dimensional environment having a respective set of virtual content corresponding to the respective type of exercise when a set of conditions is met without requiring further user input (e.g., further user input to select or navigate the view of the three-dimensional environment and/or the set of virtual content corresponding to the respective type of exercise). Improves usability of the device by performing an action when a set of conditions are met without requiring further user input, and also reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the second view of the three-dimensional environment (e.g., view 7408, view 7410, etc., of FIGS. 7O-7P) includes a virtual representation of a first user (e.g., user 7002 of FIGS. 7N-7P, or another user, etc.) that is shown performing a first type of exercise in competition with the first user (e.g., based on the first user's previous best performance, based on the first user's preset configuration for the first type of exercise, etc.). In some embodiments, the third view of the three-dimensional environment includes a virtual representation of the first user that is shown performing a second type of exercise in competition with the first user (e.g., based on the first user's previous best performance, based on the first user's preset configuration for the second type of exercise, etc.). Displaying a second view of the three-dimensional environment including a virtual representation of the first user shown competing against the first user to perform the first type of exercise provides the user with improved visual feedback (e.g., improved visual feedback regarding the first type of exercise, improved visual feedback regarding the characteristics of the first user's performance of the first type of exercise, etc.). Providing improved feedback improves usability of the device, as well as reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

In some embodiments, the second view of the three-dimensional environment (e.g., view 7408, view 7410, etc., of FIGS. 7O-7P) includes a virtual representation of at least a second user, different from the first user (e.g., user 7002 of FIGS. 7N-7P, or another user, etc.), shown to be performing the first type of exercise in competition with the first user. In some embodiments, the third view of the three-dimensional environment includes a virtual representation of at least a second user, different from the first user, shown to be performing the second type of exercise in competition with the first user. Displaying the second view of the three-dimensional environment, including a virtual representation of at least a second user, different from the first user, shown to be performing the first type of exercise in competition with the first user, provides the user with improved visual feedback (e.g., improved visual feedback that the at least second user is also performing the first type of exercise, improved visual feedback regarding the user's performance of the first type of exercise relative to the second user's performance of the first type of exercise, etc.). Providing improved feedback improves the usability of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

It should be understood that the particular order described for the operations in FIG. 12 is merely an example, and that the described order is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, and 11000) are also applicable in a similar manner to method 12000 described above in connection with FIG. 12. For example, the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described above with reference to method 12000 may optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, controls, movements, references, three-dimensional environments, display generating components, surfaces, representations of physical objects, virtual objects, and/or animations described herein with reference to other methods described herein (e.g., methods 8000, 9000, 10000, and 11000), the details of which are not repeated here for the sake of brevity.

The operations described above with reference to Figures 8, 9A-9B, 10, 11, and 12 are optionally performed by components shown in Figures 1-6. In some embodiments, aspects/operations of methods 8000, 9000, 10000, 11000, and 12000 may be interchanged, substituted, and/or added between these methods. For the sake of brevity, the details thereof will not be repeated here.

The foregoing has been described with reference to specific embodiments for purposes of explanation. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular uses contemplated.

Claims (20)

1. A method comprising:
A computer system in communication with a first display generation component and one or more first input devices, comprising:
Displaying a first computer-generated experience at a first immersive level;
receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level , the first computer-generated experience at the first immersion level including virtual content;
In response to receiving the biometric data corresponding to the first user,
displaying the first computer-generated experience at a second immersion level including more virtual content than the first immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, the first computer-generated experience displayed at the second immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level;
continuing to display the first computer-generated experience at the first immersion level according to a determination that the biometric data corresponding to the first user does not satisfy the first criterion; and
A method comprising: receiving first updated biometric data corresponding to the first user while displaying the first computer-generated experience at the second immersive level;
In response to receiving the first updated biometric data corresponding to the first user,
displaying the first computer-generated experience at a third immersion level including more virtual content than the second immersion level in accordance with a determination that the first updated biometric data corresponding to the first user satisfies a second criterion different from the first criterion, the first computer-generated experience displayed at the third immersion level occupying a larger portion of the field of view of the first user than the first computer-generated experience displayed at the second immersion level;
continuing to display the first computer-generated experience at the second immersion level in accordance with a determination that the first updated biometric data corresponding to the first user satisfies the first criterion and does not satisfy the second criterion; and
The method of claim 1 , comprising: receiving second updated biometric data corresponding to the first user while displaying the first computer-generated experience at the second immersive level;
in response to receiving the second updated biometric data corresponding to the first user,
displaying the first computer-generated experience at the first immersion level in accordance with a determination that the second updated biometric data corresponding to the first user does not satisfy the first criterion satisfied for transitioning to displaying the first computer-generated experience at the second immersion level;
The method of claim 1 , comprising: 4. The method of claim 1, wherein the biometric data includes a respiration rate of the first user, and the first criterion includes a criterion that is met if the respiration rate of the first user is below a first threshold respiration rate. The method of claim 1 , wherein the first criterion comprises a criterion that is met if the biometric data meets one or more pre-set thresholds for at least a threshold time. Displaying the first computer-generated experience at the first immersive level includes displaying virtual content at respective first positions corresponding to locations of one or more first portions of a physical environment while maintaining display of a representation of one or more second portions of the physical environment;
4. The method of claim 1, wherein displaying the first computer-generated experience at the second immersive level comprises displaying virtual content at the respective first positions corresponding to the locations of the one or more first portions of the physical environment and at respective second positions corresponding to at least some of the one or more second portions of the physical environment. in response to receiving the biometric data corresponding to the first user and in accordance with a determination that changes in the biometric data corresponding to the first user are progressing toward satisfying the first criterion;
4. The method of claim 1, further comprising: gradually reducing visual emphasis of at least a portion of a representation of a physical environment that was visible through the first display generation component while the first computer-generated experience is displayed at the first immersion level; and displaying the first computer-generated experience at the second immersion level comprises displaying virtual content of the first computer-generated experience at a position corresponding to the portion of the representation of the physical environment such that the portion of the representation of the physical environment ceases to be visible through the first display generation component. in response to receiving the biometric data corresponding to the first user and in accordance with a determination that changes in the biometric data corresponding to the first user are progressing toward satisfying the first criterion;
4. The method of claim 1, further comprising: modifying visual characteristics of at least a portion of a representation of a physical environment that was visible through the first display generation component while the first computer-generated experience was displayed at the first immersive level by an amount corresponding to the change in the biometric data corresponding to the first user. in response to receiving the biometric data corresponding to the first user and in accordance with a determination that changes in the biometric data corresponding to the first user are progressing toward satisfying the first criterion;
4. The method of claim 1, further comprising: extending a display of virtual content over at least a portion of a representation of a physical environment that was visible through the first display generation component while the first computer-generated experience is displayed at the first immersion level and by an amount corresponding to the change in the biometric data corresponding to the first user. 4. The method of claim 1, wherein the first criterion comprises a criterion that is met if the first user performs less than a threshold amount of movement of a first type when the biometric data is being received. detecting a first type of movement being performed by the first user while displaying the first computer-generated experience at the second immersive level;
in response to detecting the first type of movement being performed by the first user;
re-displaying the first computer-generated experience at the first immersion level in accordance with a determination that the movement of the first type exceeds a preset threshold movement amount;
The method according to any one of claims 1 to 3, comprising: detecting a first type of movement being performed by the first user while displaying the first computer-generated experience at the second immersive level;
in response to detecting the first type of movement being performed by the first user;
4. The method of claim 1, further comprising: switching from displaying the first computer-generated experience at the second immersive level from a first perspective to displaying the first computer-generated experience at the second immersive level from a second perspective different from the first perspective in accordance with a determination that the first type of movement exceeds a preset threshold movement amount. The method of any one of claims 1 to 3, wherein the transition from displaying the first computer-generated experience at the first immersion level to displaying the first computer-generated experience at the second immersion level is a discontinuous transition that occurs at a time corresponding to a time when the first criterion is met. 4. The method of claim 1, wherein the first computer-generated experience displayed at the first immersion level depicts a first virtual environment having a first number of dimensions , and the first computer-generated experience displayed at the second immersion level depicts a second virtual environment having a second number of dimensions , the second number being greater than the first number . Displaying the first computer-generated experience at the first immersive level includes displaying the first computer-generated experience with at least a first visual characteristic that changes in accordance with changes in the biometric data received while displaying the first computer-generated experience at the first immersive level;
4. The method of claim 1, wherein displaying the first computer-generated experience at the second immersive level comprises displaying the first computer-generated experience with at least a second visual characteristic that changes in accordance with changes in the biometric data received while displaying the first computer-generated experience at the second immersive level. In response to receiving the biometric data corresponding to the first user,
4. The method of claim 1, further comprising: changing an audio output mode from a first audio output mode to a second audio output mode in accordance with a determination that the biometric data corresponding to the first user satisfies the first criterion, the first audio output mode having fewer computational control variables than the second audio output mode. 1. A computer system comprising:
A first display generation component;
one or more input devices;
one or more processors;
and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising:
Displaying a first computer-generated experience at a first immersive level;
receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level , the first computer-generated experience at the first immersion level including virtual content;
In response to receiving the biometric data corresponding to the first user,
displaying the first computer-generated experience at a second immersion level including more virtual content than the first immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, the first computer-generated experience displayed at the second immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level;
continuing to display the first computer-generated experience at the first immersion level according to a determination that the biometric data corresponding to the first user does not satisfy the first criterion; and
A computer system comprising instructions for: The computer system of claim 17, wherein the one or more programs include instructions for performing the method of any one of claims 2 to 16. 1. A computer program comprising instructions, which when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to:
Displaying a first computer-generated experience at a first immersive level;
receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level , the first computer-generated experience at the first immersion level including virtual content;
In response to receiving the biometric data corresponding to the first user,
displaying the first computer-generated experience at a second immersion level including more virtual content than the first immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, the first computer-generated experience displayed at the second immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level;
continuing to display the first computer-generated experience at the first immersion level according to a determination that the biometric data corresponding to the first user does not satisfy the first criterion; and
A computer program product for causing a computer to perform operations including: 20. The computer program of claim 19, further comprising instructions which, when executed by the computer system, cause the computer to perform the method of any one of claims 2 to 16.

Images