WO2018227098A1 - External camera assisted virtual reality - Google Patents

Abstract

A camera may be configured to provide a real-time image in a virtual reality (VR) session. A VR system may discover available external cameras. The VR system may determine positions of the external cameras. The active camera may be selected automatically based on at least one of the positions of the external cameras, a motion or environmental change, or an object detection. The active camera may be updated periodically based on one or more of user movement tracking information, user gesture, or a user input device. The VR system may establish a video connection with the active camera. Images from the active camera may be received and displayed during the VR session. The images may comprise an inset view showing a self-view of the VR user or inset view showing the VR user environment.

Description

EXTERNAL CAMERA ASSISTED VIRTUAL REALITY

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application is a non-provisional filing of, and claims benefit under 35 U.S.C. §119(e) from, U.S. Provisional Patent Application Serial No. 62/517,617, filed June 9, 2017, entitled "External Camera Assisted Virtual Reality", which is incorporated herein by reference in its entirety.

BACKGROUND

[0002] 360-degree video is a rapidly growing format emerging in the media industry. 360-degree video is enabled by the growing availability of virtual reality (VR) devices. 360-degree video may provide the viewer a new sense of presence. When compared to rectilinear video {e.g., 2D or 3D), 360-degree video may pose difficult engineering challenges on video processing and/or delivery. Enabling comfort and/or an immersive user experience may call for high video quality and/or very low latency. The large video size of 360-degree video may be an impediment to delivering the 360-degree video in a quality manner at scale.

SUMMARY

[0003] Systems, methods, and instrumentalities are disclosed for a camera configured to provide a live/real-time image {e.g., surrounding view and/or self-view of a VR user via an active camera) in a VR session. A VR system may register one or more available external cameras. The external cameras may be detached from a VR headset. The external cameras may be attached to the VR headset. When the VR system discovers and/or registers one or more available external cameras, the VR system may receive, determine and/or calibrate positions of the external cameras. The VR system may select an active camera among the available external cameras. The active camera may be selected automatically based on at least one of the positions of the external cameras, a motion or environmental change, or an object detection and/or identification. The active camera may be updated periodically based on one or more of a VR user movement tracking information, a user gesture, and/or a user input device. The active camera may be periodically updated based on an active range of the VR user. The VR system may establish a video connection with the active camera. Images from the active camera may be received and displayed during the VR session. The images may comprise an inset view showing a self-view of the VR user. The images may comprise the inset view showing the VR user environment. [0004] Some embodiments operate with the use of cameras that are not attached to a user's head- mounted display. In one such method, a position of the head-mounted display is tracked. Respective positions of each of a plurality of external video cameras unattached to the head-mounted display are determined (e.g. using a discovery process). One of the external video cameras is selected (e.g.

automatically) based on the position of the head-mounted display relative to the positions of the external video cameras. Video captured by the selected external video camera is displayed on the head-mounted display.

[0005] In some embodiments, selection of one of the external video cameras is made such that video captured by the selected external video camera includes a view of a user of the head-mounted display.

[0006] In some embodiments, the orientation of the head-mounted display is tracked, and selection of one of the external video cameras is further made such that the selected external video camera is within a predetermined region with respect to the orientation of the head-mounted display.

[0007] In some embodiments, the orientation of the head-mounted display is tracked, and selection of one of the external video cameras is made such that video captured by the selected external video camera includes a view in a predetermined direction (e.g. a forward view or a rearward view) relative to the orientation of the head-mounted display.

[0008] In some embodiments, respective orientations of each of the external video cameras are determined, and selection of one of the external video cameras is further based in part on the orientations of the external video cameras (e.g. so as to select a camera oriented toward the user, or to select a camera oriented along the same direction as the direction of the head-mounted display).

[0009] In some embodiments, the selection of one of the external video cameras is based in part on respective distances between the head-mounted display and the external video cameras. For example, in some embodiments, the selection of one of the external video cameras is made such that the selected external video camera is within a predetermined range of distances from the head-mounted display.

[0010] In some embodiments, the head-mounted display is a virtual reality display, and displaying the video captured by the selected external video camera includes displaying the video captured by the selected external video camera as an inset in virtual reality content.

[0011] In some embodiments, the information indicating the respective positions of each of the plurality of external video cameras is periodically updated.

[0012] In some embodiments, the selection of one of the external video cameras is based in part on one or more of the following camera parameters of the external video cameras: spatial resolution, frame rate, and field of view. In some embodiments, the selection of one of the external video cameras is based at least in part on input of user preferences. [0013] Some embodiments operate with the use of cameras that are are attached to a user's head- mounted display. In one such method performed at a head-mounted display, video is captured from at least one external camera attached to the head-mounted display. A selection of a view of interest is received from a user. From the captured video, a portion of the captured video corresponding to the view of interest is selected. On the head-mounted display, the selected portion of the captured video is displayed as an inset in virtual reality content. In some such embodiments, the captured video is 360-degree video. In such embodiments, selecting a portion of the captured video may include extracting the selected portion from the 360-degree video.

[0014] In some embodiments, the view of interest is specified relative to an orientation of the head- mounted display. In other embodiments, the view of interest is specified relative to an absolute orientation. In embodiments in which the view of interest is specified relative to an absolute orientation, the method may further include tracking the orientation of the head-mounted display, where the tracked orientation of the head-mounted display is used to maintain the view of interest in the selected portion of the captured video.

[0015] Further described in the present disclosure are systems that include a processor and a non- transitory computer storage medium that stores instructions operative to perform the functions described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. 1 illustrates an example viewport video displayed on a head mounted display (HMD).

[0017] FIG. 2 illustrates an example 3D coordinates of a sphere.

[0018] FIG. 3A illustrates an example sphere sampling in longitudes (cp) and latitudes (Θ) in

equirectangular projection.

[0019] FIG. 3B illustrates an example with equirectangular projection.

[0020] FIG. 4A illustrates an example 3D geometry of cubemap relative to a sphere.

[0021] FIG. 4B illustrates an example picture with cubemap projection [e.g., using 4 X 3 face arrangement).

[0022] FIG. 5A illustrates an example VR headset with a constellation of LEDs.

[0023] FIG. 5B illustrates an example sensor which may be used to determine the position of a VR headset.

[0024] FIG. 6 illustrates an example inset view of a VR user {e.g., self-view).

[0025] FIG. 7 illustrates an example inset view of VR user's actual environment. [0026] FIG. 8 illustrates an example utilizing one or more external cameras {e.g., 4 detached external cameras) in a camera-assisted VR system.

[0027] FIG. 9 illustrates an example of external camera assisted VR system.

[0028] FIG. 10 illustrates an example of selecting an active camera using an active range.

[0029] FIGs. 11 A and B illustrate examples of updating an active camera based on movement tracking. FIG. 11 A illustrates front of the VR facing camera #1 at first time instance, and FIG. 11 B illustrates front of the VR facing camera #2 at second time instance.

[0030] FIG. 12 illustrates an example of generating a 2D viewport image from a 360-degre video using a rectilinear projection.

[0031] FIG. 13 illustrates an example of generating a view of interest (VOI) on an external camera from a complete 360-degree video.

[0032] FIG. 14 illustrates an example of generating a VOI on an external camera from a partial 360- degree video.

[0033] FIG. 15 illustrates an example of moving and/or zooming VOI via a user input device.

[0034] FIG. 16A is a system diagram of an example communications system in which one or more disclosed embodiments may be implemented.

[0035] FIG. 16B is a system diagram of an example wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 16A.

[0036] FIG. 16C is a system diagram of an example radio access network and an example core network that may be used within the communications system illustrated in FIG. 16A.

[0037] FIG. 16D is a system diagram of another example radio access network and an example core network that may be used within the communications system illustrated in FIG. 16A.

[0038] FIG. 16E is a system diagram of another example radio access network and an example core network that may be used within the communications system illustrated in FIG. 16A.

DETAILED DESCRIPTION

[0039] A detailed description of illustrative embodiments will now be described with reference to the various Figures. Although this description provides a detailed example of possible implementations, it should be noted that the details are intended to be exemplary and in no way limit the scope of the application. Virtual Reality.

[0040] Virtual reality (VR) has applications in areas including, but not limited to healthcare, education, social networking, industry design/training, games, movies, shopping, entertainment, and the like. VR may bring an immersive viewing experience. VR may create a virtual environment that may surround a viewer and may provide a true sense of "being there" for the viewer. The quality and/or user experience of one or more aspects in a VR processing chain, including capturing, processing, display and/or applications may be improved. For example, on the capturing side, VR system may use one or more camera systems to capture a scene from one or more different views of the surrounding environment {e.g., 6-12 views from a 360-degree video camera which may comprise an array of individual cameras). One or more captured views may be stitched together to form a 360-degree video in high resolution {e.g., 4K or 8K).

[0041] In some cases, 360-degree video is a video signal encompassing an entire sphere and/or having a field of view of 360 degrees horizontally and 180 degrees vertically. A 360-degree video may be obtained and/or produced via camera captures. A 360-degree video may be obtained and/or produced via other means. For example, in VR gaming, a 360-degree video may be rendered using computer graphics technology. In a VR-enabled Operating System (OS), the entire user interface may be rendered into a 360- degree video which surrounds the user.

[0042] On the client or user side, the VR systems may include a computation platform {e.g., a desktop personal computer (PC) or a compact mobile embedded system such as a phone), a head mounted display (HMD), a VR controller, and/or head tracking sensors. The computation platform may receive and/or decode the 360-degree video content and/or other immersive graphic content and may render a viewport for display. FIG. 1 illustrates an example viewport video displayed on an HMD. As shown in FIG. 1 , two pictures, one for each eye, may be rendered and/or magnified through lenses for better viewing, and may be displayed on the HMD for stereoscopic viewing. The head tracking sensors may keep {e.g., constantly) track of the viewer's head orientation. The orientation information may be provided to the computation platform to determine the current viewpoint and/or to render the corresponding viewport picture for display. Certain HMD headsets may include a motherboard and one or more position tracking sensors {e.g., a magnetometer, a gyroscope, and/or an accelerometer). Certain HMD headsets may include a separate stand-alone positional tracker that, through its sensors {e.g., wireless sensors), monitors a series of infrared LEDs embedded in the headset to track user's positions.

[0043] Certain HMD headsets {e.g., high performance HMD headsets) may be too big and/or not convenient to wear. Certain high-performance headsets may be connected {e.g., to a PC-based computing platform) via wires, which may subject the users to be in a constrained range of movement. Certain untethered VR devices may be available. For example, certain untethered VR devices may be based on a smartphone platform. Certain untethered VR devices may include a headset into which a VR-enabled smartphone may be plugged and/or a VR controller may be connected.

[0044] Complete immersion in VR may cause a user uneasiness. The user may become completely detached from the external surrounding. A rendered view of a virtual environment may be combined with an imagery from the surrounding real world {e.g., augmented reality, mixed reality, and/or merged reality).

[0045] Augmented reality (AR) may take a view of the real world and may add a digital information and/or data on top of the VR. The added information may be a textual information about a certain object in the real world and/or more complex information such as a simulated image and/or video. AR may be delivered through a sensor-packed wearable device {e.g., smart glasses). With a specialized wearable, such as smart glasses, AR may get additional digital information without checking another device. AR may be used for some specialized work in the enterprise. For example, specialized glasses may give a manufacturing worker instructions on how to inspect and/or maintain a piece of equipment.

[0046] Mixed reality may let a user see the real world overlaid with virtual objects. If the rendering quality of the virtual objects is sufficiently high, the user may not be able to distinguish the virtual objects from the surrounding real world. The virtual objects may be positioned in the real world and may be rendered to blend naturally into the real world. As the user moves around in the mixed reality environment, the relative distance between the user and a virtual object may change in the same way as the relative distance change between the user and a real object. For example, mixed reality may use a specialized wearable device that uses sensors to track user movement and/or a holographic projection display to mix virtual objects with the real-world surroundings to provide a mixed reality gaming experience.

[0047] For merged reality, a user may wear a headset. The user may experience an immersive virtual environment while staying within context of the real surroundings. For example, the user may see their own hands and/or may use their hands to interact with objects in the virtual world. Collision detection and/or avoidance capabilities may allow the user to see people around them when they get close and may see a wall that they may about to run into. A camera array may scan a room and may make a 3D map of the room and/or the objects in the room. The 3D computer model created by the cameras may fuse virtual graphics and/or objects in the real world. For merged reality, an untethered device may be used. The untethered device may include a computing platform, sensors and/or cameras incorporated into a device.

[0048] Some embodiments described herein operate to open a window to reality {e.g., the real surroundings) by merging one or more real-time camera images into the VR environment.

360-Degree Video.

[0049] Certain cameras may capture images on a 2D plane. A video created by the captured images on a 2D plane may be referred to as rectilinear video. The video signals from the rectilinear video may have a limited field of view (FOV). A video format {e.g., 360-degree video) may be provided. The 360-degree video may capture visual information beyond the rectilinear 2D video and may be used for VR, mixed reality, and/or merged reality systems.

[0050] 2D video may be captured on a flat surface. 360-degree video may be captured using one or more multi-camera arrays and/or fisheye cameras. The multiple scenes from different cameras may be processed using image stitching technology to obtain a video signal that covers the entire sphere, spanning 360 degrees horizontally and 180 degrees vertically. FIG. 2 illustrates an example 3D coordinates of a sphere. As shown in FIG. 2, sphere sampling may be in longitudes (cp) and latitudes (Θ) in the 3D [X, Y, Z] coordinate system. After capturing and stitching, the spherical video signal may be projected to a 2D planar surface with a geometry conversion process. For example, an equirectangular projection format and/or a cubemap projection format may be used as examples of projection formats for the geometry conversion process. FIG. 3A illustrates an example sphere sampling in longitudes (cp) and latitudes (Θ) in

equirectangular projection. FIG. 3B illustrates an example picture with equirectangular projection. FIG. 4A illustrates an example 3D geometry of cubemap relative to a sphere. FIG. 4B illustrates an example picture with cubemap projection [e.g., using 4 X 3 face arrangement). Other format projections, such as octahedron, icosahedron, segmented sphere, unicube, equal-area, cylinder, and/or the like, may be used.

[0051] A window of reality may be created by merging a real-time captured image from an external camera with a 360-degree immersive content. The 360-degree immersive content may be a 360-degree video. For example, 360-degree video content may be streamed to a device or locally stored on a device. The 360-degree video content may be decoded and/or rendered adaptively to a user in response to user movements and/or corresponding HMD/sensor orientation changes. The 360-degree immersive content may be other immersive graphical content. For example, the 360-degree immersive content may be a VR application, game, and/or operating system having 2D and/or 3D models of content. The 2D and/or 3D models of content may be arranged and/or rendered in order to present a viewport to the user and may be adapted in accordance with user movements and/or corresponding HMD/sensor readings. Merging realtime captured imagery from a local camera may be configured to other VR, mixed reality, and/or merged reality experiences.

[0052] A rectilinear video may be embedded into a 360-degree video. For example, a rectilinear 2D video may be copied onto a face of a cubemap. The cubemap may be converted to other projection formats [e.g., equirectangular) using a projection-format conversion approach.

Position and Orientation Sensing.

[0053] Certain VR headset devices may provide an application program interface (API) to obtain a position and/or orientation of the headset. The information may include the headset's location in the 3D space [e.g., X, Y, Z coordinates) and the headset's orientation information parameters [e.g., yaw, pitch roll). The position information may be obtained from one or more sensors and/or sensor fusion

technologies.

[0054] For example, an inertial measuring unit (IMU) sensor may be used to provide position information. The IMU may include one or more electronic sensors such as accelerometers, gyroscopes, and/or magnetometers. The IMU may measure a device's velocity, orientation, and/or gravitational forces. In virtual and/or augmented reality, IMUs may be used to perform rotational tracking for the HMDs and/or the input devices. The IMU may measure the rotational movements of the pitch, yaw, and/or roll, providing 3 degrees of freedom (DOF) of a 6DOF tracking system.

[0055] The IMUs on a VR headset may respond quickly. For example, the IMU may sample at 1000 Hz and may report positions at 500 Hz. IMUs may drift and may cause linear position measurement errors on the order of meters per second. Certain VR headsets may use a second type of sensor. For example, the VR headset may use one or more optical sensors to provide reliable linear [e.g., X, Y, and/or Z) position information.

[0056] FIG. 5A illustrates an example VR headset 500 with a constellation of LEDs, e.g. LEDs 502. FIG. 5B illustrates a sensor 550 which may be used together with the VR headset {e.g., the sensor may be referred to as a camera). The LED in a VR headset may blink in the infrared spectrum in sequence with a known pattern. The camera may receive the light from the LEDs. An image sensor may be used for optical sensor(s) in the camera. A 3D model may be built up from the detected points of light [e.g., using the known pattern to the camera) and may map translationally and/or rotationally to the origin. Tracking the y and/or z coordinates {e.g., shown in the 3D XYZ coordinate system in FIG. 2) may be performed by monitoring the LED constellation as it moves relative to the view area of the camera. For the x coordinate [e.g., depth), tracking may be enabled. For example, when the headset moves closer, the size of the constellation grid in the view area may get bigger. When the headset moves away, the size of the constellation grid in the view area may get smaller.

[0057] The camera may track the LED patterns and may use the patterns to identify orientation [e.g., yaw, pitch, and/or roll) and/or linear position {e.g., X, Y, and/or Z). The camera may not deliver smooth position and/or orientation estimates for VR [e.g., due to slower camera sensor readings and/or occlusion/blocking issues). Sensor fusion technologies may pair one or more IMUs with a camera to obtain orientation and/or position information at high rate for VR.

Issues Addressed in Some Embodiments.

[0058] Certain VR devices may cut off the connection to the external world when a user puts the device on and starts a VR session. Certain merged reality devices may allow the user to stay connected with the surrounding real world while being immersed in a virtual world. Such devices may use integrated cameras built into the headset. Integrated cameras built into a headset may have limited range of view for the user. For example, as objects are moving farther away from the headset, the objects may become more difficult to distinguish. Certain merged reality devices may operate by constantly scanning the surrounding environment with a front-facing sensor array and/or constructing a 3D model of the surrounding space. Sophisticated sensors and/or higher computing resources {e.g., than a normal camera view) may be provided on certain merged reality devices.

[0059] A system may use one or more cameras {e.g., external or internal cameras) to assist a VR experience. During a VR session using a headset system, a user may become immersed in the virtual world and may lose a connection to the real world. The system using one or more external cameras may display images captured from the external cameras within the 360-degree video and/or other immersive content displayed during a VR session to provide the user with a window of reality. 2D cameras and/or 360- degree cameras may be used for the system described herein. Imagery from available cameras may be integrated into the VR experience in order to improve the user's connection with the real world. For example, imagery from the cameras may be combined with streamed or downloaded VR video content, real time VR communication, interactive games, and/or VR applications that may provide an immersive view of content. The integrated imagery techniques described herein may be built into a VR application or may be provided as a background function or service {e.g., supported by the VR-enabled OS).

[0060] A rectilinear video and/or 360-degree video content may be merged and/or displayed in real-time. The rectilinear video may be converted and may fit within a cubemap face {e.g., the video may not span more than one face). A rectilinear video to 360-degree video conversion may be provided {e.g., regardless of how the rectilinear images and/or the 360-degree video are captured and/or generated).

External Camera Assisted VR.

[0061] In some embodiments, one or more external cameras may be used to help a user stay aware of the surrounding space while being immersed in a virtual environment. The VR headset may be tethered {e.g., with a limited range of movement). The VR headset may be untethered with a larger range of movement. The external cameras may be detached from the headset. The external cameras may be attached to the headset.

[0062] FIG. 6 illustrates an example inset view 602 of a VR user {e.g., self-view). As shown in FIG. 6, the inset 602 in the display 604 seen by the VR user may be a frame that shows a self-view of the user.

The inset view may be intended to alleviate the discomfort associated with wondering "What do I look like?" while interacting with the VR experience. The inset view may be a third person view of the user that may be implemented by merging images from one or more cameras external to the headset into the VR display.

For example, the external cameras may be configured to provide the self-view of the user. Other cameras [e.g., that may exist in the space and/or that may be discovered and/or used by the VR system) may be configured to provide a self-view of the user. For example, the space may be equipped with a smart security camera. The space may include a personal computer with a web cam. The space may have a gaming system that includes a camera system having a view of the space. The space may have one or more cameras of the same or of different types. The available cameras may be discovered and/or may be used to provide a view of the user engaging in VR using the approaches described herein.

[0063] The self-view of the VR user may be displayed to a specialized AR wearable device {e.g., smart glasses) when the user is not in a VR session. One or more external cameras may be configured to provide the self-view of the user, as described herein. The self-view may be displayed to the devices {e.g., as an inset image). For example, a user may be wearing smart glasses and walking. The user may see a regular view of the reality through the smart glasses, like normal glasses. When one or more external cameras capture the self-view of the user, the user may see the captured self-view {e.g., as an inset image) along with the regular view of the reality {e.g., similar to FIG. 6).

[0064] FIG. 7 illustrates an example inset view 702 of a VR user's actual environment. The inset view 702 shown in FIG. 7 may be the view of the environment that the VR user would see if the user was looking through the headset. The inset view 702 may be a view from an external camera not attached to the headset. The user may be able to see the environment around the user without disrupting the VR experience. For example, the user may see the user's hands and/or some parts of user's body. For other example, the inset view may let the user see other people in the room. The inset view may assist in avoiding obstacles and/or accidents during the VR experience. The cameras may be mounted on the VR headset. For example, the cameras may be built into the HMD device, or the cameras may be external and may be attached to the HMD device.

[0065] The view of the environment of the VR user may be displayed to a specialized AR wearable device {e.g., smart glasses) when the user is not in a VR session. One or more external cameras may be configured to provide the environment view of the user, as described herein. The environment view may be displayed to the devices {e.g., as an inset image). For example, a user may be wearing smart glasses and walking. The user may see a regular view of the reality through the smart glasses, like normal glasses. When one or more external cameras capture the environment view of the user, the user may see the captured environment view {e.g., as an inset image) along with the regular view of the reality {e.g., similar to FIG. 7). The user may not need to turn their head to see left, right, and/or behind and may look at the inset image in the device while walking.

[0066] One or more of the following may be considered for selecting and/or enabling the inset view. For example, the camera image may be automatically selected when the VR experience starts. The camera image may be selected by the VR user by taking some action with a remote control and/or gesture system. The camera image may be selected [e.g., continuously) at a preconfigured update rate based on where the headset is pointed. One or more images may be inset {e.g., simultaneously) based on one or more cameras, processing a wide angle, and/or 360-degree image. One or more images may be saved when the VR experience starts. The images may show a front, back, right, and/or left view. Some or all images may be inset {e.g., a front, back, right, and/or left view). An inset may cycle through an image with different views {e.g., a front, back, right, and/or left view) at a certain rate. A subset of images may be displayed and/or used to cycle through some or all captured images. A combination of some or all images, an inset, and/or a subset of images described herein may be used.

Detached External Cameras.

[0067] FIG. 8 illustrates an example utilizing one or more external cameras {e.g., four detached external cameras) in a camera-assisted VR system. As shown in FIG. 8, an external camera {e.g., detachable external camera) may be positioned in the front (camera 802), back (camera 804), left (camera 806), and/or right (camera 808), relative to the headset 810, at a certain distance. Some or all devices may be in the same room. For example, the external cameras may be positioned on the side walls, and the VR headset may be in the middle of the room. The cameras may share position and/or orientation information with a camera selection module 812, and this information may be updated on a continual basis (e.g. periodically, or in response to a detected movement).

[0068] The camera selection module 812 may implement predetermined or user-selected criteria for the selection of one or more active cameras. The user may choose to receive the images captured by one or more external cameras. The user may choose to have captured images displayed together with the 360- degree video 814 and/or other immersive content being displayed in the VR headset. The displayed image may be displayed using a window 816 within the viewport 818 (similar to the inset window as shown in FIG. 6 and/or FIG. 7). The camera or cameras that have been selected (e.g. automatically) to receive images may be referred to as the active cameras.

[0069] FIG. 9 illustrates an example of external camera assisted VR system. At the start of a VR session

902, the available external cameras and the headset may report to each other of their respective locations.

For example, when a VR session starts, the VR system {e.g., the computing platform that drives the VR experience) may communicate {e.g., over a local network) to discover available external cameras and/or other devices which may have camera capabilities {e.g., that may function as external cameras). In some cases, a fixed set of cameras may be assumed and may be available during an entire VR session {e.g., new camera may not come into a session and existing cameras may not leave during the session). For fixed set of cameras, the discovery process may be executed once initially. A non-fixed set of cameras may be provided and may be available during an entire VR session {e.g., camera(s) may come into a session and camera(s) may leave during the session). A combination of fixed set of cameras and non-fixed set of cameras may be provided. For example, during a preconfigured time or event of the VR session, the VR system may have non-fixed set of cameras. During a preconfigured time or event, the VR system may change to a fixed set of cameras. For example, during the first 10 minutes of the VR session, the VR system may allow a flexible set of cameras. During this time, the camera(s) may come into a session and/or camera(s) may leave during this period. After the 10 minutes, the VR system may allow a fixed set of cameras. Camera(s) may not come into the VR session and the camera(s) may not leave until the VR session is over. The discovery process may occur periodically in order to update the set of external cameras, which may be accessible to the VR system. One or more protocols may support device discovery and/or capability exchange within a network, for example, universal plug and play (UPnP).

[0070] The available cameras and/or camera-capable devices may go through a handshake stage 904 [e.g., at the beginning of a VR session) to be registered. The available cameras and/or camera-capable devices may report location and/or orientation information to the VR system in step 906, if the location and/or orientation information is known. For example, the camera may have the capability to discover its location and/or orientation relative to the physical space and/or relative to the VR HMD. One or more localization systems may be used to enable the capability of a camera device to discover its location and/or orientation. For example, the cameras may be equipped with LEDs visible to the external sensor of an HMD. The VR system may determine the location of the cameras relative to the HMD. For other example, the space may be equipped with location beacons visible to the one or more camera devices, the VR system, and/or the HMD system. For another example, the user may label the cameras {e.g., a set of smart home security cameras) with approximate location information {e.g., location metadata and/or a location label) during an initial setup and/or installation stage. The approximate location labeling may be done {e.g., manually), for example, by using the HMD {e.g., with a front camera view enabled for peek-through) to interactively move through the space and to identify locations and/or orientations of the external cameras {e.g., using gestures to point out the camera locations within the space, or by successively positioning the HMD in front of each camera at the approximate camera orientation in order to register each camera position). The cameras may be assumed to be in fixed positions {e.g., wall mounted), and the manual location setting may be done {e.g., once) when the system is first set up. The use of cameras in fixed positions a described herein is one example, but other non-fixed positions and/or movable positions of the cameras may be used for the external-camera-assisted VR systems described herein.

[0071] A camera's field of view (FOV) may be used to properly overlay a rectilinear video into a 360- degree video. The camera's FOV may be measured in the horizontal direction, and the vertical FOV may be calculated using the camera's aspect ratio. Different cameras may have different FOV and/or aspect ratios. During the handshake stage, camera parameters, such as FOV and/or aspect ratios, may be reported by the available cameras to the VR system and/or headset. [0072] The location of a device may be determined and/or calibrated [e.g., in step 906 after the initial handshake) within the entire space {e.g., within the room). An active camera may be selected in step 908. For example, an active camera may be selected based on the user's preference. The VR headset may start receiving and/or displaying images from the active camera in step 912. The images from the active camera may be combined with 360-degree video and/or other immersive content, depending on the VR application or experience in which the user is engaged. The positions of each device may be updated [e.g., periodically). The VR system and/or the headset may determine in step 914 whether the active camera has changed. If the active camera has changed, a new active camera may communicate with the VR system and/or headset [e.g., to establish a connection and/or to provide video content to the VR system). The VR system and/or headset may establish a video connection 910 between the headset and the new active camera. The VR headset may start receiving and/or displaying images from the new active camera in step 912. The process may continue until the VR session ends.

[0073] The VR headset {e.g., through the VR user) may specify which of the available camera(s) it may want to receive video from. For example, one or more may be provided.

[0074] A camera, which may be a fixed camera, may be specified as an active camera. If a fixed camera is specified as an active camera, updating the active camera may be skipped [e.g., in the decision logic), as one active camera may be registered from start to end of a VR session.

[0075] An active range may be specified such that the system may choose a camera [e.g., adaptively) based on the active range as the VR user moves around in the space. For example, the user may choose the front {e.g., the area in front of the HMD) as the active range. Using the 3D coordinate system shown in FIG. 2, the front may be defined with a range of [-45°, 45°] in two directions {e.g., Θ and/or φ may be in the range of [-45°, 45°]). For example, the range of [-45°, 45°] in two directions may be referred to as pitch and yaw range. One or more available cameras that fall within {e.g., or that may provide a view of space within) the user-defined {e.g., pitch and/or yaw) range may be selected by the VR system as an active camera. The 2D imagery from the selected camera may be merged into the 360-degree video and/or other immersive content. The merged 2D imagery and the 360-degree video and/or other immersive content may be shown to the user. The range may be specified by the distance from the camera to the VR headset. For example, a camera that is 2 to 3 feet away from the VR headset may be selected as an active camera. The user may specify an upper limit and/or a lower limit of the distance, depending on if the user cares about monitoring close-by objects and/or remote objects. The user may sort the distances and/or may select the farthest away or closest camera. A combination of pitch and/or yaw range and distance range may be chosen. For example, the user may specify the farthest away camera that is in the front view to be the active camera. [0076] FIG. 10 illustrates an example of selecting an active camera using an active range. As shown in FIG. 10, a front facing active range 1000 may be between 2 feet to 3 feet from the VR headset. Four cameras may be located near the VR headset. In this example, Cameras 1002, 1003, and 1004 are not chosen as an active camera as the cameras may be located too far away [e.g., camera 1003), too close {e.g., camera 1002), or not in the front view {e.g., cameras 1003 and 1004). Camera 1001 is within the active range 1000 (depicted by the shaded region). The active range preference specified by the user results in selection of camera 1001 as the active camera.

[0077] If the VR system chooses a camera based on the active range as described herein, the headset may update the active camera {e.g., periodically) based on the headset's movement. FIGs. 11 A and B illustrate an example of updating an active camera based on movement tracking. As shown in FIG. 11 A, the user may select the front view 1110 as the active range, resulting in selection of camera 1101) As the user's head turns in the clockwise direction, the active region 1110 likewise rotates to encompass camera 1102. The active camera may change from camera 1101 at the first time instance (FIG. 11 A) to camera 1102 at the second time instance (FIG. 11 B). In FIGs. 11 A and 11 B, the user may receive video of the front of the VR headset and/or part of the user's surroundings {e.g., objects situated behind the user's head). If the user selects the farthest-away camera as the active camera, as the user moves around, the distance between the user and some or all cameras may be updated {e.g., periodically). A sorted list of cameras may be kept according to the distances. If the order in the sorted list changes such that a new camera is located the farthest away, the new camera may be selected and/or activated as a new active camera.

[0078] If the VR system chooses a camera based on the active range as described herein, two or more cameras may meet the criteria for a user's active camera. For example, if camera 1002 in FIG. 10 is located farther away to be in a shaded region, the newly positioned camera 1002 may fall into the active range {e.g., along with camera 1001). The number of active cameras may be limited. In this case, during the handshake stage in FIG. 9, the cameras may report technical parameters to the VR headset {e.g., along with the position information). The technical parameters may include spatial resolution, frame rate, field of view, output format including supported codecs, transport format supported, and/or the like. Based on the technical parameters, the VR headset may form an ordered list of preferred cameras. For example, the VR headset may prefer cameras that have wider field of view, higher resolution, and/or higher frame rate. The VR headset may use {e.g., may prefer to use) high efficiency video coding (HEVC) codec or H.264 codec. The VR headset may take a combination of the factors {e.g., from technical parameters) into account when forming the ordered list. Based on the ordered list of preferred cameras, when more than one camera is eligible to be the active camera, the camera with the highest preference {e.g., from the list of preferred cameras) may be selected by the VR headset as the active camera. For other example, the VR headset may be configured to show the images from one or more eligible cameras for the active camera {e.g., in a rotational manner).

[0079] As shown in FIG. 9, data about some or all of the camera's position may be exchanged between the external cameras and the VR headset during the handshake and/or calibration stage. If the external cameras are static [e.g., fixed), the camera positions may not be updated. For example, the VR headset may update the relative positions between itself and the external cameras based on its own movement that may be tracked. If the cameras are moving cameras, position information may be updated {e.g., periodically). For example, a camera may be mounted on a device, such as a robot or a drone. If two or more VR users are operating in a space [e.g., the same physical space), the HMD-mounted camera of a first user may be discovered and may be used as an external camera to provide camera views to the VR device of a second user. For the active cameras, position information and/or the video signal may be sent to the VR headset. The video captured at the active camera may be compressed, may be sent to the VR system and/or headset [e.g., via wired or wireless connection), may be decompressed at the VR system and/or headset, and/or may be displayed inside of the immersive environment.

Establishing External Camera Positions.

[0080] To establish the positions of the external cameras, position sensors may be installed on the external cameras. For example, optical sensors may be installed on the external cameras and may be used to establish external camera positions. The XYZ coordinates of the external cameras may be established by communicating with the VR headset. The sampling frequency of the optical sensors may be lower than the optical sensors on the VR headset. For example, for static external cameras, camera positions may be established once during session initialization and/or handshake. The external cameras may not be equipped with IMU-type sensors, as the external cameras may be more stationary. When the external camera positions are established, the external camera positions relative to the VR headset may be known.

The active camera may be determined based on user configuration {e.g., as shown in FIG. 9).

[0081] The external camera positions may be established using a Bluetooth connection. The real-time location systems (RTLS) technology in Bluetooth may be used to track and/or identify the location of objects in real-time using nodes and/or tags attached to or embedded in the objects being tracked.

Readers that receive and/or process the wireless signals from the tags may determine the location of objects. Besides position information, Bluetooth may be used to exchange information during the handshake session, for example, regarding the FOV parameters, aspect ratios, spatial resolutions, and/or the like for each external camera. Transmitting Video from External Cameras.

[0082] Video from the active external camera may be transmitted to the VR system and/or headset and may be overlaid with the 360-degree video being displayed in the HMD. A wired connection may be established for transmitting video from external cameras. A wireless connection, such as a WiFi connection, may be established for transmitting video from external cameras. The external camera may encode and/or transmit the video when some changes {e.g., significant changes) occur. For example, if an object moves, the external camera may encode and/or transmit the video.

Example Use Case: Mixed-Reality Shooter Game.

[0083] As an example of an application for systems and methods disclosed herein, a laser-tag like game may be considered. The game may be created in a real physical space where two or more users may play in the same space using VR or mixed reality headsets and proxy weapons. The physical space may provide a sense of touch [e.g., actual walls and/or obstacles that the user may bump into or hide behind), while the VR or mixed reality headset may provide graphics that may transform the user to a distant war zone or another planet. The user may see the physical walls and/or obstacles {e.g., enhanced via the graphic display), and may see and/or interact with [e.g., shoot at) other players that the user may encounter in the physical space.

[0084] The game space may be outfitted with cameras at various locations, and the cameras may provide a look-ahead view. A player may see which enemies or teammates may be hiding in the direction that the user may be traveling. For example, a first player may express a preference that a camera in front of the user and at a distance between 8 and 15 feet from the user be selected to provide an inset look- ahead view. The inset look-ahead view may give the user some advance warning of enemies the user may be about to encounter. As the user moves about in the game space, the camera view [e.g., look-ahead view) may appear, disappear, and/or change from one camera to other camera based on the user's own movements, as described herein.

[0085] A second player may express a preference that a camera behind the user and showing a view of the space behind the user be selected to provide an inset rear view, in order to give the user a view of any approaching enemies from behind. The rear camera view for the second player may appear, disappear, and/or change from one camera to another camera based on the user's movements, according to camera selection techniques described herein.

Example Use Case: Maintaining a Self-View.

[0086] For another example application, a VR system may allow the user to specify that an external view of the user may be maintained as the user moves from one location to another location within a space. The

VR system may determine {e.g., adaptively) which of the available cameras (if any) may provide a current view of the user and may select [e.g., adaptively) a camera as the active camera. Selection of a self-view camera may be based on whether the VR user is visible in the camera view {e.g., rather than the distance and/or the orientation of the camera location relative to the HMD).

[0087] If camera locations and/or orientations are known, the VR system and/or headset may track the location of the HMD {e.g., using a x, y, z coordinate location tracking system) and may use 3D geometry to determine which of the available cameras have a view of the user based on the current location of the HMD.

[0088] If camera locations and/or orientations are not known, the VR system and/or headset may process 2D video content from the available cameras in order to determine which of the camera views have a view of the VR user. For example, movement may be identified within the 2D video content, and the movement {e.g., which may represent movement of the user and/or the HMD) may be compared to the HMD's internal {e.g., IMU) sensors to determine whether such movement exhibits correlation. If the detected movement is correlated to the HMD sensor readings, the camera view which provided the 2D video may be determined to include a view of the VR user and/or the HMD. Processing the 2D video content from the available cameras may be repeated {e.g., periodically). The system may determine which of one or more available cameras may have a view of the VR user at a given time, and the system may switch to that camera as the active camera so that the user may have a self-view when such view is available. A camera, which may provide a 2D video content, may be utilized, and the camera's location and/or orientation information may not be communicated from the camera {e.g., may not be available).

[0089] Processing a 2D video from one or more available cameras in order to identify the VR user and/or the HMD may not need a full set of video frames from some or all cameras to be continuously processed. The VR system and/or headset may periodically {e.g., once every 5 or 10 seconds) select a range of video frames {e.g., a half-second time range) from the currently active camera to process the frames. The selected frames may be compared to the HMD sensor readings as described herein. The selected frames may be compared to the HMD sensor readings in order to check whether the VR user {e.g., the HMD) is visible within the video from the currently active camera. If the sensor readings show sufficient movement and the movement is not detectable within the selected range of 2D video frames from the active camera, the VR system and/or headset may begin to poll other available cameras {e.g., on a rotating basis) to determine whether the 2D video from other cameras have movement correlated to the HMD sensor readings. A new active camera providing a self-view may be determined.

[0090] As a side process, if the VR user {e.g., the HMD) is detected within a camera view and the detected view of the VR user is correlated to the HMD sensor readings as described herein, the orientation and/or approximate distance {e.g., or scale of the VR user within the camera view) may be determined. Based on the orientation and/or approximate distance, an approximate location of the camera may be derived. The approximate location may be used in an active camera selection process as described herein. The location and/or orientation information of a camera may be determined by the VR system and/or headset using image processing and/or motion detection. If the camera is not capable of reporting its location and/or orientation, the imaging processing and/or motion detection described herein may be applied.

Attached External Cameras.

[0091] One or more external cameras may be mounted on a VR headset. For example, one or more external cameras may be mounted on top of a VR headset and may be referred to as attached external cameras. An attached external camera may be built into an HMD device. An attached external camera may be added to a device [e.g., as a peripheral after purchasing the device).

[0092] For attached external cameras, the handshake, registration, and/or camera calibration procedures shown in FIG. 9 may be skipped.

[0093] The relative position between the VR headset and the external cameras may not change {e.g., external cameras may be attached to the VR headset). Periodic update of the active camera may be skipped. If a user manually selects a different camera as an active camera, periodic update may be employed.

[0094] A wired connection may be used to send video data between the camera and the headset.

[0095] The attached camera may be a 360-degree video camera capable of representing a spherical video surrounding the user. The 360-degree video camera may be a unit mounted {e.g., a single unit mounted high) on the VR headset and may have a view in some or all directions around the user. The 360- degree video camera may be one or more cameras arrayed around the headset {e.g., one or more front mounted cameras and/or one or more cameras mounted around the perimeter support and/or headband of the HMD). The 360-degree attached camera and/or camera array may capture video in any direction of interest to the user. The user may specify a direction of interest by specifying the center of the view of interest (VOI) and/or the range of the VOL The center and/or range of the VOI parameters may be specified using, for example, angular values of Θ and/or φ shown in FIG. 2 {e.g., similar to a viewport specification). The VOI center may be specified as a point on the sphere, and the range may be specified as width and height of the VOI using yaw and pitch angles. To receive and display the images within the VOI, one or more of the following may be used.

[0096] The external camera may receive information about the VOI {e.g., the center and/or the range). FIG. 13 illustrates an example of generating a VOI on an external camera from a complete 360-degree video. As illustrated in FIG. 13, the external 360 video camera may capture (1302), stitch (1304), and/or produce a 360-degree video. The external camera may extract the VOI from the 360-degree video (1306), e.g., using a rectilinear projection as shown in FIG. 12. FIG. 12 illustrates an example of generating a 2D viewport image from a 360-degre video using a rectilinear projection. The viewport image may be encoded (1308) and sent to a VR headset (1310). The VR headset may receive and may display images corresponding to the VOI, together with the immersive 360-degree video. The amount of video data to be sent from the external camera to the VR system and/or headset may be minimized {e.g., an image corresponding to the view of interest may be sent). The rectilinear video may be combined with 360-degree video on the VR headset. When view of interest changes, low latency may occur.

[0097] A 360-degree video camera may include an array of one or more cameras, and each camera may capture images from a specific direction. The external camera may select images from the cameras that cover the VOI and may perform stitching on the selected images to produce a partial 360-degree video. The external camera may apply rectilinear projection on the partial 360-degree video to produce the VOI. FIG. 14 illustrates an example of generating a VOI on an external camera from a partial 360-degree video. The processing at the external camera may be reduced {e.g., as the external camera may use a partial 360-degree video). In the method of FIG. 14, multi-cameras capture video (1402). Based on the selected view of interest, it is determined (1404) which camera images from the video are to be used to render the selected view of interest. Image stitching (1406) is performed to obtain at least a partial 360- degree video. The selected view of interest is extracted (1408) from the partial 360-degree video. The view of interest is encoded (1410) and sent to the VR headset (1412).

[0098] Stitching images may be performed at the VR headset. The 360-degree video camera may send the images from the cameras covering the VOI to the VR headset {e.g., directly). The 360-degree video camera may send camera position parameters for stitching operations to the VR headset. A partial spherical video may be produced from stitching at the VR headset. The VOI may be extracted from the spherical video at the VR headset. The amount of video data to be transferred may be increased.

[0099] As shown in FIG. 13 and/or FIG. 14, the external camera may receive VOI parameters to perform VOI extraction from the 360-degree video. The range of the VOI {e.g., width and height or yaw and pitch angles, respectively) may stay the same during a VR session {e.g., the entire VR session). The VOI range may be specified {e.g., one time) upon VR session initialization, and VOI center may be changed during the VR session. To specify the VOI center, the user may use one or more of user input devices, such as a remote control, a specialized VR control, a gesture control, and/or the like. The user may use the user input device to move the location of the VOI. For example, the user may use one or more of the user input devices to drag the VOI across the sphere. FIG. 15 illustrates an example of moving and/or zooming VOI via a user input device. As shown in FIG. 15, the first VOI may be centered at CO close to the equator. Using a user input device, such as a remote control and/or gesture control, the user may drag the VOI to be centered at C1. The user may shift the user's observation of the surrounding from what is in the front to what is to the left of the user. A user input device {e.g., remote control and/or gesture control) may be used to manipulate the range of the VOI. For example, the user may shift the center of the VOI and may change the range of VOI, e.g., simultaneously using the user input device, such as a gesture control. In FIG. 15, the user may shift the VOI from CO [e.g., close to the equator) to C2 [e.g., close to the north pole), and may expand the size of the VOI. The user may shift the user's observation of the surrounding from what is in the front to what is at the top.

[0100] Headset orientation information may be used to control the VOI. For example, certain devices may provide headset orientation information {e.g., pitch, yaw, and/or roll) through an API. The headset orientation information may be sent to the external 360-degree camera and may be used to select the center of the VOI. The center of VOI may move along with the user's head motion in a synchronized manner, without needing additional input from remote control or gesture control.

[0101] In FIG. 13 and/or FIG. 14, images corresponding to the VOI may be sent from the external 360- degree camera to the VR headset. Video coding [e.g., video compression) may be applied to reduce the bandwidth for representing the VOI images. Certain video coding may employ temporal motion compensation prediction (MCP) approaches. MCP approaches may create dependency among images in the temporal domain. Some or all intra coding {e.g., video coding without using MCP), raw, and/or non- coded images may be sent to allow VOI to move freely.

[0102] The VOI may be specified relative to the HMD orientation. For example, the user may specify a view forward and to the right. An active portion of the 360-degree attached camera may be defined which points forward and to the right relative to the orientation of the HMD. If the user walks or turns, the camera view that the user would see may be forward and to the right. For example, the camera view may change relative to the user's new location and/or orientation.

[0103] The VOI may be specified relative to an absolute direction and/or an absolute location within the space. For example, the user may use a user interface, which may feature a front camera view to frame a portion of the room [e.g., a couch on which the user's dog is sleeping), and the user may set this absolute frame as the desired view. When the desired direction and/or view is specified, the VR system and/or headset may obtain and display the appropriate subset of the 360-degree attached video camera content and may display the specified content area [e.g., the couch with sleeping dog), regardless of how the user turns or moves about within the room. For example, the user may be initially facing the couch with the sleeping dog, and a portion of the front camera of the 360-degree attached camera array may provide the user an inset view of the selected area. As the user turns away from the couch with the sleeping dog, the VR system and/or headset may detect the change in orientation of the HMD and may shift {e.g., smoothly as the user turns) which area from the 360-degree attached camera's view represents the active camera viewport [e.g., to maintain the original framed area in the inset view). For some positions, the active camera viewport may be derived from a camera {e.g., single component camera) in the 360-degree attached camera array. For other positions, the active camera viewport may be derived from two or more component camera views stitched together. The user may maintain {e.g., given the user movement) a stable view of an absolute direction and/or an absolute area of the physical space as the user moves around in a VR experience.

Camera Selection Examples.

[0104] A user may express preferences for what inset camera view may be provided, and/or under what circumstances an inset camera view may be shown to the user. Based on the user preferences, the VR system and/or headset may select an active camera or an active camera viewport to display to the user during a user's VR experience. Content from the selected active camera [e.g., a subset of the content for a 360-degree camera) may be obtained and/or merged with 360-degree video content and/or other immersive VR content as appropriate to the user's current VR experience. User preference may specify one or more of camera locations, camera orientations, an orientation or range of orientations relative to the VR user {e.g., relative to the HMD), a distance or range of distances relative to the VR user, and/or an absolute orientation or location in physical space which may be maintained in the inset view. User preference may request that a self-view of the VR user be maintained. For example, the VR system and/or headset may switch among the available cameras to maintain the self-view based on the user preference.

[0105] A user may specify that a real-time camera view may pop-up in the case of a change of environment and/or when motion is detected within the environment. For example, the room in which the user is playing a VR game may have been empty and quiet when the user first put on the VR headset. A few minutes later, the user's toddler may have come into the room dropping toys which may present a hazard to the VR user. If the user specifies an alert and/or pop-up view due to motion and/or change in environment, the inset view may appear when the motion and/or change is detected, and the user may be alerted to potential hazard. For a HMD-mounted camera and/or an attached 360-degree camera, the VR system and/or headset may receive the video output from the camera and may analyze the video frames for motion and/or changes, compensating for global motion [e.g., using HMD sensor input). When unexpected motion and/or change in the environment is detected, the system may display the video from the HMD-mounted camera, and the user may be made aware of the change. If the attached camera is a 360-degree camera, the system may select a viewport area based on the location of the detected motion and/or the detected changes and may display the viewport containing the detected motion and/or the detected changes to the user.

[0106] For external [e.g., separate) cameras, the VR system and/or headset may obtain a video from each of the available cameras {e.g., on a rotating or periodic basis) and may analyze the video for motion and/or environmental changes. The processing may distinguish between the motion of the VR user [e.g., which may not trigger the inset view to appear) and the motion of other people or objects within the space {e.g., which may trigger the inset view to appear). The VR system and/or headset may have location and/or orientation information of each camera and may compare such metadata to the known location of the VR user [e.g., as tracked by the VR system by location tracking sensors) in order to exclude the area of the camera view occupied by the VR user from the motion and/or environmental change detection analysis. If location and/or orientation information for some or all of the cameras are not available, the VR system and/or headset may compare the detected motion to the HMD sensor readings in order to identify {e.g., based on correlation between motion and sensor readings) whether the area of motion in a particular camera view is due to motion of the VR user. If the detected motion is due to the VR user, the

corresponding area of the camera view may be excluded from the analysis.

[0107] When motion and/or environmental change is detected in a camera view and the motion and/or environmental change does not correspond to the VR user, the camera in which the motion and/or environmental change is detected may be set to the active camera, and the view from the active camera may be shown to the user.

[0108] The user may specify a particular camera, location, and/or area to be monitored for motion and/or environmental changes. The detection of motion and/or environmental changes may be configured to a selected camera. For example, if the user has eight security cameras distributed within a house, the user may specify that motion and/or environmental change from the security camera which has a view of the front door may be used. For example, when someone enters through the front door, the change may be detected, and the inset view of that camera [e.g., having the front door view) may be displayed to the user. Motion and/or environmental changes from the other seven security cameras may not cause any change in the active camera selection.

[0109] The VR system and/or headset {e.g., or the camera devices) may have machine vision capabilities and may be capable of object detection and/or identification. The user may set one or more detailed preferences of what types of detected objects and/or object motion may trigger an active camera view to be displayed to the user in a VR session. For example, the user may be playing a VR game in a closed room with a door, and the user may specify that a camera view showing the door area may be displayed to the user when the door opens. The VR system and/or headset may analyze the available camera views to determine which views can see the door and may detect when the door opens. If the system and/or headset detects the door opening, the VR system and/or headset may select a camera with a view of the door as the active camera and may display the active camera view {e.g., the view of the door) as an inset view for the user.

[0110] For other example, the user may express a preference that if another person, a specific person, or the user's pet dog is detected in the available camera views, the detected view may be shown as an inset to the user. Additional Implementation Features.

[0111] One or more approaches described herein may be programmed into a VR application and/or game. One or more approaches described herein may be included in a VR operating system and may be supported on the VR system [e.g., computing platform) and/or the VR headset.

[0112] The VR operating system may provide one or more services and/or functions described herein and may make such services and/or functions available to a VR application which may show camera views {e.g., views from external available cameras, or from an attached 360-degree camera) to the user. For example, one or more of the following services may be provided by the VR operating system and may be available to a VR application via an API set.

[0113] One or more external cameras may be discovered. Capabilities, locations, orientations, FOV parameters, and/or the like properties of the discovered external cameras may be obtained. An external camera may be selected [e.g., by an application) to display current view [e.g., as an overlay) to the user. The position and/or size of the external camera display [e.g., inset display) may be set within the user's view. The external camera display {e.g., inset display) may be turned on or off. A preference of selecting an active camera from among one or more available cameras by the VR OS may be specified. For example, the preference may be set according to one or more of the following: an orientation or orientation range, a distance or distance range, an absolute location of the camera in the space, a relative location of the camera compared to that of the user {e.g., the HMD), a detection of motion or environmental change within the camera view, and/or the like. The active camera selection may be turned on or off. For example, when the active camera selection is on, the VR OS may monitor the user's movement and/or orientation, may select active camera views, and may display the selected views to the user, according to the preferences set by the VR application. The application may set preferences for how active camera selection may be managed by the VR OS. The VR OS may manage the details of the active camera selection process as described herein.

[0114] The VR operating system may provide the capability to access one or more external camera views as a user-facing feature. The camera display function may be available {e.g., as an overlay managed by the VR OS) for any VR application, game, and/or experience the user may be engaged in. The VR OS may provide a user interface. The user may configure the preferences and may configure the VR OS to select the active camera (or the active portion of a 360-degree camera) among one or more available cameras.

[0115] The user interface may have one or more of the following features. A list of the available camera devices {e.g., the result of a camera and/or device discovery process) may be displayed. The list may include capabilities and/or other properties {e.g., location, orientation, resolution, and/or the like) of each available camera and/or device. A current view from each camera may be accessed. Accessing the current view from a camera may allow the user to get a feel for the location and/or orientation of the camera. The user interface may specify which cameras may be allowed and which cameras may be disallowed from using as the active camera. The allowed active camera may be displayed to the user. A particular external camera or a portion of an attached 360-degree camera view may be selected [e.g., manually). The selected external camera or camera view may be displayed to the user. The size and/or location of the inset camera view may be specified by the user, and the inset camera view may be displayed to the user. The inset camera view may be turned on or off. User preferences may be specified. The user preferences may include preferences for how the VR OS may select an active camera from among one or more available cameras. For example, preference may be set according to one or more of the following: an orientation or orientation range, a distance or distance range, an absolute location of the camera in the space, a relative location of the camera compared to that of the user [e.g., the HMD), a detection of motion and/or environmental change within the camera view, a detected object, person, or event, and/or the like. Active camera selection may be turned on or off. When active camera selection is on, the VR OS may monitor the user's movement and/or orientation, may select active camera views, and may display the selected views to the user, according to the preferences set by the VR user in the user interface described herein.

Rectilinear Video to 360-Degree Video Conversion.

[0116] A rectilinear video may be projected to a 360-degree video represented in equirectangular projection (ERP) format. The width of the ERP picture may be denoted as W and the height as H. The width of the rectilinear viewport video may be denoted as WVP and the height as HVP. The center of the viewport may be denoted to be at (cpc, 9c), and the range of the viewport may be denoted to be (CPR, 9R). The range of the viewport may be set equal to the FOV of the active external camera. On the sphere shown in FIG. 12, the viewport may correspond to an area on the sphere in the range of (cpc, 9c) ± (CPR/2,

[0117] The conversion process may be applied in one or more of the following approaches. For a position (m, n) on the ERP picture, where m may be 0 < m < W and n may be 0 < n < H, the position (m, n)'s corresponding (cp, Θ) position and (Χ', Υ', Ζ') coordinates may be calculated using one or more of the following equations.

u = (m + 0.5) / W, 0 < m <W (1) v = (n + 0.5) / H, 0 < n < H (2) cp = (u - 0.5) * (2 * TT) (3) θ = (0.5 - ν) * π (4) [0118] If (φ, Θ) is outside of the range of (cpc, 9c) ± (CPR/2, 9R/2), the sample value S(m, n) on the ERP picture may be set to a default value [e.g., gray), as the position is outside of the viewport. When the sample value S(m, n) on the ERP picture is set to a default value, the position (m, n)'s corresponding (φ, Θ) position and (Χ', Υ', Ζ') coordinates may be calculated.

[0119] (φ, Θ) may be calculated by subtracting (cpc, 9c) from (φ, Θ). For example, cp may be set to cp - cpc, and Θ may be set to Θ - 9c

[0120] The corresponding coordinate (rr , n') on the viewport may be calculated using equation (5) and/or equation (6).

m' may be 0 < m' < WVP, and n' may be 0 < n' < HVP.

[0121] The non-linear warping effect in 2D-to-3D and/or 3D-to-2D coordinate conversion may exist. The calculated (rrf, n') may be out of bounds of the rectilinear video picture. If the calculated (rr , n') is out of bounds of the rectilinear video picture, the sample value S(m, n) on the ERP picture may be set to the default value, and the process may be repeated by calculating the position (m, n)'s corresponding (cp, Θ) position and (Χ', Υ', Ζ') coordinates as described herein.

[0122] If the calculated (rr , n') is inside of the rectilinear video picture, the sample value on the viewport at (m', n') may be denoted as Svp(m', n'). If (m', n') is at a fractional position, interpolation to the neighboring samples at integer positions may be applied to derive the sample value Svp(m', n'). The sample value S(m, n) on the ERP picture may be set to Svp(m', n').

[0123] The one or more approaches described herein may be extended to other projection formats. When other projection formats are used to represent a 360-degree video, equations (1), (2), (3), and/or (4) may be replaced with the 2D-to-3D coordinate conversion equations for the projection format being used.

Examples of Networks for Implementation of Embodiments.

[0124] FIG. 16A is a diagram of an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, efc, to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), and the like. [0125] As shown in FIG. 16A, the communications system 100 may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, and/or 102d (which generally or collectively may be referred to as WTRU 102), a radio access network (RAN) 103/104/105, a core network 106/107/109, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d may be configured to transmit and/or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, consumer electronics, and the like.

[0126] The communications systems 100 may also include a base station 114a and a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, and/or the networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements.

[0127] The base station 114a may be part of the RAN 103/104/105, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, e.g., one for each sector of the cell. In another embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and, therefore, may utilize multiple transceivers for each sector of the cell.

[0128] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 115/116/117, which may be any suitable wireless communication link {e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface

115/116/117 may be established using any suitable radio access technology (RAT). [0129] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 103/104/105 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115/116/117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink Packet Access (HSDPA) and/or High-Speed Uplink Packet Access (HSUPA).

[0130] In another embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 115/116/117 using Long Term Evolution (LTE) and/or LTE-Advanced (LTE-A).

[0131] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.16 {e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0132] The base station 114b in FIG. 16A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell or femtocell. As shown in FIG. 16A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the core network 106/107/109.

[0133] The RAN 103/104/105 may be in communication with the core network 106/107/109, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. For example, the core network 106/107/109 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication. Although not shown in FIG. 16A, it will be appreciated that the RAN 103/104/105 and/or the core network 106/107/109 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 103/104/105 or a different RAT. For example, in addition to being connected to the RAN 103/104/105, which may be utilizing an E-UTRA radio technology, the core network 106/107/109 may also be in communication with another RAN (not shown) employing a GSM radio technology.

[0134] The core network 106/107/109 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common

communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 112 may include wired or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another core network connected to one or more RANs, which may employ the same RAT as the RAN 103/104/105 or a different RAT.

[0135] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities, e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102c shown in FIG. 16A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0136] FIG. 16B is a system diagram of an example WTRU 102. As shown in FIG. 16B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment. Also, embodiments contemplate that the base stations 114a and 114b, and/or the nodes that base stations 114a and 114b may represent, such as but not limited to transceiver station (BTS), a Node-B, a site controller, an access point (AP), a home node-B, an evolved home node-B (eNodeB), a home evolved node-B (HeNB), a home evolved node-B gateway, and proxy nodes, among others, may include some or all of the elements depicted in FIG. 16B and described herein.

[0137] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 16B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0138] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station {e.g., the base station 114a) over the air interface 115/116/117. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals.

[0139] In addition, although the transmit/receive element 122 is depicted in FIG. 16B as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 {e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 115/116/117.

[0140] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11 , for example.

[0141] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128 {e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown). [0142] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries {e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0143] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information {e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 115/116/117 from a base station {e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0144] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[0145] FIG. 16C is a system diagram of the RAN 103 and the core network 106 according to an embodiment. As noted above, the RAN 103 may employ a UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 115. The RAN 103 may also be in communication with the core network 106. As shown in FIG. 16C, the RAN 103 may include Node-Bs 140a, 140b, 140c, which may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 115. The Node-Bs 140a, 140b, 140c may each be associated with a particular cell (not shown) within the RAN 103. The RAN 103 may also include RNCs 142a, 142b. It will be appreciated that the RAN 103 may include any number of Node-Bs and RNCs while remaining consistent with an embodiment.

[0146] As shown in FIG. 16C, the Node-Bs 140a, 140b may be in communication with the RNC 142a. Additionally, the Node-B 140c may be in communication with the RNC142b. The Node-Bs 140a, 140b, 140c may communicate with the respective RNCs 142a, 142b via an lub interface. The RNCs 142a, 142b may be in communication with one another via an lur interface. Each of the RNCs 142a, 142b may be configured to control the respective Node-Bs 140a, 140b, 140c to which it is connected. In addition, each of the RNCs 142a, 142b may be configured to carry out or support other functionality, such as outer loop power control, load control, admission control, packet scheduling, handover control, macrodiversity, security functions, data encryption, and the like.

[0147] The core network 106 shown in FIG. 16C may include a media gateway (MGW) 144, a mobile switching center (MSC) 146, a serving GPRS support node (SGSN) 148, and/or a gateway GPRS support node (GGSN) 150. While each of the foregoing elements are depicted as part of the core network 106, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[0148] The RNC 142a in the RAN 103 may be connected to the MSC 146 in the core network 106 via an luCS interface. The MSC 146 may be connected to the MGW 144. The MSC 146 and the MGW 144 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices.

[0149] The RNC 142a in the RAN 103 may also be connected to the SGSN 148 in the core network 106 via an luPS interface. The SGSN 148 may be connected to the GGSN 150. The SGSN 148 and the GGSN 150 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between and the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0150] As noted above, the core network 106 may also be connected to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[0151] FIG. 16D is a system diagram of the RAN 104 and the core network 107 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the core network 107.

[0152] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a.

[0153] Each of the eNode-Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the uplink and/or downlink, and the like. As shown in FIG. 16D, the eNode-Bs 160a, 160b, 160c may communicate with one another over an X2 interface. [0154] The core network 107 shown in FIG. 16D may include a mobility management gateway (MME) 162, a serving gateway 164, and a packet data network (PDN) gateway 166. While each of the foregoing elements are depicted as part of the core network 107, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[0155] The MME 162 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may also provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM or WCDMA.

[0156] The serving gateway 164 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via the S1 interface. The serving gateway 164 may generally route and forward user data packets to/from the WTRUs 102a, 102b, 102c. The serving gateway 164 may also perform other functions, such as anchoring user planes during inter-eNode B handovers, triggering paging when downlink data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0157] The serving gateway 164 may also be connected to the PDN gateway 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0158] The core network 107 may facilitate communications with other networks. For example, the core network 107 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the core network 107 may include, or may communicate with, an IP gateway {e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the core network 107 and the PSTN 108. In addition, the core network 107 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[0159] FIG. 16E is a system diagram of the RAN 105 and the core network 109 according to an embodiment. The RAN 105 may be an access service network (ASN) that employs IEEE 802.16 radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 117. As will be further discussed below, the communication links between the different functional entities of the WTRUs 102a, 102b, 102c, the RAN 105, and the core network 109 may be defined as reference points.

[0160] As shown in FIG. 16E, the RAN 105 may include base stations 180a, 180b, 180c, and an ASN gateway 182, though it will be appreciated that the RAN 105 may include any number of base stations and ASN gateways while remaining consistent with an embodiment. The base stations 180a, 180b, 180c may each be associated with a particular cell (not shown) in the RAN 105 and may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 117. In one embodiment, the base stations 180a, 180b, 180c may implement MIMO technology. Thus, the base station 180a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a. The base stations 180a, 180b, 180c may also provide mobility management functions, such as handoff triggering, tunnel establishment, radio resource management, traffic classification, quality of service (QoS) policy enforcement, and the like. The ASN gateway 182 may serve as a traffic aggregation point and may be responsible for paging, caching of subscriber profiles, routing to the core network 109, and the like.

[0161] The air interface 117 between the WTRUs 102a, 102b, 102c and the RAN 105 may be defined as an R1 reference point that implements the IEEE 802.16 specification. In addition, each of the WTRUs 102a, 102b, 102c may establish a logical interface (not shown) with the core network 109. The logical interface between the WTRUs 102a, 102b, 102c and the core network 109 may be defined as an R2 reference point, which may be used for authentication, authorization, IP host configuration management, and/or mobility management.

[0162] The communication link between each of the base stations 180a, 180b, 180c may be defined as an R8 reference point that includes protocols for facilitating WTRU handovers and the transfer of data between base stations. The communication link between the base stations 180a, 180b, 180c and the ASN gateway 182 may be defined as an R6 reference point. The R6 reference point may include protocols for facilitating mobility management based on mobility events associated with each of the WTRUs 102a, 102b, 102c.

[0163] As shown in FIG. 16E, the RAN 105 may be connected to the core network 109. The

communication link between the RAN 105 and the core network 109 may defined as an R3 reference point that includes protocols for facilitating data transfer and mobility management capabilities, for example. The core network 109 may include a mobile IP home agent (MIP-HA) 184, an authentication, authorization, accounting (AAA) server 186, and a gateway 188. While each of the foregoing elements are depicted as part of the core network 109, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[0164] The MIP-HA may be responsible for IP address management, and may enable the WTRUs 102a, 102b, 102c to roam between different ASNs and/or different core networks. The MIP-HA 184 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The AAA server 186 may be responsible for user authentication and for supporting user services. The gateway 188 may facilitate interworking with other networks. For example, the gateway 188 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. In addition, the gateway 188 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[0165] Although not shown in FIG. 16E, it will be appreciated that the RAN 105 may be connected to other ASNs and the core network 109 may be connected to other core networks. The communication link between the RAN 105 the other ASNs may be defined as an R4 reference point, which may include protocols for coordinating the mobility of the WTRUs 102a, 102b, 102c between the RAN 105 and the other ASNs. The communication link between the core network 109 and the other core networks may be defined as an R5 reference, which may include protocols for facilitating interworking between home core networks and visited core networks.

[0166] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, WTRU, terminal, base station, RNC, or any host computer.

Claims

CLAIMS What is Claimed:

1. A method comprising:

tracking a position of a head-mounted display;

determining respective positions of each of a plurality of external video cameras unattached to the head-mounted display;

selecting one of the external video cameras based on the position of the head-mounted display relative to the positions of the external video cameras; and

on the head-mounted display, displaying video captured by the selected external video camera.

2. The method of claim 1 , wherein selection of one of the external video cameras is made such that video captured by the selected external video camera includes a view of a user of the head-mounted display.

3. The method of any of claims 1-2, further comprising tracking an orientation of the head-mounted display, wherein selection of one of the external video cameras is further made such that the selected external video camera is within a predetermined region with respect to the orientation of the head-mounted display.

4. The method of claim 1 , further comprising tracking an orientation of the head-mounted display, wherein selection of one of the external video cameras is made such that video captured by the selected external video camera includes a view in a predetermined direction relative to the orientation of the head-mounted display.

5. The method of any of claims 1 -4, further comprising determining respective orientations of each of the external video cameras, wherein selection of one of the external video cameras is further based in part on the orientations of the external video cameras.

6. The method of any of claims 1-5, wherein the selection of one of the external video cameras is based in part on respective distances between the head-mounted display and the external video cameras.

7. The method of claim 6, wherein the selection of one of the external video cameras is made such that the selected external video camera is within a predetermined range of distances from the head-mounted display.

8. The method of any of claims 1-7, further comprising performing a discovery process to detect the

plurality of external video cameras.

9. The method of any of claims 1-8, wherein the head-mounted display is a virtual reality display, and wherein displaying the video captured by the selected external video camera comprises displaying the video captured by the selected external video camera as an inset in virtual reality content.

10. The method of any of claims 1-9, further comprising periodically updating the respective positions of each of the plurality of external video cameras.

11. The method of any of claims 1-10, wherein the selection of one of the external video cameras is based in part on at least one of spatial resolution, frame rate, and field of view of the external video cameras.

12. The method of any of claims 1-11, wherein the selection of one of the external video cameras is based at least in part on input of user preferences.

13. A method performed at a head-mounted display, the method comprising:

capturing video from at least one external camera attached to the head-mounted display;

receiving from a user a selection of a view of interest;

selecting, from the captured video, a portion of the captured video corresponding to the view of interest; and

on the head-mounted display, displaying the selected portion of the captured video as an inset in virtual reality content.

14. The method of claim 13, wherein the captured video is 360-degree video, and wherein selecting a portion of the captured video comprises extracting the portion from the 360-degree video.

15. The method of any of claims 13-14, wherein the view of interest is specified relative to an orientation of the head-mounted display.

16. The method of any of claims 13-14, wherein the view of interest is specified relative to an absolute orientation, the method further comprising tracking an orientation of the head-mounted display, wherein the tracked orientation of the head-mounted display is used to maintain the view of interest in the selected portion of the captured video.

17. The method of any of claims 13-14, wherein the view of interest is specified relative to an absolute location.

18. A system comprising a processor and a non-transitory computer storage medium storing instructions operative to perform the functions of any of claims 1-17.