KR102779068B1 - Systems and methods for augmented reality - Google Patents

Abstract

An augmented reality display system includes an electromagnetic field emitter for emitting a known magnetic field in a known coordinate system. The system also includes an electromagnetic sensor, the electromagnetic sensor measuring a parameter related to a flux of the electromagnetic sensor due to the known magnetic field. The system further includes a depth sensor for measuring a distance in the known coordinate system. The system also includes a controller for determining pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system, based at least in part on the distance measured by the depth sensor and the parameter related to the flux measured by the electromagnetic sensor. The system also includes a display system for displaying virtual content to a user, based at least in part on the pose information of the electromagnetic sensor relative to the electromagnetic field emitter.

Description

SYSTEMS AND METHODS FOR AUGMENTED REALITY

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62/292,185, filed February 5, 2016 and U.S. Provisional Patent Application Serial No. 62/298,993, filed February 23, 2016. This application is also a continuation-in-part of U.S. Provisional Patent Application Serial No. 15/062,104, filed March 5, 2016, which claims the benefit of U.S. Provisional Patent Application Serial No. 62/128,993, filed March 5, 2015 and U.S. Provisional Patent Application Serial No. 62/292,185, filed February 5, 2016. This application is also related to U.S. Provisional Patent Application Serial No. 62/301,847, filed March 1, 2016. The foregoing applications are hereby incorporated by reference in their entirety into this application.

[0002] The present disclosure relates to systems and methods for localizing the position and orientation of one or more objects in the context of augmented reality systems.

[0003] Modern computing and display technologies have enabled the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in such a way that they appear or can be perceived as being real. Virtual reality, or "VR" scenarios typically involve the presentation of digital or virtual image information that is not transparent to other actual, real-world visual inputs; augmented reality, or "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation to a visualization of the real world around the user.

[0004] For example, referring to FIG. 1 , an augmented reality scene (4) is depicted, wherein a user of the AR technology sees a real-world park-like setting (6) featuring people, trees, buildings, and a concrete platform (1120) in the background. In addition to these items, the user of the AR technology also perceives that he "sees" a robot statue (1110) standing on the real-world platform (1120), and a flying cartoon avatar character (2) that appears to be an anthropomorphic bumblebee (even though these elements (2, 1110) do not exist in the real world). As it turns out, the human visual perception system is very complex, and it is challenging to produce VR or AR technologies that enable a comfortable, natural-feeling, rich presentation of virtual imagery elements among other virtual or real-world imagery elements.

[0005] For example, head-worn AR displays (or helmet-mounted displays, or smart glasses) are typically at least loosely coupled to the user's head, and thus move as the user's head moves. When the user's head motions are detected by the display system, the data being displayed can be updated to account for changes in head pose.

[0006] For example, if a user wearing a head-worn display views a virtual representation of a three-dimensional (3D) object on the display and walks around an area where the 3D object appears, the 3D object may be re-rendered for each viewpoint, giving the user the perception that he or she is walking around an object that occupies real space. If the head-worn display is used to present multiple objects in a virtual space (e.g., a rich virtual world), measurements of head pose (i.e., the position and orientation of the user's head) may be used to re-render the scene to match the user's dynamically changing head position and orientation, thereby providing an increased sense of immersion in the virtual space.

[0007] In AR systems, detection or calculation of head pose may enable the display system to render virtual objects so that they appear to occupy real-world space in a manner that is perceptible to the user. Additionally, detection of the position and/or orientation of a real-world object, such as a handheld device (which may also be referred to as a "totem"), a haptic device, or other real-world physical object, relative to the user's head or the AR system may also facilitate the display system in presenting display information to the user so that the user can efficiently interact with certain aspects of the AR system. As the user's head moves around the real-world, the virtual objects may be re-rendered as a function of the head pose, such that the virtual objects appear to remain stationary relative to the real world. At least for AR applications, the spatial placement of virtual objects relative to physical objects (e.g., presented so that they appear spatially close to the physical objects in two or three dimensions) may be an important issue. For example, head movements can significantly complicate the placement of virtual objects in terms of the surroundings. This may be true whether the view is captured as an image of the surroundings and then projected or displayed to the end user, or whether the end user perceives the view of the surroundings directly. For example, head movements may cause the end user's field of view to change, which may require updates to the positions where various virtual objects are displayed in the end user's field of view. Additionally, head movements may occur within a wide variety of ranges and speeds. Head movement speed may vary not only between different head movements, but also within or across a range of a single head movement. For example, head movement speed may initially increase (e.g., linearly or non-linearly) from a starting point and decrease as an end point is reached, achieving a maximum speed somewhere between the starting and ending points of the head movement. Rapid head movements may even surpass the ability of certain display or projection technologies to render images that appear to the end user as uniform and/or smooth motion.

[0008] Head tracking accuracy and latency (i.e., the time elapsed between the time a user moves his or her head and the time an image is updated and displayed to the user) have been challenges for VR and AR systems. In particular, for display systems that fill a significant portion of the user's visual field with virtual elements, it is critical that head tracking accuracy be high and overall system latency be very low from the initial detection of head motion to the update of the light delivered to the user's visual system by the display. If latency is high, the system can create a mismatch between the user's vestibular system and visual sensory system, creating user perception scenarios that can lead to motion sickness or simulator sickness. If system latency is high, the apparent positions of virtual objects will appear unstable during rapid head motions.

[0009] In addition to head-worn display systems, other display systems may benefit from accurate, low-latency head pose detection. These include head-tracked display systems, where the display is not worn on the user's body, but rather mounted on, for example, a wall or other surface. A head-tracked display may act as a window onto a scene, and as the user moves his or her head relative to the "window," the scene is re-rendered to match the user's changing viewpoint. Other systems include head-worn projection systems, where the head-worn display projects light onto the real world.

[00010] Additionally, to provide a realistic augmented reality experience, AR systems may be designed to interact with the user. For example, multiple users may play a ball game with a virtual ball and/or other virtual objects. One user may "catch" the virtual ball and throw the ball back to another user. In another embodiment, a totem (e.g., a real bat communicatively coupled to the AR system) may be provided to the first user to hit the virtual ball. In other embodiments, a virtual user interface may be presented to the AR user to allow the AR user to select one of many options. The user may use totems, haptic devices, wearable components, or simply touch the virtual screen to interact with the system.

[00011] Detecting the user's head pose and orientation and detecting the physical locations of real objects in space can enable AR systems to display virtual content in an effective and enjoyable manner. However, these capabilities are key to AR systems, but are difficult to achieve. In other words, the AR system must recognize the physical locations of real objects (e.g., the user's head, a totem, a haptic device, a wearable component, the user's hand, etc.) and correlate the physical coordinates of the real objects to virtual coordinates corresponding to one or more virtual objects being displayed to the user. This requires highly accurate sensors and sensor recognition systems that can track the position and orientation of the one or more objects at high speed. Current approaches do not perform localization to satisfactory speed or accuracy standards.

[00012] Therefore, there is a need for a better localization system in the context of AR and VR devices.

[00013] Embodiments of the present invention relate to devices, systems and methods for enabling virtual reality and/or augmented reality interaction for one or more users.

[00014] In one embodiment, an augmented reality (AR) display system includes an electromagnetic field emitter for emitting a known magnetic field in a known coordinate system. The system also includes an electromagnetic sensor, the electromagnetic sensor measuring a parameter related to a flux of the electromagnetic sensor due to the known magnetic field. The system further includes a depth sensor for measuring a distance in the known coordinate system. The system also includes a controller for determining pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system based at least in part on the distance measured by the depth sensor and the parameter related to the flux measured by the electromagnetic sensor. The system also includes a display system for displaying virtual content to a user based at least in part on the pose information of the electromagnetic sensor relative to the electromagnetic field emitter.

[00015] In one or more embodiments, the depth sensor is a passive stereo depth sensor.

[00016] In one or more embodiments, the depth sensor is an active depth sensor. The depth sensor can be a texture projection stereo depth sensor, a structured light projection stereo depth sensor, a time of flight depth sensor, a LIDAR depth sensor, or a modulated emission depth sensor.

[00017] In one or more embodiments, the depth sensor includes a depth camera having a first field of view (FOV). The AR display system may also include a world capture camera, the world capture camera having a second FOV that at least partially overlaps the first FOV. The AR display system may also include a picture camera, the picture camera having a third FOV that at least partially overlaps the first FOV and the second FOV. The depth camera, the world capture camera, and the picture camera may each have different first resolutions, second resolutions, and third resolutions. The first resolution of the depth camera may be sub-VGA, the second resolution of the world capture camera may be 720p, and the third resolution of the picture camera may be 2 megapixels.

[00018] In one or more embodiments, the depth camera, the world capture camera, and the photo camera are configured to capture a first image, a second image, and a third image, respectively. The controller can be programmed to segment the second image and the third image. The controller can be programmed to segment the second image and the third image, and then fuse the second image and the third image to generate a fused image. Measuring the distance in a known coordinate system can include generating a hypothetical distance by analyzing the first image from the depth camera, and generating the distance by analyzing the hypothetical distance and the fused image. The depth camera, the world capture camera, and the photo camera can form a single integrated sensor.

[00019] In one or more embodiments, the AR display system also includes an additional localization resource for providing additional information. The pose information of the electromagnetic sensor with respect to the electromagnetic field emitter in a known coordinate system can be determined at least in part based on the additional information provided by the additional localization resource, parameters related to the distance measured by the depth sensor and the magnetic flux measured by the electromagnetic sensor.

[00020] In one or more embodiments, the additional localization resource can include a WiFi transceiver, an additional electromagnetic emitter, or an additional electromagnetic sensor. The additional localization resource can include a beacon. The beacon can emit radiation. The radiation can be infrared radiation, and the beacon can include an infrared LED. The additional localization resource can include a reflector. The reflector can reflect the radiation.

[00021] In one or more embodiments, the additional localization resource may include a cellular network transceiver, a RADAR emitter, a RADAR detector, a LIDAR emitter, a LIDAR detector, a GPS transceiver, a poster having a known detectable pattern, a marker having a known detectable pattern, an inertial measurement unit, or a strain gauge.

[00022] In one or more embodiments, the electromagnetic field emitter is coupled to a mobile component of the AR display system. The mobile component can be a handheld component, a totem, a head-mounted component housing the display system, a torso-worn component, or a belt pack.

[00023] In one or more embodiments, the electromagnetic field emitter is coupled to an object in a known coordinate system such that the electromagnetic field emitter has a known position and a known orientation. The electromagnetic sensor can be coupled to a mobile component of the AR display system. The mobile component can be a handheld component, a totem, a head-mounted component housing the display system, a torso-worn component, or a belt pack.

[00024] In one or more embodiments, the pose information includes a position and orientation of the electromagnetic sensor relative to the electromagnetic field emitter in a known coordinate system. The controller can analyze the pose information to determine the position and orientation of the electromagnetic sensor in the known coordinate system.

[00025] In another embodiment, a method for displaying augmented reality includes emitting a known magnetic field in a known coordinate system using an electromagnetic field emitter. The method also includes measuring a parameter related to a flux of the electromagnetic sensor due to the known magnetic field using an electromagnetic sensor. The method further includes measuring a distance in the known coordinate system using a depth sensor. The method also includes determining pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system based at least in part on the distance measured using the depth sensor and the parameter related to the flux measured using the electromagnetic sensor. The method also includes displaying virtual content to a user based at least in part on the pose information of the electromagnetic sensor relative to the electromagnetic field emitter.

[00026] In one or more embodiments, the depth sensor is a passive stereo depth sensor.

[00027] In one or more embodiments, the depth sensor is an active depth sensor. The depth sensor can be a texture projection stereo depth sensor, a structured light projection stereo depth sensor, a time-of-flight depth sensor, a lidar depth sensor, or a modulated emission depth sensor.

[00028] In one or more embodiments, the depth sensor includes a depth camera having a first field of view (FOV). The depth sensor may also include a world capture camera, the world capture camera having a second FOV that at least partially overlaps the first FOV. The depth sensor may also include a photographic camera, the photographic camera having a third FOV that at least partially overlaps the first FOV and the second FOV. The depth camera, the world capture camera, and the photographic camera may each have different first resolutions, second resolutions, and third resolutions. The first resolution of the depth camera may be less than VGA, the second resolution of the world capture camera may be 720p, and the third resolution of the photographic camera may be 2 megapixels.

[00029] In one or more embodiments, the method further comprises capturing a first image, a second image and a third image using the depth camera, the world capture camera and the photo camera, respectively. The method may further comprise segmenting the second image and the third image. The method may further comprise fusion of the second image and the third image after segmenting the second image and the third image to generate a fused image. The step of measuring a distance in a known coordinate system may comprise generating a hypothesis distance by analyzing the first image from the depth camera and generating a distance by analyzing the hypothesis distance and the fused image. The depth camera, the world capture camera, and the photo camera may form a single integrated sensor.

[00030] In one or more embodiments, the method also includes determining pose information of the electromagnetic sensor relative to the electromagnetic field emitter in a known coordinate system, based at least in part on additional information provided by additional localization resources, parameters relating to a distance measured using the depth sensor and a magnetic flux measured using the electromagnetic sensor.

[00031] In one or more embodiments, the additional localization resource may include a WiFi transceiver, an additional electromagnetic emitter, or an additional electromagnetic sensor. The additional localization resource may include a beacon. The method may also include a step of the beacon emitting radiation. The radiation may be infrared radiation, and the beacon may include an infrared LED. The additional localization resource may include a reflector. The method may also include a step of the reflector reflecting the radiation.

[00032] In one or more embodiments, the additional localization resource may include a cellular network transceiver, a radar emitter, a radar detector, a lidar emitter, a lidar detector, a GPS transceiver, a poster having a known detectable pattern, a marker having a known detectable pattern, an inertial measurement unit, or a strain gauge.

[00033] In one or more embodiments, the electromagnetic field emitter is coupled to a mobile component of the AR display system. The mobile component may be a handheld component, a totem, a head-mounted component housing the display system, a torso-worn component, or a belt pack.

[00034] In one or more embodiments, the electromagnetic field emitter is coupled to an object in a known coordinate system such that the electromagnetic field emitter has a known position and a known orientation. The electromagnetic sensor can be coupled to a mobile component of the AR display system. The mobile component can be a handheld component, a totem, a head-mounted component housing the display system, a torso-worn component, or a belt pack.

[00035] In one or more embodiments, the pose information includes a position and orientation of the electromagnetic sensor relative to the electromagnetic field emitter in a known coordinate system. The method may also include analyzing the pose information to determine a position and orientation of the electromagnetic sensor in the known coordinate system.

[00036] In another embodiment, an augmented reality display system includes a handheld component coupled to an electromagnetic field emitter, wherein the electromagnetic field emitter emits a magnetic field. The system also includes a head-mounted component having a display system for displaying virtual content to a user. The head-mounted component is coupled to an electromagnetic sensor, wherein the electromagnetic sensor measures a parameter related to a magnetic flux of the electromagnetic sensor due to the magnetic field, wherein a head pose of the head-mounted component in a known coordinate system is known. The system further includes a depth sensor that measures a distance in the known coordinate system. The system also includes a controller communicatively coupled to the handheld component, the head-mounted component, and the depth sensor. The controller receives a parameter related to the magnetic flux of the electromagnetic sensor from the head-mounted component and a distance from the depth sensor. The controller determines a hand pose of the hand-held component based at least in part on the distance measured by the depth sensor and the parameter related to the magnetic flux measured by the electromagnetic sensor. The system modifies the virtual content displayed to the user based at least in part on the hand pose.

[00037] In one or more embodiments, the depth sensor is a passive stereo depth sensor.

[00038] In one or more embodiments, the depth sensor is an active depth sensor. The depth sensor can be a texture projection stereo depth sensor, a structured light projection stereo depth sensor, a time-of-flight depth sensor, a lidar depth sensor, or a modulated emission depth sensor.

[00039] In one or more embodiments, the depth sensor includes a depth camera having a first field of view (FOV). The AR display system may also include a world capture camera, the world capture camera having a second FOV that at least partially overlaps the first FOV. The AR display system may also include a photographic camera, the photographic camera having a third FOV that at least partially overlaps the first FOV and the second FOV. The depth camera, the world capture camera, and the photographic camera may each have different first resolutions, second resolutions, and third resolutions. The first resolution of the depth camera may be less than VGA, the second resolution of the world capture camera may be 720p, and the third resolution of the photographic camera may be 2 megapixels.

[00040] In one or more embodiments, the depth camera, the world capture camera, and the photo camera are configured to capture a first image, a second image, and a third image, respectively. The controller can be programmed to segment the second image and the third image. The controller can be programmed to segment the second image and the third image, and then fuse the second image and the third image to generate a fused image. Measuring the distance in a known coordinate system can include generating a hypothesis distance by analyzing the first image from the depth camera, and generating the distance by analyzing the hypothesis distance and the fused image. The depth camera, the world capture camera, and the photo camera can form a single integrated sensor.

[00041] In one or more embodiments, the AR display system also includes an additional localization resource for providing additional information. The controller determines a hand pose of the handheld component based at least in part on parameters related to the additional information provided by the additional localization resource, the distance measured by the depth sensor, and the magnetic flux measured by the electromagnetic sensor.

[00042] In one or more embodiments, the additional localization resource can include a WiFi transceiver, an additional electromagnetic emitter, or an additional electromagnetic sensor. The additional localization resource can include a beacon. The beacon can emit radiation. The radiation can be infrared radiation, and the beacon can include an infrared LED. The additional localization resource can include a reflector. The reflector can reflect the radiation.

[00043] In one or more embodiments, the additional localization resource may include a cellular network transceiver, a radar emitter, a radar detector, a lidar emitter, a lidar detector, a GPS transceiver, a poster having a known detectable pattern, a marker having a known detectable pattern, an inertial measurement unit, or a strain gauge.

[00044] In one or more embodiments, the electromagnetic handheld component is a totem. The hand pose information can include a position and orientation of the handheld component in a known coordinate system.

[00045] Additional and further objects, features, and advantages of the present invention are set forth in the detailed description, drawings, and claims.

[00046] The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale, and that elements of similar structures or functions are represented by like reference numerals throughout the drawings. In order to better appreciate how the various embodiments of the present invention achieve the above-recited and other advantages and objects, a more detailed description of the inventions briefly described above will now be provided by reference to specific embodiments of the invention, which specific embodiments are illustrated in the accompanying drawings. Understanding that these drawings illustrate only typical embodiments of the present invention and are therefore not to be considered limiting of the scope of the present invention, the present invention will be described and illustrated with additional features and details by using the accompanying drawings.
[00047] FIG. 1 illustrates a plan view of an AR scene displayed to a user of an AR system according to one embodiment.
[00048] FIGS. 2A to 2D illustrate various embodiments of wearable AR devices.
[00049] FIG. 3 illustrates an exemplary embodiment of a wearable AR device interacting with one or more cloud servers of an AR system.
[00050] Figure 4 illustrates an exemplary embodiment of an electromagnetic tracking system.
[00051] FIG. 5 illustrates an exemplary method for determining the position and orientation of sensors, according to one exemplary embodiment.
[00052] FIG. 6 illustrates an exemplary embodiment of an AR system having an electromagnetic tracking system.
[00053] Figure 7 illustrates an exemplary method for delivering virtual content to a user based on a detected head pose.
[00054] FIG. 8 illustrates a schematic diagram of various components of an AR system according to one embodiment, having an electromagnetic transmitter and an electromagnetic sensor.
[00055] FIGS. 9A through 9F illustrate various embodiments of control and quick release modules.
[00056] Figure 10 illustrates a simplified embodiment of a wearable AR device.
[00057] FIGS. 11A and 11B illustrate various embodiments of the arrangement of electromagnetic sensors on head-mounted AR systems.
[00058] Figures 12a through 12e illustrate various embodiments of ferrite cubes to be coupled to electromagnetic sensors.
[00059] Figures 13a through 13c illustrate various embodiments of data processors for electromagnetic sensors.
[00060] Figure 14 illustrates an exemplary method using an electromagnetic tracking system to detect head and hand poses.
[00061] Figure 15 illustrates another exemplary method using an electromagnetic tracking system to detect head and hand poses.
[00062] FIG. 16A illustrates a schematic diagram of various components of an AR system according to another embodiment, having a depth sensor, an electromagnetic transmitter, and an electromagnetic sensor.
[00063] FIG. 16b illustrates a schematic diagram of various components and various fields of view of an AR system according to another embodiment, having a depth sensor, an electromagnetic transmitter, and an electromagnetic sensor.

[00064] Referring to FIGS. 2a-2d , several general configuration options are illustrated. In the sections of the detailed description that follow the discussion of FIGS. 2a-2d , various systems, subsystems, and components are presented to address the goals of providing a high-quality, comfortably perceived display system for human VR and/or AR.

[00065] As illustrated in FIG. 2A, a user (60) of the AR system is depicted wearing a head mounted component (58) featuring a frame (64) structure coupled to a display system (62) positioned in front of the user's eyes. A speaker (66) is coupled to the frame (64) in the depicted configuration and positioned adjacent to the user's ear canal (in one embodiment, another speaker, not depicted, is positioned adjacent the other ear canal of the user to provide stereo/shapeable sound control). The display (62) is operatively coupled (68) to a local processing and data module (70) which may be mounted in a variety of configurations, such as fixedly attached to a frame (64), fixedly attached to a helmet or hat (80) as shown in the embodiment of FIG. 2b, built into headphones, removably attached to the torso (82) of the user (60) in a backpack-style configuration as shown in the embodiment of FIG. 2c, or removably attached to the hips (84) of the user (60) in a belt-coupled style configuration as shown in the embodiment of FIG. 2d.

[00066] A local processing and data module (70) may include a power-efficient processor or controller as well as digital memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data, which data is (a) captured from sensors operatively coupled to the frame (64), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros, and/or (b) obtained and/or processed using a remote processing module (72) and/or remote data storage (74), possibly for passage to a display (62) following such processing or retrieval. The local processing and data module (70) may be operatively coupled (76, 78) to a remote processing module (72) and a remote data store (74), for example, via wired or wireless communication links, such that these remote modules (72, 74) are operatively coupled to one another and available as resources to the local processing and data module (70).

[00067] In one embodiment, the remote processing module (72) may include one or more relatively powerful processors or controllers configured to analyze and process data and/or image information. In one embodiment, the remote data storage (74) may include a relatively large-scale digital data storage facility, which may be available via the Internet or other networking configuration of a “cloud” resource configuration. In one embodiment, all data is stored and all computations are performed in the local processing and data module, allowing for completely autonomous use from any remote modules.

[00068] Referring now to FIG. 3 , a schematic diagram illustrates coordination between cloud computing assets (46) and local processing assets, which may reside, for example, on a head mounted component (58) coupled to the user's head (120) and a local processing and data module (70) coupled to the user's belt (308) (and thus, the component (70) may also be referred to as a "belt pack (70)"), as shown in FIG. 3 . In one embodiment, cloud (46) assets, such as one or more server systems (110), are operatively coupled (115) directly (40, 42) to one or both of the local computing assets, such as processor and memory configurations, coupled to the user's head (120) and belt (308), as described above, for example, via wired or wireless networking (wireless being preferred for mobility, wired being preferred for certain high-bandwidth or high-data-volume transfers that may be required). These computing assets local to the user may also be operatively coupled to each other via wired and/or wireless connection configurations (44), such as the wired coupling (68) discussed below with reference to FIG. 8 . In one embodiment, to maintain the low-inertia and small size subsystem mounted on the user's head (120), the primary transmission between the user and the cloud (46) may be via a link between the belt-mounted subsystem (308) and the cloud, with the head-mounted subsystem (120) being data-tethered to the belt-based subsystem primarily using a wireless connection, such as an ultra-wideband ("UWB") connection, such as is currently used in personal computing peripheral connectivity applications.

[00069] Using efficient local and remote processing coordination, and a suitable display device for the user, such as the user interface or user display system (62) illustrated in FIG. 2a , or variations thereof, aspects of the world relative to the user's current real or virtual location can be transmitted or "delivered" to the user and updated in an efficient manner. In other words, the map of the world can be continually updated in a storage location that may partially reside on the user's AR system and partially reside on cloud resources. The map (also referred to as a "passable world model") can be a large database containing raster imagery, 3-D and 2-D points, parametric information about the real world, and other information. As more and more AR users continually capture information about their real-world environment (e.g., via cameras, sensors, IMUs, etc.), the map becomes increasingly accurate and complete.

[00070] In a configuration such as the one described above, where there is a world model that can reside on and be distributed from cloud computing resources, such a world can be "passable" to one or more users in a relatively low bandwidth form, which is preferable to trying to pass around real-time video data, etc. The augmented experience of a person standing near a statue (i.e., as shown in FIG. 1 ) can be informed by the cloud-based world model, a subset of which can be passed to the person and their local display device to complete the view. A person sitting at a remote display device, which can be as simple as a personal computer sitting on a desk, can efficiently download the same section of information from the cloud and render it on their display. In effect, a person physically present in a park near the statue can call a friend who is far away to take a walk in the park, who joins them via virtual and augmented reality. The system needs to know where the roads are, where the trees are, where the statues are, but once that information is in the cloud, the joining friend can download aspects of the scenario from the cloud and then start walking around in augmented reality, limited to people who are actually in the park.

[00071] 3-D points can be captured from the environment, the poses (i.e., vector and/or origin position information with respect to the world) of the cameras capturing those images or points can be determined, and these points or images can be "tagged" or associated with this pose information. The points captured by the second camera can then be utilized to determine the pose of the second camera. In other words, the second camera can be orientated and/or localized based on comparisons with the tagged images from the first camera. This knowledge can then be utilized to extract textures, create maps, and generate a virtual copy of the real world (so that there are two cameras registered around it).

[00072] At this base level, in some embodiments, the human-worn system can be utilized to capture both 2-D images and 3-D points that generated points, and these points and images can be transmitted to cloud storage and processing resources. These can also be cached locally (i.e., caching the tagged images) along with the embedded pose information; and the cloud can then provision (i.e., in an available cache) the tagged 2-D images (i.e., tagged with the 3-D pose) along with the 3-D points. If the user is observing something dynamic, the user can also transmit additional information about the motion to the cloud (e.g., if viewing another person's face, the user can take a texture map of the face and push it up at an optimized frequency, even if the rest of the surrounding world is essentially static). More information on object recognizers and passable world models can be found in U.S. patent application Ser. No. 14/205,126, entitled "System and method for augmented and virtual reality," which is herein incorporated by reference in its entirety along with the following additional references related to augmented and virtual reality systems, such as those developed by Magic Leap, Inc. of Fort Lauderdale, Florida: U.S. patent application Ser. No. 14/641,376; U.S. patent application Ser. No. 14/555,585; U.S. patent application Ser. No. 14/212,961; U.S. patent application Ser. No. 14/690,401; U.S. patent application Ser. No. 13/663,466; and U.S. patent application Ser. No. 13/684,489.

[00073] In order to capture points that can be used to generate a "passable world model", it is helpful to know precisely the user's position, pose, and orientation relative to the world. More specifically, the user's position should be localized to a granular degree, since it may be important to know the user's head pose as well as his or her hand pose (e.g., if the user is grasping a handheld component, making a gesture, etc.). In one or more embodiments, GPS and other localization information may be utilized as inputs to this processing. Highly accurate localization of the user's head, totems, hand gestures, haptic devices, etc. is important for displaying appropriate virtual content to the user.

[00074] One approach to achieving high-precision localization may involve the use of electromagnetic fields coupled with electromagnetic sensors strategically placed on the user's AR headset, belt pack, and/or other assistive devices (e.g., totems, haptic devices, gaming devices, etc.). Electromagnetic tracking systems typically include at least an electromagnetic field emitter and at least one electromagnetic field sensor. The sensors can measure electromagnetic fields having a known distribution. Based on these measurements, the position and orientation of the field sensor relative to the emitter is determined.

[00075] Referring now to FIG. 4 , an exemplary system diagram of an electromagnetic tracking system (such as those developed by organizations such as Polhemus(RTM), Inc. of Colchester, Vermont, a Biosense(RTM) division of Johnson & Johnson Corporation, and manufactured by Sixense (RTM) Entertainment, Inc. of Los Gatos, California, and other tracking companies) is illustrated. In one or more embodiments, the electromagnetic tracking system includes an electromagnetic field emitter (402) configured to emit a known magnetic field. As illustrated in FIG. 4 , the electromagnetic field emitter can be coupled to a power supply (e.g., current, batteries, etc.) for providing power to the emitter (402).

[00076] In one or more embodiments, the electromagnetic field emitter (402) includes several coils that generate magnetic fields (e.g., at least three coils positioned perpendicular to each other to generate magnetic fields in the x, y, and z directions). The magnetic fields are used to establish a coordinate space. This allows the system to map the positions of the sensors with respect to a known magnetic field, and helps determine the position and/or orientation of the sensors. In one or more embodiments, the electromagnetic sensors (404a, 404b, etc.) may be attached to one or more real-world objects. The electromagnetic sensors (404) may include smaller coils in which a current may be induced via the emitted electromagnetic field. Typically, the "sensor" components (404) may include small coils or loops, such as a set of three differently-oriented (i.e., oriented orthogonally with respect to one another, for example) coils coupled together within a small structure, such as a cube or other container, such that these small coils or loops are positioned/oriented to capture incoming magnetic flux from a magnetic field emitted by the emitter (402), and by comparing the currents induced through these coils, and by knowing the relative positioning and orientation of the coils with respect to one another, the relative position and orientation of the sensor with respect to the emitter can be calculated.

[00077] One or more parameters of the behavior of "inertial measurement unit" (IMU) components and coils operably coupled to the electromagnetic tracking sensors may be measured to detect a position and/or orientation of the sensor (and the object to which the sensor is attached) with respect to a coordinate system to which the electromagnetic emitter is coupled. Of course, such a coordinate system may be transformed into a world coordinate system to determine a position or pose of the electromagnetic emitter in the real world. In one or more embodiments, multiple sensors may be used with respect to the electromagnetic emitter to detect the position and orientation of each of the sensors within the coordinate space.

[00078] It should be recognized that in some embodiments, the head pose may already be known based on sensors on the head-mounted component of the AR system, and the SLAM analysis may be performed based on image data and sensor data captured via the head-mounted AR system. However, it may be important to know the position of the user's hand (e.g., a handheld component such as a totem) with respect to the known head pose. That is, it may be important to know the hand pose with respect to the head pose. Once the relationship between the head (assuming the sensors are positioned on the head-mounted component) and the hand is known, the position of the hand with respect to the world (e.g., world coordinates) can be easily computed.

[00079] The electromagnetic tracking system can provide positions in three directions (i.e., X, Y and Z directions) and additionally two or three orientation angles. In one or more embodiments, measurements of the IMU can be compared to measurements of the coils to determine the position and orientation of the sensors. In one or more embodiments, both electromagnetic (EM) data and IMU data can be combined with various other sources of data, such as cameras, depth sensors, and other sensors, to determine the position and orientation. This information can be transmitted to the controller (406) (e.g., wirelessly, Bluetooth, etc.). In one or more embodiments, the pose (or position and orientation) can be reported at a relatively high refresh rate compared to conventional systems. Typically, the electromagnetic emitter is coupled to a relatively stable, large object, such as a table, operating table, wall, or ceiling, and one or more sensors are coupled to smaller objects, such as medical devices, handheld gaming components, etc. Alternatively, various features of the electromagnetic tracking system can be employed to create a configuration in which changes in position and/or orientation, or deltas, between two objects moving in space with respect to a more stable global coordinate system can be tracked, as described below with reference to FIG. 6 ; in other words, a variation of the electromagnetic tracking system can be utilized to track position and orientation deltas between a head-mounted component and a handheld component, while the head pose with respect to the global coordinate system (i.e., the room environment localized to the user) is determined in another manner, such as by SLAM ("simultaneous localization and mapping") techniques using outward-capturing cameras that can be coupled to the head-mounted component of the system, as illustrated in FIG. 6 .

[00080] The controller (406) can control the electromagnetic field generator (402) and also capture data from various electromagnetic sensors (404). It should be appreciated that the various components of the system can be coupled to one another via any electro-mechanical or wireless/Bluetooth means. The controller (406) can also include data regarding a known magnetic field, and a coordinate space relative to the magnetic field. This information can then be used to detect the position and orientation of the sensors relative to the coordinate space corresponding to the known electromagnetic field.

[00081] One advantage of electromagnetic tracking systems is that they can produce very accurate tracking results with minimal latency and high resolution. Additionally, electromagnetic tracking systems do not necessarily have to rely on optical trackers, and sensors/objects that are not in the user's line of sight can be easily tracked.

[00082] It should be recognized that the strength of the electromagnetic field (v) drops as a cubic function of the distance (r) from the coil transmitter (e.g., the electromagnetic field emitter (402)). Therefore, an algorithm based on the distance from the electromagnetic field emitter may be desired. The controller (406) may be configured with such algorithms to determine the position and orientation of the sensor/object at various distances from the electromagnetic field emitter. Given the rapid decrease in the strength of the electromagnetic field with moving further away from the electromagnetic emitter, the best results with respect to accuracy, efficiency and low latency may be achieved at closer distances. In typical electromagnetic tracking systems, the electromagnetic field emitter is powered by current (e.g., a plug-in power supply) and has sensors positioned within a 20 ft radius from the electromagnetic field emitter. A shorter radius between the sensors and the electromagnetic field emitter may be more desirable for many applications, including AR applications.

[00083] Referring now to FIG. 5 , an exemplary flow diagram illustrating the functionality of a typical electromagnetic tracking system is briefly described. At 502, a known electromagnetic field is emitted. In one or more embodiments, the magnetic field emitter may generate magnetic fields, and each coil may generate an electric field in a direction (e.g., x, y, or z). The magnetic fields may be generated in any waveform. In one or more embodiments, each of the axes may oscillate at a slightly different frequency. At 504, a coordinate space corresponding to the electromagnetic field may be determined. For example, the controller (406) of FIG. 4 may automatically determine the coordinate space around the emitter based on the electromagnetic field. At 506, the behavior of coils of sensors (which may be attached to a known object) may be detected. For example, the current induced in the coils may be calculated. In other embodiments, the rotation of the coils, or any other quantifiable behavior, may be tracked and measured. At 508, this behavior can be used to detect positions and orientations of the sensor(s) and/or known objects. For example, the controller (406) can consult a mapping table that correlates the behavior of the coils of the sensors to various positions or orientations. Based on these calculations, a position in coordinate space, along with the orientation of the sensors, can be determined. In some embodiments, pose/position information can be determined from the sensors. In other embodiments, the sensors can communicate data detected by the sensors to the controller, and the controller can consult the mapping table for determined pose information relative to a known magnetic field (e.g., coordinates for a handheld component).

[00084] In the context of AR systems, one or more components of the electromagnetic tracking system may need to be modified to enable accurate tracking of the mobile components. As described above, tracking the user's head pose and orientation is important in many AR applications. Accurate determination of the user's head pose and orientation allows the AR system to display appropriate virtual content to the user. For example, a virtual scene may include a monster hiding behind a real building. Depending on the pose and orientation of the user's head relative to the building, the view of the virtual monster may need to be modified so that a realistic AR experience is provided. That is, the position and/or orientation of a totem, haptic device, or some other means for interacting with the virtual content may be important to enabling the AR user to interact with the AR system. For example, in many gaming applications, the AR system must detect the position and orientation of a real object relative to the virtual content. That is, when displaying a virtual interface, the positions of the totem, the user's hand, the haptic device, or any other real-world objects configured for interaction with the AR system must be known with respect to the displayed virtual interface, in order for the system to understand commands, etc. Conventional localization methods, including optical tracking and other methods, typically suffer from high latency and low resolution issues, which make rendering virtual content difficult in many augmented reality applications.

[00085] In one or more embodiments, the electromagnetic tracking system discussed in connection with FIGS. 4 and 5 may be adapted to an AR system to detect the position and orientation of one or more objects with respect to an emitted electromagnetic field. Conventional electromagnetic systems tend to have large and bulky electromagnetic emitters (e.g., 402 of FIG. 4), which is problematic in AR devices. However, smaller electromagnetic emitters (e.g., in the millimeter range) may be used to emit a known electromagnetic field in the context of an AR system.

[00086] Referring now to FIG. 6 , as illustrated, an electromagnetic tracking system may be integrated with an AR system, wherein an electromagnetic field emitter (602) is integrated as part of a handheld controller (606). In one or more embodiments, the handheld controller may be a totem to be used in a gaming scenario. In other embodiments, the handheld controller may be a haptic device. In still other embodiments, the electromagnetic field emitter may be simply integrated as part of a belt pack (70). The handheld controller (606) may include a battery (610) or other power supply that powers the electromagnetic field emitter (602). It should be appreciated that the electromagnetic field emitter (602) may also include or be coupled to an IMU component (650) configured to assist in determining positioning and/or orientation of the electromagnetic field emitter (602) relative to other components. This may be particularly important in cases where both the electromagnetic field emitter (602) and the sensors (604) are mobile. As illustrated in the embodiment of FIG. 6, placing the electromagnetic field emitter (602) on the handheld controller rather than on the belt pack ensures that the electromagnetic field emitter does not compete for resources on the belt pack, but rather uses its own battery source on the handheld controller (606).

[00087] In one or more embodiments, the electromagnetic sensors (604) may be positioned at one or more locations on the user's headset (58), along with other sensing devices, such as one or more IMUs or additional flux capturing coils (608). For example, as illustrated in FIG. 6 , the sensors (604, 608) may be positioned on either side of the headset (58). Since these sensors (604, 608) are engineered to be somewhat smaller (and thus may be less sensitive in some cases), having multiple sensors may improve efficiency and precision.

[00088] In one or more embodiments, one or more sensors may also be positioned on the belt pack (620) or on any other part of the user's body. The sensors (604, 608) may communicate wirelessly or via Bluetooth with a computing device (607) (e.g., a controller) that determines the pose and orientation of the sensors (604, 608) (and the AR headset (58) to which the sensors (604, 608) are attached) in relation to the known magnetic field emitted by the electromagnetic field emitter (602). In one or more embodiments, the computing device (607) may reside on the belt pack (620). In other embodiments, the computing device (607) may reside on the headset (58) itself, or even on the handheld controller (606). The computing device (607) can receive measurements from the sensors (604, 608) and determine the position and orientation of the sensors (604, 608) in relation to a known electromagnetic field emitted by the electromagnetic field emitter (602).

[00089] In one or more embodiments, the computing device (607) may ultimately include a mapping database (632) (e.g., a passable world model, coordinate space, etc.) to detect poses and determine coordinates of real and virtual objects, and may even be connected to the cloud resources (630) and the passable world model. The mapping database (632) may be consulted to determine position coordinates of the sensors (604, 608). The mapping database (632) may reside on the belt pack (620) in some embodiments. In the embodiment illustrated in FIG. 6 , the mapping database (632) resides on the cloud resource (630). The computing device (607) communicates wirelessly with the cloud resource (630). The determined pose information along with the points and images collected by the AR system may then be communicated to the cloud resource (630) and then added to the passable world model (634).

[00090] As described above, conventional electromagnetic emitters may be too bulky for AR devices. Therefore, the electromagnetic field emitter may be engineered to be compact using smaller coils compared to conventional systems. However, considering that the strength of the electromagnetic field decreases as a cubic function of the distance from the electromagnetic field emitter, a shorter radius (e.g., about 3-3.5 ft) between the electromagnetic sensors (604) and the electromagnetic field emitter (602) may reduce power consumption compared to conventional systems such as those described above in FIG. 4.

[00091] In one or more embodiments, this aspect may be utilized to extend the life of the battery (610) that may power the controller (606) and the electromagnetic field emitter (602). That is, in other embodiments, this aspect may be utilized to reduce the size of the coils that generate the magnetic field in the electromagnetic field emitter (602). However, to obtain the same strength of the magnetic field, the power may need to be increased. This enables the compact electromagnetic field emitter unit (602) to fit compactly into the handheld controller (606).

[00092] Several other changes can be made when using an electromagnetic tracking system in AR devices. While this pose reporting rate is somewhat good, AR systems may require a much more efficient pose reporting rate. To this end, IMU-based pose tracking can be used in the sensors. Crucially, the IMUs should be kept as stable as possible to increase the efficiency of the pose detection process. The IMUs can be engineered to remain stable for up to 50-100 milliseconds. It should be recognized that some embodiments may utilize an external pose estimator module that may enable pose updates to be reported at a rate of 10-20 Hz (i.e., IMUs may drift over time). By keeping the IMUs stable at a reasonable rate, the rate of pose updates can be dramatically reduced to 10-20 Hz (as compared to the higher frequencies in conventional systems).

[00093] Another way to conserve power in the AR system would be if the electromagnetic tracking system could run at a 10% duty cycle (e.g., only pinging for ground truth every 100 milliseconds). This would mean that the electromagnetic tracking system would wake up every 10 milliseconds out of every 100 milliseconds to generate a pose estimate. This directly translates into power savings, which could ultimately impact the size, battery life, and cost of the AR device.

[00094] In one or more embodiments, this reduction in duty cycle may be strategically exploited by providing two (not shown) handheld controllers rather than just one. For example, a user may be playing a game that requires two totems, etc. That is, in a multi-player game, two users may have their own totems/handheld controllers to play the game. If two controllers are used rather than one (e.g., symmetric controllers for each hand), the controllers may operate with offset duty cycles. The same concept may also be applied to controllers utilized by two different users playing a multi-player game, for example.

[00095] Referring now to FIG. 7 , an exemplary flow diagram illustrating an electromagnetic tracking system in the context of AR devices is described. At 702, a handheld controller emits a magnetic field. At 704, electromagnetic sensors (located in the headset, belt pack, etc.) detect the magnetic field. At 706, the position and orientation of the headset/belt are determined based on the behavior of the coils/IMUs in the sensors. At 708, pose information is communicated to a computing device (e.g., in the belt pack or headset). At 710, optionally, a mapping database (e.g., a passable world model) may be consulted to correlate real world coordinates with virtual world coordinates. At 712, virtual content may be delivered to a user in the AR headset. It should be appreciated that the flow diagram described above is for illustrative purposes only and should not be construed as limiting.

[00096] Advantageously, using an electromagnetic tracking system similar to that outlined in FIG. 6 enables pose tracking (e.g., head position and orientation, positions and orientations of other controllers and totems). This allows the AR system to project virtual content with higher accuracy and very low latency compared to optical tracking techniques.

[00097] Referring to FIG. 8, a system configuration featuring a number of sensing components is illustrated. Referring to FIGS. 9a-9f , a physical multicore lead is illustrated here, which also features a control and quick release module (86) described below, a local processing and data module (70), such as a head-mounted wearable component (58) operably coupled (68) to a belt pack. The local processing and data module (70) is operably coupled (100) to a handheld component (606), here by a wireless connection, such as low power Bluetooth; the handheld component (606) may also be operably coupled (94) directly to the head-mounted wearable component (58), such as by a wireless connection, such as low power Bluetooth. In general, where IMU data is transmitted to coordinate pose detection of various components, high-frequency coupling, such as in the range of hundreds or thousands of cycles per second or more, may be desirable; for example, tens of cycles per second may be adequate for electromagnetic localization sensing, for example, with sensor (604) and transmitter (602) pairings. Also depicted is a global coordinate system (10) representing stationary objects in the real world around the user, such as a wall (8). Cloud resources (46) may also be operatively coupled (42, 40, 88, 90) to the local processing and data module (70), the head-mounted wearable component (58), resources that may be coupled to the wall (8), or other items that are fixed relative to the global coordinate system (10), respectively. Resources coupled to a wall (8) or having known positions and/or orientations with respect to the global coordinate system (10) may include a WiFi transceiver (114), an electromagnetic emitter (602) and/or receiver (604), a beacon or reflector (112) configured to emit or reflect a given type of radiation, such as an infrared LED beacon, a cellular network transceiver (110), a radar emitter or detector (108), a lidar emitter or detector (106), a GPS transceiver (118), a poster or marker having a known detectable pattern (122), and a camera (124). The head-mounted wearable component (58) features similar components as illustrated, in addition to illumination emitters (130) configured to assist the camera (124) detectors, such as infrared emitters (130) for an infrared camera (124); Additionally, the head-mounted wearable component (58) is characterized by having one or more strain gauges (116) on the head-mounted wearable component (58), the one or more strain gauges (116) being fixedly coupled to the frame or mechanical platform of the head-mounted wearable component (58) and configured to determine a deflection of such platform between components such as the electromagnetic receiver sensors (604) or the display elements (62), wherein it may be valuable to understand whether a bend in the platform has occurred, for example, in a thin portion of the platform, such as above the nose on the glasses-like platform as illustrated in FIG. 8 . The head-mounted wearable component (58) is also characterized by a processor (128) and one or more IMUs (102). Each of the components is preferably operably coupled to the processor (128). A handheld component (606) and a local processing and data module (70) featuring similar components are illustrated. As illustrated in FIG. 8, due to the numerous sensing and connectivity means, such a system would likely be heavy, power hungry, large, and relatively expensive. However, for illustrative purposes, such a system could be utilized to provide a very high level of connectivity, system component integration, and position/orientation tracking. For example, with such a configuration, the various main mobile components (58, 70, 606) could be localized with respect to their positions relative to a global coordinate system using WiFi, GPS, or cellular signal triangulation; beacons, electromagnetic tracking (as described above), radar, and lidar systems could still provide additional position and/or orientation information and feedback. Markers and cameras could also be utilized to provide additional information regarding relative and absolute position and orientation. For example, various camera components (124), such as those illustrated as coupled to the head-mounted wearable component (58), may be utilized to capture data that may be utilized in simultaneous localization and mapping protocols, or “SLAM,” to determine where the head-mounted wearable component (58) is and how it is oriented relative to other components.

[00098] Referring to FIGS. 9A-9F , various aspects of the control and quick release module (86) are illustrated. Referring to FIG. 9A , the two outer housing components are coupled together using a magnetic coupling configuration that may be enhanced with mechanical latching. Buttons (136) for operation of the associated system may be included. FIG. 9B illustrates a partial cutaway view showing the buttons (136) and the lower top printed circuit board (138). Referring to FIG. 9C , the buttons (136) and the lower top printed circuit board (138) are removed, revealing the female contact pin array (140). Referring to FIG. 9D , an opposing portion of the housing (134) is removed, revealing the lower printed circuit board (142). As shown in FIG. 9E , the lower printed circuit board (142) is removed, revealing the male contact pin array (144). Referring to the cross-sectional view of FIG. 9f, at least one of the male pins or the female pins is configured to be spring-loaded such that they can be pressed along the longitudinal axis of the respective pin; the pins may be referred to as "pogo pins" and typically comprise a highly conductive material, such as copper or gold. When assembled, the illustrated configuration mates the 46 male pins with the female pins, and the entire assembly can be quick-released in half by manually pulling them apart and overcoming the magnetic interface (146) load that can be developed using north and south magnets oriented around the perimeter of the pin arrays (140, 144). In one embodiment, an approximately 2 kg load from compressing the 46 pogo pins is reacted with a closing holding force of approximately 4 kg. The pins within the array can be separated by about 1.3 mm and can be operably coupled to various types of conductive lines, such as twisted pairs or other combinations to support USB 3.0, HDMI 2.0, I2S signals, GPIO and MIPI configurations, and, in one embodiment, high current analog lines and grounds configured for up to about 4 amps/5 volts.

[00099] Referring to FIG. 10, it would be helpful to have a minimized component/feature set that allows for a relatively slim head mounted component, such as the head mounted component (58) featured in FIG. 10, and minimizes the weight and volume of the various components. Accordingly, various permutations and combinations of the various components illustrated in FIG. 8 may be utilized.

[000100] Referring to FIG. 11a, an electromagnetic sensing coil assembly (604) is illustrated (i.e., three individual coils coupled to the housing) coupled to the head mounted component (58); this configuration adds additional geometry to the overall assembly, which may be undesirable. Referring to FIG. 11b, rather than housing the coils in a box or single housing as in the configuration of FIG. 11a, the individual coils may be integrated into various structures of the head mounted component (58), as illustrated in FIG. 11b. For example, the x-axis coil (148) may be positioned in one portion of the head mounted component (58), such as in the center of the frame. Similarly, the y-axis coil (150) may be positioned in another portion of the head mounted component (58), such as on either bottom side of the frame. Similarly, the z-axis coil (152) may be placed on another portion of the head mounted component (58), e.g., on either top side of the frame.

[000101] FIGS. 12a-12e illustrate various configurations for characterizing a ferrite core coupled to an electromagnetic sensor to increase field sensitivity. Referring to FIG. 12a, the ferrite core may be a solid cube (1202). While the solid cube (1202) may be the most effective in increasing field sensitivity, it may also be the heaviest as compared to the remaining configurations illustrated in FIGS. 12b-12e. Referring to FIG. 12b, a plurality of ferrite disks (1204) may be coupled to the electromagnetic sensor. Similarly, referring to FIG. 12c, a solid cube having a uniaxial air core (1206) may be coupled to the electromagnetic sensor. As illustrated in FIG. 12c, an open space (i.e., air core) may be formed in the solid cube along one axis. This may reduce the weight of the cube while still providing the required field sensitivity. In another embodiment, referring to FIG. 12d, a solid cube having a three-axis air core (1208) may be coupled to the electromagnetic sensor. In this configuration, the solid cube is hollow along all three axes, significantly reducing the weight of the cube. Referring to FIG. 12e, a ferrite rod having a plastic housing (1210) may also be coupled to the electromagnetic sensor. It should be appreciated that the embodiments of FIGS. 12b-12e may be utilized to provide lighter weight and mass savings than the solid core configuration of FIG. 12a, as discussed above.

[000102] Referring to FIGS. 13A-13C , time division multiplexing (“TDM”) may also be utilized to reduce mass. For example, referring to FIG. 13A , a conventional local data processing configuration for a 3-coil electromagnetic receiver sensor is illustrated, wherein analog currents come from each of the X, Y, and Z coils (1302, 1304, 1306), go to a pre-amplifier (1308), through a bandpass filter (1310), through a PA (1312), through an analog-to-digital converter (1314), and ultimately to a digital signal processor (1316). Referring to the transmitter configuration of FIG. 13B and the receiver configuration of FIG. 13C , time division multiplexing may be utilized to share hardware so that each coil sensor chain does not require its own amplifiers, etc. This can be accomplished via the TDM switch (1320) illustrated in FIG. 13b, which allows for processing of signals from and to multiple transmitters and receivers using the same set of hardware components (amplifiers, etc.). In addition to saving hardware overhead by multiplexing and eliminating sensor housings, signal-to-noise ratios can be increased by having more than one set of electromagnetic sensors, each set being relatively small compared to a single larger coil set; furthermore, the low-side frequency limitations typically required to have multiple sensing coils in close proximity can be improved to allow for improved bandwidth requirements. Additionally, there can be a tradeoff with multiplexing in that multiplexing typically spreads out the reception of radio frequency signals over time, which typically results in dirtier signals; thus, larger coil diameters may be required in multiplexed systems. For example, while a multiplexed system may require a cubic coil sensor box with a lateral dimension of 9 mm, a non-multiplexed system may require only a cubic coil box with a lateral dimension of 7 mm for similar performance; thus, there may be trade-offs in minimizing geometry and mass.

[000103] In other embodiments where a particular system component, such as a head mounted component (58), features two or more sets of electromagnetic coil sensors, the system can be configured to selectively utilize the sensor and emitter pairings that are closest to each other to optimize the performance of the system.

[000104] Referring to FIG. 14, in one embodiment, after the user powers on his or her wearable computing system (160), the head mounted component assembly may capture (162) a combination of IMU and camera data (e.g., camera data used for SLAM analysis, e.g., on a belt pack processor where greater raw processing horsepower may be present) to determine and update head pose (i.e., position and orientation) relative to a real-world global coordinate system. The user may also activate (164) the handheld component to play an augmented reality game, for example, wherein the handheld component may include an electromagnetic transmitter operably coupled to one or both of the belt pack and head mounted component (166). One or more sets of electromagnetic field coil receivers (i.e., a set being three differently-oriented individual coils) are coupled to the head-mounted component to capture flux from the transmitter, which flux can be utilized (168) to determine a position or orientation difference (or "delta") between the head-mounted component and the handheld component. The combination of the head-mounted component assisting in determining pose with respect to a global coordinate system and the handheld assisting in determining the relative position and orientation of the handheld with respect to the head-mounted component allows the system to generally determine where each component is with respect to the global coordinate system, and thus the user's head pose and handheld pose can be preferably tracked (170) with relatively low latency for presentation of augmented reality image features and interactions that utilize movements and rotations of the handheld component.

[000105] Referring to FIG. 15, an embodiment is illustrated somewhat similarly to that of FIG. 14, except that the system has more sensing devices and configurations available to assist in determining poses of both the head mounted component and the handheld component (172 and 176, 178) such that the user's head pose and handheld pose can be tracked (180) for presentation of augmented reality image features and interactions using movements and rotations of the handheld component, preferably with relatively low latency.

[000106] Specifically, after the user powers on his or her wearable computing system (160), the head mounted component captures a combination of IMU and camera data for SLAM analysis to determine and update head pose with respect to a real-world global coordinate system. The system may be further configured (172) to detect the presence of other localization resources in the environment, such as Wi-Fi, cellular, beacons, radar, lidar, GPS, markers, and/or other cameras, which may be connected to various aspects of the global coordinate system or to one or more movable components.

[000107] The user may also activate the handheld component (174), for example, to play an augmented reality game, wherein the handheld component may include an electromagnetic transmitter operably coupled to one or both of the belt pack and the head mounted component (176). Other localization resources may also be similarly utilized. One or more sets of electromagnetic coil receivers (e.g., a set being three differently-oriented individual coils) coupled to the head mounted component may be used to capture magnetic flux from the electromagnetic transmitter. This captured magnetic flux may be utilized (178) to determine a position or orientation difference (or “delta”) between the head mounted component and the handheld component.

[000108] Thus, the user's head pose and handheld pose can be tracked (180) with relatively low latency for interaction with an AR system using movements or rotations of the handheld component and/or for presentation of AR content.

[000109] Referring to FIGS. 16A and 16B , various aspects of a configuration similar to that of FIG. 8 are illustrated. The configuration of FIG. 16A differs from the configuration of FIG. 8 in that, in addition to a depth sensor of the lidar (106) type, the configuration of FIG. 16A features, for illustrative purposes, a generic depth camera or depth sensor (154) (which may be, for example, a stereo triangulation style depth sensor (such as a passive stereo depth sensor, a texture projection stereo depth sensor, or a structured light stereo depth sensor) or a time-of-flight style depth sensor (such as a lidar depth sensor or a modulated emission depth sensor)); Additionally, the configuration of FIG. 16a has an additional forward facing "world" camera (124) (which may be a grayscale camera having a 720p range resolution sensor performance) as well as a relatively high-resolution "photo camera" (156) (which may be, for example, a full color camera having a 2 megapixel or greater resolution sensor performance). FIG. 16b illustrates a partial orthogonal view of the configuration of FIG. 16a for illustrative purposes, as further described below with reference to FIG. 16b.

[000110] Referring back to FIG. 16A and the stereo vs. time-of-flight style depth sensors discussed previously, while each of these depth sensor types can be utilized in the wearable computing solutions disclosed herein, each has various advantages and disadvantages. For example, many depth sensors have challenges with black surfaces and shiny or reflective surfaces. Passive stereo depth sensing is a relatively simple method of triangulating depth with a depth camera or sensor, but can be challenging when a wide field of view (“FOV”) is required and can require relatively significant computing resources; additionally, this sensor type can have challenges with edge detection, which can be important for certain use cases at hand. Passive stereo can have challenges with textureless walls, low-light situations, and repeating patterns. Passive stereo depth sensors are available from manufacturers such as Intel (RTM) and Aquifi (RTM). Stereo with texture projection (also known as "active stereo") is similar to passive stereo, but the texture projector broadcasts projection patterns into the environment, and the more textures being broadcast, the greater the accuracy available for triangulating the depth. Active stereo can also be relatively computationally intensive, presents challenges when large FOVs are required, and can be somewhat suboptimal at edge detection, but it addresses some of the challenges of passive stereo in that it is effective with textureless walls, performs well in low light, and generally does not have the problems that repeating patterns have. Active stereo depth sensors are available from manufacturers such as Intel (RTM) and Aquifi (RTM). Stereo systems with structured light, such as those developed by Primesense, Inc. (RTM) and available under the trade name Kinect (RTM), as well as those available from Mantis Vision, Inc. (RTM), are specialized in that they typically utilize a single camera/projector pairing, with the projector configured to broadcast a pattern of dots that is known a priori. Essentially, the system knows the pattern that is being broadcast, and the system knows that the variable to be determined is depth. Such configurations can be relatively efficient in terms of compute load, and can be challenging in scenarios with wide FOV requirements as well as scenarios with ambient light and patterns broadcast from other nearby devices, but can be quite effective and efficient in many scenarios. Modulated time-of-flight type depth sensors, such as those available from PMD Technologies (RTM), A.G. and SoftKinetic Inc. (RTM), wherein the emitter can be configured to transmit a wave of amplitude-modulated light, such as a sine wave; In some configurations, a camera component, which may be positioned adjacent or even overlapping, receives a returning signal on each of the pixels of the camera component, and depth mapping can be determined/computed. Such configurations may have relatively compact geometry, high accuracy, and low compute load, but may present challenges with respect to image resolution (e.g., at the edges of objects), multi-path errors (e.g., when the sensor is aimed at a reflective or shiny corner, and the detector ends up receiving more than one return path, resulting in some depth detection aliasing). Direct time of flight sensors, which may also be referred to as lidar as mentioned above, are available from suppliers such as LuminAR(RTM) and Advanced Scientific Concepts, Inc.(RTM). With these time-of-flight configurations, a pulse of light (e.g., a picosecond, nanosecond or femtosecond long light pulse) is typically transmitted such that it bathes the world around it in this light ping; each pixel on the camera sensor then waits for that pulse to return, and given the speed of light, the distance to each pixel can be calculated. These configurations can have many of the advantages of modulated time-of-flight sensor configurations (no baseline, relatively wide FOV, high accuracy, relatively low compute load, etc.) as well as relatively high frame rates, such as tens of thousands of hertz. They can also be relatively expensive, have relatively low resolution, are sensitive to bright light, and are susceptible to multi-path errors; and they can also be relatively large and heavy.

[000111] Referring to FIG. 16b, a partial plan view is shown for illustrative purposes featuring the user's eyes (12), as well as cameras (14) (e.g., infrared cameras) having fields of view (28, 30) and light or radiation sources (16) (e.g., infrared) directed toward the eyes (12) to enable eye tracking, observation and/or image capture. The three outward-facing world-capture cameras (124) are shown having their FOVs (18, 20, 22), as well as the depth camera (154) and its FOV (24), and the photo camera (156) and its FOV (26). The depth information obtained from the depth camera (154) may be enhanced by using data from the overlapping FOVs and other outward-facing cameras. For example, the system might end up with something like a sub-VGA image from the depth sensor (154), a 720p image from the world cameras (124), and sometimes a 2 megapixel color image from the photo camera (156). Such a configuration would have five cameras sharing a common FOV, three of which have heterogeneous visible spectrum images, one in color, and the third having relatively low resolution depth. The system could be configured to segment the grayscale and color images, fuse those images, create a relatively high resolution image from them, take stereo correspondences, use the depth sensor to provide hypotheses about stereo depth, and use the stereo correspondences to obtain a more refined depth map (which could be much better than what was available from the depth sensor alone). These processes could be run on local mobile processing hardware, or could be run using cloud computing resources, perhaps in conjunction with data from other people in the area (say, two people sitting across a table from each other), resulting in a fairly refined mapping. In other embodiments, all of the sensors could be combined into a single, integrated sensor to achieve this functionality.

[000112] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. The examples are provided to illustrate the broader applicable aspects of the present invention. Various changes may be made and equivalents may be substituted for the invention described herein without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process operation(s) or step(s) to the objective(s), spirit or scope of the present invention. Additionally, as will be recognized by those skilled in the art, each of the individual variations described and illustrated herein has individual components and features that can be readily distinguished from or combined with features of any of the other several embodiments without departing from the scope and spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[000113] The present invention includes methods that can be performed using target devices. The methods can include an act of providing such a suitable device. Such providing can be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, position, set up, activate, power-up, or otherwise operate the device required by the target method. The methods mentioned herein can be performed in any order of the mentioned events, as well as in any order of the mentioned events that is logically possible.

[000114] Exemplary aspects of the present invention have been described above, along with details regarding material selection and fabrication. As to other details of the present invention, these may be recognized in connection with the above-referenced patents and publications, as well as generally known to or by those skilled in the art. The method-based aspects of the present invention may equally be applied with respect to additional operations, as is conventionally or logically employed.

[000115] Moreover, while the present invention has been described with reference to several examples selectively incorporating various features, the invention is not limited to what is described or shown when considering each variation of the invention. Various changes may be made to the invention described and equivalents may be substituted (whether recited herein or not for the sake of some brevity) without departing from the true spirit and scope of the invention. Furthermore, where a range of values is provided, it is understood that every intervening value between the upper and lower limits of that range, and any other stated or intervening value within that stated range, is encompassed within the invention.

[000116] It is also contemplated that any optional feature of the described variations of the invention may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are plural of the same items. More specifically, as used herein and in the claims associated therewith, the singular includes plural referents unless specifically stated otherwise. In other words, use of the singular allows for "at least one" of the subject item in the claims as well as in the description above associated with the present disclosure. It is further noted that such claims may be drafted to exclude any optional element. Therefore, this statement is intended to serve as antecedent basis for use of such exclusive terminology as "solely," "only," etc., or use of a "negative" limitation in connection with the recitation of claim elements.

[000117] Without using such exclusive terminology, the term "comprising" in the claims associated with this disclosure should allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims or whether the addition of a feature can be deemed to modify the nature of an element recited in such claims. Except as specifically defined herein, all technical and scientific terms used herein are to be given as broad a commonly understood meaning as possible while still maintaining claim validity.

[000118] The scope of the present invention is not limited to the examples and/or subject matter described herein, but rather only by the scope of the claim language associated with this disclosure.

Claims (143)

As an augmented reality (AR) display system,
A handheld component coupled to an electromagnetic field emitter that emits a magnetic field;
A head-mounted component having a display system for displaying virtual content to a user, the head-mounted component being coupled to an electromagnetic sensor for measuring a parameter related to magnetic flux in the electromagnetic sensor resulting from a magnetic field, wherein a head pose of the head-mounted component in a known coordinate system is known;
A depth sensor for measuring a distance in the known coordinate system, wherein the depth sensor comprises a depth camera having a first field of view (FOV);
A world capture camera having a second FOV at least partially overlapping with the first FOV;
A photographic camera having a third FOV at least partially overlapping with the first FOV and the second FOV; and
A controller communicatively coupled to the handheld component, the head mounted component, and the depth sensor, wherein the controller receives parameters related to magnetic flux in the electromagnetic sensor from the head mounted component and receives the distance from the depth sensor.
Including,
The controller determines a hand pose of the handheld component based at least in part on parameters related to the magnetic flux measured by the electromagnetic sensor and the distance measured by the depth sensor,
The depth camera, the world capture camera and the photo camera are configured to capture a first image, a second image and a third image, respectively.
The above controller
Segmenting the second and third images, and
It is programmed to fuse the second image and the third image to generate a fused image after segmenting the second image and the third image,
Measuring the above distance in the above known coordinate system is:
generating a hypothetical distance by analyzing the first image from the depth camera; and
Generating said distance by analyzing said hypothesis distance and said fused image
An AR display system comprising: In the first paragraph,
An AR display system, wherein the depth camera, the world capture camera, and the photo camera each have different first resolutions, second resolutions, and third resolutions. In the second paragraph,
An AR display system, wherein the first resolution of the depth camera is sub-VGA, the second resolution of the world capture camera is 720p, and the third resolution of the photo camera is 2 megapixel. In the first paragraph,
An AR display system, wherein the depth camera, the world capture camera and the photo camera form a single integrated sensor. In the first paragraph,
An AR display system, wherein the hand pose comprises a position and orientation of the handheld component in the known coordinate system. In the first paragraph,
The above world capture camera is an AR display system that is a grayscale camera. In the first paragraph,
The above photo camera is a color camera, AR display system. In the first paragraph,
An AR display system further comprising analyzing the fused image to generate multiple stereo correspondences. In Article 8,
An AR display system, wherein generating the distance by analyzing the hypothesis distance and the fused image includes analyzing the hypothesis distance and the plurality of stereo correspondences. As an augmented reality (AR) display system,
A handheld component comprising an electromagnetic field emitter that emits a magnetic field;
A head-mounted component having a display system for displaying virtual content to a user, the head-mounted component including a first electromagnetic sensor for measuring a first parameter related to a first magnetic flux in the first electromagnetic sensor originating from the magnetic field, wherein a head pose of the head-mounted component in a known coordinate system is known;
A body-worn component comprising a second electromagnetic sensor for measuring a second parameter related to a second magnetic flux in a second electromagnetic sensor originating from said magnetic field;
A controller communicatively coupled to the handheld component, the head-mounted component and the body-worn component, wherein the controller receives first parameters and second parameters related to first magnetic flux and second magnetic flux in the first electromagnetic sensor and the second electromagnetic sensor, respectively, from the head-mounted component and the body-worn component, and receives a distance in the known coordinate system;
Additional localization resources that provide additional information.
Including,
An AR display system, wherein the controller determines at least a position and an orientation of the handheld component with respect to the known coordinate system based at least in part on first parameters and second parameters related to the first magnetic flux and the second magnetic flux measured by the first electromagnetic sensor and the second electromagnetic sensor, the distance, and additional information provided by the additional localization resource. In Article 10,
The above additional localization resource is an AR display system including a WiFi transceiver. In Article 10,
An AR display system, wherein the additional localization resource comprises an additional electromagnetic emitter. In Article 10,
An AR display system, wherein the additional localization resource comprises an additional electromagnetic sensor. In Article 10,
The additional localization resource is an AR display system including a cellular network transceiver. In Article 10,
The additional localization resource is an AR display system including a RADAR emitter. In Article 10,
An AR display system, wherein the additional localization resource comprises a RADAR detector. In Article 10,
The additional localization resource is an AR display system including a LIDAR emitter. In Article 10,
The additional localization resource is an AR display system including a LIDAR detector. In Article 10,
The above additional localization resource is an AR display system including a GPS transceiver. In Article 10,
An AR display system, wherein the additional localization resource comprises a poster having a known detectable pattern. In Article 10,
An AR display system, wherein the additional localization resource comprises a marker having a known detectable pattern. In Article 10,
An AR display system, wherein the additional localization resource comprises an inertial measurement unit. In Article 10,
An AR display system, wherein the additional localization resource comprises a strain gauge. In Article 10,
An AR display system, wherein the display system displays virtual content to the user based at least in part on a position and orientation of the handheld component determined with respect to the known coordinate system. In Article 10,
An AR display system, wherein the additional localization resource comprises a reflector. In Article 25,
The above reflector is an AR display system that reflects radiation. In Article 10,
The additional localization resource is an AR display system including a beacon. In Article 27,
The above beacon is an AR display system that emits radiation. In Article 28,
An AR display system wherein the above radiation is infrared radiation and the beacon includes an infrared LED. As an augmented reality (AR) display system,
A handheld component comprising an electromagnetic field emitter that emits a magnetic field;
A head-mounted component comprising a display system and a head-mounted electromagnetic sensor for detecting first position information from the electromagnetic field emitter;
A belt pack comprising a belt pack electromagnetic sensor for detecting second position information from the electromagnetic field emitter;
Additional localization resources providing third-party location information;
A controller for receiving first position information, second position information, and third position information from the head-mounted component, the belt pack, and the additional localization resource to determine position and orientation with respect to a known coordinate system.
Including,
The above third location information is selected from the group consisting of Wi-Fi information, cellular information, RADAR information, LIDAR information, GPS information, camera information, electromagnetic information, radiation information, inertial information, and strain information,
An AR display system, wherein the third location information is linked to the known coordinate system. In Article 30,
The above additional localization resource is an AR display system including a WiFi transceiver. In Article 30,
An AR display system, wherein the additional localization resource comprises an additional electromagnetic emitter. In Article 30,
An AR display system, wherein the additional localization resource comprises an additional electromagnetic sensor. In Article 30,
The additional localization resource is an AR display system including a cellular network transceiver. In Article 30,
The additional localization resource is an AR display system including a RADAR emitter. In Article 30,
An AR display system, wherein the additional localization resource comprises a RADAR detector. In Article 30,
The additional localization resource is an AR display system including a LIDAR emitter. In Article 30,
The additional localization resource is an AR display system including a LIDAR detector. In Article 30,
The above additional localization resource is an AR display system including a GPS transceiver. In Article 30,
An AR display system, wherein the additional localization resource comprises a poster having a known detectable pattern. In Article 30,
An AR display system, wherein the additional localization resource comprises a marker having a known detectable pattern. In Article 30,
An AR display system, wherein the additional localization resource comprises an inertial measurement unit. In Article 30,
The additional localization resource is an AR display system including a strain gauge. In Article 30,
An AR display system, wherein the display system displays virtual content to a user wearing the head-mounted component based at least in part on a position and orientation determined with respect to the known coordinate system. In Article 30,
An AR display system, wherein the additional localization resource comprises a reflector. In Article 45,
The above reflector is an AR display system that reflects radiation. In Article 30,
The additional localization resource is an AR display system including a beacon. In Article 47,
The above beacon is an AR display system that emits radiation. In Article 48,
An AR display system wherein the above radiation is infrared radiation and the beacon includes an infrared LED.
delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete