CN104011788A - Systems and methods for augmented and virtual reality - Google Patents

Abstract

One embodiment is directed to a system for enabling two or more users to interact within a virtual world that includes virtual world data, the system comprising a computer network comprising one or more computing devices, the one or more computing devices comprising: a memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user such that the second user can experience the first portion from the second user's location such that aspects of the first user's virtual world are efficiently communicated to the second user.

Description

Systems and methods for augmented and virtual reality

relevant application data

This application claims the benefit of US Provisional Application Serial No. 61/552,941 filed October 28, 2011 under 35 USC Section 119. The aforementioned application is hereby incorporated by reference in its entirety into the present application.

technical field

The present invention generally relates to systems and methods configured to facilitate interactive virtual or augmented reality environments for one or more users.

Background technique

Virtual and augmented reality environments are generated (in part) by computers using data describing the environment. Such data can describe, for example, various objects that a user can perceive and interact with. Examples of these objects include: objects rendered and displayed for the user to see, audio played for the user to hear, and tactile (tactile) feedback for the user to feel. Users can perceive and interact with virtual and augmented reality environments through various visual, auditory and tactile components.

Contents of the invention

One embodiment is directed to a system for enabling two or more users to interact within a virtual world including virtual world data, the system comprising a computer network comprising one or more computing devices, the One or more computing devices comprising: memory, processing circuitry, and software at least partially stored in said memory and executable by said processing circuitry to process at least a portion of said virtual world data; wherein said virtual world data at least the first portion originates from the first user's virtual world local to the first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to the second user such that the second user can access the The second user's location to experience the first portion is such that aspects of the first user's virtual world are efficiently communicated to the second user. The first user and second user may be in different physical locations or substantially in the same physical location. At least a portion of the virtual world may be configured to change in response to changes in the virtual world data. At least a portion of the virtual world may be configured to change in response to physical objects perceived by the user device. A change in the virtual world data may indicate that a virtual object has a predetermined relationship to the physical object. Changes in the virtual world data may be provided to a second user device for presentation to the second user according to the predetermined relationship. The virtual world may be operable to be rendered by at least one of a computer server or user equipment. The virtual world may be presented in a two-dimensional format. The virtual world may be presented in a three-dimensional format. The user device may be operable to provide an interface for enabling interaction between a user and the virtual world in an augmented reality mode. The user device may be operable to provide an interface for enabling interaction between a user and the virtual world in a virtual reality mode. The user device may be operable to provide an interface for enabling interaction between the user and the virtual world in a combination of augmented and virtual reality modes. The virtual world data may be transmitted over a data network. The computer network may be operable to receive at least a portion of the virtual world data from a user device. At least a portion of the virtual world data transmitted to the user device may include instructions for generating at least a portion of the virtual world. At least a portion of the virtual world data may be transmitted to the gateway for at least one of processing or distribution. At least one of the one or more computer servers may be operable to process virtual world data distributed by the gateway.

Another embodiment is directed to a system for a virtual and/or enhanced user experience in which an avatar is animated based at least in part on data on a wearable device with optional input from pitch changes and facial recognition software.

Another embodiment is directed to a system for virtual and/or augmented user experience where camera poses or viewpoint orientations and vectors can be placed anywhere in a world sector.

Another embodiment is directed to a system for a virtual and/or augmented user experience where the world or portions thereof can be rendered at various and selectable scales for an observing user.

Another embodiment is directed to a system for virtual and/or augmented user experience, where features (such as points or parametric lines in addition to pose-tagged images) can be used as base data for a world model from which software Robot or object recognizers can be used to create parametric representations of real-world objects, labeling source features for mutual inclusion in segmented object and world models.

Description of drawings

FIG. 1 illustrates a representative embodiment of the disclosed system for facilitating interactive virtual or augmented reality environments for multiple users.

FIG. 2 illustrates an example of user equipment for interacting with the system illustrated in FIG. 1 .

Figure 3 illustrates an example embodiment of a mobile wearable user device.

4 illustrates an example of objects seen by a user when the mobile wearable user device of FIG. 3 is operating in an enhanced mode.

5 illustrates examples of objects seen by a user when the mobile wearable user device of FIG. 3 is operating in a virtual mode.

6 illustrates examples of objects seen by a user when the mobile wearable user device of FIG. 3 is operating in a hybrid virtual interface mode.

Figure 7 illustrates an embodiment in which two users located in different geographic locations each interact with the other user and a common virtual world through their respective user devices.

FIG. 8 illustrates the embodiment of FIG. 7 extended to include an embodiment using a haptic device.

Figure 9A illustrates an example of a hybrid mode interfacing, where a first user interfaces with a digital world in a hybrid virtual interface mode, and a second user interfaces with the same digital world in a virtual reality mode.

9B illustrates another example of hybrid mode interfacing, where a first user interfaces with a digital world in hybrid virtual interface mode, and a second user interfaces with the same digital world in augmented reality mode.

Figure 10 illustrates an example illustration of a user's view when interfacing with the system in augmented reality mode.

11 illustrates an example illustration of a user's view showing virtual objects triggered by physical objects when the user in augmented reality mode interfaces with the system.

Figure 12 illustrates one embodiment of an integrated augmented and virtual reality configuration in which one user in an augmented reality experience visualizes another user's representation in a virtual reality experience.

Figure 13 illustrates one embodiment of an augmented reality experience configuration based on time and/or unexpected events.

Figure 14 illustrates one embodiment of a user display configuration suitable for a virtual and/or augmented reality experience.

Figure 15 illustrates one embodiment of on-premises and cloud-based computing coordination.

Figure 16 illustrates various aspects of the registration configuration.

Detailed ways

Referring to FIG. 1 , system 100 is representative hardware for implementing the processes described below. Such a representative system includes a computing network 105 consisting of one or more computer servers 110 connected by one or more high bandwidth interfaces 115 . Servers in a computing network do not have to be co-located. Each of the one or more servers 110 includes one or more processors for executing program instructions. The server also contains memory for storing program instructions and data for use and/or generation by processes executed by the server at the direction of the program instructions.

Computing network 105 communicates data between servers 110 and between servers and one or more user devices 120 over one or more data network connections 130 . Examples of such data networks include, but are not limited to, any or all types of public and private data networks (both mobile and wired), including, for example, the interconnection of many such networks commonly referred to as the Internet. No particular medium, topology, or protocol is intended to be implied by this diagram.

User devices are configured to communicate directly with any server in computing network 105 or servers 110 . Alternatively, user device 120 communicates with remote server 110 and, optionally, through a specially programmed local gateway for processing data and/or for passing data between network 105 and one or more local user devices 120 140 communicates locally with other user equipment.

As illustrated, gateway 140 is implemented as a single hardware component that includes a processor for executing software instructions and a memory for storing software instructions and data. The gateway has its own wired and/or wireless connection to the data network for communicating with the server 110 comprising the computing network 105 . Alternatively, gateway 140 can be integrated with user equipment 120, which is worn or carried by the user. For example, gateway 140 may be implemented as a downloadable software application that is installed and runs on a processor contained in user device 120 . In one embodiment, gateway 140 provides one or more user access to computing network 105 via data network 130 .

Servers 110 each contain, for example, working memory and storage devices for storing data and software programs, microprocessors for executing program instructions, graphics for rendering and generating graphics, images, video, audio, and multimedia files processors and other special purpose processors. Computing network 105 may also include facilities for storing data accessed, used, or created by servers 110 .

A software program running on the server and optionally the user device 120 and the gateway 140 is used to generate a digital world (also referred to herein as a virtual world) with which the user interacts with the user device 120 . The digital world is represented by data and processes that describe and/or define virtual non-existent entities, environments and conditions that can be presented to a user via user device 120 for user experience and interaction. For example, a certain type of object, entity, or item that appears to physically appear when instantiated in a scene viewed or experienced by a user may include its appearance, its behavior, how the user is allowed to interact with it, and other characteristics description of. Data used to create the environment of a virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and/or describe the virtual environment. Additionally, data defining various conditions governing the operation of a virtual world may include, for example, laws of physics, temporal, spatial relationships, and other data that may be used to define and/or create virtual conditions governing the operation of a virtual world (including virtual objects) .

Unless the context dictates otherwise, an entity, object, condition, characteristic, behavior, or other characteristic of the digital world is generally referred to herein as an object (eg, a digital object, a virtual object, a rendered physical object, etc.), unless the context dictates otherwise. Objects can be any type of animated or non-animated object, including but not limited to buildings, plants, vehicles, people, animals, creatures, machines, data, video, text, pictures, and other users. Objects can also be defined in the digital world for storing information about items, behaviors or conditions that are actually present in the physical world. In this document, data describing or defining an entity, object or item, or storing its current state, will generally be referred to as object data. This data is processed by the server 110 or, depending on the implementation, by the gateway 140 or the user device 120 to instantiate an instance of the object and render the object in an appropriate manner for the user to experience through the user device.

Programmers who develop and/or curate digital worlds create or define objects and the conditions under which they are instantiated. However, the digital world can allow others to create or modify objects. Once an object is instantiated, the state of the object may be allowed to be changed, controlled, or manipulated by one or more users experiencing the digital world.

For example, in one embodiment, the development, production and management of the digital world is generally provided by one or more system administration programmers. In some embodiments, this may include: the development, design and/or execution of storylines, themes and events in the digital world, and through various forms of events and media (such as, for example, film, digital, web, mobile, enhanced reality and live entertainment) narrative distribution. Systems administration programmers also deal with the technical administration, moderation, and planning of the digital world and its associated user communities, as well as other tasks typically performed by network administrators.

Users interact with one or more digital worlds using some type of local technology device, generally denoted as user device 120 . Examples of such user devices include, but are not limited to, smartphones, tablet devices, heads-up displays (HUDs), game consoles, or any other device capable of communicating data and providing an interface or display to a user, and combinations of such devices. In some embodiments, user device 120 may include local peripherals or input/output components (such as, for example, keyboards, mice, joysticks, game controllers, tactile interface devices, motion capture controllers, optical tracking devices (such as those that can available from Leap Motion Corporation, or those available from Microsoft under the trade name Kinect (RTM), audio devices, speech devices, projection systems, 3D displays, and holographic 3D contact lenses) or in communication therewith.

An example of a user device 120 for interacting with the system 100 is illustrated in FIG. 2 . In the example embodiment shown in FIG. 2 , a user 210 may interface with one or more digital worlds through a smartphone 220 . The gateway is implemented by a software application 230 stored on and running on the smartphone 220 . In this particular example, data network 130 comprises a wireless mobile network connecting user equipment (ie, smartphone 220 ) to computer network 105 .

In one implementation of the preferred embodiment, the system 100 is capable of supporting a large number of simultaneous users (e.g., millions of users), each using some type of user equipment 120 to interface with the same digital world or with multiple digital world connection.

User equipment provides the user with means to enable visual, auditory and/or physical interaction between the user and the digital world generated by the server 110, including other users and objects (real or virtual) presented to the user. The interface provides the user with the ability to see, hear, or otherwise perceive a rendered scene, as well as interact with the scene in real time. The manner in which the user interacts with the rendered scene may be dictated by the capabilities of the user's device. For example, if the user device is a smart phone, user interaction may be achieved by the user touching the touch screen. In another example, if the user device is a computer or game console, a keyboard or game controller can be used for user interaction. The user device may contain additional components that enable user interaction, such as sensors, where objects and information (including gestures) detected by the sensors may be provided as input representing user interaction with the virtual world using the user device.

The rendered scene may be presented in various formats such as, for example, two-dimensional or three-dimensional visual displays (including projectors), sound, and tactile or tactile feedback. The rendered scene may be interfaced by the user in one or more modes including, for example, augmented reality, virtual reality, and combinations thereof. The format of the rendered scene and the interface mode may be specified by one or more of: user equipment, data processing capabilities, user equipment connectivity, network capacity, and system workload. The real-time nature of interaction and data exchange with a large number of users simultaneously with the digital world is enabled by computing network 105, server 110, gateway component 140 (optional), and user equipment 120.

In one example, computing network 105 consists of a large-scale computing system having single and/or multiple core servers (ie, server 110 ) over high-speed connections (eg, high-bandwidth interface 115 ). Computing network 105 may form a cloud or mesh network. Each of the servers contains memory, or is coupled with computer readable memory for storing software for implementing data to create, design, change or manipulate objects of the digital world. These objects and their instances can be dynamic, coming in and out, changing over time and in response to other conditions. Examples of dynamic capabilities of objects are generally discussed herein with respect to various embodiments. In some embodiments, each user interfaced with system 100 is also represented as an object and/or collection of objects within one or more digital worlds.

Servers 110 within computing network 105 also store computing state data for each of the digital worlds. Computational state data (also referred to herein as state data) may be an integral part of object data, and generally defines the state of an instance of an object at a given moment in time. Thus, computing state data may change over time and may be affected by actions of one or more users and/or programmers maintaining system 100 . When a user affects computing state data (or other data comprising the digital world), the user directly changes or otherwise manipulates the digital world. If the digital world is shared with or interfaced by other users, the user's actions may affect the digital world experienced by other users interacting with the digital world. Thus, in some embodiments, changes to the digital world made by a user will be experienced by other users interfacing with the system 100 .

In one embodiment, data stored in one or more servers 110 within computing network 105 is transmitted or deployed to one or more user devices 120 and/or gateway component 140 at high speed and low latency. In one embodiment, the object data shared by the server may be complete data or may be compressed, and contain instructions for recreating the entire object data on the user side, by the user's local computing device (e.g., gateway 140 and/or user device 120) for rendering and visualization. In some embodiments, software running on servers 110 of computing network 105 can adapt the objects (or any data exchanged by computing network 105 ) in the digital world it generates and sends to specific Data of the user equipment 120. For example, when a user interacts with the digital world through user device 120, server 110 may identify the particular type of device being used by the user, the connectivity of the device, and/or the available bandwidth between the user device and the server, and adjust the size of the device appropriately. And balance the data to be delivered to the device to optimize user interaction. An example of this could include reducing the size of transmitted data to low-resolution quality so that the data can be displayed on a particular user device with a low-resolution display. In a preferred embodiment, computing network 105 and/or gateway component 140 renders the interface at a rate sufficient to operate at 15 frames per second or higher, and at a resolution of high definition or higher to user device 120 Delivery data.

Gateway 140 provides local connectivity to computing network 105 for one or more users. In some embodiments, gateway 140 may be implemented by a downloadable software application running on user device 120 or another local device, such as the local device shown in FIG. 2 . In other embodiments, gateway 140 may be implemented by a hardware component (with appropriate software/firmware stored on the component, with a processor) that communicates with (but does not communicate with) user device 120 incorporated or attached to user equipment 120), or merged with user equipment 120. Gateway 140 communicates with computing network 105 via data network 130 and provides data exchange between computing network 105 and one or more local user devices 120 . As discussed in more detail below, gateway component 140 may include software, firmware, memory, and processing circuitry, and may be capable of processing data communicated between network 105 and one or more local user devices 120 .

In some embodiments, gateway component 140 monitors and adjusts the rate of data exchanged between user device 120 and computer network 105 to allow optimal data processing capabilities for a particular user device 120 . For example, in some embodiments, gateway 140 buffers and downloads both static and dynamic aspects of the digital world, even those beyond the field of view presented to the user through an interface connected to the user device. In such embodiments, instances of static objects (structured data, software implementing methods, or both) may be stored in memory (local to gateway component 140, user device 120, or both), and The local user's current position (as indicated by data provided by computing network 105 and/or user device 120 ) is referenced. Instances of dynamic objects (which may include, for example, intelligent software agents and objects controlled by other users and/or local users) may be stored in a cache memory. Dynamic objects representing two-dimensional or three-dimensional objects within a scene presented to a user can, for example, be decomposed into component shapes, such as static shapes that move but do not change, and dynamic shapes that change. The portion of the dynamic object that changes can be updated by a real-time threaded high-priority data stream from the server 110 managed by the gateway component 140 over the computing network 105 . As one example of a prioritized threaded data flow, data within the 60 degree field of view of the user's eyes may be given higher priority than more peripheral data. Another example includes prioritizing dynamic characters and/or objects within the user's field of view over static objects in the background.

In addition to managing data connections between computing network 105 and user device 120 , gateway component 140 may store and/or process data that may be presented to user device 120 . For example, in some embodiments, gateway component 140 may receive from computing network 105 compressed data, such as describing graphical objects to be rendered for viewing by a user, and perform advanced rendering techniques to ease transmission from computing network 105 to The data payload of the user equipment 120. In another example, where gateway 140 is a stand-alone device, gateway 140 may store and/or process data for a local instance of an object, rather than transmitting the data to computing network 105 for processing.

Referring now also to FIG. 3, the digital world can be experienced by one or more users in various formats (depending on the capabilities of the user equipment). In some embodiments, user device 120 may include, for example, a smartphone, tablet device, head-up display (HUD), game console, or wearable device. Generally, a user device will include a processor for executing program code stored in memory on the device, coupled to a display, and a communication interface. An example embodiment of a user device comprising a mobile wearable device, ie, a head-mounted display system 300, is illustrated in FIG. According to an embodiment of the present disclosure, the head mounted display system 300 includes a user interface 302 , a user perception system 304 , an environment perception system 306 and a processor 308 . Although processor 308 is shown in FIG. The components are integrated, or can be integrated into other system 100 components such as gateway 140, for example.

The user device presents the user with an interface 302 for interacting with and experiencing the digital world. Such interactions may involve the user and the digital world, one or more other users interfaced with the system 100, and objects within the digital world. Interface 302 generally provides visual and/or audio sensory input (and, in some embodiments, physical sensory input) to the user. Thus, the interface 302 may include a speaker (not shown) and a display component 303 that, in some embodiments, enables a stereoscopic three-dimensional display and/or a three-dimensional display that embodies the more natural characteristics of the human visual system. In some embodiments, display assembly 303 may include a transparent interface (such as a clear OLED) that, when in the "off" setting, enables Optically corrects the view of the physical environment around the user. As discussed in more detail below, interface 302 may contain additional settings to allow for various visual/interface capabilities and functions.

In some embodiments, user perception system 304 may include: one or more sensors 310 operable to detect certain characteristics, characteristics, or information about an individual user wearing system 300 . For example, in some embodiments, sensor 310 may include: a camera or optical detection/scanning circuitry capable of detecting optical characteristics/measurements of the user in real-time, such as, for example, one or more of the following: pupil constriction/dilation, per pupil Angle measurement/positioning, secondary spherocity, eye shape (eg change in eye shape over time), and other anatomical data. Such data may provide or be used to calculate information (eg, the user's visual focus) that may be used by head mounted system 300 and/or interface system 100 to optimize the user's viewing experience. For example, in one embodiment, sensors 310 may each measure the rate at which the pupils of each eye of the user contract. Such data may be transmitted to processor 308 (or gateway component 140 or server 110 ), where it is used to determine, for example, a user's response to a brightness setting of interface display 303 . Interface 302 may be adjusted according to the user's response by, for example, dimming display 303 if the user's response indicates that the zero degree level of display 303 is too high. User perception system 304 may contain other components in addition to those described above or illustrated in FIG. 3 . For example, in some embodiments, user perception system 304 may include a microphone for receiving voice input from a user. The user perception system may also include: one or more infrared camera sensors, one or more visible light spectrum camera sensors, structured light emitters and/or sensors, infrared light emitters, coherent light emitters and/or sensors, gyroscopes, Accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and haptic interfaces.

Context awareness system 306 includes one or more sensors 312 for obtaining data from the physical environment around the user. Objects or information detected by the sensors may be provided as input to the user device. In some embodiments, such input may represent user interaction with the virtual world. For example, a user viewing a virtual keyboard on a desktop can use his fingers to gesture as if it were typing on the virtual keyboard. The motion of the finger movement can be captured by the sensor 312 and provided to the user device or system as input, where the input can be used to change the virtual world or create new virtual objects. For example, finger movements can be recognized (using a software program) as typing, and the gesture of the recognized typing can be combined with the known positions of the virtual keys on the virtual keyboard. The system can then render a virtual monitor displayed to the user (or other users interfaced with the system), where the virtual monitor displays text typed by the user.

The sensor 312 may comprise, for example, a generally outward-facing camera or a scanner for interpreting scene information, for example by constantly and/or intermittently projecting infrared structured light. The environment awareness system 306 may be used to map one or more elements of the physical environment around the user by detecting or registering the local environment, including static objects, dynamic objects, people, gestures and various lighting, atmospheric and acoustic conditions. Thus, in some embodiments, environment awareness system 306 may comprise: image-based three-dimensional reconstruction software embedded in a local system (eg, gateway component 140 or processor 308 ) and operable to digitally reconstruct images detected by sensor 312 One or more objects or information of . In an exemplary embodiment, the context awareness system 306 provides one or more of: motion capture data (capture gesture recognition), depth perception, facial recognition, object recognition, unique object feature recognition, voice/audio recognition and processing , sound source localization, noise reduction, infrared or similar laser projection, and monochrome and/or color CMOS sensors (or other similar sensors), field of view sensors, and various other optically enhanced sensors. It should be appreciated that the context awareness system 306 may include other components in addition to those described above or illustrated in FIG. 3 . For example, in some embodiments, environment awareness system 306 may include a microphone for receiving audio from the local environment. The user perception system may also include: one or more infrared camera sensors, one or more visible light spectrum camera sensors, structured light emitters and/or sensors, infrared light emitters, coherent light emitters and/or sensors, gyroscopes, Accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and haptic interfaces.

As noted above, in some embodiments, processor 308 may be integrated with other components in head-mounted system 300, integrated with other components in interface system 100, or may be a separate device as shown in FIG. wearable or separate from the user). Processor 308 may be connected to various components and/or Components in interface system 100 . Processor 308 may include: memory modules, integrated and/or additional graphics processing units, wireless and/or wired Internet connectivity, and the ability to Gateway component 140) codec and/or firmware that transforms data into image and audio data, where image/video and audio can be presented to the user via interface 302.

Processor 308 handles data processing for various components in headset system 300 and data exchange between headset system 300 and gateway 140 and, in some embodiments, computing network 105 . For example, processor 308 may be used to buffer and process data streams between the user and computing network 105, thereby enabling a smooth continuous and high-fidelity user experience. In some embodiments, processor 308 may be powerful enough to achieve 8 frames per second at 320x240 resolution to 24 frames per second at high definition resolution (1280x720), or higher, such as 60-120 frames per second and 4k Process data at rates anywhere between 10k+ resolution and 50,000 frames per second. Additionally, processor 308 may store and/or process data that may be presented to a user rather than streaming from computing network 105 in real time. For example, in some embodiments, processor 308 may receive compressed data from computing network 105 and perform advanced rendering techniques (such as lighting or shading) to alleviate the data load transmitted from computing network 105 to user device 120 . In another example, processor 308 may store and/or process local object data instead of transmitting the data to gateway component 140 or computing network 105 .

In some embodiments, head mounted system 300 may incorporate various settings or modes that allow for various visual/interface capabilities and functions. The mode may be selected manually by the user, or automatically by a component in the headset system 300 or the gateway component 140 . As previously mentioned, one example of the head-mounted system 300 includes an "off" mode in which the interface 302 provides substantially no digital or virtual content. In the off mode, the display assembly 303 may be transparent, enabling an optically correct view of the physical environment surrounding the user with little-to-no optical distortion or computational overlap.

In one example embodiment, head mounted system 300 includes: an "augmented" mode in which interface 302 provides an augmented reality interface. In enhanced mode, interface display 303 may be substantially transparent, allowing the user to view the local physical environment. Simultaneously, virtual object data provided by computing network 105, processor 308, and/or gateway component 140 is presented on display 303 in conjunction with the physical local environment.

FIG. 4 illustrates an example embodiment of objects seen by a user when interface 302 is operating in an enhanced mode. As shown in FIG. 4 , interface 302 presents physical objects 402 and virtual objects 404 . In the embodiment illustrated in FIG. 4 , physical objects 402 are real physical objects that exist in the user's local environment, while virtual objects 404 are objects created by system 100 that are displayed via user interface 302 . In some embodiments, the virtual object 404 may be displayed at a fixed orientation or location within the physical environment (e.g., a virtual monkey standing next to a particular landmark located in the physical environment), or may be displayed to the user as a Objects related to the orientation of the interface display 303 (eg, a virtual clock or thermometer visible in the upper left of the display 303).

In some embodiments, the virtual object may be prompted to move away from, or triggered by, an object physically present inside or outside the user's field of view. The virtual object 404 is prompted to leave the physical object 402 or the virtual object 404 is triggered by the physical object 402 . For example, the physical object 402 may actually be a stool, and a virtual object 404 such as a virtual animal standing on the stool may be displayed to the user (and in some embodiments, other users interfaced with the system 100). In such embodiments, environment awareness system 306 may use, for example, software and/or firmware stored in processor 308 to recognize various features and/or shape patterns (captured by sensors 312) to identify physical object 402 as stool. These identified shape patterns (such as, for example, the top of a stool) can be used to trigger placement of virtual object 404 . Other examples include: walls, tables, furniture, cars, buildings, people, floors, plants, animals - any object that can be seen can be used to trigger an augmented reality experience that has some relationship to the object or objects.

In some embodiments, the particular virtual object 404 that is triggered may be selected by the user or automatically by other components in the head mounted system 300 or by the interface system 100 . Additionally, in embodiments where the virtual object 404 is automatically triggered, specific physical objects 402 (or features thereof) may be selected based on prompting the virtual object 404 to move away from a specific physical object 402 (or a feature thereof) or triggering the virtual object 404 . Virtual object 404 . For example, if the physical object is identified as a diving board extending over a pool, the triggered virtual object could be a creature wearing a snorkel, bathing suit, flotation device, or other related item.

In another example embodiment, head mounted system 300 may include a "virtual" mode in which interface 302 provides a virtual reality interface. In virtual mode, the physical environment is omitted from display 303 and virtual object data provided by computing network 105 , processor 308 and/or gateway component 140 is presented on display 303 . Omitting the physical environment may be accomplished by physically blocking the visual display 303 (eg, via an overlay) or by features of the interface 302 where the display 303 transitions to an opaque setting. In the virtual mode, live and/or stored visual and audio sensations may be presented to the user through the interface 302, and the user experiences and interacts with the digital world (digital objects, other users, etc.) through the virtual mode of the interface 302. Therefore, the interface provided to the user in the virtual mode is composed of virtual object numbers including the virtual digital world.

FIG. 5 illustrates an example embodiment of a user interface when the head mounted interface 302 is operating in a virtual mode. As shown in FIG. 5, the user interface presents a virtual world 500 composed of digital objects 510, which may include: atmosphere, weather, terrain, buildings, and people. Although not illustrated in FIG. 5 , digital objects may also include, for example, plants, vehicles, animals, creatures, machines, artificial intelligence, location information, and any other objects or information that define virtual world 500 .

In another example embodiment, the headset 300 may include a "hybrid" mode in which various features in the headset 300 (as well as features in the virtual and augmented modes) may be combined to create a or multiple custom interface patterns. In one example custom interface mode, the physical environment is omitted from the display 303 and virtual object data is presented on the display 303 in a manner similar to the virtual mode. However, in this example custom interface pattern, the virtual objects can be completely virtual (that is, they do not exist in the local physical environment) or they can be rendered as virtual objects in the interface 302 in place of real objects of physical objects. local physics object. Thus, in this particular customization mode (referred to herein as a hybrid virtual interface mode), live and/or stored visual and audio sensations can be presented to the user through the interface 302, as well as user experience and integration including fully virtual objects and rendered interact with the digital world of physical objects.

Figure 6 illustrates an example embodiment of a user interface operating in accordance with a hybrid virtual interface mode. As shown in Figure 6, the user interface presents a virtual world 600 consisting of fully virtual objects 610 and rendered physical objects 620 (renderings of objects that would otherwise physically appear in the scene). Pursuant to the example illustrated in FIG. 6 , rendered physical objects 620 include: building 620A, ground 620B, and platform 620C, and are shown with bold outlines 630 to indicate to the user that the objects are rendered. Additionally, the complete virtual object 610 includes: an additional user 610A, a cloud 610B, a sun 610C, and a flame 610D on a platform 620C. It should be appreciated that complete virtual objects 610 may contain, for example, atmosphere, weather, terrain, buildings, people, plants, vehicles, animals, creatures, machines, artificial intelligence, location information, and Any other objects or information rendered by objects in the local physical environment. In contrast, rendered physical objects 620 are real local physical objects rendered as virtual objects in interface 302 . A bold outline 630 represents one example of a physical object used to indicate rendering to the user. As such, methods other than those disclosed herein may be used to indicate rendered physical objects as such.

In some embodiments, rendered physical objects 620 may be detected using sensors 312 of environment awareness system 306 (or using other devices such as motion or image capture systems) and detected by, for example, software stored in processing circuitry 308 and/or Firmware is converted into digital object data. Thus, when a user interfaces with the system 100 in the hybrid virtual interface mode, various physical objects may be displayed to the user as rendered physical objects. This may be particularly useful for allowing a user to interface with the system 100 while still being able to safely navigate the local physical environment. In some embodiments, a user may be able to selectively remove or add rendered physical objects to interface display 303 .

In another example user interface mode, the interface display 303 may be substantially transparent, allowing the user to view the local physical environment while various local physical objects are displayed to the user as rendered physical objects. The custom interface mode of this example is similar to the enhanced mode, except that one or more of the virtual objects may be rendered physical objects as described above with respect to the previous example.

The example custom interface modes described above represent several example embodiments of the various custom interface modes that can be provided by the promiscuous mode of the headset system 300 . Accordingly, various other custom interface modes may be created from various combinations of the features and functions provided by the components in head mounted system 300 and the various modes described above without departing from the scope of the present disclosure.

The embodiments discussed herein merely describe a few examples for providing an interface operating in off, augmented, virtual, or promiscuous modes, and are not intended to limit the functionality of the components of the head mounted system 300 or the scope or content of the respective interface modes. . For example, in some embodiments, virtual objects may contain data displayed to the user (time, temperature, altitude, etc.), objects created and/or selected by the system 100, objects created and/or selected by the user, or even represent An object for other users that interface with the system 100 . Additionally, virtual objects may contain extensions of physical objects (eg, avatars grown from physical platforms), and may be visibly connected to, or disconnected from, physical objects.

Virtual objects can also be dynamic and change over time, according to various relationships (e.g., position, distance, etc.) may vary depending on various variables specified in the software and/or firmware of system 300, gateway component 140, or server 110. For example, in some embodiments, a virtual object may respond to user equipment or components thereof (e.g., a virtual ball moves when a haptic device is placed next to the virtual ball), physical or verbal user interaction (e.g., when a user approaches the virtual creature runs away when the virtual creature runs away, or the virtual creature speaks when the user speaks to it), a chair is thrown at the virtual creature and the creature avoids the chair, other virtual objects (for example, when the first virtual creature sees the first virtual creature Two virtual creatures, the first virtual creature reacts), physical variables (such as location, distance, temperature, time, etc.), or other physical objects in the user's environment (for example, when a physical car passes by, shown as Virtual creatures standing in physical streets become flattened).

The various modes discussed herein may apply to user equipment other than head mounted system 300 . For example, an augmented reality interface may be provided via a mobile phone or tablet device. In such embodiments, the phone or tablet may use a camera to capture the physical environment near the user, and virtual objects may be overlaid on the phone/tablet's display screen. Additionally, a virtual mode can be provided by displaying the digital world on the display screen of the phone/tablet. Thus, using the components of the phone/tablet described herein as well as other components connected to or used in conjunction with the user device, these patterns can be intermixed to create various custom interfaces as described above. For example, a promiscuous virtual interface mode may be provided by a computer monitor, television screen, or other device that lacks a camera that operates in conjunction with a motion or image capture system. In this example embodiment, the virtual world can be viewed from a monitor/screen, and object detection and rendering can be performed by a motion or image capture system.

FIG. 7 illustrates an example embodiment of the present disclosure in which two users located in different geographic locations each interact with other users and a common virtual world through their respective user devices. In this embodiment, two users 701 and 702 are throwing a virtual ball 703 (a type of virtual object) back and forth, where each user is able to observe the effects of the other users in the virtual world (e.g., each user observes to a virtual ball that changes direction, a virtual ball grabbed by another user, etc.). Because the movement and position of the virtual ball (i.e., virtual ball 703) is tracked by server 110 in computing network 105, in some embodiments, system 100 may communicate to users 701 and 702 relative to each user's ball 703 exact location and timing of arrival. For example, if a first user 701 is located in London, user 701 may throw a ball 703 at a rate calculated by system 100 at a second user 702 located in Los Angeles. Accordingly, the system 100 can communicate to the second user 702 (eg, via email, text message, instant message, etc.) the exact time and location of the ball's arrival. As such, the second user 702 can use his device to see the ball 703 arrive at the specified time and location. One or more users may also use geographic location mapping software (or similar) to track the one or more virtual objects as they virtually traverse the globe. An example of this could be a user wearing a three-dimensional head-mounted display looking up in the sky and seeing a virtual airplane flying overhead superimposed on the real world. The virtual aircraft may be piloted by the user, by an intelligent software agent (software running on the user's device or gateway), other users who may be local and/or remote, and/or any combination of these.

As previously mentioned, the user device may include a haptic interface device that provides feedback to the user (e.g., resistance, vibration, light, sound, etc.) ). For example, the embodiments described above with respect to FIG. 7 may be extended to include the use of a haptic device 802 as shown in FIG. 8 .

In this example embodiment, haptic device 802 may be displayed as a baseball bat in the virtual world. When the ball 703 arrives, the user 702 can wave the haptic device 802 towards the virtual ball 703 . If the system 100 determines that the virtual stick provided by the haptic device 802 has made "contact" with the ball 703, the haptic device 802 may vibrate or provide other feedback to the user 702, and the virtual stick 703 may follow the direction of the ball in the direction calculated by the system 100. The velocity, direction and timing of detection of contact to the stick bounces off the virtual stick.

In some embodiments, the disclosed system 100 can facilitate mixed-mode interfacing, where multiple users can interface with a common virtual world using different interfacing modes (e.g., enhanced, virtual, promiscuous, etc.) (contains dummy objects within it). For example, a first user interfacing with a particular virtual world in a virtual interface mode may interact with a second user interfacing with the same virtual world in an augmented reality mode.

Figure 9a illustrates a first user 901 (interfacing with the digital world of system 100 in hybrid virtual interface mode) and a first subject 902 for a second user 901 (interfacing with the same digital world of system 100 in full virtual reality mode) Appears to be an example of a virtual object to user 922. As described above, when interfacing with the digital world via the hybrid virtual interface mode, local physical objects (eg, first user 901 and first object 902 ) can be scanned and rendered as virtual objects in the virtual world. The first user 901 may, for example, be scanned by a motion capture system or similar device, as well as in the virtual world (by software/firmware stored in the motion capture system, gateway component 140, user device 120, system server 110, or other device) is rendered as a first rendered physical object 931 . Similarly, the first object 902 may be scanned, for example, by the context awareness system 306 of the head mounted interface 300, and in the virtual world (by software/ firmware) is rendered as a second rendered physical object 932. In the first part 910 of Figure 9A, the first user 901 and the first object 902 are shown as physical objects in the physical world. In the second part 920 of FIG. 9A, the first user 901 and the first object 902 are shown as they would appear to a second user 922 in full virtual reality mode connected to the same digital world interfaced with the system 100. are the first rendered physics object 931 and the second rendered physics object 932 .

FIG. 9B illustrates another example embodiment of hybrid mode interfacing, where a first user 901 interfaces with the digital world in hybrid virtual interface mode, and a second user 922 interfaces with the digital world in augmented reality mode, as described above. The same digital world (and the second user's physical local environment 925) is connected. In the embodiment in FIG. 9B, a first user 901 and a first object 902 are located at a first physical location 915, and a second user 922 is located at a different second physical location 925 separated by some distance from the first location 915 . In this embodiment, virtual object 931 and virtual object 932 may be translated in real time (or near real time) to a location within the virtual world corresponding to second location 925 . Thus, the second user 922 can observe and interact with rendered physical objects 931 and 932 representing the first user 901 and first object 902 respectively in the second user's physical local environment 925 .

FIG. 10 illustrates an example illustration of a user's view when interfacing with system 100 in augmented reality mode. As shown in FIG. 10, the user sees a local physical environment (ie, a city with multiple buildings) and a virtual character 1010 (ie, a virtual object). 2D visual objects (e.g., billboards, postcards, or magazines) and/or one or more 3D frames of reference (such as buildings, cars, people, animals, aircraft, parts of buildings, and/or any 3D physical objects, virtual objects, and/or combinations thereof) to trigger the orientation of the avatar 1010. In the example illustrated in FIG. 10 , known locations of buildings in a city may provide registration fiducials and/or information and key features for rendering virtual character 1010 . Additionally, the user's geospatial location (e.g., provided by GPS, attitude/orientation sensors, etc.) or mobile location relative to the building may include data used by the computing network 105 to trigger the display of the virtual character(s) 1010 data transmission. In some embodiments, data for displaying virtual character 1010 may include rendered character 101 and/or instructions (performed by gateway component 140 and/or user device 120) for rendering virtual character 1010 or a portion thereof. In some embodiments, if the user's geospatial location is unavailable or unknown, server 110, gateway component 140, and/or user device 120 may still use estimates that estimate where particular virtual and/or physical objects may be located. Algorithm to display the virtual object 1010 using the user's last known position as a function of time and/or other parameters. This can also be used to determine the orientation of any virtual objects in case the user's sensors become occluded and/or experience other malfunctions.

In some embodiments, a virtual character or virtual object may comprise a virtual statue, where rendering of the virtual statue is triggered by a physical object. For example, referring now to FIG. 11 , a virtual figurine 1110 may be triggered by a real physical platform 1120 . The triggering of the figurine 1110 may be in response to a virtual object or feature (eg, fiducial, design feature, geometry, pattern, physical location, height, etc.) detected by the user device or other components of the system 100 . When the user views the platform 1120 without using a user device, the user sees the platform 1120 without the figurine 1110 . However, when the user views the platform 1120 through the user equipment, the user sees the statue 1110 on the platform 1120 as shown in FIG. 11 . The statue 1110 is a virtual object, and thus may be fixed, animated, change over time or relative to the user's viewing orientation, or even vary depending on which particular user is viewing the statue 1110. For example, if the user is a child, the figurine might be a dog; but if the viewer is an adult male, the figurine might be the large robot shown in FIG. 11 . These are examples of user adherence and/or state adherence experiences. This would enable one or more users to perceive one or more virtual objects independently and/or in combination with physical objects, as well as experience customized and personalized versions of the virtual objects. Statue 1110 (or portions thereof) may be rendered by various components of the system including, for example, software/firmware installed on a user device. The virtual object (ie, figurine 1110 ) forms a relationship with the physical object (ie, platform 1120 ) using data indicative of the position and pose of the user device, in conjunction with registered features of the virtual object (ie, figurine 1110 ). For example, the relationship between one or more virtual objects and one or more physical objects can be a function of distance, location, time, geographic location, proximity to one or more other virtual objects, and/or include any kind of Any other functional relationship of virtual and/or physical data. In some embodiments, image recognition software in the user device may also enhance digital-to-physical object relationships.

The interactive interfaces provided by the disclosed systems and methods can be implemented to facilitate various activities, such as, for example, interacting with one or more virtual environments and objects, interacting with other users, and experiencing various forms of media content , containing commercials, concerts, and movies. Thus, the disclosed system facilitates user interaction such that users not only watch or listen to media content, but actively participate in and experience the media content. In some embodiments, user participation may involve changing existing content or creating new content to be rendered in one or more virtual worlds. In some embodiments, media content and/or user-created content may be themed around mythological creations of one or more virtual worlds.

In one example, musicians (or other users) may create musical content to be rendered to users interacting with a particular virtual world. Music content can include, for example, various tracks, EPs, albums, videos, short films and concert performances. In one example, many users may interface with the system 100 to simultaneously experience a virtual concert performed by musicians.

In some embodiments, the generated media may contain a unique identification code associated with a particular entity (eg, band, artist, user, etc.). The code can be in the form of a set of alphanumeric characters, a UPC code, a QR code, a two-dimensional image trigger, a three-dimensional physical object characteristic trigger or other digital indicia, as well as sound, image and/or both. In some embodiments, the code may also be embedded with digital media to which system 100 may be used to interface. A user may obtain the code (eg, via payment of a fee) and redeem the code to access media content produced by the entity associated with the identification code. Media content may be added to or removed from the user interface.

In one embodiment, in order to avoid the computation and bandwidth of transferring video data in real time or near real time with low latency from one computing system to another, such as from a cloud computing system to a local processor coupled to a user , parametric information about various shapes and geometries can be transferred and used to define surfaces, while textures can be transferred and added to these surfaces to bring about static or dynamic details, such as re Bitmap-based video details of a person's face are mapped to the geometry of the face. As another example, if the system is configured to recognize a person's face, and knows that the person's avatar is located in the augmented world, the system can be configured to transmit the relevant world information and The information, after which the rest is transferred to a local computing system, such as the one depicted in FIG. Bitmaps - all of this at an order of magnitude below the bandwidth associated with the initial setup transfer or delivery of live video. Thus, cloud-based and on-premises computing assets can be used in an integrated manner such that cloud processing does not require relatively low-latency computing, and locally processing assets with low-latency processing is a very precious task, and in such cases, due to this The form of data transferred to the local system is preferably transmitted at a relatively low bandwidth, in the form of an aggregate amount of data (ie, parametric information, textures, etc. for real-time video of everything).

Referring ahead to FIG. 15 , collaboration between cloud computing assets ( 46 ) and local computing assets ( 308 , 120 ) is schematically illustrated. In one embodiment, cloud (46) assets are operatively directly connected, such as via a wired or wireless network (wireless is preferred for mobility, wired is preferred for certain high-bandwidth or high-data-volume transfers), as may be desired. coupled (40, 42) to one of local computing assets (120, 308) such as processor and memory configurations that may be installed in a structure configured to couple with a user head (120) or strap (308) or both. These computing assets locally located at the user may also be operably coupled to each other via wired and/or wireless connectivity arrangements (44). In one embodiment, in order to maintain the low inertia and small size of the head-mounted subsystem (120), the primary transfer between the user and the cloud (46) may be via a link between the belt-based subsystem (308) and the cloud. way, using wireless connectivity (such as ultra-wideband (“UWB”) connectivity, as currently used, for example, in personal computing peripheral connectivity applications) to tie the primary data of the head-mounted subsystem (120) to a band-based subsystem (308).

Using efficient local and remote processing collaboration, and an appropriate display device for the user, such as the user interface 302 or user "display device" featured in FIG. 3, the display device 14 described below with reference to FIG. 14, or variations thereof, Aspects of the world about the user's current real or virtual location can be transferred or "transmitted" to the user and updated in an efficient manner. Indeed, in one embodiment, one person using a virtual reality system ("VRS") in augmented reality mode and another person using VRS in fully virtual mode in order to explore the same world local to the first person, These two users may experience each other in various ways as users located in each other's worlds. For example, referring to FIG. 12 , a scene similar to that described with reference to FIG. 11 is depicted, plus a virtualization of a second user's avatar 2 from a full virtual reality scene flying through the described augmented reality world. That is, the scene depicted in FIG. 12 can be experienced and displayed in first-person augmented reality—except for real physical elements near the local world in the scene (such as the ground, buildings in the background, figurine platforms 1120 ) are also shown with two augmented reality elements (figure 1110 and the second person's flying bumblebee avatar 2). Dynamic updates may be used to allow the first person to visualize the process of the second person's avatar 2, such as the avatar 2 flying through the first person's local world.

Furthermore, using a configuration as described above, where there is a world model that can reside on cloud computing resources and can be distributed from there, preferably trying to distribute real-time video data or the like, such worlds can be distributed at relatively low cost. A form of bandwidth is "deliverable" to one or more users. The augmented experience for people standing near the statue (i.e., as shown in FIG. 12 ) can be informed by a cloud-based world model, a subset of which can be sent down to them and their local display device to complete the view . Someone sitting at a remote display device (which could be as simple as a personal computer sitting on a desk) can efficiently download the same portion of the information from the cloud and have it rendered on their display. In fact, a person who is in fact present in a park located near a statue can go for a walk in that park with a remotely located friend joined by virtual and augmented reality. The system will need to know where the streets are, where the trees are, where the statues are - but with that information on the cloud, the joining friend can download aspects of the scene from the cloud, and then start walking, as if they were actually in the park people about local augmented reality.

Referring to FIG. 13 , an embodiment based on time and/or other contingency parameters is depicted in which a virtual and/or augmented reality interface (such as the user interface 302 or user display device having the features depicted in FIG. 3 , described below with reference to FIG. 14 ) Display device 14, or a combination thereof) in conjunction with the person using the system (4) and entering the coffee establishment to order a cup of coffee (6). VRS can be configured to use perception and data collection capabilities (locally and/or remotely) to provide display enhancements in augmented and/or virtual reality for that person, such as highlighting the location of doors in a coffee establishment or related Bubbles window for coffee menu (8). When a person receives a cup of coffee that he has ordered or after the system detects some other relevant parameter, the system can be configured to display (10) one or more time-based augmented or virtual realities in a local environment with a display device Images, video and/or sound, such as Madagascar jungle scenes from walls and ceilings, with or without jungle sounds and other effects, static or dynamic. Based on timing parameters (i.e., 5 minutes after a full coffee cup has been identified and handed to the user; 10 minutes after the user has been identified walking through the front door of the establishment, etc.) or other parameters (such as by when the user When the last sip of coffee is ingested from the cup, the system recognizes that the user has finished the coffee by noticing that the coffee cup is pointing upside down - or the system recognizes that the user has left the front door of the establishment (12), to the user's such Rendering can be aborted.

Referring to Figure 14, one embodiment of a suitable user display device (14) is shown, comprising a display lens (82) mounted to the user's head or eye via a housing or frame (84). Display lens (82) may comprise one or more transparent mirrors positioned through housing (84) in front of user's eye (20) and configured to reflect projected light into eye (20) and facilitate beam shaping , while also allowing transmission of at least some light from the local environment in an augmented reality configuration (in a virtual reality configuration, it may be desirable for the display system 14 to be able to block substantially all light from the local environment, such as through a dark blackout, blocking curtains, full black LCD panel mode or whatnot). In the depicted embodiment, two wide field of view machine vision cameras (16) are coupled to the housing (84) to image the environment around the user; in one embodiment, these cameras (16) are dual capture visible light/ Infrared camera. As shown, the depicted embodiment also includes a pair of laser scanning shaped wavefront (i.e., for depth) light projections with a display mirror and optics configured to project light (38) into the eye (20) module. The depicted embodiment also includes two miniature infrared cameras (24) paired with infrared light sources (26, such as light emitting diodes "LEDs") configured to track the user's eyes (20) to support rendering and user enter. The system (14) also features a sensor assembly (39) which may include X, Y and Z axis accelerometer capabilities as well as magnetic compass and X, Y and Z axis gyroscope capabilities, preferably at relatively high frequencies such as 200Hz) to provide data. The depicted system (14) also includes a head gesture processor (36), such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), and/or ARM processor (Advanced Reduced Instruction Set Machine), which can A user's head pose is calculated in real-time or near real-time from the wide-field image information output by the capture device (16). Also shown is another processor (32) configured to perform digital and/or analog processing to derive pose from gyroscope, compass and/or accelerometer data from the sensor assembly (39). The depicted embodiment also features a GPS (37, Global Positioning Satellite) subsystem to aid in posture and positioning. Finally, the depicted embodiment includes a rendering engine (34), which may feature hardware running a software program configured to provide rendering information local to the user for the user's view of the world to facilitate operation of the scanner and imaged into the user's eyes. The rendering engine (34) may be operatively coupled (81, 70, 76/78, 80; i.e., via wired or wireless connectivity) to the sensor gesture processor (32), image gesture sensor (36), eye tracking camera (24 ) and a projection subsystem (18) such that the light of the rendered augmented and/or virtual reality object is projected using the scanning laser device (18) in a manner similar to a retinal scanning display. The wavefront of the projection beam (38) can be bent or focused to match the desired focal distance of the augmented and/or virtual reality object. A tiny infrared camera (24) can be used to track the eyes to support rendering and user input (ie where the user is looking, how deeply he is focusing; as discussed below, the edge of the eye can be used to estimate the depth of focus). GPS (37), gyroscope, compass and accelerometer (39) can be used to provide coarse and/or fast pose estimation. Camera (16) images and gestures combined with data from associated cloud computing resources can be used to map the local world, as well as share user views with virtual and augmented reality communities. While most of the hardware in the featured display system (14) is depicted in FIG. , but the depicted hardware components may be mounted or housed within other components, such as a mounting strap component, as shown, for example, in FIG. 3 . In one embodiment, all components of the system (14) characterized in FIG. 14 (except image gesture processor (36), sensor gesture processor (32), and rendering engine (34)) are directly coupled to Communication between the housing (84), and the last three components above, and the remaining components of the system (14) may be through wireless communication (such as ultra-wideband) or wired communication. The depicted housing (84) is preferably head-mountable and wearable by the user. It may also feature speakers (such as those that can be inserted into the user's ears and used to provide sounds to the user that may be associated with an augmented or virtual reality experience, such as the jungle sounds mentioned with reference to FIG. It can be used to capture the user's local voice) as a feature.

With regard to the projection of light (38) into the user's eye (20), in one embodiment, a miniature camera (24) can be used to measure where the center of the user's eye (20) is geometrically heading, generally its Corresponds to the orientation or "depth of focus" of the focus of the eye (20). The three-dimensional surface of all points towards which the eyes tend is called the "binocular field of vision". The focal distance can have a finite number of depths or can be infinitely variable. Projected light from the gaze convergence distance appears to be focused to the subject eye (20), while light in front or behind the gaze convergence distance is blurred. Furthermore, it has been found that spatially coherent light having a beam diameter of less than about 0.7 millimeters is correctly resolved by the human eye, regardless of where the eye is focused; given this understanding, to create the illusion of an appropriate depth of focus, one can Eye gaze convergence is tracked using a tiny camera (24), and the rendering engine (34) and projection subsystem (18) can be used to render all objects at or near the binocular view in focus, and at All other objects are defocused at different degrees (ie, with intentionally created blur). Transmissive light directing light devices configured to project relevant light into the eye may be provided by suppliers such as Lumus Corporation. Preferably, the system (14) renders to the user at a frame rate of about 60 frames per second or higher. As mentioned above, preferably a miniature camera (24) can be used for eye tracking, and the software can be configured to capture not only gaze convergence geometry but focus position cues for use as user input. Preferably such systems are configured with brightness and contrast suitable for day and night use. In one embodiment, such a system preferably has a latency of less than about 20 milliseconds for virtual object alignment, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arcminute, which is about the limit of the human eye . The display system (14) can be integrated with a positioning system, which can involve GPS elements, optical tracking, compasses, accelerometers, and/or other data sources to aid in orientation and pose determination; positioning information can be used to facilitate positioning in the user's view of the relevant world (i.e., such information will facilitate the glasses knowing where they are relative to the real world).

Other suitable display devices include, but are not limited to: desktop and mobile computers, smart phones, software and hardware features may additionally be used to facilitate or simulate three-dimensional perspective (e.g., in one embodiment, the frame may be removably coupled to Smartphone, frame featuring a subset of 200Hz gyroscope and accelerometer sensors, two small machine vision cameras with wide field of view lenses, and an ARM processor - to emulate some of the functionality of the configuration characterized in Figure 14 ), platform computers, tablet computers that can be enhanced as described above for smartphones, tablet computers enhanced with additional processing and perception hardware, head-mounted systems (which use smartphones and/or tablet Computers to display enhanced and virtual viewpoints (visual accommodation via magnification optics, mirrors, contact lenses or light-structuring elements), non-transparent light-emitting element displays (LCD, OLED, vertical cavity surface emitting lasers, steered laser beams, etc. ), transparent displays (which allow humans to see both the natural world and artificially generated images (e.g., light-guiding optical elements, transparent and polarized OLEDs illuminating near-focus contact lenses, controlling laser beams, etc.)), having light-emitting elements Contact lenses (such as those available under the trademark Ioptik RTM from Innovega Corporation of Belleville, Washington, having light-emitting elements; they can be combined with specialized complementary eyewear components), with light-emitting An implantable device for the Element, and an implantable device for an optical receiver that emulates the human brain.

Using systems such as those depicted in FIGS. 3 and 14 , three-dimensional points can be captured from the environment, and the pose of the camera capturing those images or points can be determined (i.e., with vector and/or source orientation information about the world) so that These points or images can be "tagged" with or associated with this pose information. The points captured by the second camera can then be used to determine the pose of the second camera. That is, one can orient and position a second camera based on a comparison with the marked image from the first camera. This knowledge can then be used to extract textures, make maps, and create a virtual copy of the real world (because then there are two cameras around the real world being registered). So at the base level, in one embodiment, you have a person wearing a system that captures a 3D point and generates a 2D image of that point, and those points and images can be sent out to cloud storage and processing resources. They can also be cached locally (i.e., cache labeled images) with embedded pose information; so that the cloud can have ready (i.e., in available cache) labeled 2D images (i.e., labeled with 3D poses), together with 3D points. If the user is looking at something dynamic, he can also send additional information about the motion up to the cloud (for example, if looking at another person's face, the user can get a texture map of that face and at an optimized frequency to push that texture up even though the surrounding world is otherwise largely static).

The cloud system can be configured to save some points as reference points only for poses, to reduce overall pose tracking calculations. In general, it may be desirable to have some silhouette feature to be able to track major items in the user's environment as the user moves around the room, such as walls, tables, etc., and the user may wish to be able to "share" the world and make some other users walk Go into that room and see the dots too. Such useful and critical points can be called "fiducial points" because they are quite useful as anchor points - they are related to features that can be recognized by machine vision and can be seen from different pieces of user hardware consistently and repeatedly. related to the features extracted from the world. Therefore, preferably these benchmarks can be saved to the cloud for further use.

In one embodiment, it is preferable to have a relatively even distribution of fiducials throughout the relevant world, as they are the sorts of items that cameras can easily use to identify locations.

In one embodiment, the associated cloud computing configuration may be configured to periodically collate the database of 3D points and any associated metadata to use the best data from various users for both fiducial refinement and world creation. By. That is, the system can be configured to obtain an optimal data set by using input from various users viewing and performing functions within the relevant world. In one embodiment, the database is inherently fractal - the cloud delivers higher resolution information to such users as they move closer to objects. As the user maps objects closer, this data is sent to the cloud, and if the new 3D point and image-based texture maps are better than those previously stored in the database, the cloud can Add them to the database. All can be configured to happen simultaneously from many users.

As noted above, augmented or virtual reality experiences may be based on recognizing certain types of objects. For example, it is important to understand that certain objects have depth in order to recognize and understand such objects. Recognizer software objects ("recognizers") can be deployed on cloud or local resources to specifically help recognize various objects on one or both platforms as users navigate data in the world. For example, if the system has data for a world model including a 3D point cloud and pose-labeled images, and has a table with a cluster of points on it and an image of the table, it may not be able to ascertain that it is a table as a human would know it is a table. What is observed is (actually) the table. That is, an image showing a large portion of a table at certain three-dimensional points in space and somewhere off in space may not be sufficient to immediately recognize that it is a table that is being viewed. To aid in this identification, a specific object recognizer can be created which will take in the raw 3D point cloud, segment a set of points, and eg extract the plane of the top surface of a table. Similarly, a recognizer can be created to segment a wall from a 3D point, so that the user can change the wallpaper or remove part of the wall in virtual or augmented reality, as well as have an entrance to another room (which in the real world is not actually exists there). Such recognizers operate within the data of a world model, and can be thought of as software "robots" that crawl the world model and imbue that world model with semantic information, or about an ontology that is thought to exist between points in space Argument. Such recognizers or software robots can be configured so that their entire existence is about walking around the relevant world of the data and finding what it thinks is a wall or a chair or other item. They can be configured to mark a set of points as having the functional equivalence of "this set of points belongs to a wall", and can include a combination of point-based algorithms and image analysis of marking gestures for mutually informing systems about what is it.

Object recognizers can be created for many purposes of various utility depending on the viewing angle. For example, in one embodiment, a supplier of coffee, such as Starbucks, may invest in creating an accurate identifier of Starbucks' coffee cups within the relevant world of data. Such a recognizer can be configured to crawl the world of data large and small in search of Starbucks coffee cups, so when operating in the relevant nearby space (i.e., likely, in a Starbucks wholesale store just around the corner , offering coffee to the user while looking at his Starbucks mug for a period of time), they can be segmented out and the user identified. With the cup segmented out, it can be quickly identified when the user moves it across his desk. Depending on the computing resources available, such recognizers may be configured to run or operate not only on cloud computing resources and data, but also on local resources and data, or both cloud and local. In one embodiment, there is a global copy of the world model on the cloud with millions of users contributing to that global model, but for a small world or subworld like a particular individual's office in a particular town, the global Most of the world will not care what the office looks like, so the system can be configured to collate data and move to locally cached information (which is considered most locally relevant to a given user). In one embodiment, for example, when a user walks up to a table, relevant information (such as a segmentation of a particular cup on his table) may be configured to reside only on his local computing resources and not on the cloud, since Objects identified as frequently moving objects (such as a cup on a table) do not have to burden the cloud model and the transfer burden between cloud and local resources. Accordingly, cloud computing resources may be configured to segment three-dimensional points and images, thus separating permanent (i.e., generally non-moving) objects from movable objects, and this may affect where associated data will remain and where processed, removes the processing burden from the wearable/local system for some kind of data about the more permanent object, allows one processing of a location (the location can then be shared with an unlimited number of other users), allows multiple The data source simultaneously builds a database of fixed and movable objects in specific physical locations, and segments objects from the background to create object-specific fiducials and texture maps.

In one embodiment, the system may be configured to ask the user for input regarding the identity of certain objects (e.g., the system may present the user with a question such as "Is that a Starbucks coffee mug?") so that the user may train the system and Allows the system to associate semantic information with objects in the real world. Ontologies can provide guidance on what objects segmented from the world can do, how they behave, etc. In one embodiment, the system may feature a virtual or real keypad, such as a wirelessly connected keypad, connectivity to a smartphone's keypad, or the like, to facilitate some user input to the system.

The system can be configured to share basic elements (walls, windows, table geometry, etc.) to get images and upload those images to the cloud. The cloud then starts populating old and new data sets, and is able to run optimization routines and establish benchmarks that exist on individual objects.

GPS and other positioning information can be used as input to such processing. Additionally, other computing systems and data (such as a person's online calendar or Facebook account information) can be used as input (e.g., in one embodiment, cloud and/or local systems can be configured to to analyze the contents of the user's schedule so that over time, information can be moved from the cloud to the user's local system in preparation for the user's arrival time in a given destination).

In one embodiment, tags such as QR codes and the like can be inserted into the world for use in non-statistical gesture computation, security/access control, communication of special information, spatial messaging, non-statistical object recognition, etc.

In one embodiment, cloud resources may be configured to transfer digital models of real and virtual worlds between users, with modes rendered by individual users based on parameters and textures, as described above with reference to "transferable worlds." This reduces the bandwidth associated with delivering real-time video, allows the rendering of virtual viewpoints of a scene, and allows millions or more users to participate in a virtual collection while sending each other the data they need to view (such as video) , because their views are rendered from their local computing resources.

A virtual reality system ("VRS") may be configured to register a user's position and field of view (together referred to as a "pose") through one or more of: computer vision using real-time metrics from the camera, simultaneous localization and mapping techniques , maps, and data from sensors such as gyroscope, accelerometer, compass, barometer, GPS, radio signal strength triangulation, signal time-of-flight analysis, LIDAR range, RADAR range, odometry, and sonar range. The wearable device system can be configured to simultaneously map and orient. For example, in an unknown environment, the VRS may be configured to gather information about the environment, determine fiducial points suitable for user pose calculations, other points for world modeling, images for providing a texture map of the world. The fiducials can be used to optically calculate the pose. When the world is mapped with more detail, more objects can be segmented out and given their own texture maps, but the world is still preferably represented with simple polygons at low spatial resolution using low resolution texture maps. Other sensors, such as those discussed above, can be used to support such modeling efforts. The world can be inherently irregular in shape, in that moving or otherwise seeking a better view (via viewpoint, "overseeing" mode, zooming in, etc.) requires high-resolution information from cloud resources. Moving closer to objects captures higher resolution data, and this can be sent to the cloud, which can calculate new data and/or insert new data at interstitial positions in the world model data.

Referring now to FIG. 16, the wearable system can be configured to capture image information and extract fiducials and identification points (52). The wearable local system may calculate the gesture using one of the gesture calculation techniques mentioned below. The cloud (54) can be configured to use graphics and fiducials to segment three-dimensional objects from a more static three-dimensional background; the images provide texture maps for objects and the world (textures can be real-time video). Cloud resources (56) may be configured to store and serve static fiducials and textures for world registration. Cloud resources can be configured to manage point clouds of optimal point density for registration. The cloud resource can be configured (62) to use all available points and textures to generate a fractal solid model of the object; the cloud can manage point cloud information for optimal fiducial point density. The cloud resource (64) can be configured to ask the user for training about the density of segmented objects and the world; the ontology database can use the responses to imbue the objects and the world with actionable attributes.

The specific modes of registration and mapping below are characterized by the term "O-pose", which signifies a pose determined from an optical or camera system; Combination of data from instruments, compasses, accelerometers, etc.); and "MLC," which stands for cloud computing and data management resources.

1. Orientation: Make a basic map of the new environment

Purpose: Build pose if environment is not mapped or (equivalent) or if not connected to MLC.

· Extract points from images, track them frame by frame, and use S-pose to measure fiducial points.

Use S-pose because there is no reference point

• Benchmarks based on persistent filtering.

• This is the most basic mode: it will always work for low precision poses. Using little time and some relative motion, it will establish a minimal set of fiducial points for O-pose and/or mapping.

• Get out of this mode once the O-position is reliable.

2. Mapping and O-Pose: Mapping the Environment

Purpose: Build high-accuracy poses, map the environment, and provide maps (with images) to the MLC.

• Compute the O-pose from mature world fiducials. Use S-pose as a check for O-pose solutions and for accelerated computation (O-pose is a non-linear gradient search).

• Maturity benchmarks may come from the MLC, or those benchmarks determined locally.

· Extract points from images, track frame by frame, use O-poses to triangulate fiducials.

· Benchmarks based on persistent filter differences

Provides fiducials and pose-marked images to the MLC

• The last three steps do not need to happen in real time.

3. O-posture: determine the posture

Objective: To establish highly accurate poses in already mapped environments using minimal processing power.

• Use historical S-poses or O-states (n-1, n-2, n-3, etc.) to estimate the pose at n.

• Use the pose at n to project the fiducial into the image captured at n, then create an image occlusion from this projection.

• Extract points from occluded regions (significantly reduces processing burden by searching/extracting points from only occluded subsets of the image).

· Compute O-pose from extracted points and mature world fiducials.

• Use the S-pose and O-pose at n to estimate the pose at n+1.

• Option: Provide pose-tagged images/videos to the MLC cloud.

4. Super Res: Determine super-resolution images and reference points

Purpose: Create super-resolution images and fiducials.

• Composite pose-tagged images to create super-resolution images.

• Use of super-resolution images to enhance estimation of fiducial orientations.

• Iterative O-pose estimation from super-resolution fiducials and images.

• Option: Loop the above steps on the wearable (in real time) or (MLC) (for a better world).

In one embodiment, a VLS system may be configured to have certain basic functions, as well as functions facilitated by "apps" or applications that may be distributed through the VLS to provide certain specific functions. For example, the following applications can be installed to the subject VLS to provide specific functionality.

Application of art drawing. Artists create image transformations that represent the world they see. Users are able to use these transformations and thus see the world "through" the artist's eyes.

Desktop modeling application. Objects that the user "builds" from physical objects that are placed on the table.

virtual presence applications. Users transmit a virtual model of the space to other users, who then use the avatars to move around the space.

Avatar emotion app. Measurement of subtle pitch changes, monitoring head movement, body temperature, heartbeat, etc., animating subtle effects on virtual presence avatars. Digitizes a person's state information and transmits it to a remote avatar using less bandwidth than video. Additionally, such data is mappable to sentient non-human avatars. For example, an avatar of a dog can show excitement by wagging its tail based on the inflection of excitement.

As opposed to sending everything back to a server, an efficient mesh-type network might be desirable for moving data. However, many mesh networks have suboptimal performance because the orientation information and topology are not well characterized. In one embodiment, the system can be used to determine the location of all users with high accuracy, and thus a mesh network configuration can be used for high performance.

In one embodiment, the system can be used for searching. Using augmented reality, for example, users will generate and leave content about many aspects of the physical world. Much of this content is not textual, and thus not easily searchable by typical methods. The system can be configured to provide facilities for tracking personal and social networking content for search and reference purposes.

In one embodiment, if the display device tracks two-dimensional points through successive frames, then a vector-valued function is fitted to the temporal evolution of those points, possibly simplifying at any point in time (e.g., between frames) Or a vector-valued function at some point in the near future (by advancing the vector-valued function forward in time). This allows creating high-resolution post-processing, as well as predicting future poses before the next image is actually captured (e.g. doubling the speed of registration is possible without doubling the camera frame rate).

For body-fixed rendering (as opposed to head-fixed or world-fixed rendering), an accurate view of the body is desired. Rather than measuring the body, it is possible in one embodiment to derive the position of the user's head from its average orientation. If the user's face is pointing forward most of the time, a multi-day average of the head orientation will reveal that direction. Combined with the gravity vector, this provides a reasonably stable coordinate system for body-fixed rendering. Using current measurements of head orientation relative to such a long-duration coordinate system allows consistent rendering of objects on or near the user's body - without the use of additional instruments. For an implementation of this embodiment, the average value of a single register of the head orientation vector may be enabled, and the current sum of the data divided by delta-t will give the current average head orientation. Keeping five or so registers starting at day n-5, n-4, n-3, n-2, n-1 allows the use of rolling averages from the previous "n" days.

In one embodiment, the scene may be zoomed out and presented to the user in a smaller than actual space. For example, where there is a scene that must be rendered in a huge space (ie, such as a football field), there may not be an equivalently large space to render, or such a large space may be inconvenient for the user. In one embodiment, the system can be configured to reduce the scale of the scene so that the user can view it in small form. For example, one might have a god's eye view video game, or World Cup soccer game, played in an unscaled field - or a scaled down field and presented on the living room floor. The system can be configured to only shift the rendered perspective, scale and associated adaptation distance.

The system is also configured to draw user attention to specific items by manipulating the focus of virtual or augmented reality objects, by highlighting them, varying contrast, brightness, scale, etc.

Preferably, the system can be configured to implement the following modes:

Open space rendering:

Grab keypoints from a structured environment, then use ML rendering to fill in between spaces.

• Potential venues: stages, output spaces, large interior spaces (stadiums).

Object wrapper:

· Recognize 3D objects in the real world and then enhance them.

• "Recognition" here means recognizing a three-dimensional object (blob) as an anchor image with sufficiently high accuracy.

• There are two types of recognition: 1) the type of classified object (eg, "face"); 2) the specific instance of the classified object (eg, Joe, a person).

· Create recognition software objects for various things (walls, ceilings, floors, faces, roads, sky, skyscrapers, low-rise buildings, tables, chairs, cars, road signs, billboards, doors, windows, bookshelves, etc.).

• Some markers are Type I, and have generic functionality, eg "Put my video on that wall", "That's a dog".

Other recognizers are Type II, and have specific functions, such as "My TV is 3.2 feet from the ceiling on the wall in _my_ living room", "That's Fido" (which is more capable version of the generic recognizer)

Establishing recognizers as software objects allows metered release of functionality, and finer-grained control of experiences

Body-Centric Rendering

• Render virtual objects fixed to the user's body.

• Something should float around the user's body, like a digital tool belt.

• This requires knowing where the body is, not just the head. Reasonable accuracy in body orientation can be obtained by having a long-term average of the orientation of the user's head (the head is generally pointing forward parallel to the ground).

• The simple case is an object floating around the head.

Transparency / Plane

· For Type II identified objects, a planar view is shown

• Link identified objects of type II to an online database of three-dimensional models.

• One should start with objects that have commonly available 3D models such as cars and public facilities.

virtual presence

· Depicting remote human avatars into open spaces.

ο Subset of "Open Space Rendering" (above).

o Users create a rough geometry of the local environment, and iteratively send both geometry and texture maps to other users.

o Users must grant permission to other users to enter their environment.

ο Subtle sound queues, gesture tracking, and head movements are sent to remote avatars. Animate the avatar based on these fuzzy inputs.

ο Above minimizes bandwidth.

Make the wall an "entrance" to another room

o As with other methods, geometry and texture maps are delivered.

o Instead of showing an avatar in a local room, designate identified objects (eg, walls) as entrances to other environments. In this way, multiple people can sit in their own room and see other people's environment "through" the wall.

virtual viewpoint

Create a dense digital model of an area as a set of cameras (people) view the scene from different viewpoints. This rich digital model can be rendered from any vantage point that at least one camera can see.

• Examples. People at the wedding. The scene is jointly modeled by all participants. The recognizer distinguishes and texture maps stationary objects from moving objects (e.g. walls have a stable texture map, people have a more frequent moving texture map.)

· Scenes can be rendered from any viewpoint using rich digital models that update in real time. Participants in the back can fly through the air to the front row for a better view.

• Participants can show their moving avatar, or have their perspective hidden.

• Participants outside the venue can use their avatars or find "seats" invisibly if the organizer allows.

• Very likely to require very high bandwidth. Conceptually, high-frequency data streams through the crowd over high-speed local wireless. Low frequency data are from MLC.

• It is simple to establish an optimal routing path for the local network since all participants have high accuracy position information.

messaging

· Simple silent messaging may be desired

• For this and other applications, it may be desirable to have a finger chord keyboard.

• Haptic glove solutions can provide enhanced performance.

Full Virtual Reality (VR)

- With the visual system dimmed, shows a view that is not covered in the real world.

· The registration system still needs to be based on the head position.

· "Couch mode" allows the user to fly.

"Walking Mode" re-renders real-world objects as virtual ones, so users don't get confused with the real world.

· Rendering body parts is necessary for suspend disbelief. This means having methods for tracking and rendering body parts in FOV.

• Non-transparent visors are a form of VR that have many image enhancement advantages without the possibility of direct overlay.

Wide FOV, possibly even the ability to see the rear

· Various forms of "super vision": telescope, perspective, infrared, magic eye, etc.

In one embodiment, the system for virtualizing and/or augmenting the user experience is configured such that it can be based at least in part on having input from sources such as speech pitch change analysis and facial recognition analysis (as performed by associated software modules) The data on the wearable device animates the remote avatar associated with the user. For example, referring back to FIG. 12 , facial recognition could be based on a smile on the user's face, or based on a friendly tone of voice or speaking (such as by voice input configured to analyze to a microphone (which can capture voice samples locally from the user) software to determine), making the bee avatar (2) animate to have a friendly smile. Additionally, the avatar character may be animated in such a way that the avatar appears to express some emotion. For example, in an embodiment where the avatar is a dog, a happy smile or tone of voice detected by the human user's local system may be represented in the avatar as an avatar of a dog wagging its tail.

Various exemplary embodiments of the invention are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate the broader applicability aspects of the invention. Various changes may be made and equivalents may be substituted for the invention described without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, combination of substances, process, process action(s) or step(s) to the objective(s), spirit or scope of the invention. Furthermore, as those skilled in the art will appreciate, each of the individual variations described and illustrated herein has discrete components and features that can be incorporated without departing from the scope or spirit of the invention. The and features may be readily separated from or combined with features of any of the other several embodiments. All such modifications are intended to come within the scope of the claims associated with this disclosure.

The present invention encompasses methods that may be performed using the subject device. The method may include an act of providing such suitable equipment. Such provisioning may be performed by end users. That is, the act of "providing" merely requires the end user to obtain, access, approach, locate, set, activate, raise, or otherwise act to provide the equipment necessary for the subject method. The methods described herein can be carried out in any order of events described which is logically possible, as well as in said order in time.

Exemplary aspects of the invention have been set forth above, along with details regarding material selection and fabrication. As for other details of the present invention, these details can be learned in conjunction with the above-cited patents and publications and are known or understood by those skilled in the art. In terms of additional acts as commonly or logically used, the same may apply with respect to the method-based aspects of the invention.

Furthermore, by having described the invention with reference to several examples optionally incorporating various features, the invention is not limited to those aspects described or indicated according to what is contemplated with respect to each variation of the invention. Various changes may be made to the invention as described, and equivalents employed (whether recited herein or not included for reasons of brevity) without departing from the true spirit and scope of the invention. ) to replace. Additionally, where ranges of values are provided herein, it is understood that each intervening value (between the upper and lower limits of that range), as well as any other stated or intervening values in that stated range values are encompassed within the present invention.

Furthermore, it is contemplated that any optional feature of variations of the invention described may be stated and claimed independently or in combination with one or more of the features described herein. References to an item in the singular include the possibility that there may be a plurality of the same item. More specifically, as used herein and in the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural referents unless otherwise clearly stated. That is, in the description above and the claims associated with this disclosure, use of the articles allows for "at least one" of the subject item. It should also be noted that such claims may be drafted to exclude any optional elements. As such, this statement is intended to be used as an antecedent basis for use of such exclusive terms (eg, "only," "only," and the like) in connection with the recitation of claimed elements, or use of a "negative" limitation.

In the absence of the use of such exclusive terms, the term "comprising" in claims associated with the present disclosure shall allow the inclusion of any additional elements - whether or not a given number of elements are recited in such claims, or The addition of features may be considered to alter the nature of the elements recited in such claims. Except as expressly defined herein, all technical and scientific terms used herein are to be given the broadest possible meaning as commonly understood while maintaining claim validity.

The breadth of the present invention is not limited by the examples provided and/or the subject description, but rather only by the scope of the language of the claims associated with this disclosure.

Claims (18)

1. for making two or more users to carry out a mutual system in the virtual world that comprises virtual world data, comprising:

Computer network, it comprises one or more computing equipments, and described one or more computing equipments comprise: storer, treatment circuit and be stored at least in part in described storer and carried out to process by described treatment circuit the software of at least a portion of described virtual world data;

At least first of wherein said virtual world data derives from the first user virtual world of first user this locality, and wherein said computer network is operationally to first described in the user device transmissions for presenting to the second user, make described the second user to experience described first from described the second user's position, make the aspect of the virtual world of described first user be sent efficiently to described the second user.

2. system according to claim 1, wherein said first user and the second user are in different physical locations.

3. system according to claim 1, wherein said first user and the second user are substantially in Same Physical position.

4. system according to claim 1, at least a portion of wherein said virtual world changes in response to the variation of described virtual world data.

5. system according to claim 1, at least a portion of wherein said virtual world changes in response to the physical object by described subscriber equipment perception.

6. system according to claim 5, wherein the described variation in described virtual world data represents that virtual objects and described physical object have predetermined relationship.

7. system according to claim 6, wherein offers the described variation in described virtual world data the second subscriber equipment for presenting to described the second user according to described predetermined relationship.

8. system according to claim 1, wherein said virtual world operationally at least one in computer server or subscriber equipment is played up.

9. system according to claim 1, wherein presents described virtual world with two-dimensional format.

10. system according to claim 1, wherein presents described virtual world with 3 dimensional format.

11. systems according to claim 1, wherein said subscriber equipment operationally provides interface in augmented reality pattern, between user and described virtual world, to carry out alternately for making it possible to.

12. systems according to claim 1, wherein said subscriber equipment operationally provides interface in virtual reality pattern, between user and described virtual world, to carry out alternately for making it possible to.

13. systems according to claim 11, wherein said subscriber equipment operationally provides interface between user and described virtual world, to carry out alternately in the combination of enhancing and virtual reality pattern for making it possible to.

14. systems according to claim 1 are wherein transmitted described virtual world data on data network.

15. systems according to claim 1, wherein said computer network operationally receives at least a portion from the described virtual world data of subscriber equipment.

16. systems according to claim 1, at least a portion that is wherein transferred to the described virtual world data of described subscriber equipment comprises for generating the instruction of at least a portion of described virtual world.

17. systems according to claim 1, are wherein transferred at least a portion of described virtual world data at least one for processing or in distributing of gateway.

18. systems according to claim 17, at least one computer server in wherein said one or more computer servers is operationally processed the virtual world data of being distributed by described gateway.