KR20260018858A - Create interactive and immersive virtual and augmented reality environments that correspond to digital twins of real-world elements. - Google Patents

Abstract

A method and system for generating a digital twin of a live event for an interactive virtual environment. The system comprises a server configured to receive a video data stream and an audio data stream. The server operates a computing module configured to analyze the video data stream and corresponding metadata, determine a spatial depth between images of the video data stream, match objects in each image to a coordinate matrix in a virtual space, match a 3D audio field to the coordinate matrix in the virtual space, and generate a digital twin of the live event based on the images, the corresponding metadata, the spatial depth, the set of predetermined coordinates, and the coordinate matrix. The server is configured to generate a plurality of digital representations derived from a plurality of orientations of the live event, including at least one 360-degree digital representation and a plurality of unidirectional digital representations.

Description

Create interactive and immersive virtual and augmented reality environments that correspond to digital twins of real-world elements.

The present technology relates to electronic systems and methods for creating interactive virtual or augmented reality environments that correspond to digital twins of real-life elements, the real-life elements being particularly relevant to live events, including live events in the performing arts, music, and sports.

Augmented reality (AR) and virtual reality (VR) applications are well-known in this field. Some of these applications use AR technology to enhance real-world experiences with digital overlays. Others digitize real-world elements to simulate them in VR. For example, it is known that digital elements aimed at simulating the laws of physics are integrated into the metaverse, a digital virtual world, allowing users to interact with objects similar to real-world objects. This is well-known in the field of digital games.

However, digitizing large-scale live events presents numerous technical challenges, making it difficult to compute and distribute them to multiple users. Advances in these technologies could provide suitable solutions for the future. In general, numerous patent documents describe various aspects of augmented reality and virtual reality technologies.

For example, U.S. Patent No. 10,650,590, issued May 12, 2020, to Pankaj N Topiwala et al. for “Method and system for fully immersive virtual reality,” teaches a method and system that uses a grid of video sensors spanning an area and extensive signal processing to generate a model-based view of reality. System-level components for grid-based synchronous capture, point cloud generation and refinement, morphology, polygonal tiling and surface representation, texture mapping, data compression, and user-directed signal processing are used to generate a virtualized world viewable from any location within the area, at any time within the capture interval, in any viewing direction, and on user demand. This data stream is transmitted for short-range network-based transmission and 5G. Finally, the virtualized world, because it is inherently model-based, is integrated with augmentation (or subtraction) to create a harmonious, photorealistic blend of the real and synthetic worlds. It provides a fully immersive mixed reality world with full interactivity supported using gestures.

For example, U.S. Patent No. US11,196,964, issued December 7, 2021, to Cevat Yerli for "Merged reality live event management system and method," teaches an accurate and flexible merged reality system and method configured to enable remote viewing and participation in a real or virtual event. In the merged reality system, at least one portion of the real or virtual world can be replicated or streamed into corresponding sub-universes contained within the virtual world system, wherein some of the sub-universes contain events that guests can view and interact with from one or more associated guest physical locations. Other virtual elements, such as purely virtual objects or graphical representations of applications and games, may also be included in the virtual world system. The virtual objects include logic, virtual data, and models that provide their own computing capabilities and autonomous behavior. The system enables guests to virtually visit, interact, and transact within the event via the virtual world system.

For example, U.S. Patent No. US11,202,037, issued December 14, 2021, to Cevat Yerli for "Virtual presence system and method through merged reality," teaches a virtual presence merged reality system comprising a server comprising at least one processor and a memory comprising a data storage that stores a persistent virtual world system comprising one or more virtual replicas of real-world elements. The virtual replicas provide self-computing capabilities and autonomous operation. The persistent virtual world system comprises a virtual replica of a physical location hosting a live event, wherein the persistent virtual world system is configured to communicate over a network with a plurality of connected devices that include sensing mechanisms configured to capture real-world data of the live event to enable updating the persistent virtual world system. The system allows guests to virtually visit, interact, and transact within the live event via the persistent virtual world system. Computer-implemented methods thereof are also provided.

For example, U.S. Patent Application No. US20200401576 to Cevat Yerli, published December 24, 2021, for "Interacting with real-world items and corresponding databases through a virtual twin reality," teaches a system comprising at least one cloud server of a cloud server computer system comprising at least one processor and memory, the memory storing a persistent virtual world system comprising one or more virtual objects comprising virtual data and models. The virtual objects include one or more of a virtual twin, a purely virtual object, or an application, wherein at least one of the virtual objects represents a store of real-world items connected to a periodically updated database associated with products in at least one store. Users can access the store through the persistent virtual world system via a user device that enables interaction with and between elements within the store.

For example, U.S. Patent No. US11,245,872, issued February 8, 2022 to Cevat Yerli for 'Merged reality spatial streaming of virtual spaces' teaches a merged reality system comprising at least one server storing a virtual world system comprising one or more virtual objects, the virtual objects comprising at least a first location, a second location, and virtual replicas of real-world elements within the at least the first and second locations. The at least one server is configured to receive real-world data from real-world elements within the first and second locations from a plurality of connected devices communicating with the at least one server over a network; use the real-world data from the first and second locations in the virtual world system to enrich and synchronize the corresponding real-world elements with the virtual replicas; and overlap and stream at least a portion of the real-world data from the second location onto, for example, one or more surfaces of the virtual replica of the first location.

For example, U.S. Patent No. 1,032,588, issued June 8, 2021, to Haritaoglu et al., for "Method and apparatus for spatially enhanced adaptive bitrate live streaming for 360 degree video playback," discloses a device and method for delivering a spatially enhanced live streaming experience for virtual reality or 360 degree live streaming video. A live streaming video signal is encoded into multiple streams of various resolutions. Portions of the high-resolution video stream corresponding to fields of view within the full 360-degree view are merged with lower-resolution video streams. The resulting video stream is referred to as a spatially adaptive video stream. The multiple spatially adaptive video streams are generated to provide a high-resolution field of view across the full 360 degrees. When a viewer looks in different directions, the video player plays one of the spatially adaptive video streams depending on the direction from which the viewer is looking.

In another example, U.S. Patent Publication No. 2020/0082389, filed September 9, 2019 and published March 12, 2020, to inventor Regev, entitled “Payment system for augmented, mixed, or virtual reality platforms integrated with cryptocurrency wallet,” discloses an improved electronic reality system (e.g., an augmented reality, mixed reality, and/or virtual reality system) that integrates a cryptocurrency wallet to transfer funds between senders and receivers based on interactions performed in the electronic reality system, such as an interaction where an image of an item to be purchased or a representation of funds to be transferred is dragged and dropped from an AR display to an image of a cryptocurrency wallet.

Moreover, companies such as Google Inc. and Apple Inc. are working on devices aimed at enabling users to use virtual and augmented reality applications, such as Google Glass, Apple Glass, the Apple AR/VR Headset, and Magic Leap VR. Other companies are also developing various virtual and augmented reality devices for users. These devices have application programming interfaces (APIs) that give developers access to create compatible virtual and augmented reality applications.

Previous inventions are known to suffer from a number of drawbacks, including a lack of sufficient computing power to host data-heavy virtual reality environments, insufficient flexibility in controlling data traffic, and the high cost of implementing large-scale live events.

What is needed is a method and/or system that addresses one or more of the problems described herein and/or one or more problems recognizable to one of ordinary skill in the art having read the present disclosure.

The purpose of this technology is to provide an efficient method and system for creating interactive virtual or augmented reality environments that correspond to digital twins of real-world elements and/or live events, and for effectively integrating these interactive virtual reality environments into the metaverse. Compared to existing systems and methods, the method and system should be reliable, easy to install, cost-effective, and more accessible to internet users.

Another objective of the present invention is to provide an efficient method and system for adapting the technology to create interactive virtual or augmented reality environments for digitizing live events in real time using spatial web computing. The method and system should be more reliable, easier to install, more cost-effective, and more accessible to Internet users than existing systems and methods.

Therefore, the purpose of the present technology is to improve at least some of the inconveniences present in the prior art.

The present invention relates to generating graphical representations of physical spaces and/or large-scale events from video feeds for the purpose of generating stable digital virtual reality representations, which are capable of handling a sufficient amount of traffic to allow a user to access said digital virtual reality representations and interact with digital elements of the environment generated by said digital virtual reality representations.

Embodiments of the present technology were developed based on the researcher's understanding of at least one technical issue related to conventional approaches for creating and maintaining digital virtual or augmented reality. The engineer considered the possibilities of applying distributed ledger technology, blockchain technology, and automation techniques related to video game environment programming.

For example, currently known conventional systems do not appear to take into account the advancements in modern computing power and the security improvements of blockchain technology.

The above researchers further found that it would be beneficial to provide a system that could utilize several independent server network configurations connected to the same network to generate a digital representation of a spatial environment.

In the context of this specification, unless specifically provided otherwise, the words "first," "second," "third," etc., when used grammatically as adjectives, are used as adjectives solely to allow us to distinguish the nouns they modify from one another, and not to describe any particular relationship between those nouns. Thus, for example, it should be understood that the use of the terms "first" server or "second" server is not intended to imply any particular function, order, type, age, hierarchy, or ranking (for example) of/among said servers, nor is their use (by itself or in combination) intended to imply that there must necessarily be any "first" or "second" server in any given situation. Furthermore, as discussed elsewhere in this specification, reference to a "first" element and a "second" element does not preclude the two elements from being the same actual, real-world element. Thus, for example, in some instances, the "first" or "second" server may be the same software and/or hardware, while in other instances they may be different software and/or hardware.

According to a first broad aspect of the present technology, a system is provided for creating a digital twin in an interactive virtual or augmented reality environment. The digital twin corresponds to a physical live event, for example, a live performance, a live performing arts event, a sporting event, a concert, a public event, a private event, etc. The digital twin may also correspond to a portion of the live event, for example, any real-life element of the live event whose parameters have been recorded and digitized, such as the geometry of a concert hall, an event venue, an entertainment venue, an entertainment park, a stadium, an arena, or a performance by an artist, athlete, or their group, or a portion of an exhibition displayed in the concert hall or stadium, etc. The system comprises a plurality of video capture devices connected to a network. Each video capture device is adapted to operate within a set of predetermined coordinates of the live event. The set of coordinates defines the physical space in which the live event occurs. Each video capture device is adapted to be controlled remotely. The video capture devices may be controlled independently of one another or as a group of devices. The control may be pre-programmed into the system or may be partially or fully controlled by an operator. Each of the video capture devices is configured to: Adapted to capture videos of the above live event. It is understood that the video capture devices may include any device suitable for collecting data that can be digitized to create a graphical representation of the virtual reality environment. For example, the video capture devices may be video cameras, infrared cameras, special (range) measuring cameras, drones, etc. It is understood by those skilled in the art that these specific methods, while within the scope of the present invention, may be developed and modified in the future as technology develops.

For ease of understanding, the term "video" in this specification corresponds to any data that can be captured by a video capture device and used to generate a graphical representation of the virtual reality environment.

Continuing with the first broad aspect of the present technology, each captured video has corresponding metadata. Metadata, as used herein, is data that provides information about other data but does not provide the content of that data. For example, corresponding metadata for a video may include the location of the capture of the video, the type of device used to capture the video, any encoder or library, or other data that may be useful for creating a digital twin based on the captured video, as well as other data that may be associated with the video, such as a description of the artist, athlete, or performance captured in the video.

The system also has a plurality of audio recording devices adapted to record one or more audio tracks of the live event, and to record portions of the live event, such as music, voices, ambient sounds, etc. The plurality of audio recording devices may also be adapted to record not only a 3-D audio field of the live event, but also different intensities of different audio tracks at different locations in the physical space where the live event occurs.

The system also has a server or a plurality of servers configured to receive data from each video capture device and each audio recording device. The server operates a computing module. The module is configured to analyze the captured video and the corresponding metadata, including but not limited to determining a spatial depth between images captured by different video capture devices, matching each captured video to a coordinate matrix of the physical space in which the live event is occurring, matching the 3-D audio field to the coordinate matrix of the physical space, and generating a digital twin of at least a portion of the live event based on the captured video, the corresponding metadata, the spatial depth, the set of predetermined coordinates, and the coordinate matrix. The server is configured to generate a plurality of digital representations derived from a plurality of orientations of the live event, including at least one 360-degree digital representation and a plurality of unidirectional digital representations, each digital representation corresponding to a viewing angle within the digital twin in the interactive virtual reality environment. The server may use any suitable computing technology to determine dimensions of the physical space in which the live event is occurring, determine shapes existing within the physical space in which the live event is occurring as well as distances between each of these shapes, and generate a digital twin comprising a graphical representation of the physical space in which the live event is occurring, shapes existing within the physical space in which the live event is occurring, and distances between each of these shapes. The system is further configured to transmit the digital representation and the audio to a receiving device of a user participating in an interactive virtual reality environment of the live event.

In another aspect of the present technology, the system is further configured to receive data from a user's device to feed data to the system, whereby the data may correspond to the user's interaction with the virtual reality environment, including, for example, data relating to the user's location, the user's movements, and the user's gaze.

The device may be any wearable device with integrated motion capture sensors, artificial intelligence glasses, virtual reality glasses, artificial intelligence earphones for binaural sound, virtual reality headset, mobile phone, computer, laptop, smartwatch, game console, and cross-network device.

It is understood that the user's device is not claimed as part of this patent application, and that any reference to the user, the user's device, or data generated by the user's device and transmitted to the system is not part of the system, and is merely mentioned for ease of understanding without limiting the scope of the claims.

In another aspect of the technology, the module is further configured to determine which real-life elements are suitable to be included in the graphical representation of the digital twin in the virtual reality environment. In this way, the module is configured to determine which real-life elements that may be present in the captured video are to be omitted from the digital representation and which real-life elements are to be retained as part of the digital representation. The module may use various parameters to determine which real-life elements to digitize and retain within the digital twin. For example, the module may determine the user's field of view and retain only those real-life elements that are within the user's line of sight. For example, the module may estimate the computational complexity of digitizing a real-life element to retain within the digital twin, and if the complexity exceeds a certain threshold, the computing module will omit the real-life element, i.e., will not be digitized and will not be added/retained within the digital twin.

In another aspect of the technology, the digital representation is transmitted to the device in real time or at a delayed time after the live event. For example, a live event may be digitized to create a digital twin in a virtual reality environment that users can access in real time. Other events may be digitized to create a digital twin in a virtual reality environment that users can access after the live event has passed, thus creating a persistent virtual reality environment that users can access at any time, allowing users to participate in any part of the live event they wish.

In another aspect of the technology, the digital representation, audio recording, corresponding metadata, and corresponding coordinate matrix data are parsed by the system and stored in a database for at least partial on-demand redistribution. For example, the digital twin in the virtual reality environment may be hosted on an independent server and added to different metaverses running on various protocols. In this way, any portion of the digital twin of the live event may be used to create a virtual reality environment accessible to users of one or more metaverses.

In another aspect of the above technology, the digital representation, audio recording, corresponding metadata, and corresponding coordinate matrix data are stored in a distributed manner using blockchain technology. It is understood that the digital twin may be stored in a distributed or centralized manner, depending on the desired technical characteristics of the virtual reality environment. In some cases, it may be technically advantageous to store the digital twin in a distributed manner using blockchain technology, with the goal of allowing for the decentralized use of resources to increase the computing power needed to create and execute the virtual reality environment of the digital twin of the live event. For example, this may be advantageous in allowing users to access the virtual reality environment of the digital twin of the live event in real time.

In another aspect of the technology, non-fungible tokens are assigned to at least a portion of the digital twin, including the digital representation, the audio recording, the corresponding metadata, and the corresponding coordinate matrix data. For example, a first non-fungible token may be assigned to a first audio recording, a second non-fungible token may be assigned to a second audio recording, a third non-fungible token may be assigned to a third audio recording, a fourth non-fungible token may be assigned to a first video, a fourth non-fungible token may be assigned to a second video, a fifth non-fungible token may be assigned to the first metadata, and a sixth non-fungible token may be assigned to the first coordinate matrix data set. The non-fungible tokens may be generally abbreviated as NFTs.

In some implementations, it is understood that fungible tokens may also be used by the system as a means of payment, exchange, or communication between users or between the system and users.

In another aspect of the technology, the system includes a payment processor adapted to identify a non-fungible token purchased by a user and assign corresponding non-fungible token rights to the user. For example, the payment processor may be integrated with the system, or a third-party payment processor may communicate with the system via a network. The term "payment processor" is used herein in the broadest sense possible and is understood to encompass any technology that operates with monetary instruments and non-monetary instruments such as cryptocurrencies, digital coupons, discount cards, points, etc. The purchase of a non-fungible token (NFT) is understood herein to be understood by those skilled in the art of using non-fungible tokens, and such unique rights may be associated with the NFT, and these rights may be conveyed to its purchaser by any suitable computing protocol, including any suitable blockchain protocol.

In another aspect of the above technology, the system assigns a non-fungible token to the digital twin of the live event. For example, it is possible to assign permissions to the entire digital twin, such that the owner of the NFT of the entire digital twin can control the assignment of additional sub-permissions to portions of the digital twin. For example, it is possible to assign permissions to the entire digital twin, such that the owner of the NFT of the entire digital twin can control the assignment of additional sub-permissions to portions of the digital twin to multiple users, also known as fractionalized ownership.

In another aspect of the technology, the system utilizes artificial intelligence to generate graphical representations. For example, the artificial intelligence algorithm may be used to determine how to approximate the digitized shapes and dimensions of real-world elements to create a digital twin of the live event before assembling the entire digital twin. For example, the artificial intelligence may approximate the shape of the stage, the appearance of the performers, the size of the stadium, the dimensions of the athletes, the seating arrangements of the audience, the dance floor, the mash pit, the crowd, the dining area with tables and chairs, walls, windows, balconies, musical instruments, sports equipment, and more.

In another aspect of the technology, the system generates a data stream that is received by a user device, the data stream instructing the user device to generate an augmented reality effect for the user, the augmented reality effect comprising (a) a graphical projection on a physical element or as a hologram and an audio stream of at least a portion of a digital twin of the live event or real-life element, (b) a graphical representation and an audio stream for the user of at least a portion of the digital twin of the live event or real-life element, or (c) a combination of (a) and (b) as described herein. Thereby, the augmented reality effect generated for the user enables the user device to convey to the user an experience of a physical location that is digitized into the digital twin augmented reality environment.

According to a second broad aspect of the technology, a method for creating an interactive virtual reality environment in a metaverse is provided. The environment comprises a plurality of digital twins of real-world elements digitized into graphical representations. The plurality of digital twins may comprise at least one of a visual element, an audio element, a spatial element, and a tactical element. For example, the visual element may be a graphical representation of captured video, i.e., after the captured video has been processed and digitized into a suitable file format that can be integrated into the metaverse. For example, the visual element may be a graphical representation, such as a video game, or a graphical representation, such as a computer-generated virtual reality. The visual element may be a graphical representation that a user can see through the eyes of their avatar when viewing the metaverse from various angles through their user device. For example, the audio element may be an audio track that a user hears through their user device. The spatial element may be computer code that determines, for example, the distance a user's avatar must travel from one location to another in the virtual reality environment of the metaverse, or the direction in which the avatar can look, throw an object, or hear a sound. A tactile element may be, for example, computer code that generates commands to a user's device to simulate tactile sensations that the user may experience through the user's device interface while interacting with the metaverse. The tactile sensations may be, for example, vibrations, pushes, electrical pulses, temperature differences, etc. The tactile element may be generated by a mechanism integrated into the user's device and may be controlled by the system by sending commands to the device to engage the user, for example, when the user touches a vibrating object in the metaverse, the user's device may vibrate accordingly. The virtual reality environment of the digital twin may be connected to distributed ledger technology to facilitate complex computing tasks and/or communication between the user's device and the metaverse hosting servers. The method comprises the following steps:

(i) A step of creating a digital twin of the above real-life element by executing the following:

(a) capturing video to generate the visual element which is the digital representation, recording audio to generate an audio element which is its corresponding digital audio element, determining tactile characteristics of an object within the digital twin to generate a tactile element, and measuring distances and spatial coordinates between shapes and/or spaces in a virtual reality environment of the digital twin to generate a spatial element, and

(b) recording metadata corresponding to one of the visual element, the audio element, the spatial element and the tactile element;

(ⅱ) a step of storing the digital twin on a server;

(ⅲ) a step of assigning tokens to at least one of the visual element, the audio element, the spatial element and the tactile element;

(ⅳ) a step of connecting the server to a metaverse hosting server;

(ⅴ) a step of integrating the digital twin into the metaverse by creating the virtual reality environment corresponding to the digital twin;

(vi) providing a user device with access to the virtual reality environment for an interactive experience within the virtual reality environment by allowing the user to interact with at least one of a visual element, an audio element, a spatial element, and a tactile element.

In another aspect of the above technology, the step of storing the digital twin on the server further includes the step of storing the digital twin in a distributed manner using blockchain technology.

In another aspect of the above technology, the method further comprises communicating token data with the user device. For example, the token data may be used for user interaction with at least one of a visual element, an audio element, a spatial element, and a tactile element, and/or for authorization management.

In another aspect of the above technology, the token is a non-fungible token.

In another aspect of the technology, the step of providing access to the virtual reality environment to the user's device comprises receiving commands from the user's wearable device having an integrated motion capture sensor, artificial intelligence glasses, virtual reality glasses, artificial intelligence earphones for binaural sound, virtual reality headset, mobile phone, computer, laptop, smartwatch, gaming console and cross-network devices by the metaverse hosting server or by a pre-programmed server.

In another aspect of the technology, the step of measuring distance and spatial coordinates comprises determining at least one of a shape of the visual element, a distance between the first visual element and the second visual element, and a velocity associated with the moving visual element.

In another aspect of the above technology, the method further comprises the step of connecting a payment processor adapted to identify a purchased token and a corresponding authorization of the purchased token.

In another aspect of the technology, the step of providing access to the virtual reality environment comprises the step of allowing the user to launch his or her digital self into any location within the virtual reality environment.

In another aspect of the technology, the method further comprises the step of generating a portion of the metaverse based on a virtual reality environment of the digital twin.

In another aspect of the above technology, the method further comprises the step of generating a plurality of virtual reality environments corresponding to the plurality of digital twins.

In another aspect of the above technology, the method further comprises the step of integrating a plurality of digital twins into the virtual reality environment.

In another aspect of the above technology, the method further comprises selecting real-world elements for digitization within the digital twin. For example, the step may include determining which real-world elements will be digitized and added as digital elements within the virtual reality environment, and which elements that are part of the captured video, recorded audio, measured spatial data, and measured tactile data will be omitted from the digital twin and/or deleted from the final virtual reality environment generated within the metaverse. In another embodiment, the step of selecting real-world elements comprises generating an approximation of the real-world elements, which is represented as a visual element, an audio element, a tactile element, a spatial element, or a combination thereof. The approximation may be coarse or detailed, depending on the desired characteristics of the virtual reality environment.

According to a third broad aspect of the technology, a method is provided, which is operated by a system for generating a 3-D graphical representation in a virtual reality environment. The 3-D graphical representation corresponds to a real-world element or a live event. The system is connected to a network. The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server operates a computing module configured to analyze the captured video, the recorded audio, and the corresponding recorded metadata, as well as to digitize objects that appear in the images of the captured video and are at least partially identified by the corresponding metadata to generate the 3-D graphical representation of the real-world element or the live event. The method comprises the following steps:

a. A step of generating a 3-D graphical representation model based on objects shown in an image of the captured video, based on determining distances between objects shown in an image of the captured video, and based on analyzed corresponding metadata;

b. a step of linking an audio track to an element of the 3-d graphic representation;

c. A step of storing a 3-d graphic representation in the above server;

d. A step of connecting the above server to a metaverse hosting server;

e. A step of integrating the 3-D graphic representation into the metaverse by creating the virtual reality environment corresponding to the 3-D graphic representation;

f. A step of providing access to the virtual reality environment to the user's device.

The method can also generate a coordinate matrix of a physical space corresponding to the real-life element, and generate a 3-D graphical representation of at least a portion of the real-life element.

The method may also determine tactile characteristics of objects depicted in images of the captured video and assign these characteristics to the 3-D graphical representation for future interactions with the user's device or the user's avatar.

The method can also determine spatial coordinates of the 3-D graphical representation and assign these coordinates to objects depicted in the images of the captured video.

The method may also enable an interactive experience within the virtual reality environment by allowing a user to interact with a digital twin of an object depicted in an image of the captured video.

According to a fourth broad aspect of the present technology, a system is provided for generating a digital twin in an interactive virtual reality environment corresponding to at least one real-world element. The system is connected to a network. The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server operates a computing module configured to (a) analyze the captured video, the recorded audio, and the corresponding recorded metadata, (b) determine shapes of objects in the captured video and distances between objects in the captured video, and (c) determine a coordinate matrix of a 3-D virtual space comprising a digital twin of the real-world element.

According to a fifth broad aspect of the present technology, a system is provided for integrating an interactive virtual reality environment corresponding to a digital twin of a real-world element. The system is connected to a network. The system includes a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device. The server operates a computing module configured to (a) analyze the captured video, the recorded audio, and the corresponding recorded metadata to determine a spatial depth between the real-world element captured by the video capture device and recorded by the audio device or recorded by the metadata device, (b) approximate the shapes of the real-world element, (c) determine the real-world element suitable for digitization, and (d) generate a digital twin of at least a portion of the real-world element. The server is configured to transmit the data to a metaverse hosting server to integrate the interactive virtual reality environment into the metaverse.

According to a sixth broad aspect of the present technology, a system is provided for generating a computational space that integrates augmented reality, mixed reality, virtual reality, geofencing, cryptocurrency and other technologies to provide a segmented database of digitized spaces that mimic, dedicate and digitize all events classified as live music events, live performing arts and live sporting events.

In another embodiment of the above technology, a system and method for providing a shared virtual or augmented reality environment is provided, whereby a digital overlay skin of a user's avatar can be viewed by other participants in the shared virtual or augmented reality environment.

In another embodiment of the above technology, a system and method are provided for notifying attendees in an augmented or virtual reality environment of upcoming related events for music, live entertainment and sporting events that include dynamic pop-ups and/or advertisements.

In another embodiment of the above technology, a system and method are provided for integrating an application programming interface into the digital twin that allows communication via a protocol compatible with a "Play To Earn Gaming Framework" integrated into the metaverse.

In another embodiment of the above technology, a system and method are provided that allows participants to purchase event-related collectibles, event tickets, or other items in an augmented or virtual reality environment that includes dynamic pop-ups and/or advertisements.

In another aspect of the technology, the metaverse event may differ from the live event corresponding to the digital twin. The virtual or augmented reality immersive event may be hosted in multiple virtual or augmented reality environments in addition to the virtual or augmented reality environment of the live event venue that is recorded, digitized, and broadcast as the main digital twin of the live event. For example, the digital metaverse event space may have a secondary virtual or augmented reality environment generated by the system (100), which may be represented in the metaverse as another room or another location, where virtual objects related and/or unrelated to the main live event may be presented. For example, such virtual objects may include digital NFT art galleries generated by the methods described herein, whereby these digital NFT art galleries are offsets in the metaverse of the original venue corresponding to the digital twin of the live event virtual or augmented reality environment.

In another aspect of the technology, the technology includes an augmented reality environment configured to interact with a real-world physical environment, such as a building, an urban space, a park, a room, a stadium, etc. The augmented reality environment operates on a server having a computer processor and a database. The server is connected to a network so that the augmented reality environment can communicate with multiple user devices (e.g., smartphones, computers, AR/VR headsets, etc.). The augmented reality environment can communicate with the user devices via any suitable computer program and/or mobile application. The user devices are configured to display what is captured by a camera of the user device, and at least one digital overlay is added to the image by the augmented reality environment. In one embodiment, a camera and/or LiDAR system associated with the user device is operable to generate a point cloud that maps the space. The point cloud provides not only an image of the environment, but also depth and relative angles of objects within the environment for mapping the environment into three-dimensional space. Methods for generating LiDAR point clouds are described in U.S. Patent Publication Nos. 20200/0158869 and 2019/0244378, each of which is incorporated herein by reference in its entirety.

In another aspect of the above technology, the virtual or augmented reality environment is configured to record the profile of a user interacting with it. Each user profile has an associated digital wallet configured to store user-related data, such as preferences, age, gender, information about one or more cryptocurrencies, and/or one or more currencies.

In another aspect of the technology, the virtual or augmented reality environment is associated with a predetermined cryptocurrency configured to be used to purchase goods within the virtual or augmented reality environment, pay for events within the environment, place bets on events and/or games within the environment, and/or access specific areas within the environment. For example, the cryptocurrency may be a cryptocurrency native to the virtual or augmented reality environment and/or the metaverse, and is operable only to be exchanged within the virtual or augmented reality environment. In one embodiment, the area in which the cryptocurrency can be used is defined by at least one geofence.

In another aspect of the technology, different areas of the virtual or augmented reality environment are marked with unique identifiers, such as ID numbers, cryptocurrency tokens, QR codes, barcodes, etc. For example, the identifiers may be images, sounds, three-dimensional objects, point cloud representations, etc.

Data within or associated with a virtual or augmented reality environment is understood to be stored in a centralized or distributed database on a server hosting the virtual or augmented reality environment. Every virtual object within the virtual or augmented reality environment is indexed by a unique identifier. In some embodiments, the unique identifier may be associated with information regarding relative light levels, elevation, location color, location purpose (e.g., game room, store, hallway, etc.), and/or its relative position relative to other unique identifiers. Information associated with the unique identifier is used to determine which digital overlay is provided over the corresponding unique identifier.

In another aspect of the above technology, the virtual or augmented reality environment enables selection of a geolocation as a destination. When the geographic location is selected, the virtual or augmented reality environment automatically determines a point of origin of the selection (e.g., where the user is when the destination geographic location is selected) through the collection of sensor data from at least one geographic location sensor within the user device making the selection. In another embodiment, the point of origin of the selection is determined via received signal strength indication (RSSI) between the user device and a plurality of beacons (e.g., based on WI-FI, BLUETOOTH, WIMAX, etc.) using methods known in the art, such as those described in U.S. Patent Publication No. 2017/0295461, which is incorporated herein by reference in its entirety. Using the selected destination geographic location and the determined origin geographic location, the augmented reality platform automatically generates an optimal route (e.g., a traffic-aware shortest route) between the two locations using any method known in the art, such as those described in U.S. Patent No. 9,886,036, which is incorporated herein by reference in its entirety.

In another aspect of the above technology, geofencing technology is utilized. The geofences described herein are created as described in prior art, such as U.S. Patent Nos. 10,375,514, 10,841,734, 10,834,212, and 10,979,849, each of which is incorporated herein by reference in its entirety.

In another aspect of the technology, the virtual or augmented reality environment recognizes another user based on facial recognition of the user, using any facial recognition technology known in the art, such as described in U.S. Patent No. 9,275,269, which is incorporated herein by reference in its entirety.

The virtual or augmented reality environment operates on a computer system connected to a network and includes a plurality of computing devices, a server, and a database. The server is configured to communicate with the plurality of computing devices via the network. The server typically includes a processing unit having an operating system, as is known in the art. The operating system executes computer programs to create and operationally maintain the virtual or augmented reality environment. The database is typically used to store data, memory, and programs necessary to operate the operating system and to create and maintain the virtual or augmented reality environment.

In the context of this specification, unless specifically provided otherwise, a "server" is a computer program running on suitable hardware that receives requests over a network (e.g., from a device) and performs the requests or causes requests to be performed. The hardware may be a single physical computer or a single physical computer system, but need not be both in the context of the present disclosure. In this context, the use of the expression "server" is not intended to imply that all operations (e.g., received commands or requests) or any particular operation are received, performed, or caused to be performed by the same server (i.e., the same software and/or hardware); it is intended that any number of software components or hardware devices may be involved in receiving/transmitting, performing, or causing to be performed any operation or request, and the results of any operation or request may be included therein; furthermore, all such software and hardware may be one server or multiple servers, both of which are encompassed within the expression "at least one server."

In the context of this specification, unless specifically provided otherwise, a "module" is a computer program that runs on suitable hardware and performs or causes some computational task to be performed. The term "module" has a broader technical meaning than "server" and is not bound to any specific hardware. The term "module" may refer to one or more modules, and is not limited to any combination of modules or hardware used therewith.

In the context of this specification, unless specifically provided otherwise, a "device" is any computer hardware capable of executing software appropriate for the relevant task at hand. Thus, some (non-limiting) examples of devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network equipment such as routers, switches, and gateways. It should be noted that a device operating as a user device in this context is not excluded from operating as a server or module with respect to another device. The use of the expression "device" does not exclude multiple devices used to receive/transmit, perform, or cause to be performed any operation or request, or the result of any operation or request, or any method step described herein.

In the context of this specification, unless specifically provided otherwise, a "database" is any structured collection of data, regardless of its specific structure, database management software, or computer hardware on which the data is stored, implemented, or otherwise rendered available for use. A database may reside on the same hardware as the processor that stores or uses the information contained in the database, or it may reside on separate hardware, such as a dedicated server or multiple servers.

In the context of this specification, unless specifically provided otherwise, the term "information" encompasses any nature or type of information that can be stored in a database. Accordingly, such information includes, but is not limited to, audiovisual works (e.g., images, movies, sound recordings, presentations, etc.), data (e.g., location data, numerical data, etc.), text (e.g., opinions, comments, questions, messages, etc.), documents, spreadsheets, etc.

In the context of this specification, unless specifically provided otherwise, the expression "component" means including software (as appropriate to the particular hardware context) sufficient and necessary to achieve the particular function(s) referenced.

In the context of this specification, unless specifically provided otherwise, the expressions “digital representation” and “graphical representation” are used interchangeably.

In this specification, unless specifically stated otherwise, the expression "computer usable information storage media" is intended to include media of any nature and type, including RAM, ROM, disks (CD-ROM, DVD, floppy disks, hard drives, etc.), USB keys, solid state drives, tape drives, etc.

Each embodiment of the present technology has at least one of the aforementioned objectives and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology resulting from attempts to achieve the aforementioned objectives may not satisfy the aforementioned objectives and/or may satisfy other objectives not specifically recited herein.

Additional and/or alternative features, aspects and advantages of the implementation of the present technology will become apparent from the following description, the accompanying drawings and the appended claims.

For a better understanding of the present technology, as well as other aspects and additional features of the technology, reference is made to the following description to be used in conjunction with the accompanying drawings, wherein:
FIG. 1 is a schematic diagram of a system for communicating with a metaverse and user devices implemented according to one embodiment of the present technology;
FIG. 2 is a schematic diagram of an alternative implementation of a system for communicating with user devices according to one embodiment of the present technology;
FIG. 3 is a schematic diagram of another alternative implementation of a system for communicating with user devices according to one embodiment of the present technology;
FIG. 4 is a schematic diagram of another alternative implementation of a system for communicating with user devices according to one embodiment of the present technology;
FIG. 5 is a block diagram of another alternative method implemented according to one embodiment of the present technology;
FIG. 6 is a block diagram of another alternative method implemented according to one embodiment of the present technology;
FIG. 7 is a block diagram of another alternative method implemented according to one embodiment of the present technology;
FIG. 8 is a block diagram illustrating a method for creating a 3-D digital twin implemented according to one embodiment of the present technology;
FIG. 9 is a block diagram of another method for creating a 3-D digital twin implemented according to one embodiment of the present technology;
FIG. 10 is a block diagram of another method for creating a 3-D digital twin implemented according to one embodiment of the present technology;
FIG. 11 is a block diagram illustrating a method of user interaction with a virtual reality environment implemented according to one embodiment of the present technology;
FIG. 12 is a schematic diagram of a user device implemented according to one embodiment of the present technology.

Reference will now be made in detail to certain specific examples of embodiments of the present invention, including certain modes of carrying out the invention, which the inventors consider suitable for understanding the technology of the present invention. Examples of specific embodiments are illustrated in the accompanying drawings. While the present technology has been described in conjunction with these specific embodiments, it will be understood that the present invention is not intended to be limited to the embodiments described. On the contrary, it is intended to cover alternatives, modifications, and equivalents that may be included within the scope of the present invention as defined by the appended claims.

The techniques and mechanisms of the present technology may be described within the context of the present technology. However, it should be noted that the techniques and mechanisms of the present technology apply to a variety of combinations of aspects and are not limited to merely schematic examples and embodiments. In the following description, specific details are presented to provide a thorough understanding of the present technology. Certain exemplary embodiments of the present technology may be implemented without some or all of these specific details. In other instances, well-known process operations have not been described in detail to avoid unnecessarily obscuring the present technology.

The present disclosure describes flexible virtual and augmented reality systems and methods configured to enable remote generation of fully immersive virtual reality environments and partially immersive augmented reality environments that users can choose to view and/or participate in. The virtual and augmented reality systems are generated by merging the digitization of real-world elements and spaces with the creation of artificial objects within the real-world environment itself. At least one portion of the real or virtual world may be replicated or streamed to a corresponding space in the metaverse, respectively, to create sub-universes and/or sub-spaces of the larger virtual world. The sub-universes and/or sub-spaces may be adapted to host digital twins of live events that users can view and/or interact with from one or more associated user physical locations. Virtual elements, which are purely virtual objects and graphical representations of applications, games, and characters, may be generated by the system or by third-party resources accessible to the virtual world over a network.

Figure 1 illustrates an exemplary embodiment of a system (100). The system (100) is configured to create digital twins of real-world elements and/or live events, digitize them in real time, and create interactive 3D digital representations that are assembled into 3D models that can be integrated into a metaverse and/or sub-universe or virtual space as a virtual or augmented reality environment. The system (100) may be hosted on a single server or a combination of servers. In this embodiment, the system (100) has a device control and data acquisition server (101) and a computing server (106). It will be appreciated by those skilled in the art that the servers (101, 106) may each be a cloud-based computing configuration hosting multiple servers and/or multiple modules, as described in more detail below, and/or a blockchain technology configuration hosted on a distributed ledger configured to execute multiple modules, as described in more detail below.

The above system (100) is connected to a network, for example, the Internet, for wireless or wired communication and for processing by at least one mobile communication computing device. Alternatively, the wireless and wired communications and connections between the devices and components described herein include wireless network communications such as WI-FI, WIMAX (WORLDWIDE INTEROPERABILITY FOR MICROWAVE ACCESS), Radio Frequency (RF) communication including RF identification (RFID), Near Field Communication (NFC), BLUETOOTH including BLUETOOTH LOW ENERGY (BLE), ZIGBEE, Infrared (IR) communication, cellular communications, satellite communications, USB (Universal Serial Bus), Ethernet communications, communications via fiber optic cables, coaxial cables, twisted pair cables, and/or any other type of wireless or wired communications. The system (100) may be a virtualized computing system capable of executing any or all aspects of the software and/or application components presented herein on the device control and data acquisition server (101) and the computing server (106). In certain aspects, the computer The system (100) may be implemented using hardware or a combination of software and hardware, on a dedicated computing device, integrated into another entity, or distributed across multiple entities or computing devices.

The device control and data collection server (101) and the computing server (106) may be any suitable electronic device including at least a processor and a memory, such as a server, a blade server, a mainframe, a mobile phone, a personal digital assistant (PDA), a smart phone, a desktop computer, a netbook computer, a tablet computer, a workstation, a laptop, and other similar computing devices. The components, their connections and relationships, and their functions depicted herein are intended to be exemplary only and are not intended to limit the implementation of the invention described and/or claimed in this application. For example, the device control and data collection server (101) and/or the computing server (106) may include components such as a processor, a system memory having random access memory (RAM) and read-only memory (ROM), and a system bus coupling the memory to the processor (not shown). The above device control and data collection server (101) and the computing server (106) may be configured to include components such as a storage device for storing an operating system and one or more application programs, a network interface unit, and/or an input/output controller (not shown). Each component is operable to be coupled to one another via at least one bus. The input/output controller is operable to receive and process input from or provide output to a number of other devices, including but not limited to alphanumeric input devices, mice, electronic styluses, display units, touch screens, signal generating devices (e.g., speakers), or printers. The processor may be a general purpose microprocessor (e.g., a central processing unit (CPU)), a graphics processing unit (GPU), a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate or transistor logic, a discrete hardware component, or any other suitable entity or combination thereof that can perform computations, process instructions for execution, and/or perform other manipulations of information.

It should be understood that there may be multiple processors, multiple buses, and multiple types of memories (e.g., a combination of DSPs and microprocessors, multiple microprocessors, or one or more microprocessors combined with a DSP core). Multiple computing devices may be connected, each providing a portion of the required operations (e.g., a server bank, a group of blade servers, or a multiprocessor system). Alternatively, some steps or methods may be implemented by circuitry specific to a given function.

The system (100) generates instructions operable to be implemented in hardware, software, firmware, or any combination thereof. A computer-readable medium is operable to provide volatile or non-volatile storage for one or more sets of instructions, such as an operating system, data structures, program modules, applications, or other data implementing any one or more of the methodologies or functions described herein. The computer-readable medium is operable to include memory, a processor, and/or a storage medium, and may be a single medium or multiple media (e.g., a centralized or distributed computer system) that store one or more sets of instructions. A non-transitory computer-readable medium includes any computer-readable medium, the only exception being a transitory radio signal itself. The instructions may be transmitted or received over a network via a network interface unit, which includes any transmission medium operable to include a modulated data signal, such as a carrier wave or other transport mechanism. The term "modulated data signal" means a signal having one or more of its characteristics changed or set in such a manner as to encode information within the signal.

Storage devices and memory include, but are not limited to, volatile and nonvolatile media such as cache, RAM, ROM, EPROM, EEPROM, FLASH memory or other solid state memory technology; disks (e.g., digital versatile disks (DVD), HD-DVD, BLU-RAY, compact disks (CD) or CD-ROM) or other optical storage; magnetic cassettes, magnetic tape, magnetic disk storage, floppy disks or other magnetic storage devices; or any other medium that can be used to store computer readable instructions and that can be accessed by the system (100).

The above device control and data collection server (101) has a live data collection module (105) and a control module (105a). It is understood that the data collection module (105) may be located in one device control and data collection server (101), and the control module (105a) may be located in another device control and data collection server (101).

The control module (105a) controls a plurality of video capture devices (102), a plurality of audio recording devices (103), and a plurality of sensors (104). It should be understood that any number of video capture devices (102), audio capture devices (103), and sensors (104) may exist, from one to hundreds. The number of video capture devices (102), audio capture devices (103), and sensors (104) depends on the physical space and/or real-world elements that need to be captured and digitized. Sometimes, a single video capture device (102), a single audio capture device (103), and a single sensor (104) will suffice. In other cases, a single or multiple video capture devices (102) without any other devices will suffice. In other cases, a single or multiple audio capture devices (103) without any other devices will suffice. The permutations of the video capture device (102), audio capture device (103), and sensors (104) will depend on the amount of data stream that may be required to digitize the live event or real-world elements. The video capture device (102) may include any type of camera, such as a video camera, an infrared camera, an ultraviolet camera, etc. The audio capture device (103) may include a device that may be used to map the space and/or record any type of sound produced at the live event. The sensors (104) may include motion sensors, position sensors, proximity sensors, thermo sensors, accelerometers, etc. The sensors (104) are typically used as a source of auxiliary data streams that may help identify the type, shape, and distance and coordinates of objects in the physical space where the live event occurs, as well as the location and movement of performers, athletes, decorations, equipment, etc., and also include luminance, texture, temperature, and other characteristics of the objects that are to be digitized for inclusion in a 3D model of the virtual or augmented reality environment.

The video capture device (102), the audio recording device (103), and the sensor (104) are remotely controlled by the control module (105a). Typically, the video capture device (102), the audio recording device (103), and the sensor (104) each have a predetermined location in the physical space where the live event occurs, have known coordinates within the physical space, have a known recording range, have a known recording angle (e.g., field of view), have a known sensitivity, have a known motion range, and so on. These known parameters are combined herein as corresponding metadata. In this way, the video capture device (102), the audio recording device (103), and the sensor (104) each have metadata corresponding to that particular device. The video capture device (102), the audio recording device (103), and the sensor (104) are each calibrated for each live event, and these calibration parameters are also part of the metadata.

The live data collection module (105) of the device control and data collection server (101) is configured to receive data streams from the video capture device (102), the audio recording device (103), and the sensor (104) during the recording of a live event. In some embodiments of the present technology, different configurations of these devices may be arranged to record data of a live event. For example, the video capture device (102) may be integrated with the audio recording device (103) and the sensor (104). As such, it may be a single device, such as a smartphone, a video camera, or a drone, that records video, audio, position, movement, acceleration, etc. In other configurations, a video camera that records video and audio may be present, but its position and its movements within the physical space of the live event venue may be measured by a separate sensor (104), which may be positioned on a tripod or drone, or otherwise attached to a device that moves the camera. In other configurations, the video camera movements may be pre-recorded, and the video camera will move along a specific trajectory. In other configurations, the audio may be recorded by an audio recording device (103) independent of the video capture device (102) and the sensor (104), for example, such an audio recording device (103) may be a microphone attached to a performer at a live event. It is understood that there may be multiple audio recording devices (103), such as microphones, as there may be sources of sound, such as musical instruments, performers, audience members, etc.

Those skilled in the art will appreciate that any number of configurations of the video capture device (102), audio recording device (103), and sensor (104) are possible and are within the scope of the present technology. Regardless of such configuration, it is understood that the live data collection module collects a sufficient number of data streams that may include captured video, recorded audio, and sensor data (e.g., luminance, motion, position, acceleration, temperature) to generate a 3D model of the physical space of the live event venue and/or the performers/athletes/audience/staff participating in the live event.

The live data acquisition module (105) receives all data streams in a digital format suitable for processing by the system (100). The data streams are indexed by the system (100) and then stored in the live data acquisition module (105). Each individual data stream is identified by all necessary information known to those skilled in the art, such as, for example, file type, file size, information about the device that recorded the data, corresponding metadata, etc. The information for identifying the data streams depends on the computer model used by the computing server (106) to analyze the data streams and generate a 3D model of the live event physical space with real-world elements integrated into the 3D model to create a digital twin of the virtual or augmented reality environment.

The computing server (106) receives data from the device control and data collection server (101). The computing server (106) can process the received data using the following two approaches:

(a) the computing server (106) will supplement the data received from the live data collection module (105) with data/information located in the stored data module (107) as illustrated in FIGS. 1, 2 and 3, and the computing server (106) will apply one or more techniques to the data of the live data collection module (105) supplemented by the data stored in the stored data module (107) to create a 3D model and a digital twin of the live event; or

(b) The computing server (106) will not supplement the data received from the live data collection module (105) with information located in the stored data module (107) as illustrated in FIG. 4, and the computing server (106) will only apply one or more techniques for generating the 3D model and the digital twin of the live event to the data received from the live data collection module (105).

Technologies for creating digital twins of 3D models and live events include at least one of the following:

(a) Image-based modeling and rendering techniques (IBMR) can be computed in the image-based rendering module 108, the method of which is described in Manuel M. Oliveira, Image-Based Modeling and Rendering Techniques: A Survey. Instituto de Informatica, UFRGS, Caixa Postal 15064, CEP 91501-970, Porto Alegre, RS, Brasil; Heung-Yeung Shum and Sing Bing Kang, A Review of Image-based Rendering Techniques. Microsoft Research, each of which is incorporated herein by reference in its entirety.

(b) Photogrammetry and close range photogrammetry techniques can be computed in the photogrammetry module 109, and the methods are described in the book Close Range Photogrammetry; Principles, techniques and applications by T Luhmann, S Robson, S Kyle and I Harley, published by Whittles Publishing, Dunbeath Mains Cottages, Dunbeath, Caithness KW6 6EY, Scotland, UK, 2006, ISBN 1-870325-50-8; Surendra Pal Singh, Kamal Jain, V. Ravibabu Mandla, A new approach towards image based virtual 3D city modelling by using close range photogrammetry, Department of Geodesy, Department of Civil Engineering, Indian Institute of Technology, Roorkee, the entire contents of each of which are herein incorporated by reference.

(c) Hybrid techniques include multi-view 3D reconstruction techniques based on SFM, combinations of IBMR and close-range photogrammetry, laser scanning techniques, computer vision techniques, etc., which can be computed in the hybrid module 110, and these methods are at least partially described in the following documents, each of which is incorporated herein by reference in its entirety.

a. Lei Gao, Yingbao Zhao, Jingchang Han, Huixian Liu. Research on Multi-View 3D Reconstruction Technology Based on SFM. School of Electrical Engineering. Department of Electrical Engineering, Hebei University of Science and Technology, Shijiazhuang 050018, China;

b. Shen, XL; Dou, Y.; Mills, S.; Eyers, DM; Feng, H.; Huang, Z. "Distributed sparse bundle adjustment algorithm based on three-dimensional point partition and asynchronous communication". Front. Inf. Technol. Electron. Eng. 2018, 19, 889-904;

c. Crosilla, F.; Beinat, A.; Fusiello, A.; Maset, E.; Visintini, D. Basics of computer vision. In Advanced Procrustes Analysis Models in Photogrammetric Computer Vision; Springer International Publishing: Cham, Switzerland, 2019;

d. DeTone, D.; Malisiewicz, T.; Rabinovich, A. Superpoints: Self-supervised interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, Utah, USA, June 18–23, 2018; pp. 224–236.

e. Zhu, S.; Shen, T.; Zhou, L.; Zhang, R.; Wang, J.; Fang, T.; Quan, L. Parallel structure from motion from local increment to global averaging. arXiv 2017, arXiv:1702.08601;

f. Schonberger, J.L.; Hardmeicr, H.; Sattler, T.; Pollefeys, M. Comparative evaluation of hand-crafted and learned local features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 21–26, 2017; pp. 1482–1491;

g. Bian, J.W.; Lin, W.Y.; Matsushita, Y.; Yeung, S.K.; Nguyen, T.D.; Cheng, M.M. GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 21–26, 2017; pp. 4181–4190;

h. Sweeney, C.; Sattler, T.; Hollerer, T.; Turk, M.; Pollefeys, M. Optimizing the viewing graph for structure-frommotion simplification. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, December 7–13, 2015;

i. Wilson, K.; Snavely, N. Robust global translation using 1dsfm. In Proceedings of the European Conference on Computer Vision (Springer); Springer: Cham, Switzerland, 2014;

j. Sweeney, C.; Fragoso, V.; Hollerer, T.; Turk, M. Large-scale SfM using distributed camera models. In Proceedings of the 2016 4th International Conference on 3D Vision (3DV), Stanford, CA, USA, October 25–28, 2016; pp. 230–238;

k. Schonberger, J.L.; Frahm, J.M. Rethinking architecture in motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, June 27–30, 2016;

l. Yao, Y.; Luo, Z.; Li, S.; Fang, T.; Quan, L. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September 8–14, 2018; pp. 767–783;

m. Knapitsch, A.; Park, J.; Zhou, Q.Y.; Koltun, V. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph(Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans). 2017, 36, 78-90;

n. Berger, M.; Tagliasacchi, A.; Seversky, L.M. A survey of surface reconstruction from point clouds. Comput. Graph. Comput. Graph. Forum 2017, 36, 301-329;

o. Zhu, S.; Zhang, R.; Zhou, L.; Shen, T.; Fang, T.; Tan, P.; Quan, L. Very large-scale global SfM by distributed motion averaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, June 18–23, 2018;

p. Kasten, Y.; Geifman, A.; Galun, M.; Basri, R. GPSFM: Global projective sfm using algebraic constraints on multi-view fundamental matrices. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June 15–20, 2019;

q. Liu, H.; Zhang, G.; Bao, H. Robust keyframe-based monocular SLAM for augmented reality. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality, ISMAR Adjunct 2016, Mérida, Mexico, September 19–23, 2016;

r. Ke, T.; Roumeliotis, S.I. An efficient algebraic solution to the perspective-three-point problem. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 21–26, 2017; pp. 7225–7233.

s. Cefalu, A.; Haala, N.; Fritsch, D. Hierarchical structure from motion combining global image orientation and structureless bundle adjustment. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2017, 42, 535–542;

t. Wang, R.; Lin, J.; Li, L.; Xiao, Z.; Hui, Y.; Xin, Y. A revised orientation-based correction method for SfM-MVS point clouds of outcrops using ground control planes with marks. J. Struct. Geol. 2021, 143, 104266;

u. Dugan, U.; Sangsoo, L. Microscopic structure from motion (SfM) for microscale 3D surface reconstruction. Sensors (Microscopic structure from motion (SfM) for microscale 3D surface reconstruction. Sensors) 2020, 20, 5599;

v. Khalil, M.; Ismanto, I.; Fu'ad, M.N. 3D reconstruction using structure from motion (SFM) algorithm and multi-view stereo (MVS) based on computer vision. IOP Conf. Ser. Mater. Sci. Eng. 2021, 1073, 012066;

w. Mali, V.K.; Venu, P.; Nagaraj, M.K.; Kuiry, S.N. Demonstration of structure-from-motion (SfM) and multi-view stereo (MVS) close-range photogrammetry technique for scour hole analysis. Sa dhana 2021, 46, 227;

x. Yu, Q.; Yang, C.; Wei, H. Part-wise AtlasNet for 3D point cloud reconstruction from a single image. Knowl.-Based Syst. 2022, 242, 108395;

y. Mur-Artal, R.; Tardos, J.D. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans.Robot. 2017, 33, 1255-1262.

The computing server (106) also has an artificial intelligence computing module (111) (AI computing module (111)) that can be used to enhance data received from the live data collection module (105) with information that facilitates identification of objects, shapes, and depths of real-life elements captured in images of live events received from the video capture device (102). The AI computing module (111) can function in conjunction with the stored data module (107) to target and match similar real-life elements captured in images from the data stream of the video capture device (102) with known sample digitized objects within a library of the stored data module (107). For example, the shape and form of a particular chair captured as an image by the video capture device (102) and processed by the live data acquisition module (105) may be targeted and matched to the shape and form of a similar chair in the stored data module (107), since the dimensions, textures and digital representations of these chairs were stored in the stored data module (107) prior to the capture of the live event by the video capture device (102), thereby facilitating the processing of the digitization of the chair in the 3D model of the digital twin.

Figure 1 illustrates that the computing server (106) has a 3D model rendering module (112). The 3D model rendering module (112) creates the digital twin based on a digital representation of the live event venue itself as well as digital representations of real-life elements present at the live event venue.

After the digital twin is generated by the computing server (106), the system (100) generates a virtual or augmented reality environment, and the virtual or augmented reality environment is then communicated to the metaverse of the metaverse server (113) or configured within the metaverse. The metaverse server (113) may be configured as a server that operates as a centralized database or a disturbed ledger.

The metaverse server (113) communicates with the user device (114) to provide a virtual or augmented reality environment of the digital twin of the live event to the user device (114).

In some embodiments, the virtual or augmented reality environment of the digital twin of the live event may be stored on the metaverse server (113). In other embodiments, only a portion of the virtual or augmented reality environment of the digital twin of the live event may be stored on the metaverse server (113), and other portions of the virtual or augmented reality environment of the digital twin of the live event are uploaded to the metaverse server (113) following a request between the system (100) and the metaverse server (113).

FIG. 2 illustrates an embodiment in which a system (100) includes a video capture device (102), an audio recording device (103), and a sensor (104). In this embodiment, the system (100) has an integrated set of video capture devices (102), audio recording devices (103), and sensors (104), which may be located at a given location, such as a stadium, and which is designed to create a 3D model of the event there by creating a digital twin of the live event occurring there.

FIG. 3 illustrates an embodiment of a system (100) that creates a virtual or augmented reality digital twin of a live event hosted within the system (100) and enabling the user device (114) to communicate with the system (100) via the communication module (115).

It is understood that the system (100), the metaverse server (113), the communication module (115), and the user device (114) communicate via a network, which may be the Internet, Ethernet, or the like. The video capture device (102), the audio recording device (103), and the sensor (104) may also communicate with the control and data collection server (101) via a network or in any other manner known in the art.

Figure 4 illustrates an exemplary embodiment that does not have a stored data module (107). In this embodiment, the AI computing module (111) can access any additional data that may be required to generate a 3D model via the Internet.

Figure 5 illustrates an exemplary embodiment of a method (500) for creating a digital twin for creating a virtual or augmented reality environment according to the present technology. The method (400) comprises the following exemplary steps:

a) capturing video and generating the visual element, recording audio and generating the audio element, determining tactile characteristics and generating the tactile element, and measuring distance and spatial coordinates and generating the spatial element;

b) recording metadata corresponding to one of the visual element, the audio element, the spatial element and the tactile element;

c) a step of creating a digital twin of the above real-life element;

d) a step of storing the digital twin on a server;

e) assigning tokens to at least one of the visual element, the audio element, the spatial element and the tactile element;

f) Step of connecting the server to the metaverse hosting server;

g) a step of integrating the digital twin into the metaverse by creating a virtual reality environment corresponding to the digital twin; and

h) providing the user device with access to the virtual reality environment corresponding to the digital twin.

Figure 6 illustrates an exemplary embodiment of a method (500) for creating a virtual or augmented reality environment with multiple digital twins of real-world elements to which cryptocurrency tokens are assigned. The virtual or augmented reality environment of the method (500) is integrated into a metaverse that provides access to multiple user devices. The method comprises the following steps:

a) receiving a digital video stream having corresponding metadata, receiving a digital audio stream having corresponding metadata, and receiving supplementary data from a sensor;

b) generating (1) from images in said video stream that determine a video element, (2) from audio stream audio elements, and (3) from a combination of data tactile elements and spatial elements;

c) creating digital twins of a plurality of real-life elements captured from images of the video stream, and assigning corresponding video elements, audio elements, tactile elements, and spatial elements to each digital twin;

d) a step of storing references to each digital twin created in a distributed manner in the blockchain;

e) assigning cryptocurrency tokens to at least one digital twin visual;

f) creating a virtual or augmented reality environment hosting the digital twin;

g) a step of integrating a virtual or augmented reality environment into the metaverse; and

h) A step of providing access to the virtual reality environment corresponding to the user device.

The method (600) assigns cryptocurrency tokens to different virtual elements of the virtual or augmented reality environment. In this specification, a digital twin refers to a virtual object, which may include any of the following: a digitized portion of a live performance, a digitized song or act, a specific set of seats in a digitized performance, specific angles of view in a digitized performance, a specific object that may be a digital twin of a real-world object or a purely virtual object such as a dragon, an avatar, a fireball, etc., a sequence of digitized images of the performance, a specific digital space corresponding to an actual physical space at a live event venue, and any other virtual object within the virtual or augmented reality environment. The cryptocurrency tokens may be fungible tokens or non-fungible tokens.

In some embodiments of the system (100), cryptocurrency payment processing systems are also integrated into a generated virtual or augmented reality environment, allowing a user to interact with a user device to obtain a digital twin of a cryptocurrency token within the virtual or augmented reality environment, as well as the actual virtual or augmented reality environment itself. To facilitate this, the system (100) assigns permanent or semi-permanent references to each digital twin within the virtual or augmented reality environment of a blockchain. Those skilled in the art will appreciate that the system (100) can assign references to any blockchain currently in existence or that may exist in the future. Assigning the references includes providing unique identifiers to the digital twin of a given blockchain.

The virtual or augmented reality environment may also be hosted on a distributed ledger and fully operational on blockchain technology. The computing power of the computing server (106) would then be distributed across multiple machines, each performing a separate task that would be recorded on the blockchain and accessible by the system (100) upon request.

FIG. 7 illustrates a method (700) for enabling a user to communicate token data via a blockchain between a user device and a virtual or augmented reality environment within a metaverse after the virtual or augmented reality environment has been created by the system (100). It is understood that many metaverses and many virtual or augmented reality environments may only allow the user to see and hear the digital twin within the virtual or augmented reality environment, thereby limiting other interactions between the user and the metaverse or virtual or augmented reality environment. In some cases, this may optimize the computing power of the communication module (115) that communicates with the server (113) hosting the metaverse or the user device (114).

FIG. 8 illustrates an exemplary embodiment of a method (800) for creating a digital twin of a virtual or augmented reality environment of a live event, wherein the system (100) initially creates a 3D model of the live event venue with digital representations of an appropriate number of real-world elements of the venue and virtual objects integrated into the virtual or augmented reality environment by the system (100) or a metaverse server (113). In some cases, it may be useful to first create the entire 3D virtual or augmented reality environment of the live event, and then add audio elements to the virtual or augmented reality environment. The audio elements may be added differently to different parts of the virtual or augmented reality environment of the live event, for example, sounds may be quiet in some locations of the virtual or augmented reality environment of the live event, while others may be louder. Additionally, the audio may be segmented into different tracks, whereby each track corresponds to a separate source of sound. As exemplified by the method (800), the audio track corresponding to an audio element within the virtual or augmented reality environment of the live event may be added after the digital twin is created. In some embodiments, the actual file containing the audio track may not be hosted on the server of the system (100) and may be connected to it via a network.

In some embodiments, the system (100) may store captured video data streams, audio data streams, and data streams from sensors, and in other embodiments, the system (100) may use these data streams to create a 3D model of a virtual or augmented reality environment of a live event, and it is understood that the actual video, audio, and sensor data may be stored on a separate server, and the system (100) may connect its computing module (106) thereto to create real-world elements and digital representations of the live event.

FIG. 9 illustrates an exemplary embodiment of a method (900) for creating a 3D model digital twin of a real-life element by the computing server (106), comprising the following steps:

a) receiving video, audio, and sensor data streams;

b) Live data processing stage;

c) Depth/range processing stage;

d) AI analysis and data enhancement processing steps;

e) Mesh generation step;

f) Point cloud creation step;

g) 3D model approximation step;

h) Viewpoint creation step;

i) Compression/transmission stage

It will be understood that one skilled in the art can modify the above method (900) to remain within the scope of the present technology.

FIG. 10 illustrates an exemplary embodiment of a method (1000) for generating a 3D model of a digital twin of a real-life element or a live event venue from a multi-view image based on a multi-view 3D reconstruction technique.

FIG. 11 illustrates an exemplary embodiment of a method (1100) of a user device interacting with a virtual or augmented reality environment. It is understood that any suitable model of user device interaction with the virtual or augmented reality environment is within the scope of the present technology.

FIG. 12 illustrates an exemplary embodiment of a user device (1200). The user device (1200) is illustrated as a virtual reality headset, but the user device may also include a smartphone, smart glasses, a computer, a tablet, a gamepad, etc. (not illustrated herein).

In some embodiments, it is understood that the system (100) may have a 3D model of the physical space of the venue where the live event takes place stored in the stored data module (107). As such, the system (100) will only capture video, audio, and sensor data streams related to actual performances occurring at the venue. The device control and data collection server (101) will operate the live data collection module (105) to capture video, audio, and sensor data streams of performers, athletes, and/or decorations and animations present at the live event venue. The computing server (106) will then match the 3D model of the digital twin of the physical space of the venue stored in the stored data module (107) with the digital twin generated by any one of the image-based rendering module (108), the photogrammetry module (109), and/or the hybrid module (110). The AI computing module (111) may be activated to facilitate and/or enhance compatibility between a pre-stored 3D digital twin of the physical space and a generated 3D digital twin of the live event. In some embodiments, the use of the AI computing module (111) may be omitted. The 3D model rendering module (112) generates a final 3D model of the digital twin corresponding to the physical space of the location and the physical space of the live event, which creates a virtual or augmented reality environment.

The computing server (106) allows the system (100) to generate the virtual or augmented reality environment with sufficient accuracy in real time or near real time by incorporating the ability to determine which real-life elements of the live event will be digitized, which will not be digitized by the system (100), and which will be digitized by the system (100) during the course of the live event and integrated into the virtual or augmented reality environment at a given time. This technique can be applied to selectively identify real-time elements that require a large computational load to be digitized. For example, fireworks, fires, fountains, etc., can be omitted from being digitized entirely and instead replaced with virtual objects from a library of virtual objects stored in the stored data module (107) or available via the Internet.

In some embodiments of the present technology, the system (100) can generate a 3D model of a digital twin of a large physical location, such as a stadium or concert hall, for a live event. The generated virtual or augmented reality environment will virtually correspond to the size of the location. When a user places their avatar in the virtual or augmented reality environment, the user can experience the live event from the location of their avatar. Multiple users may have their avatars in the virtual or augmented reality environment, and each user may have different permissions assigned to them for interacting within the virtual or augmented reality environment. To determine which permissions a user has and how they can interact with the virtual or augmented reality environment, the system (100) can generate virtual geofences around each avatar of each user or group of users. The virtual geofences can also be generated at different locations within the virtual or augmented reality environment, whereby each location may have similar or different permissions assigned to it. Additionally, the geofences can be associated with cryptocurrency tokens. In this way, the geofence can allow users to purchase or exchange virtual objects within the geofence using the token. Additionally, to enter a specific geofence, the user may be required to purchase or exchange cryptocurrency tokens, similar to what a regular user might do in real life, for example, to enter a VIP area at a live event.

It is understood that the system (100) may utilize any suitable geofencing technology or equivalent, including but not limited to the use of a global positioning system (GPS) satellite network and/or local radio frequency identifiers (such as Wi-Fi nodes or Bluetooth beacons), to create a virtual boundary around a location. The virtual or augmented reality environment geofence may be created using the coordinate system of the metaverse, as well as the matrix and other types of boundary calculators. The geofence may then be paired with a software application on the user device or with a cryptocurrency token that responds to the boundary in some manner, such as dictated by parameters of the metaverse or the virtual or augmented reality environment.

The system (100) may be designed to operatively record a live event, create a digital twin thereof, and transmit the data to the metaverse for the purpose of AR/VR/mixed reality broadcasting. Thus, examples of user devices (1200) include devices that enable AR/VR/mixed reality broadcasting. The system (100) and methods described herein provide a technical solution for the combination of attaching a live event broadcast or portion of a broadcast to a digital twin and creating a virtual reality or augmented reality environment therewith.

In some embodiments, the systems and methods described herein enable the attachment of digital twins and broadcasts, or portions thereof, to a blockchain using geofencing or equivalent techniques, along with possible combinations of cryptocurrency technologies, including NFTs. This allows the system (100) to integrate any suitable combination of "Play to Earn" functionality for use in a metaverse live event into a virtual reality or augmented reality environment. It is understood that any combination of NFTs attached to virtual objects integrated into the virtual reality or augmented reality environment of the live event may be present.

In some embodiments, the live data collection module (105) is configured to operate within a WEB3 framework and generate data suitable for effective transmission to the metaverse.

In some embodiments, the blockchain technology is used by the system (100) to enable cataloging and redistribution of virtual objects within the virtual reality or augmented reality environment during or after a live event.

As described above, the system (100) may use any suitable technique for 3D modeling to create a digital twin of a real-life element.

The various techniques and mechanisms of the present technology will sometimes be described in the singular for clarity. However, it should be noted that some embodiments include multiple iterations of the technique or multiple instantiations of the mechanism, unless otherwise noted. For example, the system uses a processor in various contexts. However, it will be appreciated that the system could utilize multiple processors, cloud computing, distributed ledger technology, multi-core processors, video cards or graphics accelerators, quantum computers, or any combination thereof, and still remain within the scope of the present invention. Furthermore, the techniques and mechanisms of the present technology will sometimes describe a connection between two entities. It should be noted that a connection between two entities does not necessarily imply a direct, unobstructed connection, as various other entities may exist between the two entities. For example, a processor may be connected to memory, but it will be appreciated that various bridges and controllers may exist between the processor and the memory. Consequently, unless otherwise noted, a connection does not necessarily imply a direct, unobstructed connection.

Although numerous specific details are set forth in the above description, embodiments of the present invention may be practiced without these specific details. Well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring the understanding of this description. Terms such as "embodiment," "various embodiments," and the like indicate that the described embodiment(s) may include certain features, structures, or characteristics, but not all embodiments necessarily include the particular features, structures, or characteristics. Some embodiments may not have some, all, or all of the features described with respect to other embodiments. "Connected" may indicate that elements are in direct physical or electrical contact with one another, and "coupled" may indicate that elements cooperate or interact with one another, but may or may not be in direct physical or electrical contact. Furthermore, while similar or identical numbers may be used to designate the same or similar parts in different drawings, doing so does not imply that all drawings including similar or identical numbers constitute a single or identical embodiment.

Those skilled in the art will understand that when the present description refers to "receiving data" from a user, the electronic device that receives the data from the user can receive an electronic (or other) signal from the user. Those skilled in the art will also understand that displaying the data to the user via a user-graphical interface (such as a screen of the electronic device) can include sending a signal to the user-graphical interface, and that the signal containing the data can be manipulated and at least a portion of the data can be displayed to the user using the user-graphical interface.

Some of these steps and signal transmission-reception are well known in the art and, as such, have been omitted from certain portions of this description for brevity. The signal may be transmitted-received using optical means (e.g., fiber optic connections), electronic means (e.g., using wired or wireless connections), and mechanical means (e.g., pressure-based, temperature-based, or any other suitable physical parameter-based).

Modifications and improvements to the aforementioned implementations of this technology will become apparent to those skilled in the art. The foregoing description is intended to be illustrative rather than restrictive. Accordingly, the scope of this technology is intended to be limited only by the scope of the appended claims.

Claims (34)

A system for creating a digital twin in an interactive virtual reality environment,
The above digital twin corresponds to a live event,
The above system:
A plurality of video capture devices connected to a network, each video capture device being configured to capture video of the live event, each captured video having corresponding metadata;
A plurality of audio recording devices for recording audio of the above live event;
A server configured to receive data from each video capture device and each audio recording device;
Including,
The above server operates a computing module,
The above module:
Analyze the captured videos and the corresponding metadata,
Determining the spatial depth between images captured by different video capture devices,
Match each captured video to a coordinate matrix in virtual space,
Matching the above audio to the coordinate matrix of the virtual space,
configured to generate a digital twin of at least a portion of the live event based on the captured videos, the corresponding metadata, the spatial depth, a set of predetermined coordinates, and the coordinate matrix;
The above server,
Generating a plurality of digital representations from multiple orientations of the live event, including at least one 360-degree digital representation and a plurality of unidirectional digital representations, each digital representation configured to correspond to a viewing angle within the digital twin in the interactive virtual reality environment;
The system wherein the digital representations and audio can be transmitted to a device of a user participating in the interactive virtual reality environment of the live event during operation. In the first paragraph,
The system is further configured to receive data from a user's device, the data corresponding to the user's interactions with the virtual reality environment. In the second paragraph,
The system wherein the computing module is further configured to calculate a threshold value for determining real-life elements suitable to be present within the digital representation of the digital twin in the virtual reality environment. In claim 1, claim 2 or claim 3,
The system wherein the server is further configured to transmit the digital representations to the device in real-time or at a delayed time after the live event. In any one of the first to fourth paragraphs,
A system wherein the digital representations, audio recordings, corresponding metadata and corresponding coordinate matrix data are parsed by the system and stored in a database for at least partial on demand re-distribution. In any one of the first to fifth paragraphs,
A system wherein the above digital representations, audio recordings, corresponding metadata and corresponding coordinate matrix data are stored in a distributed manner using blockchain technology. In paragraph 6,
A system wherein non-fungible tokens are assigned to at least a portion of the digital twin, including the digital representations, audio recordings, corresponding metadata, and corresponding coordinate matrix data. In paragraph 7,
A system comprising a payment processor adapted to identify a non-fungible token purchased by the user and assign a corresponding non-fungible token entitlement to the user. In any one of claims 1 to 8,
The system assigns a non-fungible token to the digital twin of the live event. In any one of claims 1 to 9,
The above server is a system that uses artificial intelligence to generate digital representations. A method for creating an interactive virtual reality environment in the metaverse,
The above environment includes a digital twin of a real-life element comprising at least one of a visual element, an audio element, a spatial element, and a tactile element,
The above environment is connected to distributed ledger technology,
The above method is:
A step of creating a digital twin of said real-life element from at least one of the following:
(a) capturing video to generate the visual element, recording audio to generate the audio element, determining tactile characteristics to generate the tactile element, and measuring distance and spatial coordinates to generate the spatial element, and
(b) recording metadata corresponding to one of the visual element, the audio element, the spatial element and the tactile element;
A step of storing the above digital twin on a server;
A step of assigning tokens to at least one of the visual element, the audio element, the spatial element and the tactile element;
A step of connecting the above server to a metaverse hosting server;
A step of integrating the digital twin into the metaverse by creating the virtual reality environment corresponding to the digital twin;
providing a user device with access to the virtual reality environment corresponding to the digital twin;
A method comprising: In Article 11,
A method wherein the step of storing the digital twin on a server further includes the step of storing it in a distributed manner using blockchain technology. In claim 11 or 12,
The method further comprises the step of communicating token data with the user device. In Article 13,
A method wherein the above token is a non-fungible token. In any one of Articles 11 to 14,
A method wherein measuring distance and spatial coordinates comprises determining at least one of a shape of the visual element, a distance between the first visual element and the second visual element, and a velocity associated with the moving visual element. In any one of Articles 11 to 15,
A method further comprising the step of connecting a payment processor adapted to identify a purchased token and a corresponding authorization of the purchased token. In any one of Articles 11 to 16,
A method wherein the step of providing access to the virtual reality environment comprises the step of allowing the user to launch his or her digital self into any location within the virtual reality environment. In any one of Articles 11 to 17,
A method further comprising the step of creating a portion of the metaverse based on the virtual reality environment of the digital twin. In any one of Articles 11 to 18,
A method further comprising the step of generating a plurality of virtual reality environments corresponding to a plurality of digital twins. In any one of Articles 11 to 19,
A method further comprising the step of integrating a plurality of digital twins into said virtual reality environments. In any one of Articles 11 to 20,
A method further comprising the step of selecting real-life elements for digitization within the digital twin. A method operated by a system for generating a 3-D graphical representation in a virtual reality environment,
The above 3-D graphical representation corresponds to real-life elements, and the system is connected to a network,
The system comprises a video capture device, a corresponding metadata recording device, an audio recording device, and a server configured to receive data from the video capture device, the audio recording device, and the metadata recording device.
The server operates a computing module configured to analyze the captured videos, recorded audio and recorded corresponding metadata to digitize objects displayed in the images of the captured videos to generate a 3-D graphical representation of the real-life elements,
The above method is:
a. A step of generating a 3-D graphical representation model based on objects displayed in images of the captured video, determining distances between objects displayed in images of the captured video, and based on analyzed corresponding metadata;
b. a step of linking audio tracks to elements of the 3-d graphic representation;
c. a step of storing the 3-d graphic representation in the server;
d. A step of connecting the above server to a metaverse hosting server;
e. a step of integrating the 3-D graphic representation into the metaverse by creating the virtual reality environment corresponding to the 3-D graphic representation; and
f. providing access to the virtual reality environment to the user's device;
A method comprising: As a system for creating digital twins in an interactive virtual reality environment,
The above digital twin corresponds to real-life elements, and the system is connected to a network,
The above system:
video capture device;
Corresponding metadata recording device;
audio recording device;
A server configured to receive data from the video capture device, the audio recording device, and the metadata recording device;
Including,
A system wherein the server operates a computing module configured to analyze the captured videos, recorded audio and recorded corresponding metadata, determine shapes of objects in the captured videos and distances between objects in the captured videos, and determine a coordinate matrix of a 3-D virtual space including a digital twin of the real-life element. A system that integrates an interactive virtual reality environment corresponding to a digital twin of real-life elements into the metaverse.
The above system is connected to a network,
The above system:
video capture device;
Corresponding metadata recording device;
audio recording device;
A server configured to receive data from the video capture device, the audio recording device, and the metadata recording device;
Including,
The server (operates a computing module configured to analyze the captured videos, recorded audio and recorded corresponding metadata, determine a spatial depth between real-life elements captured by video capture devices, recorded by audio devices or recorded by metadata devices, approximate shapes of the real-life elements, determine real-life elements suitable for digitization, and generate a digital twin of at least a portion of the real-life elements.
A system wherein the server is configured to transmit data to a metaverse hosting server to integrate the interactive virtual reality environment into the metaverse. A system for creating digital twins in an interactive virtual reality environment.
The above digital twin corresponds to a live event,
The above system:
A plurality of video data streams received over a network, each video stream corresponding to a set of predetermined coordinates in the physical space of the live event and having corresponding metadata;
A plurality of audio data streams corresponding to the 3-D audio field of the above live event;
A server configured to receive the video data streams and the audio data streams;
Including,
The server that operates the computing module, the module:
Analyzing the above video data streams and the corresponding metadata,
Determining the spatial depth between images of the above video data streams,
Match the objects in each image to the coordinate matrix of the virtual space,
Matching the above 3-D audio field to the coordinate matrix of the virtual space,
configured to generate a digital twin of at least a portion of the live event based on the images, the corresponding metadata, the spatial depth, the set of predetermined coordinates, and the coordinate matrix;
The above server:
Generating a plurality of digital representations from multiple orientations of said live event, said digital representation comprising at least one 360-degree digital representation and a plurality of unidirectional digital representations, each digital representation corresponding to a viewing angle within said digital twin in said interactive virtual reality environment, and
A system configured to transmit the above digital representations and audio to a user device over a network. In Article 25,
The system is further configured to receive data from the user device, the data corresponding to the user's interactions with the virtual reality environment. In Article 26,
The system wherein the computing module is further configured to calculate a threshold value to determine a real-life element suitable for approximating digital representations of the real-life element within the graphical representation of the digital twin in the virtual reality environment. In Article 25, Article 26 or Article 27,
The system wherein the digital representations are transmitted to the device in real time or at a delayed time after the live event. In any one of Articles 25 to 28,
A system wherein the digital representations, audio, corresponding metadata and corresponding coordinate matrix data are at least partially parsed by the system and stored in a database for on demand re-distribution. In any one of Articles 25 to 29,
A system in which the above digital representations, audio, corresponding metadata and corresponding coordinate matrix data are stored in a distributed manner using blockchain technology. In Article 30,
A system wherein non-fungible tokens are assigned to at least a portion of said digital twin, said digital representations, audio recordings, corresponding metadata, and corresponding coordinate matrix data. In Article 31,
A system comprising a payment processor adapted to identify a non-fungible token purchased by the user and assign a corresponding non-fungible token entitlement to the user. In any one of Articles 25 to 32,
The system assigns a non-fungible token to the digital twin of the live event. In any one of Articles 25 to 33,
The above server is a system that uses artificial intelligence to generate digital representations.