WO2024243623A1

WO2024243623A1 - A metaverse monitoring apparatus and process

Info

Publication number: WO2024243623A1
Application number: PCT/AU2024/050548
Authority: WO
Inventors: Minzhao LYU; Rahul Dev TRIPATHI; Vijay SIMARAMAN
Original assignee: Canopus Networks Assets Pty Ltd
Current assignee: Canopus Networks Assets Pty Ltd
Priority date: 2023-05-31
Filing date: 2024-05-24
Publication date: 2024-12-05
Anticipated expiration: 2025-11-30

Abstract

A metaverse monitoring process for execution by at least one processor of a metaverse monitoring apparatus of a communications network, the process including the steps of: receiving data packets of one or more user network addresses and candidate primary network flows of metaverse application domains of known metaverse applications; processing initial data packets of the candidate primary network flows to generate, for each said candidate primary network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate primary network flow for the user network address; comparing the generated sequences of initial upstream packet sizes of the candidate primary network flows to corresponding predetermined sequences of initial upstream packet sizes of primary network flows of known metaverse application domains of each of a plurality of known metaverse applications to identify, for each said user network address, each of the candidate primary network flows for the user network address as being a corresponding primary network flow of a corresponding one of the known metaverse application domains of a corresponding one of the known metaverse applications; and responsive to determining that primary network flows of all of the known metaverse application domains of a corresponding one of the known metaverse applications have been identified for a corresponding user network address, detecting a session of the corresponding known metaverse application for the corresponding user network address.

Description

A METAVERSE MONITORING APPARATUS AND PROCESS

TECHNICAL FIELD

The present invention relates to metaverses and computer-based virtual reality environments, and in particular to an apparatus and process for detecting and classifying metaverse sessions of network users, and optionally classifying metaverse activities of those users.

BACKGROUND

The term "metaverse" refers to an immersive virtual world for social interaction in which users, generally represented by an avatar, immerse themselves in a 3D world, and interact with animate and inanimate objects without being subject to real-world limitations such as distance, ability, and strength. Unlike other single-purpose networked applications such as online gaming and social media, the metaverse aims to provide users with a holistic experience encompassing daily activities such as working in the office, conducting meetings, socializing with friends, attending classes, shopping, playing games, participating in concerts and ceremonies, and the like. To this end, metaverse platforms allow developers to create and operate their own environments and events - recent examples include a wedding ceremony, a charter school, and a university campus.

Early versions of the metaverse, embodied in platforms such as SecondLife, Minecraft, Sandbox, and Decentraland, were accessed via 2D Personal Computer screens, and consequently lacked the ability to give users immersive experiences when interacting with the virtual environment. However, recent advances in virtual reality (VR) wearable hardware, from vendors such as Oculus, HTC, and Microsoft, have enabled true immersive experiences that provide 3D visualization along with body movement sensing so that users can perceive and interact with virtual environments in a natural way. The Economist magazine predicts that the market value of VR headsets will grow from $5B in 2021 to $12B in 2026, and the VR metaverse will create new experiences spanning all sectors, including entertainment, education, healthcare, and remote work. Indeed, companies such as Meta, Google, Microsoft, and ByteDance are investing heavily into building their VR ecosystems. The telecommunications industry is becoming increasingly keen to tap into the opportunity of VR metaverses, and is starting to explore how network functionality and performance need to evolve to support such immersive services. The Telecom Infra Project has recently launched a new project group to address "metaverse readiness", and whose primary objective is to accelerate the development of solutions and architectures that enhance network readiness to support metaverse experiences, addressing technical limitations to provide greater agility, programmability, performance, and reliability in both fixed and mobile networks. Telecommunications operators such as T-Mobile and Telefonica have thrown their weight behind this, while others (including SingTel and SK Telcom) are also talking of developing offerings with high-speed connectivity and ultra-low latency that enable enhanced metaverse experience.

However, to enhance the metaverse VR experience, telecommunications operators need to go beyond a simplistic view focused only on access bandwidth and latency. In particular, they need to possess a thorough understanding of how metaverse applications behave on their networks, in order to answer technical questions such as: (a) which servers and locations does the application access data from?; (b) what are the durations, volumes, and data rates of the various data transfers; and (c) which segments of the session are most relevant to user experience? As shown below, unlike a streaming video or gaming session that involves fairly predictable content exchange with just one or a few servers, a metaverse session can be tremendously more complex, involving tens of autonomous systems, hundreds of domains, and thousands of traffic flows. Without fully understanding the application network dynamics, telecommunications providers will be unable to monitor and improve the performance of the various component blocks, such as user-generated computational elements, edge compute, and distributed token systems. This will limit their ability to identify the network components that have the most impact on user experience, and to adjust their configurations in order to properly support metaverse applications.

It is desired to overcome or alleviate one or more difficulties of the prior art, or to at least provide a useful alternative. SUMMARY

In accordance with some embodiments of the present invention, there is provided a metaverse monitoring process for execution by at least one processor of a metaverse monitoring apparatus of a communications network, the process including the steps of: receiving data packets of one or more user network addresses and candidate primary network flows of metaverse application domains of known metaverse applications; processing initial data packets of the candidate primary network flows to generate, for each said candidate primary network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate primary network flow for the user network address; comparing the generated sequences of initial upstream packet sizes of the candidate primary network flows to corresponding predetermined sequences of initial upstream packet sizes of primary network flows of known metaverse application domains of each of a plurality of known metaverse applications to identify, for each said user network address, each of the candidate primary network flows for the user network address as being a corresponding primary network flow of a corresponding one of the known metaverse application domains of a corresponding one of the known metaverse applications; and responsive to determining that primary network flows of all of the known metaverse application domains of a corresponding one of the known metaverse applications have been identified for a corresponding user network address, detecting a session of the corresponding known metaverse application for the corresponding user network address.

In some embodiments, the predetermined sequences of initial upstream packet sizes of primary network flows of the known metaverse application domains include packet sizes of SSL handshake packets and first application data of the known metaverse application domains.

In some embodiments, the predetermined sequences of initial upstream packet sizes of primary network flows of the known metaverse application domains include respective packet sizes of the following five data packets of the primary network flows of each of the known metaverse application domains: SSL client hello, crypto key exchange (CKE), crypto key selection (CCS), encrypted handshake message (EHM), and the first application data of the known metaverse application domain.

In some embodiments, the step of comparing includes determining a corresponding prefix and a corresponding domain identifier of the corresponding one of the known metaverse application domains.

In some embodiments, the step of comparing includes using a trained inference model to map, for each said user network address, a corresponding one of the generated sequences of initial upstream packet sizes to the corresponding prefix and corresponding domain identifier of the corresponding known metaverse application domain for the user network address.

In some embodiments, the process includes: receiving data packets of candidate time-critical activity flows towards metaverse application domains listening on a predetermined range of port numbers; processing the data packets of the candidate time-critical activity flows to generate, for each said candidate time-critical activity flow and corresponding user network address, a corresponding sequence of packet sizes of the candidate time-critical activity flow for the user network address; comparing the generated sequence of packet sizes of each candidate time-critical activity flow to corresponding predetermined sequences of packet sizes of time-critical activity flows towards known metaverse application domains listening on a predetermined range of port numbers to identify at least one of the candidate time- critical activity flows as being a time-critical activity flow of a corresponding one of the known metaverse application domains for the user network address.

In some embodiments, the process includes: responsive to said detecting, dynamically generating volumetric network traffic statistics for the known metaverse application for the user network address; and comparing the generated volumetric network traffic statistics to corresponding predetermined volumetric network traffic statistics of predetermined user activity states of the known metaverse application to dynamically determine a corresponding one of a plurality of predetermined user activity states of the known metaverse application domain as a current user activity state for the user network address, wherein a current user activity of the user network address is dynamically classified as the dynamically determined user activity state of the predetermined user activity states of the known metaverse application.

In some embodiments, the predetermined volumetric network traffic statistics include packet counts, volumes, and flow counts for both primary flows and time-critical activity flows.

In some embodiments, the step of comparing includes applying a stateful machine learning model.

In some embodiments, the stateful machine learning model is configured to dynamically classify the current user activity during a detected metaverse session, wherein each successive current user activity state is dynamically determined on the basis of immediately previous user activity states and current volumetric network traffic statistics for primary flows and time-critical activity flows, including packet counts, volumes, and flow counts.

In some embodiments, the process includes generating usage and performance data representing one or more of: a count of metaverse users, a count of sessions of each metaverse application, network bandwidth consumption values for respective different user activity states of each metaverse session, times spent in respective different user activity states of each metaverse session, a count of network flows and corresponding network flow types for corresponding user activity states of each metaverse session, a collection of information (one or more of IP address, service prefix, domain name, subnet, and autonomous system) of servers serving each metaverse session, and latencies between the user and the distributed servers in each metaverse session.

In some embodiments, the process includes: receiving data packets from a software-defined networking (SDN) flow switch of a communications network; and processing header fields of the received data packets to identify subsets of the data packets as belonging to the candidate network flows of the metaverse application domains of the known metaverse applications; wherein only the identified subsets of the data packets are received as the data packets of the candidate primary network flows of the metaverse application domains of the known metaverse applications. In some embodiments, the process includes a training process, including: receiving network packets of primary network flows of metaverse application domains of a session of a known metaverse application; processing initial data packets of the primary network flows to generate, for each said primary network flow, a corresponding sequence of initial upstream packet sizes of the primary network flow of the corresponding metaverse application domain; and processing the generated sequences of initial upstream packet sizes of the known metaverse application domains to generate trained models for the known metaverse application domains, the trained models being configured to classify each of a plurality of further corresponding sequences of initial upstream packet sizes of candidate primary network flows of metaverse application domains as belonging to a corresponding one of the metaverse application domains.

In accordance with some embodiments of the present invention, there is provided a metaverse monitoring training process, including: receiving network packets of primary network flows of metaverse application domains of a session of a known metaverse application; processing initial data packets of the primary network flows to generate, for each said primary network flow, a corresponding sequence of initial upstream packet sizes of the primary network flow of the corresponding metaverse application domain; processing the generated sequences of initial upstream packet sizes of the known metaverse application domains to generate trained models for the known metaverse application domains, the trained models being configured to classify each of a plurality of further corresponding sequences of initial upstream packet sizes of candidate primary network flows of metaverse application domains as belonging to a corresponding one of the metaverse application domains. In some embodiments, the training process includes: receiving data packets of known time-critical activity flows and towards known metaverse time-critical activity domains; processing the received data packets of the known time-critical activity flows to generate, for each said known time-critical activity flow, a corresponding sequence of packet sizes of the known time-critical activity flow towards the known metaverse time-critical activity domain; processing the generated sequences of packet sizes of the known time- critical activity flows towards the known metaverse time-critical activity domains to generate trained models for classifying each of a plurality of further corresponding sequences of candidate time-critical activity flows towards metaverse time-critical activity domains as being a corresponding time-critical activity flow of a corresponding one of the known metaverse time-critical activity domains.

In some embodiments, the process includes monitoring network performance of at least one of the detected metaverse applications, and based on the monitoring, causing reconfiguration of at least one networking component to improve the network performance of the at least one of the detected metaverse applications.

In some embodiments, the step of causing includes using one or more network application programming interfaces (APIs) and network slices to: (i) amend routing paths for one or more corresponding metaverse application domains, (ii) prioritise metaverse traffic flows, and/or (iii) map metaverse application traffic to network slices.

In accordance with some embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon processor-executable instructions that, when executed by at least one processor of a metaverse monitoring apparatus, cause the metaverse monitoring apparatus to execute any one of the above processes.

In accordance with some embodiments of the present invention, there is provided a metaverse monitoring apparatus of a communications network, the apparatus including components configured to execute any one of the above processes. In accordance with some embodiments of the present invention, there is provided a metaverse monitoring apparatus of a communications network, including: a network interface to receive data packets of candidate network flows of metaverse application domains; a primary network flow sequence generator configured to process initial data packets of candidate primary network flows of metaverse application domains to generate, for each said candidate primary network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate primary network flow for the user network address; and a metaverse session identifier configured to:

(i) compare, for each said candidate primary network flow and corresponding user network address, the generated sequence of initial upstream packet sizes of the candidate primary network flow to corresponding predetermined sequences of initial upstream packet sizes of primary network flows of predetermined metaverse application domains to identify the candidate primary network flow as being a primary network flow of a corresponding one of the known metaverse application domains; and

(ii) responsive to determining that primary network flows of all of the known metaverse application domains of a corresponding one of the known metaverse applications have been identified for a corresponding user network address, detect a session of the corresponding known metaverse application for the corresponding user network address.

In some embodiments, the apparatus includes: a time-critical activity network flow sequence generator configured to process initial data packets of candidate time-critical activity network flows towards metaverse application domains listening on a predetermined range of port numbers to generate, for each said candidate time-critical activity network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate time-critical activity network flow for the user network address; and a time-critical activity network flow detector configured to compare the generated sequence of initial upstream packet sizes of each candidate time-critical activity network flow to predetermined sequences of initial upstream packet sizes of predetermined time-critical activity network flows of known metaverse application domains to identify the candidate time-critical activity network flow as being a time- critical activity network flow of a corresponding one of the known metaverse application domains for the corresponding user network address.

In some embodiments, the apparatus includes: a metaverse user activity identifier configured to: responsive to said determining, dynamically generate volumetric network traffic statistics for the detected session of the known metaverse application and corresponding user network address; and compare the generated volumetric network traffic statistics to corresponding predetermined volumetric network traffic statistics of known user activity states of the known metaverse application to dynamically determine a corresponding one of the known user activity states of the known metaverse application domain as a current user activity state for the user network address, wherein a current user activity of the user network address is classified as the determined state of the known states of the known metaverse application.

In some embodiments, the metaverse user activity identifier includes a stateful classifier and a stateless classifier, and the metaverse user activity identifier is configured to dynamically determine the current user activity state by:

(i) using the stateful classifier to statefully classify the current user activity state based on a predetermined number of previously classified user activity states for the session, if available; and

(ii) using the stateless classifier to statelessly classify the current user activity state if the predetermined number of previously classified user activity states for the session is not available.

In some embodiments, the metaverse user activity identifier is configured to aggregate over time the volumetric network traffic statistics of the identified primary network flows and the identified time-critical activity flows of different user activity states of detection user sessions of known metaverse application domains for each user network address. In some embodiments, the apparatus further includes a metaverse user experience analyser configured to monitor network performance metrics, including a distribution of servers among autonomous systems, throughput, packet rate, latency, and jitter of the identified primary network flows and identified time-critical activity network flows of different user activity states of detected user sessions of detected metaverse applications for user network addresses.

In some embodiments, the apparatus further includes a network configuration component configured to cause reconfiguration of at least one networking component to improve the network performance of the at least one of the detected metaverse applications, in dependence on the corresponding network performance metrics.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the present invention are hereinafter described, by way of example only, with reference to the accompanying drawings, wherein:

Figure 1 is a schematic diagram of a communications network including a metaverse monitoring apparatus in accordance with the described embodiments of the present invention;

Figure 2 is a block diagram of the metaverse apparatus.

Figure 3 is a block diagram of software components of the metaverse apparatus of Figure 2;

Figure 4 is a flow diagram of a three-stage metaverse process in accordance with the described embodiments of the present invention, and executed by the metaverse apparatus;

Figure 5 is a block diagram illustrating user activity classification components of the metaverse apparatus;

Figure 6 is a set of screenshots of respective user activities (with state labels) in a representative walk-through of an example MultiverseVR session;

Figures 7 and 8 illustrate the complexity of meta-verse user network activities in the example MultiverseVR session walk-through: Figure 7 is a chart showing the accessed service domains ranked by total packet count, and Figure 8 is a Sankey diagram showing the distribution of network flows across different user activity states, autonomous systems, and latency ranges; Figures 9 to 11 are respective graphs of flow span, TCP packet rate, and UDP packet rate as a function of time during the MultiverseVR session walk-through;

Figures 12 and 13 are Sankey diagrams illustrating the distribution of flows across autonomous systems and latency ranges for SUE (separate user-created event) and SPE (separate provider-created event) states, respectively, of the MultiverseVR session walk-through;

Figure 14 is a Sankey diagram showing the distribution of flows across autonomous systems and latency ranges for ATs (Asset Trading states) of the MultiverseVR session walk-through;

Figure 15 is a graph of packet rate (including both TCP and UDP) as a function of time during content creation (CC) activity in a Rec Room session;

Figure 16 is a schematic diagram illustrating the systematic definition of network traffic attributes of metaverse user activity states of MultiverseVR sessions;

Figure 17 is a table illustrating normalised values of the network traffic attributes of Figure 16 as greyscale values;

Figure 18 is a graph showing measured spans of tracked flows towards primary and time-critical activity demands as a function of time for ground-truth metaverse sessions in a university network;

Figure 19 is a graph showing measured upstream packet rates of ground-truth MultiverseVR sessions in the university network;

Figure 20 is a graph of upstream packet rates of VRChat primary and time-critical activity flows from university dormitory residents during a long-weekend holiday; and

Figures 21 and 22 are bar graphs of the volume of data consumed and time spent, respectively, in VRChat, for the two available activity states HS and SUE.

DETAILED DESCRIPTION

Embodiments of the present invention include a metaverse monitoring apparatus and process that address the technical limitations of the prior art as described above by detecting and identifying metaverse sessions from general network traffic, and optionally also classify user activity of each metaverse session. The metaverse apparatus and process enable telecommunications network operators (including Internet Service Providers (ISPs)) to monitor and quantify metaverse network usage and user experience of their network users so that they can quantify and improve the performance of their networks for metaverse applications, and make better planning and provisioning decisions.

Figure 1 is a schematic diagram of a communications network including a metaverse monitoring apparatus 200 in accordance with an embodiment of the present invention, and Figure 2 is a block diagram of the metaverse apparatus 200 itself, with metaverse monitoring components 300 of the metaverse monitoring apparatus 200 shown in Figure 3. The metaverse monitoring apparatus 200 executes a metaverse monitoring process 400, as shown in Figure 4, which, inter alia, receives copies of data packets from the network, and processes them to detect and identify network flows of metaverse sessions from general network traffic by using trained inference models to identify unique characteristics of network traffic flows of metaverse applications, as described below.

In the described embodiments, the metaverse monitoring apparatus 200 is implemented using a standard computer system such as an Intel IA-64 based computer system, as shown in Figure 2, and the metaverse monitoring process 400 executed by the metaverse monitoring apparatus 200 is implemented as executable instructions of the metaverse monitoring components 300 stored on non-volatile (e.g., hard disk or solid-state drive) storage 204 associated with the computer system, as shown in Figure 2. However, it will be apparent to those skilled in the art that in other embodiments at least parts of the metaverse monitoring process 400 could alternatively be implemented in other forms, such as configuration data of one or more field programmable gate arrays (FPGAs), or as one or more dedicated hardware components, such as applicationspecific integrated circuits (ASICs), or as combinations of these various forms, for example.

In the described embodiments, the metaverse monitoring apparatus 200 includes standard computer components, including random access memory (RAM) 206, at least one processor 208, and external interfaces 210, 212, 214, all interconnected by a bus 216. The external interfaces may include universal serial bus (USB) interfaces 210, at least one of which may be connected to a keyboard and a pointing device such as a mouse 218, at least one network interface connector (NIC) 212 which connects the metaverse monitoring apparatus 200 to a communications network such as the Internet 102, and a display adapter 214, which may be connected to a display device 222 such as an LCD panel display. As shown in Figure 2, in the described embodiments the metaverse monitoring apparatus 200 also includes a number of standard software modules, including an operating system 224 such as Linux or Microsoft Windows, and structured query language (SQL) support 230 such as MySQL, available from http://www.mysql.com, which allows data to be stored in and retrieved from an SQL database 232.

As shown in Figure 3, in the described embodiments the metaverse monitoring components 300 of the apparatus 200 include a packet processing module 302 written in Golang, built over the open-source Virtual Network Functions Framework NFF-GO 304, available from https ://github.com/aream/nff-ao. In turn the NFF-GO framework utilises the DPDK (Data Plane Development Kit) 306, available from https://www.dpdk.or /, as its underlying packet processing functions.

The packet processing module 302 maintains states for flows that are candidates for metaverse primary domains or time-critical activity domains, based on their upstream packet payload sizes. The packet processing module 302 also maintains the states of detected metaverse sessions for their active flows toward primary domains and time- critical activity domains, domain names identified by Server Name Indications (SNI), packet counts, and byte counts.

A metaverse primary domain flow detection component 308 is implemented on top of the stateful packet processing module 302, and uses the upstream packet payload sizes of candidate flows to detect active primary domain flows and their domain names.

A metaverse time-critical activity domain flow detection component 310 is also implemented on top of the stateful packet processing module 302, and uses the upstream packet payload sizes of candidate flows to detect active time-critical activity domain flows and their metaverse applications.

An active metaverse session detection & user activity state classification component (also referred to herein for convenience as the "user activity component") 312 is implemented on top of the above three components 302, 308, and 310 described above. The user activity component 312 detects active metaverse session by checking whether a required list of primary domain flows with their domain names are detected by component 308. The user activity component 312 performs classification of user activity states for active metaverse sessions using pre-trained machine-learning models based on attributes extracted by packet processing modules for detected time-critical activity domain flows by component 310 and primary domain flows by component 308.

A metaverse user activity analyser component 314 aggregates volumetric attributes of classified user activity states for active metaverse sessions provided by the component 312 to generate activity profiles of sessions, users, and metaverse applications.

A metaverse user experience analyser component 316 monitors network performance metrics (including packet rate, throughput, latency, jitter, and destination server IP address) of flows belonging to active metaverse sessions detected by the component 312. The performance metrics are extracted from IP and TCP/UDP headers of packets from the metaverse flows identified by the packet processing module 302.

A network configuration component 318 uses network application programming interfaces (APIs) 320 and network slices to amend routing paths for one or more corresponding metaverse application domains, prioritise metaverse traffic flows, and/or map metaverse application traffic to network slices. The network configuration component 318 can be controlled manually by a network administrator, and can be configured to operate automatically to improve performance of one or more active metaverse applications in dependence on the corresponding performance metrics generated by the metaverse user experience analyser component 316.

In order to provide a better understanding of the present invention and its metaverse context, the most popular metaverse applications and virtual reality (VR) platforms in the market today are briefly described below, and the network behaviour of four of those metaverse applications are then described in detail.

VR Platforms and Metaverse Applications

Among VR headsets in the market today, the Oculus VR platform, owned by Meta (i.e., Facebook) is the most dominant, accounting for nearly 80% of the market. The Oculus Rift model needs to be tethered to a graphics PC, whereas the more modern Quest model is a stand-alone device. Other significant VR headset manufacturers include HTC Vive (models include Focus, Pro, and Cosmos), Microsoft HoloLens, Lenovo Explorer, and Dell Visor, with PlayStation VR and Pico VR joining this fast- expanding market recently. Table 1 below lists five popular metaverse applications emerging in the market. Horizon Worlds is Meta's flagship metaverse application, available exclusively on the Oculus VR platform. It reported about 300K monthly active users in February 2022, and uses an in-app centralized token currency. Users are allowed to create their own events (e.g., for socializing, working, education, and entertainment), and require approval from the metaverse operator before being made accessible to other users. Horizon Worlds is currently limited to the North America market.

Table 1: Five popular metaverse virtual reality (VR) Applications and their current specifications.

MultiverseVR is another popular metaverse, developed and operated by unicorn startup Clipo Labs. It allows users to buy, rent, and trade virtual properties (e.g., residential apartments, retail shops, concert halls), where they can host events such as parties, sales, and ceremonies. MultiverseVR claims to be one of the most versatile VR applications, allowing third parties to support transactions via NFTs & cryptocurrencies.

Other emerging VR metaverses include AltspaceVR, VRChat, and Rec Room, with attributes shown in Table 1. All of the metaverse applications in Table 1 are available on the Oculus VR platform. As described above, the metaverse monitoring apparatuses and processes described herein are based on identifying and recognising unique characteristics of network traffic flows of metaverse applications and user activity states of those applications. Accordingly, the inference models used to detect those unique characteristics first need to be trained. In work leading up to the invention, the inventors captured and analysed time traces of network traffic of selected meta-verse applications and user activities within those applications.

As shown in Figure 1, in order to collect traffic traces of metaverse VR sessions, a VR headset (in the described embodiments, an Oculus Quest 2 VR headset) 104 is connected to a WiFi router connected to the Internet 102 (via the inventors' university network in the described embodiments). A desktop computer 106 is also connected in the lab to generate non-metaverse traffic for the purpose of evaluating the metaverse detection processes. Traffic is tapped at the University's egress gateway 108 to the Internet 102, and filters are provisioned for selective packet captures ("PCAP"s) pertaining to the end-points of interest (such as the WiFi router 110 to which the VR headset 104 connects). During each metaverse session, a timeline of activities is carefully logged, indicating what the user was doing over each period within the corresponding metaverse.

A Representative Metaverse VR Walk-Through

For the purposes of illustration, described herein is an example of correlating metaverse user activity with network activity, using an illustrative 45-minute 'walkthrough' session of one of the most popular metaverses: the MultiverseVR application on the Oculus Quest 2 VR headset.

Figure 6 is a series of representative screenshots of consecutive user activities (with corresponding state labels shown above the screenshots) during the walk-through, with associated timelines and descriptions provided in Table 2 below. Table 2: summary of user activities during a MultiverseVR walk-through

In the evaluation session walk-through, a test user put on the headset 104, launched the MultiverseVR application console, and started the login and initialization processes in the 'home space' ("HS") (shown in Table 2 above as "HS1", at 0-159s in the timeline), which allows the user to remain online without being involved in social activities. The user then exited the home space and entered the Main Hub ("MH"), which could be a public interchange station ("MH2", 160-457s) or the public street ("MH3", 458-788s), from which the user can access portals to other events hosted either by the Multiverse provider or in other metaverse users' virtual properties. The public street hosting portal of state MH3 shows that each floor of a building is owned by a different user, each of whom can host freely accessible, invitation-only, or paid commercial events within their properties. The user can explore the events by entering the building or checking the interactive menu for nearby properties. Unlike MultiverseVR, the VRChat metaverse has not implemented social settings of MH; instead, it uses pop-up menus for this purpose.

From the public street, the user visited several separate user-created events ("SUE"s), including a movie theater and its associated media room ("SUE4", "SUE5", 789 - 1059s) in a commercial property, and a user-created shopping mall (SUE7-9, 1113- 1224s) displaying assets on various levels (transitions are made though the main hub MH6). The user browsed through a store selling watches, and chose to perform asset trading ("AT") using digital currencies. As shown in Figure 6, in AT10 (1225-1449s) pop-ups show details of the items available for trading with links to complete purchases.

The user then visited a business showroom from a not-for-profit organization helping people with addiction, and a company providing VR-based education (SUE11-13, 1450-1944s). After that, the user tried separate provider-created events ("SPE") operated by the MultiverseVR provider: specifically, a shooting game (SPE14, 1945- 2219s) and a space gallery (SPE15, 2220-2332s). The user then visited a friend's place in a residential apartment to attend an invitation-only balcony party (SUE16, 2333- 2559s), before teleporting back to the user's private space (HS17, 2560-2700s).

The various activities described above, including entering, walking, socializing, playing, shopping, educating, and entertaining captured in the 45-minute trace described above are typical of metaverse behavior, and this will be referred to as a running example in the following description. The various activities are categorised into distinct states (HS, MH, SUE, SPE, AT) in order to model user behavior and facilitate the development of network activity profiles. A sixth state, content creation ("CC"), that allows users to create their own digital assets and build virtual environments, has been excluded from the example above because it is outside the metaverse console for MultiverseVR.

Meta verse VR - Network Traffic Characteristics

In work leading up to the invention, the inventors spent tens of hours immersed in each of the metaverse applications listed in Table 1 (except for Horizon Worlds, which was not available in the inventors' country), maintaining logs of their activities, and correlating them with network packet captures. The inventors found that the user activity states introduced above for the illustrative metaverse trace are broadly applicable across the various metaverse applications (with some differences, described below). The anatomy of metaverse VR network traffic is described below, illustrating its complexity and dependence on user activity.

Distinct user activity states: All metaverse VR applications have a home space (HS), where the user enters upon login, performs initialization, and can have private time in the virtual world. In most metaverses (except VRchat), the user then enters the main hub (MH), which could be a street, plaza, or station, where they can walk around and interact with users and events. VRchat is an exception in that it does not have a main hub, and instead uses pop-up menus. All the metaverses considered allow third party separate user-created event (SUE), wherein users can attend social events (e.g., a family party, a business meeting, an online educational class, and a game) created and hosted by others. In addition, MultiverseVR operates separate provider-created events ("SPE"s), such as games, chatrooms, and virtual exhibitions, built and directly operated by the metaverse provider. MultiverseVR and Rec Room support asset trading ("AT"), where users can browse and exchange any assets such as NFTs and currencies. Content creation ("CC") is generally done by users outside of the metaverse application itself, with the exception of Rec Room, which allows in-app content creation.

Service domains accessed: To study the complex network activity within a metaverse session, the inventors find it convenient to decompose the network activity by the user activity states described above. Further, since each metaverse session can interact with a large number of domains, to make the study tractable the domains are categorized into 4 classes based on ownership and purpose: primary domains are those directly operated by the metaverse application developer; time-critical activity domains are those providing real-time synchronization of user movement and activity; cloud content domains are general-purpose content delivery networks ("CDNs") that, for example, serve video streams; and third-party service domains are for non-metaverse services such as advertisements and social media content.

With the user activity states and service domain categorizations in place, it is illustrative to first describe some high-level characteristics of the traffic collected for the representative walk-through of MultiverseVR described above. Overall, the walkthrough session accessed 87 service domains, and Figure 7 is a bar chart showing the top 15 domains ranked by their packet counts, with shading to represent their categories.

Not surprisingly, the majority of packets are exchanged with the primary MultiverseVR domain shapevrcloud, which serves the environment (buildings, rooms, streets, etc.) information as the user moves. Cloud content domains cloudfront and oculuscdn seem to serve VR platform related images and advertisements, akamaized serves the video watched by the user in the theater in SUE5, and fbcdn provides the social media feeds. UDP servers in the 172.x.x.x subnet manage time-critical activity to update user location and gesture as the users move around in the metaverse. Third party services such as unity3d, crypto, etc. provide graphics, access to third-party businesses, and crypto services. As can be seen, the number of domains involved in delivery of a metaverse VR session is an order of magnitude higher than for streaming video or gaming sessions, highlighting the challenge telecommunications operators will face to optimize and troubleshoot their networks for such applications.

Figure 8 is a Sankey diagram depicting the metaverse traffic pattern by user activity state, highlighting the autonomous systems (AS) from which the traffic is sourced, along with the measured latency, for the representative MultiverseVR walk-through session described above. Each line in the Sankey diagram represents one flow, and its width is proportional to the packet count of the flow. It is apparent that the majority of traffic corresponds to separate user-created events (SUE), since that is where users are expected to engage the most socially. Further, the latency for some of the content in SUE and asset trading (AT) is very high (even exceeding 100ms), as they are from third parties, and therefore might not be from the local cache/CDN. Network operators might therefore want to selectively optimize their routing paths or caching strategies in dependence on the user activity state.

The network traffic characteristics of each identified user activity state are described in detail below.

Home Space (HS)

HS is where a user enters MultiverseVR. The network dynamics are analysed in terms of flow setup sequence, the domains that they communicate with, and the rates/volumes within these flows. In the following description, the walk-through shown in Figure 6 is used as the primary illustrative example, highlighting different behaviors in other metaverses.

Figure 9 shows the span of the individual flows that are active during the metaverse session. Each line represents a corresponding unique 5-tuple network flow identifier, and the lines are labelled by the corresponding domain types (/.e., primary, time-critical activity, cloud content, or third-party service). It can be seen that in HS1, most of the flows are to third-party services (e.g., graph. oculus) and cloud content (e.g., oculuscdn) servers, which are used by the VR platform to fetch graphical data and exchange administrative information prior to commencing a metaverse session. Once the metaverse activity commences, flows (labelled as primary flows near the bottom of Figure 9, and near the end of HS1) are initiated to the primary metaverse domain (i.e., shapevrcloud) for MultiverseVR initialization, including user login, account settings, and asset inventory loading. More specifically, six TCP flows are initiated to prod. shapevrcloud, followed by one TCP flow to prodblob. shapevrcloud. Figures 10 and 11 show respectively the upstream/downstream TCP and UDP packet rates (measured every second) observed during the MultiverseVR session, labelled by domain type (note that the y-axis of Figure 10 uses a log scale for easier visualization). In the home state (HS), the inventors observed that around 10-100 packets are exchanged per second with the Oculus servers, while the packet rate to the primary metaverse servers is over a hundred. There is no time-critical activity in this state.

The inventors collected and analyzed similar traces for the other metaverse applications. For instance, AltSpaceVR (with its primary domain altvr) in its home space (HS) creates flows towards three prefixes in the order config, cdn-content-ingress, and account; VRChat has its sequential primary domain prefixes api, pipeline, assets; and Rec Room (with its primary domain rec) communicates with prefixes including api, auth, img, cdn, clubs, and commerce.

All TCP flows are SSL-encrypted, but the SNI can be extracted from the connection setup packets to identify the server the communication is with. The sequence of TCP connections and the payload lengths of the first few packets of each connection are used to develop the metaverse activity detection models, as described below.

In terms of network requirements, HS is the least demanding state. The activity in HS is predominantly private without interaction with other users, so is not very sensitive to latency and jitter. Further, the service domains being accessed in HS are a small set, limited to the VR platform and the metaverse operator. The low to moderate bandwidth requirements in this state, coupled with the small number of domains the content is coming from, mean that this state should not be a major concern for network operators.

Main Hub (MH)

After leaving the private home space, the user entered a public interchange station and then a public street that serves as an immersive menu to other available events, either hosted by individual users or directly created by the metaverse operator. It is also a social hub for all users who have not decided which event to go next.

As shown in MH2, MH3 and MH6 of Figure 9, there are active flows toward primary domains to fetch metaverse content such as surrounding environments. Given that MH events are developed by metaverse operators, and most of the content (e.g., city maps, building structures, and street layouts) has already been installed with the application console in the VR headset, only a small amount of dynamic content (e.g., weather in the event, and announcements from the developer) has to be downloaded from the primary domain at run-time. This leads to a fairly low packet rate and bandwidth usage for primary domain flows, as shown in Figure 10, the exception being the beginning of each MH state whereby a bulk of metaverse content is downloaded.

In addition to primary flows that fetch event content via TCP at run-time, there are few flows toward time-critical activity domains (i.e., starting from MH2 in Figure 9) synchronizing real-time user positions and avatar gestures in MH, as this state involves socialization among users. Those time-critical activity flows are purely via certain UDP ports 5055, 5056, or 5058 which are commonly used by online gaming for a similar purpose. As shown in Figure 11, the data exchange with time-critical activity domains during MH is roughly constant, at about 30 pps in the upstream direction, whereas downstream packet rates are variable, depending on the how crowded (i.e., the number of active users) the current event is. For example, Figure 11 shows variations in the downstream packet rates in state MH3 due to the changing number of nearby users.

When exploring the interchange station (MH2) and public street (MH3), the inventors also saw a number of posters, video advertisements, and third-party customized user avatars. Such content provided by commercial or residential users (e.g., owners of a building) is often fetched from third-party service domains such as ads-twitter, doubleclick, redditstatic, while some of the larger contents are downloaded from dedicated cloud content providers such as cloudfront and fbcdn. Therefore, a non- negligible numbers of TCP packets flow toward those domains in the MH states.

For a network operator, guaranteeing low latency and stable jitter in both TCP and UDP is critical for user-perceived metaverse experience, as application content and third- party content are fetched from their respective service domains at run-time, which could be quite distributed in routing paths. MH is not bandwidth demanding in TCP, because most of the metaverse content has already been installed in the local console, whereas bandwidth availability for UDP has to be sufficient and stable due to constant synchronization of user activities in MH. Separate User-Created Event (SUE)

In the representative walk-through, the user entered several separate metaverse events such as a theatre (SUE4-5), shopping mall (SUE7-9), business showrooms (SUE11-13), and a friend's apartment (SUE16), created and hosted by individual commercial and residential users.

When exploring those SUEs, event content created by users and uploaded to the metaverse service domain is constantly being fetched at run-time. Therefore, there is a burst of TCP flows toward the primary domain (i.e., shapevrcloud) during each SUE, as shown by the lines grouped near the bottom of the graph in Figure 9. The respective packet rate (i.e., the shaded region labelled as "Primary" in Figure 10) and bandwidth usage also stay at the highest level among all states. In the walk-through, the user was not always having a smooth experience during SUE states, seeing for example blank or low-resolution items for a while before they became clearer. As depicted in the Sankey diagram of Figure 12, this is probably due to the relative high latency (i.e., >50ms) and limited TCP bandwidth for some primary domain flows going through AS1 that is hosting content for those items.

Similar to the MH state described above, there are one or two concurrent UDP flows toward time-critical activity domains of the metaverse to synchronize real-time user activities. The upstream packet rate and bandwidth stay at a constant level, while the downstream ones depend on how crowded the event is. During gestural and vocal interactions with other users, there is no perceived feeling of lag and interruption, likely because the latency is low (<10ms) and sufficient bandwidth is available for the flows hosted in AS2, as shown in Figure 12.

The inventors have also seen many decorative pictures, videos, icons, and external links embedded in each of the events. For example, in the theatre (SUE4-5), the movies being played and dynamic posters on the walls are all from external sources (e.g., vimeo and gvt2). In the shopping mall (i.e., SUE7-9), each store front is a static or dynamic picture of the brand which is linked to its official site. As a result, many flows toward different third-party service and cloud content domains are initialized during SUEs; however, the number of TCP and UDP packets to such domains is fairly low, as shown in Figures 10 and 11. Figure 12 shows that those flows are sent to six different ASes, with some of them having latency higher than 100ms, which can lead to laggy and unresponsive experience for users in interacting with the items. SUE is the most important state, central to the concept of the metaverse, enhanced by third parties. It is also the most demanding of all states in terms of network resources. Sufficiently high bandwidth, low latency, and stable jitter are required for both TCP and UDP flows toward primary, time-critical activity, third-party service, and cloud content domains that all play critical roles in user-perceived experience. Moreover, the service domains involved in this state are often highly distributed among many ASes, making it challenging for network operators to optimize their routing paths to support good user experience.

Separate Provider-Created Event (SPE)

As discussed above in relation to the walk-through, apart from separate metaverse events created by users, the inventors have also explored some events that are directly created and operated by the metaverse provider, including a multi-player shooting game (SPE14) and a NASA space gallery (SPE15). Those provider-created events are often rich in content, large in space, involving mechanisms more complex than their counterparts created by individual users. They are also quite attractive to users who seek casual entertainment (e.g., gaming) and exploration (e.g., gallery), rather than socialization (e.g., meetup, conference, and ceremony) that is the main focus of SUE states.

The provider-created nature of the SPE state suggests that the corresponding event content could be mostly pre-downloaded as patches of the console application in the user's VR headset, instead of being fetched at run-time. Thus, there is no surge of new flows to the primary domain, as shown for the SPE14 and SPE15 states in Figure 9. The inventors could also observe very low profiles of packet rate and bandwidth usage (such as those shown in Figure 10) during SPEs, except their beginning periods. Similar to MH and SUE, a user is interacting with others in real-time. Therefore, one or two UDP flows toward time-critical activity domains are always active and transmitting data at a stable rate - a slightly higher packet rate and bandwidth usage is observed in the shooting game (i.e., SPE14 of Figure 11) that is more demanding on real-time synchronization than other social events.

The third-party service and cloud content domains being accessed during SPEs are primarily for VR graphics such as graph-facebook-hardware, graph. oculus, unity3d, and oculuscdn. Their packet rates and bandwidth consumption are at low levels, as shown in Figure 10. Unlike MH and SUE, there is often no or little other third-party content such as advertisements being fetched during SPEs. The user-perceived experience in SPE is not very sensitive to TCP metrics, since the packet rate and bandwidth usage for VR graphics is lower than 30pps and 10Kbps, respectively. In contrast, UDP latency, bandwidth, and jitter play important roles in user-perceived experience, because real-time activity synchronization among users is critical in SPE states. In addition, the optimization of routing paths is not very challenging for network operators because there are only few (e.g., three in the walkthrough) ASes involved, as shown in Figure 13, corresponding to primary metaverse content (AS1), time-critical activity (AS2), and VR graphics (AS3).

Asset Trading (AT)

Asset trading is the state where individual and commercial users can purchase and sell their digital (e.g., NFT) or physical assets via third-party platforms using cryptocurrency, digital tokens, or real-world money. In ATs, a user often stays static when browsing items on sale (typically through pop-up windows) without frequent interaction with the environment and other users. Therefore, very few packets are exchanged between a user's VR headset and the primary or time-critical activity domains, though those flows remain active, as shown in Figures 9 and 10. Unlike other activity states that are dominated by primary metaverse content and time-critical activities, the majority of network traffic in ATs is for third-party service and cloud content domains serving the description/advertisement of items (e.g., adsrvr, pinimg, fbcdn, and cloudflare) and currency platforms (e.g., crypto and opensea). Those service domains are often located in a diversified set of ASes, as shown in Figure 14: AS7 and AS6 are for the market domain opensea and cloud content fbcdn, respectively, while other domains belong to eight ASes, most of which have high latencies larger than 50ms.

While there are time-critical activity and primary domain flows, ATs have almost no requirement of UDP/TCP performance due to their inactivity in metaverse content/activity exchange, whereas TCP metrics toward third-party and cloud content domains are quite important. Given the diversity of ASes involved in this state, optimization of routing could be complex to achieve. Content Creation (CC)

Content creation by users in MultiverseVR is done outside of the metaverse, and thus there is no CC state during the representative walk-through. However, the Rec Room metaverse allows content creation within the application, and so the inventors conducted a session to illustrate the corresponding network behaviour. The traffic profile of the CC state is shown in Figure 15, where the user is creating an outdoor party using available building blocks. In short, the primary domain during CC occasionally has a quite spiky pattern for its upstream packet rate to update the content created by a user to the metaverse primary domain rec.

The time-critical domain flows are active during CC, but with much less packet rate and bandwidth consumption (possibly for keep-alive messages) compared with MH, SUE, and SPE, as there is no interaction with other users when creating content. No cloud content domain is observed in this state, because it does not load any content provided by other users, and there is a small amount of traffic exchanged with the third-party service domain oculus for the VR platform.

In terms of network requirements, CC requires sufficient TCP bandwidth to ensure timely content uploads from the user device to the primary metaverse domain, which could go up to 15Mbps in the representative Rec Room session.

Armed with the insights described above, the inventors developed the metaverse monitoring apparatuses and processes described herein to detect metaverse VR sessions and classify user activity states. The metaverse monitoring apparatuses and processes withstand not just HTTPS packet content encryption, but also encryption of the SNI carried in SSL client hello packets, as envisaged by HTTP/3 with encrypted client hello (ECH). They additionally eliminate all reliance on DNS, not only due to encryption, but also because many UDP flows are not preceded by DNS lookups. The metaverse monitoring apparatuses and processes instead rely on extracting a wide range of flowlevel attributes that are encryption agnostic, as described below. Metaverse activity is detected based on signatures generated from packet payload size sequences of primary and time-critical activity domain flows. Packet Payload Size Sequence Signatures of Metaverse Flows

As discussed above, flows toward primary and time-critical activity domains that are directly operated for each metaverse have unique patterns in their first several packet payload sizes, which are used to accurately detect those metaverse flows and identify their domain types. As for primary domain flows, the packet size-based signatures are robust to SNI encryption that could make their domain names and service prefixes unobservable by network operators.

Primary domain flows

At the beginning of a metaverse VR session, the user device (e.g., a VR headset) initializes a series of SSL-encrypted TCP flows to exchange administrative information (i.e., authentication and user activity tracking) with the primary domain. The first several upstream packets in those flows are identifiable by their packet payload sizes excluding Ethernet, IP and transport-layer headers. Specifically, those flows are initiated by TCP and SSL handshakes, followed by the actual application data. TCP handshakes consist of standard SYN and ACK packets with no payload content (i.e., zero byte). In an SSL handshake, the user device sends two to four upstream packets of service-specific handshake messages, including client hello, crypto key exchange, crypto key selection and encrypted handshake; the latter three messages could be delivered by either one or separate packets. In addition to the handshake processes, the first packet carrying application data also has a specific payload size, which is hardly surprising as it often serves as a fixed starting message, depending on the requested service.

Extracting flow signatures

As described above, at the start of a metaverse VR session, a series of primary domain prefixes is accessed through multiple flows from the user device, each having an important role (e.g., authentication and critical application content) for the session. The inventors developed an automatic training process (using Golang and Python in the described embodiments) that takes ground-truth traffic traces files (i.e., PCAPs) and their primary domain names (e.g., altvr for AltSpaceVR) as inputs to extract packet size sequence signatures (containing packet payload sizes of SSL handshakes and first application data) of the primary domain flows with their respective prefix names. Representative packet payload size sequences

Representative signatures of the four studied metaverses obtained from the training process are given in Table 3 below. The upstream packet payload sizes of each primary domain flow during the first HS state are listed within square brackets, labelled by their service prefixes in the order of appearance. Within each pair of square brackets, the first value is the payload size of the SSL client hello packet, the last value is the payload size of the first application data packet, and the one or more values between the first and last values are the respective payload sizes of the other SSL handshake packets (for CKE, CCS and EHM). Taking the MultiverseVR as an example, in the training trace files, a user always sends six flows toward the service prefix prod followed by one flow towards prodblobs. The first flow has 414 bytes of payload in its SSL client hello packet, 75 bytes of payload in the CKE packet, 6 bytes in the CCS packet, 45 bytes in the EHM packet, and 338 bytes in the first application data packet. However, not all primary domain flows have their SSL CKE, CCS, and EHM in separate packets. As shown in Table 3, the flow towards the pipeline prefix of AltSapceVR has all three messages in one packet with a payload size of 134, whereas the primary flows for Rec Room have their CCS and EHM messages in one packet with a payload size as 51 bytes.

Table 3: Prefix sequence and packet payload size signatures of flows toward primary domains of each metaverse application, which contain SSL client hello, crypto key exchange (CKE), crypto key selection (CCS), encrypted handshake message (EHM) and first application data.

Described below are the steps by which the upstream packet payload size signatures are used to detect active primary domain flows toward different service prefixes. The inventors found that the sequence of service prefixes accessed by the detected flows is a robust indicator of active metaverse sessions.

Time-critical activity domain flows

UDP flows toward time-critical activity domains are created during states when the user is synchronizing their positions and gestures with others. Such flows are sent to certain port numbers, and with unique payload size sequences of their first several (e.g., four to seven) upstream packets. These packets carry initial messages specific to the requested services, similar in spirit to multiplayer gaming applications. Therefore, the training process extracts the per-flow upstream packet payload size sequences for traffic destined to the various time-critical activity domains UDP ports (e.g., ports 5055, 5056, and 5058 for MultiverseVR), and uses these sequences to train the models as described above. Example packet size sequence signatures of flows toward those UDP port numbers are shown in Table 4 below.

Table 4: UDP ports and upstream packet payload size sequences of flows toward time-critical activity domains

Described below are the steps by which the described processes use the identified primary and time-critical activity domain flows to detect active metaverse sessions and to track their activity states.

Volumetric Attributes of Metaverse User Activity States

The inventors found that different states of user activity exhibit unique volumetric patterns that can be captured by structured attributes. In the described embodiments, forty attributes are generated from packet and flow statistics of metaverse sessions, as shown schematically in Figure 16 and listed in Table 5 below. However, it will be apparent to those skilled in the art in light of this disclosure that other attributes, and/or other combinations of attributes, may be used in other embodiments.

The attributes of the described embodiments cover transport-layer protocols (i.e., TCP or UDP), service domain types that are directly operated for a metaverse (i.e., primary and time-critical activity), and various measurement metrics of packets and flows. Those metrics are either directly maintained for each session (i.e., host-level statistics in Figure 16); or for all user flows that are represented by their median and standard deviation values. For example, as shown in Table 5 below, the attribute labelled Al (tcp_prim_mdn_vol) represented by the top dotted line in Figure 16 is defined as the median volume of all TCP flows toward primary domains during a monitored interval. Similarly, the last attribute A40 (udp actv pkt ct) represented by the bottom dashed line is defined as the UDP packet count toward time-critical activity domains from a metaverse session during a monitored interval (e.g., 10 seconds in the described embodiments).

Table 5: List of volumetric attributes for metaverse user activity classification.

These volumetric attributes collectively cover distinguishable volumetric characteristics of a metaverse session in different activity states. A representative set of attribute values under different states as computed from the MultiverseVR traffic traces is represented as a heat map in Figure 17 (where the values have been normalized to their maximum purely for the purpose of visualization). Each activity state has its own attribute value range as represented by shading in Figure 17, except attributes All to A30 that are omitted from the heatmap as they all remain zero for MultiverseVR. The zero-valued twenty attributes are for TCP of primary domain and UDP of time-critical activity domains, which do not exist in the four studied metaverses. However, they are reserved to capture future variations such as QUIC over UDP for primary metaverse content.

Methodology of Metaverse Session Detection & User Activity State Classification

Figure 4 is a flow diagram of a metaverse monitoring process 400 in accordance with the described embodiments of the present invention. The process receives a packet stream (e.g., from a tap or mirror) from an operational network to detect and classify metaverse user activities in three stages, each of which has a unique objective and scope of processed packet streams, and thus is equipped with its own runtime data structure (404, 414, 428) and inference models (406, 416, 422). At a high level, the first stage (left-hand section of Figure 4) identifies flows to primary domains and their service prefixes, while the second stage (middle section) identifies flows toward time- critical activity domains. The third stage (right-hand section) uses the outputs of the first two stages to identify metaverse sessions and optionally also to classify user activity states, as described in more detail below.

First Stage: Detecting Primary Domain Flows

The first stage detects metaverse flows toward primary domains. At step 402, this stage receives upstream packets from an operational network (e.g., through a packet mirroring configuration on the edge router) with filtering conditions to select TCP flows to destination port 443, the potential candidates of primary flows.

At step 404, the received packets are sorted into a runtime data structure (e.g., a hash table in the described embodiments) with two key layers indexed by user IP and flow 5-tuples. The value field of the data structure is the sequence of upstream packet sizes of this flow. At runtime (step 406), the updated packet payload size sequence of a flow is checked against a pre-trained inference model - a collection of size sequences covering SSL handshakes and first application data, as described above. Each packet size sequence in the inference model is mapped to a primary domain prefix and name. For example, the packet size sequence [409,75,6,45,269] is mapped to the config prefix of the domain name altvr. As shown in Table 3, the first large size (409 bytes) of this sequence is for SSL client hello, the three following small sizes are for CKE, CCS and EHM, and the fifth size (269 bytes) is for the first application data. A perfect match of a flow packet size sequence triggers a successful detection at step 408. An inference model is used for this purpose because there are so many (e.g., over 1000 flows in the representative 45-minute session described above) primary flows to be detected, each with its own unique packet payload size sequence. At step 410, the metadata of the detected flow, including user IP, service prefixes and flow 5-tuples, is sent to the third stage.

Second Stage: Detecting Time-Critical Activity Domain Flows

The second stage detects metaverse flows toward time-critical activity domains. At step 412, all upstream UDP packets with their destination service port numbers within a prelisted range (e.g., as shown in Table 4) obtained from ground-truth training traffic traces is received by this stage. Similar to the first stage, at step 414, a run-time data structure containing two key layers of user IP and flow 5-tuples tracks the size sequences of UDP packets in each flow, and reports the updated size sequence to a pre-trained inference model at step 416 for detection at step 418. At step 420, the metadata of detected flows (i.e., user IP, metaverse name, flow 5-tuples) that have matched port numbers and size sequences is sent to the third stage.

Third Stage: Detecting Metaverse Session and Classifying User Activity State

The third stage detects (at step 422), classifies (at step 424), and tracks (at step 426) active metaverse VR sessions to give operators visibility in their networks and/or to allow corresponding network configuration actions to be taken, as described below. The detection is based on tracking service prefixes of active primary flows, and the classification is achieved by analysing volumetric attributes of active metaverse VR sessions through a stateful approach using machine-learning models. User activity tracking is achieved by processing volumetric attributes of classified metaverse user activities and network performance metrics (in the described embodiments, these metrics being packet rate, throughput, latency, jitter, and distribution of servers of metaverse primary/time-critical activity domain flows). In dependence on these network performance metrics, the metaverse monitoring apparatus and process can perform network re-configuration actions to improve performance. In the described embodiments, these actions are performed by the network configuration component 318, which uses network application programming interfaces (APIs) 320 and network slices to: (i) map network traffic to and/or from time-critical domains to low-latency network slices, (ii) prioritise primary flows that require high bandwidth, and/or (iii) reconfigure routing paths for service domains with poor performance. However, it will be apparent to those skilled in the art in light of this disclosure that other network configuration actions may be taken in other embodiments.

Detection via prefix sequence of primary domain flows

As discussed above, primary flows always start at the beginning of each metaverse session for authentication and initialisation. Thus, in the detection models shown in Figure 4, the primary domain prefixes of active metaverse flows from each session are tracked as a detection criterion. When all primary domain prefixes that should appear in the initial HS states (i.e., as shown in Table 3) have been marked as active from the detection results of stage 1 (step 408), corresponding entries are created in a runtime data structure 426 to track the network statistics of the detected metaverse session. A detected time-critical activity domain flow (from stage 2) of a tracked metaverse session triggers the addition of a corresponding entry to the run-time data structure 426.

At step 428, volumetric statistics of the tracked flows, including their packet counts and volumes over a certain interval (e.g., 10 seconds in the described embodiments), are updated according to the arrived packets 430, which are periodically sent to the classification module to determine user activity states at step 424, as shown in Figure 5.

Classification via volumetric attributes

Using the volumetric attributes as inputs, the inventors developed a stateful approach for user activity state classification with machine-learning models (developed using a random forest algorithm in the described embodiments) for each metaverse, as shown in Figure 5. The stateful design was motivated by the inadequate performance of preliminary classification efforts using stateless machine learning models, as discussed below. The key reason for adopting the stateful approach is that each state's unique traffic patterns might not always be consistent during all intervals. For example, MH and SPE have very spiky patterns in the use of primary domains during their beginning intervals, and the CC state occasionally exhibits patterns of content uploading that might not always be captured in each monitored interval.

As shown in Figure 5, the core logic of the stateful classification is as follows. During operation, stateful machine learning ("ML") classifiers 502 infer user activity states with high confidence by considering both the traffic attributes 504 of the current interval and the previous N (N = 5 in the described embodiments) states 506. In the event that there are fewer than N previous states, or the confidence level of a result from the stateful ML classifiers 502 is determined (at step 508) to be lower than an administrator-defined threshold T (T=85% in the described embodiments). In that case, the user activity state 510 of the current interval is decided by stateless ML classifiers 512 on traffic attributes 504.

User activity monitoring and user experience tracking

For the detected active metaverse user sessions with classified activity states, their volumetric attributes are further aggregated by the user activity state analyser 314 so that a network operator can better understand the metaverse activity in the network such as data volume and user time consumed by different activity states of the detected metaverse applications, and/or so that responsive network configuration actions can be taken. In addition, the user experience analyser 316 monitors the network performance metrics of each detected primary domain flow and time-critical activity domain flow, including packet rate, throughput, latency, jitter, and destination server address, so that a network operator has visibility into metaverse user experience, and/or so that responsive network configuration actions can be taken. For example, they can see which destination servers have poor network metrics as a reference for routing optimisation. EXAMPLE

An embodiment of the metaverse monitoring apparatus and process was developed leveraging virtual network functions built on top of the DPDK framework. The apparatus processed real-time traffic streams from the inventors' university campus network, mirrored to two 10 Gbps network interfaces on a commodity server in the inventors' laboratory. The apparatus can easily be scaled up for a high-throughput network by adding more parallel computes. Detection signatures and classification models loaded on the prototype were generated by the training process described above, and the resulting models can easily be updated using new and comprehensive datasets. The efficacy of the metaverse monitoring apparatus and process in detecting and classifying ground-truth metaverse user activities from the inventors' laboratory environment, as well as their practicality to be deployed in a large network, will be apparent from the following description.

Evaluation in the Lab Environment

The accuracy of the metaverse monitoring apparatus and process for detecting and classifying ground-truth user activities was demonstrated for four metaverse VR applications in the inventors' laboratory lab network environment.

Detection Accuracy

For the purposes of evaluation, 20 sessions of each of the metaverse applications described above were launched, as well as four types of other networked applications for comparative testing, including PC visits to metaverse websites, VR games in the Oculus platform, online PC games, and downloads of metaverse VR console. The accuracies of detecting primary domain flows (stage 1), time-critical activity domain flows (stage 2), and active metaverse sessions (stage 3) are reported in Table 6 below. Each cell in the table represents a combination of true positive (TP) and false positive (FP) rates in the format of T P |F P.

First, since metaverse sessions are detected using rigorous matching criteria of domain prefix sequences from primary flows, all metaverse VR sessions were correctly detected with 100% TP rates. No other application session was misidentified as a metaverse session, resulting in 100% 10 in all four cells under the "Session" column. There is also 100% temporal coverage of each metaverse VR session by collectively considering primary and time-critical activity flows. Table 6: Detection accuracy (in the format of "true positive | false positive") for metaverse sessions, primary domain flows and time-critical activity domain flows. The accuracy for flows is reported in four categories including 'AH', 'Long' (i.e., more than 30 seconds), 'Med.' (i.e., 10 to 30 seconds) and 'Short' (i.e., less than 10 seconds).

The detection accuracy of primary and time-critical domain flows are reported for four categories of session duration, including all, long (i.e., >30s), medium (i.e., from 10s to 30s), and short (i.e., <10s). In general, flows of both domain types for all four metaverses were accurately detected, mostly with >95% TP and absolute 0 FP. It is also noted that five long and seven short non-metaverse flows were misidentified as primary flows of MultiverseVR and Rec Room, respectively. Given the large number (more than IM) of non-Meta flows in the evaluation dataset, the two FPs are labeled as 0% in Table 5. In addition, long primary flows of VRChat, short time-critical activity flows of MultiverseVR, and medium time-critical activity flows of AltSpaceVR were detected with accuracies lower than 85%. This might be due to the limitation of the training dataset that it does not provide a very comprehensive coverage of all possible flow profiles. Such short flows are often for ad-hoc items instead of major event content. Therefore, it is important to note that missing such non-critical flows does not impact the detection of active metaverse sessions. Classification Accuracy

In work leading up to the invention, the inventors initially developed stateless classifiers purely using network attributes extracted per interval, which does not have adequate accuracy especially in the CC, MH, and SPE states, which do not always have consistent network profiles. The detailed accuracy of the stateless method is discussed below.

Table 7 below shows the accuracy of the subsequently developed stateful process of the described embodiments that achieves high accuracy in user activity state classification for all studied metaverses. In short, it achieves more than or nearly 95% true-positive rate, and less than 1% false-positive rate for almost all user activity states.

Table 7: Classification accuracy (i.e., "true positive | false positive") of user activity states using the described ML-based stateful classification approach on the four studied metaverse VR applications.

In comparison, the accuracy of the prior stateless ML-based method in classifying user activity states of the four studied metaverse VR applications using the evaluation dataset is shown in Table 8 below, which lists the resulting accuracies (in the format of "true positive | true negative") for each available user activity state in different metaverses. Because each metaverse VR application might only support certain user activity states, the unavailable states are marked as in Table 8.

Table 8: Classification accuracy (i.e., "true positive | false positive") of user activity states using the stateless machine-learning classifiers on the four studied metaverse VR applications.

It is noted that that the stateless approach only covers current traffic features of each activity type without considering the past status (i.e., classification results), which can provide important patterns that only appear periodically, and thus might not consistently achieve good accuracy, especially when compared with the stateful approach described above.

Deployment Insights in a Campus Network

After evaluating the performance of the metaverse monitoring apparatus and process in a laboratory environment, the inventors demonstrated its capability in an operational network (i.e., the university campus) with millions of concurrent flows and tens of Gbps traffic throughput in real-time. The embodiment of the metaverse monitoring apparatus and process used for this purpose focuses on flows toward primary and time-critical domains that are directly operated for metaverse VR applications, but can be extended to third-party service and cloud content domains visited by active metaverse sessions. However, tracking general-purpose domain usage could be inaccurate for metaverse sessions behind NAT. Accordingly, the metaverse monitoring apparatus and process processes packets mirrored from the campus network, providing visibility into emerging metaverse VR usage (including application popularity, bandwidth usage, user habit, service domain distribution and latency) for network optimization.

Ground-truth sessions

In one evaluation period, the inventors stayed in the university dormitory (served by campus WiFi gateways) to immerse themselves in the four studied metaverses and other VR games, arbitrarily and without prior planning. After comparing their system logs, all of their sessions in the four metaverses were successfully detected and classified with accurate temporal coverage, while other VR game sessions were not misidentified. Representative network statistics of a two-hour period are shown in Figures 18 and 19, including the span of detected primary and time-critical activity flows in Figure 18, and their upstream packet rates in Figure 19.

Metaverse activity in a dormitory over a long-weekend holiday

During a long weekend from Saturday midnight to the following Tuesday, the metaverse monitoring apparatus detected ten metaverse sessions in the dormitory VLAN. They are all from one user who was heavily immersed in VRChat during the Saturday and Sunday nights (i.e., from 12 AM to 4 PM). In contrast, the user stayed in the metaverse for a relatively short duration during the daytime on Monday; perhaps they had classes on the following Tuesday.

Figure 22 visualises the upstream packet rates of the entire university network ("All"), primary flows ("VRC. Primary"), and time-critical activity flows ("VRC. Time-critical."), respectively. Figures 20 and 21 show respectively the volume consumed in Gigabytes and the time spent in hours by the dormitory metaverse sessions in the two available activity states of VRChat (i.e., HS and SUE) over the long weekend. One-third of the user time was spent in the home space, which only consumed about one-sixth of the total upstream volume. This is consistent with the inventors' finding that the SUE state is very demanding and requires prioritized network resource provisioning for premium metaverse experience.

There are tens of ASes accessed by the metaverse sessions detected in the wild. Five ASes serving primary domain content are used as representative examples. The perflow latency from each AS is shown in Table 9 below, with each column representing a corresponding latency range. It is clear that most of the flows in AS1, AS2, AS4, and AS5 had good latencies (i.e., less than 20ms), while a minor fraction of them were suffering from poor latencies (20 - 50ms). Six flows toward AS3 that operates some primary domain services of VRChat had poor latencies greater than 50ms, possibly requiring routing/caching optimization by the corresponding network operator.

Table 9: User-perceived latency of each AS involved in VR-Chat by dormitory residents (shown by number of flows and their fractions).

While there is not yet a lot of metaverse activity in the inventors' university campus network, the examples described above demonstrate that the metaverse monitoring apparatus and process provide network operators with visibility into metaverse sessions and their activity states in their networks, allowing responsive network configuration actions to be performed. In addition, it allows network operators to understand the distribution of metaverse service domains and latencies seen by users as an important reference for experience-driven network optimizations.

Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention.

Claims

CLAIMS:

1. A metaverse monitoring process for execution by at least one processor of a metaverse monitoring apparatus of a communications network, the process including the steps of: receiving data packets of one or more user network addresses and candidate primary network flows of metaverse application domains of known metaverse applications; processing initial data packets of the candidate primary network flows to generate, for each said candidate primary network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate primary network flow for the user network address; comparing the generated sequences of initial upstream packet sizes of the candidate primary network flows to corresponding predetermined sequences of initial upstream packet sizes of primary network flows of known metaverse application domains of each of a plurality of known metaverse applications to identify, for each said user network address, each of the candidate primary network flows for the user network address as being a corresponding primary network flow of a corresponding one of the known metaverse application domains of a corresponding one of the known metaverse applications; and responsive to determining that primary network flows of all of the known metaverse application domains of a corresponding one of the known metaverse applications have been identified for a corresponding user network address, detecting a session of the corresponding known metaverse application for the corresponding user network address.

2. The metaverse monitoring process of claim 1, wherein the predetermined sequences of initial upstream packet sizes of primary network flows of the known metaverse application domains include packet sizes of SSL handshake packets and first application data of the known metaverse application domains.

3. The metaverse monitoring process of claim 2, wherein the predetermined sequences of initial upstream packet sizes of primary network flows of the known metaverse application domains include respective packet sizes of the following five data packets of the primary network flows of each of the known metaverse application domains: SSL client hello, crypto key exchange (CKE), crypto key selection (CCS), encrypted handshake message (EHM), and the first application data of the known metaverse application domain.

4. The metaverse monitoring process of any one of claims 1 to 3, wherein the step of comparing includes determining a corresponding prefix and a corresponding domain identifier of the corresponding one of the known metaverse application domains.

5. The metaverse monitoring process of claim 4, wherein the step of comparing includes using a trained inference model to map, for each said user network address, a corresponding one of the generated sequences of initial upstream packet sizes to the corresponding prefix and corresponding domain identifier of the corresponding known metaverse application domain for the user network address.

6. The metaverse monitoring process of any one of claims 1 to 5, including: receiving data packets of candidate time-critical activity flows towards metaverse application domains listening on a predetermined range of port numbers; processing the data packets of the candidate time-critical activity flows to generate, for each said candidate time-critical activity flow and corresponding user network address, a corresponding sequence of packet sizes of the candidate time-critical activity flow for the user network address; and comparing the generated sequence of packet sizes of each candidate time-critical activity flow to corresponding predetermined sequences of packet sizes of time-critical activity flows towards known metaverse application domains listening on a predetermined range of port numbers to identify at least one of the candidate time-critical activity flows as being a time-critical activity flow of a corresponding one of the known metaverse application domains for the user network address.

7. The metaverse monitoring process of claim 6, including: responsive to said detecting, dynamically generating volumetric network traffic statistics for the known metaverse application for the user network address; and comparing the generated volumetric network traffic statistics to corresponding predetermined volumetric network traffic statistics of predetermined user activity states of the known metaverse application to dynamically determine a corresponding one of a plurality of predetermined user activity states of the known metaverse application domain as a current user activity state for the user network address, wherein a current user activity of the user network address is dynamically classified as the dynamically determined user activity state of the predetermined user activity states of the known metaverse application.

8. The metaverse monitoring process of claim 7, wherein the predetermined volumetric network traffic statistics include packet counts, volumes, and flow counts for both primary flows and time-critical activity flows.

9. The metaverse monitoring process of claim 7 or 8, wherein the step of comparing includes applying a stateful machine learning model.

10. The metaverse monitoring process of claim 9, wherein the stateful machine learning model is configured to dynamically classify the current user activity during a detected metaverse session, wherein each successive current user activity state is dynamically determined on the basis of immediately previous user activity states and current volumetric network traffic statistics for primary flows and time-critical activity flows, including packet counts, volumes, and flow counts.

11. The metaverse monitoring process of any one of claims 1 to 10, including generating usage and performance data representing one or more of: a count of metaverse users, a count of sessions of each metaverse application, network bandwidth consumption values for respective different user activity states of each metaverse session, times spent in respective different user activity states of each metaverse session, a count of network flows and corresponding network flow types for corresponding user activity states of each metaverse session, a collection of information (one or more of IP address, service prefix, domain name, subnet, and autonomous system) of servers serving each metaverse session, and latencies between the user and the distributed servers in each metaverse session.

12. The metaverse monitoring process of any one of claims 1 to 11, including : receiving data packets from a software-defined networking (SDN) flow switch of a communications network; and processing header fields of the received data packets to identify subsets of the data packets as belonging to the candidate network flows of the metaverse application domains of the known metaverse applications; wherein only the identified subsets of the data packets are received as the data packets of the candidate primary network flows of the metaverse application domains of the known metaverse applications.

13. A metaverse monitoring training process, including: receiving network packets of primary network flows of metaverse application domains of a session of a known metaverse application; processing initial data packets of the primary network flows to generate, for each said primary network flow, a corresponding sequence of initial upstream packet sizes of the primary network flow of the corresponding metaverse application domain; and processing the generated sequences of initial upstream packet sizes of the known metaverse application domains to generate trained models for the known metaverse application domains, the trained models being configured to classify each of a plurality of further corresponding sequences of initial upstream packet sizes of candidate primary network flows of metaverse application domains as belonging to a corresponding one of the metaverse application domains.

14. The metaverse monitoring training process of claim 13, including: receiving data packets of known time-critical activity flows and towards known metaverse time-critical activity domains; processing the received data packets of the known time-critical activity flows to generate, for each said known time-critical activity flow, a corresponding sequence of packet sizes of the known time-critical activity flow towards the known metaverse time-critical activity domain; and processing the generated sequences of packet sizes of the known time- critical activity flows towards the known metaverse time-critical activity domains to generate trained models for classifying each of a plurality of further corresponding sequences of candidate time-critical activity flows towards metaverse time-critical activity domains as being a corresponding time-critical activity flow of a corresponding one of the known metaverse time-critical activity domains.

15. The process of any one of claims 1 to 14, including monitoring network performance of at least one of the detected metaverse applications, and based on the monitoring, causing reconfiguration of at least one networking component to improve the network performance of the at least one of the detected metaverse applications.

16. The process of claim 15, wherein the step of causing includes using one or more network application programming interfaces (APIs) and network slices to: (i) amend routing paths for one or more corresponding metaverse application domains, (ii) prioritise metaverse traffic flows, and/or (iii) map metaverse application traffic to network slices.

17. A computer-readable storage medium having stored thereon processorexecutable instructions that, when executed by at least one processor of a metaverse monitoring apparatus, cause the metaverse monitoring apparatus to execute the process of any one of claims 1 to 16.

18. A metaverse monitoring apparatus of a communications network, the apparatus including components configured to execute the process of any one of claims 1 to 16.

19. A metaverse monitoring apparatus of a communications network, including: a network interface to receive data packets of candidate network flows of metaverse application domains; a primary network flow sequence generator configured to process initial data packets of candidate primary network flows of metaverse application domains to generate, for each said candidate primary network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate primary network flow for the user network address; a metaverse session identifier configured to:

20. The metaverse monitoring apparatus of claim 19, further including: a time-critical activity network flow sequence generator configured to process initial data packets of candidate time-critical activity network flows towards metaverse application domains listening on a predetermined range of port numbers to generate, for each said candidate time-critical activity network flow and corresponding user network address, a corresponding sequence of initial upstream packet sizes of the candidate time-critical activity network flow for the user network address; and a time-critical activity network flow detector configured to compare the generated sequence of initial upstream packet sizes of each candidate time- critical activity network flow to predetermined sequences of initial upstream packet sizes of predetermined time-critical activity network flows of known metaverse application domains to identify the candidate time-critical activity network flow as being a time-critical activity network flow of a corresponding one of the known metaverse application domains for the corresponding user network address.

21. The metaverse monitoring apparatus of claim 20, further including: a metaverse user activity identifier configured to: responsive to said determining, dynamically generate volumetric network traffic statistics for the detected session of the known metaverse application and corresponding user network address; and compare the generated volumetric network traffic statistics to corresponding predetermined volumetric network traffic statistics of known user activity states of the known metaverse application to dynamically determine a corresponding one of the known user activity states of the known metaverse application domain as a current user activity state for the user network address, wherein a current user activity of the user network address is classified as the determined state of the known states of the known metaverse application.

22. The metaverse monitoring apparatus of claim 21, wherein the metaverse user activity identifier includes a stateful classifier and a stateless classifier, and the metaverse user activity identifier is configured to dynamically determine the current user activity state by:

23. The metaverse monitoring apparatus of claim 21 or 22, wherein the metaverse user activity identifier is configured to aggregate over time the volumetric network traffic statistics of the identified primary network flows and the identified time-critical activity flows of different user activity states of detection user sessions of known metaverse application domains for each user network address.

24. The metaverse monitoring apparatus of any one of claims 19 to 23, further including a metaverse user experience analyser configured to monitor network performance metrics, including a distribution of servers among autonomous systems, throughput, packet rate, latency, and jitter of the identified primary network flows and identified time-critical activity network flows of different user activity states of detected user sessions of detected metaverse applications for user network addresses.

25. The metaverse monitoring apparatus of claim 24, further including a network configuration component configured to cause reconfiguration of at least one networking component to improve the network performance of the at least one of the detected metaverse applications, in dependence on the corresponding network performance metrics.