FR3145823A1 - Animation of a virtual entity associated with a first user, simulation method and virtual reality simulator - Google Patents

Abstract

Animation of a virtual entity associated with a first user, simulation method and virtual reality simulator The invention relates to the animation of a virtual entity, in particular an avatar, associated with a first user. More generally, the invention relates to the simulation of virtual reality, in particular of an immersive virtual environment but also of a mixed environment combining real objects/entities and virtual objects/entities. An object of the invention is a method of animating a virtual entity associated with a first user of a virtual reality system during the reproduction of the virtual entity, the virtual entity being composed of at least a first mobile part, the animation comprising: - determining a current virtual position of a first mobile part of a virtual entity associated with the first user according to at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system. Thus, the tracking of the first parts of the body of the first user is improved, thereby reducing the animation errors of the virtual entity associated with the first user and therefore limiting the interaction errors of the first user with virtual reality. Figure for abstract: Figure 1

Description

Animation of a virtual entity associated with a first user, simulation method and virtual reality simulator

The invention relates to the animation of a virtual entity, in particular an avatar, associated with a first user. More generally, the invention relates to the simulation of virtual reality, in particular of an immersive virtual environment but also of a mixed environment combining real objects/entities and virtual objects/entities.

State of the art

Currently, the virtual reconstruction of real hands of users of virtual and mixed reality headsets and glasses relies on the use of a front camera included in the headset or glasses.

However, for the tracking to be permanent, this technique is strongly limited by the mobility of the hands. Indeed, in the event of occultation of a hand by the other hand, wrist or arm of the user or even by another user, the fingers of this occulted hand are then no longer detected. This causes a loss of interaction capacity of the user who is also not aware of this loss of capacity because the virtual hand is always reproduced (in connection with the data captured prior to the occultation). This loss of interaction capacity causes, in addition to interaction errors, confusion and frustration of the user.

S. Golodetz, et al propose in their article “Live Collaborative Large-Scale Dense 3D Reconstruction Using Consumer-Grade Hardware”, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2018 pp.413-414 to use several cameras placed in the environment in order to reconstruct real environments by pooling the reconstructed meshes to reduce the risks of transient change in the captured environment.

In the article "What is virtual reality hand tracking?", published on 23.06.2022, https://vrx.vr-expert.com/what-is-virtual-reality-hand-tracking/, a so-called "outside-in" hand tracking is described using several external sensors fixed in a specific room and forming a tracking zone to generate a more precise representation of the person's position and movements (hands included).
The first disadvantage of these two techniques is a more complex and therefore more expensive virtual reality system due to the need to use sensors/cameras in addition to the equipment of the virtual reality headset/glasses worn by the user. These sensors/cameras being installed in a specific room and defining a fixed tracking area, this restricts the area of use of the virtual reality system by the user. Consequently, a second disadvantage is to limit the nomadism of the virtual reality headset/glasses to the tracking area. Finally, the sensors/cameras instrumenting the environment communicate with the virtual reality device tracking the hands using a remote "directory" server which allows them to be connected to the virtual reality headset/glasses. A third disadvantage is therefore to use the Internet network.

One of the aims of the present invention is to remedy drawbacks of the state of the art.

An object of the invention is a method of animating a virtual entity associated with a first user of a virtual reality system during the reproduction of the virtual entity, the virtual entity being composed of at least a first mobile part, the animation comprising:
- determining a current virtual position of a first mobile part of a virtual entity associated with the first user based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system.

Thus, the tracking of the first parts of the body of the first user is improved, thereby reducing the animation errors of the virtual entity associated with the first user and therefore limiting the interaction errors of the first user with virtual reality.

Advantageously, the animation process includes:
- estimating a current real position of at least a first part of the body of a first user of the virtual reality system based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system;
the determination of the current virtual position of the first mobile part of the virtual entity associated with the first user being carried out as a function of the estimated current real position of the first part of the body of the first user.

Advantageously, the estimation of the current actual position is furthermore a function of first data relating to the first part of the body of the first user captured by means of a first sensor worn by the first user, the first sensor and the second sensor being co-located.

Thus, the pooling of captured data further improves the monitoring of the first parts of the body of the first user.

Advantageously, the animation process includes:
- if a first part of a user's body to which the captured data relates is connected to at least a second part of the first user's body, assigning to the first user the current virtual position determined as a function of the captured data relating to at least a first part of a user's body, the captured data comprising at least one piece of data among the first captured data and the second captured data.

Thus, the invention makes it possible to remove ambiguity about the body part being tracked and reduces the risks that the first user interacts using the second virtual entity associated with the second user.

Advantageously, when the estimation of the current actual position receives several of the second data captured from several distinct second sensors worn by several distinct second users, the estimation of the current actual position comprises:
- weighting the second captured data provided by a second sensor according to a current relative position of the second sensor with respect to the first user prior to estimating the current actual position according to all of the weighted second captured data.

Thus, the invention takes into account the relevance of the captured data to disambiguate the tracking of the body parts of the first user.

Advantageously, the data captured is one or more of the following data:
- a captured image;
- an infrared image captured;
- a captured video;
- a captured movement;
- a captured depth.

Advantageously, the virtual entity is one of the following entities:
- an avatar of the first user;
- a virtual being;
- a virtual mobile device.

An object of the invention is also a virtual reality simulation method comprising:
- generating a virtual reality capable of being reproduced simultaneously by a first portable virtual reality reproduction device worn by a first user and a second portable virtual reality reproduction device worn by a second user, the virtual reality comprising at least one virtual entity; and
- animate the virtual entity associated with a first user by:
+ determining a current virtual position of a first mobile part of a virtual entity associated with the first user based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system.

Advantageously, according to an implementation of the invention, the different steps of a method according to the invention are implemented by software or computer program, this software comprising software instructions intended to be executed by a data processor of a device forming part of a virtual reality simulator and being designed to control the execution of the different steps of this method.

The invention therefore also relates to a program comprising program code instructions for executing the steps of the animation method and/or the virtual reality simulation method when said program is executed by a processor.

This program may use any programming language and may be in the form of source code, object code, or code intermediate between source code and object code, such as in partially compiled form or in any other desirable form.

An object of the invention is also a virtual reality simulator comprising:
- a virtual reality generator capable of being reproduced simultaneously by a first portable virtual reality reproduction device worn by a first user and a second portable virtual reality reproduction device worn by a second user, the virtual reality comprising at least one virtual entity;
- an animation controller of the virtual entity associated with a first user of a virtual reality system during the reproduction of the virtual entity, the virtual entity being composed of at least a first mobile part, the animation controller being able to:
+ determine a current virtual position of a first mobile part of a virtual entity associated with the first user based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality simulator.

Advantageously, the animation controller includes a tracer capable of estimating the current real position of the first part of the body of the first user.

Advantageously, the virtual reality simulator comprises a signal transmitter connected to the first portable virtual reality reproduction device, to the second portable virtual reality reproduction device, to a first sensor worn by the first user and to the second sensor, the first sensor and the second sensor being co-located, the transmitter being able to transmit virtual reality reproduction signals to the first portable virtual reality reproduction device and to the second portable virtual reality reproduction device, and to receive first data captured from the first sensor and second data captured from the second sensor.

Advantageously, the transmitter is connected to the first portable virtual reality reproduction device, to the second portable virtual reality reproduction device, to the first sensor and to the second sensor via a local network.

Thus, animation control can be performed without knowledge of the location of the first and second users. In addition, this reduces processing times and enhances the instantaneity of the animation of the virtual entity.

An object of the invention is also a virtual reality system comprising:
- a first portable reproduction device carried by a first user,
- a second portable reproduction device carried by a second user,
- a first sensor worn by the first user,
- a second sensor worn by the second user, the first sensor and the second sensor being co-located,
- a virtual reality simulator according to the invention.

Advantageously, the first sensor and the second sensor are respectively sensors integrated into the first virtual reality reproduction device and the second virtual reality reproduction device.

Advantageously, the virtual reality system comprises several distinct second sensors worn by several distinct second users.

The characteristics and advantages of the invention will appear more clearly on reading the description, given by way of example, and the figures relating thereto which represent:

, a simplified diagram of a virtual entity animation method according to the invention,

, a simplified diagram of a virtual reality simulation method according to the invention,

, a simplified diagram of a virtual reality simulator according to the invention,

, a simplified diagram of a virtual reality system according to the invention,

, a simplified diagram of the users of a virtual reality system according to the invention, when the virtual entity is a virtual representation of at least one of the users.

Virtual reality is understood to mean any reality comprising at least one virtual element: virtual object, virtual entity, virtual environment, etc. Thus, virtual reality can be an immersive virtual reality (VR), that is to say a reproduction of a computer-generated universe immersing the user who therefore only perceives virtual elements. An alternative is a virtual reality also called mixed reality (MR) combining real elements and virtual elements allowing the user to perceive and interact with both virtual elements and real elements, or even so-called mixed elements. The term mixed reality encompasses technologies ranging from augmented reality (AR), adding one or more virtual elements in a real environment, to augmented virtuality (AV) allowing the user to perceive or even interact with at least one real element in a virtual environment.

There illustrates a simplified diagram of a method of animating a virtual entity according to the invention.

The animation method EV_ANIM animates a virtual entity EV1 associated with a first user U1 of a virtual reality system during the reproduction of the virtual entity RV_RPR, the virtual entity EV1 being composed of at least a first mobile part . The EV_ANIM animation includes:
- determine VP_DT a current virtual position of a first mobile part of a virtual entity associated with the first user U1 based on at least the second data relating to the first part of the body of the first user captured by means of a second sensor c ₂ worn by a second user U2 of the virtual reality system: .

In particular, the first user U1 is a first real entity ER1 and the second user U2 is a second real entity ER2.

In particular, the EV_ANIM animation process includes:

- estimate RP_EST a current real position of at least a first part of the body of a first user of the SRV virtual reality system based on at least second data relating to the first part of the body of the first user captured by means of a second sensor c ₂ worn by a second user U2 of the virtual reality system: ;
the VP_DT determination of the current virtual position of the first moving part of the virtual entity associated with the first user U1 being carried out according to the current estimated actual position from the first part of the first user's body: .

Optionally, the EV_ANIM animation process includes:
- estimate RP_EST a current real position of at least a first part of the body of a first user of the SRV virtual reality system based on at least second data relating to the first part of the body of the first user captured by means of a second sensor c ₂ worn by a second user U2 of the virtual reality system: ;
- determine VP_DT a current virtual position of a first mobile part of a virtual entity associated with the first user U1 based on the estimated current actual position from the first part of the first user's body: .

In particular, the estimation of the current real position RP_EST is also a function of initial data relating to the first part of the body of the first user captured by means of a first sensor c ₁ worn by the first user U1: .

In particular, the EV_ANIM animation process includes:
- if a first part of a user's body to which the captured data relates is connected to at least a second part of the body of the first user , assign to the first user U1_ATT the current virtual position determined based on the captured data relating to at least a first part of a user's body : j=1, .
The data collected contain at least one piece of data among the first data captured and the second data captured .

In particular, the animation process checks whether a first part of a user's body to which the captured data relates is connected to at least a second part of the body of the first user constituting all or part of the skeleton of the first user.

In particular, when the current real position estimate RP_EST receives several of the second captured data of several separate second sensors carried by multiple separate second users , the estimation of the current real position RP_EST includes:
- weight POND the second captured data provided by a second sensor depending on a current relative position of the second sensor compared to the first user U1: , prior to the estimation of the current real position RP_EST based on all of the second weighted captured data: .

In particular, the current relative position of the second sensor relative to the first user is equal to or a function of the current position of the second sensor relative to the current position of a first sensor, when the first user is wearing a first sensor.

In particular, the captured data d _{k, k=1,2} , d ₁ , d ₂ , are one or more of the following data:
- a captured image;
- an infrared image captured;
- a captured video;
- a captured movement;
- a captured depth
- etc.

In particular, the virtual entity EV1 associated with the first user U1 is one of the following entities:
- an EV1 avatar of the first user U1;
- a virtual being, in particular a virtual animal, a virtual plant, etc.;
- a virtual mobile device, including a virtual robot, a virtual vehicle, a virtual machine, etc.
- etc.

The first user U1 and the second user U2, are co-located. Thus, a second sensor c ₂ , carried by a second U2 user, is able to capture second data relating to the first user U1 usable by the real position estimation RP_EST to estimate the current real position of at least a first mobile part of the first user .

In particular, the EV_ANIM animation includes:
- receive RCV (not shown) of the data captured d _k from one or more sensors c _k (k=1,2), in particular of the second data captured d ₂ , of at least a second sensor c ₂ , carried by a second U2 user, and possibly first data captured d1 from a first sensor _c1 worn by the first user U1.

In particular, the EV_ANIM animation includes:
- extract XTR from the captured data d _k , d1, d ₂ , data relating to at least one first mobile part of a user : .

Optionally, XTR extraction extracts captured data d _k , d ₁ , d ₂ , only data relating to the first user U1 using for example the attribution U1_ATT.

In particular, POND weighting weights the extracted captured data .

In particular, the current real position estimation RP_EST is performed based on the weighted extracted captured data.

In particular, either the generation of virtual reality RV_GN generating one or more virtual reality elements including the virtual entity EV1 associated with the first user U1, or the animation of virtual entity EV_ANIM comprises:
- move a first mobile part of the virtual entity EV1 associated with the first user U1 in the current virtual position determined .

Let the movement of the virtual entity EV_MV command to the reproduction of virtual reality RV_RPR a movement of the associated virtual entity EV1 in particular by updating the reproduction data relating to the first moving part of the associated virtual entity EV1, or by providing new reproduction data ev ₁ of the associated virtual entity EV1.

Thus, the movement of the user U1 triggers a movement of the associated virtual entity EV1. In particular, when the associated virtual entity EV1 is an avatar or a digital twin of the user U1, the movement of the virtual entity eV1 is then a twin movement of the real movement performed by the user U1, that is to say a virtual movement replicating the real movement performed by the user U1.

In particular, the virtual reality generation and/or the virtual reality reproduction are implemented by a virtual reality system SRV. In particular, the virtual reality generation RV_GN is implemented by a virtual reality simulator RV_SIMU (not shown). And, the virtual reality reproduction is, for example, implemented by one or more portable reproduction devices DRk (not shown), in particular a first portable reproduction device DR1 worn by a first user U1 and a second portable reproduction device DR2 worn by a second user U2.

In particular, the animation EV_ANIM of a virtual entity EV1 of a virtual reality system during the reproduction of the virtual entity RV_RPR allows not only an instantaneous animation of the virtual entity according to the movement mvt of the first user U1 with which it is associated, in particular of the estimated current real position of a first mobile part of the first user , but also the instantaneous reproduction of the virtual entity EV1 thus animated, in particular the reproduction of the first mobile part of the virtual entity in the current virtual position determined . Thus, immediately as soon as the user makes a movement, the virtual entity is animated according to this movement of the user and the virtual entity thus animated is reproduced.

Conversely, a virtual entity EV2 (not illustrated) can be associated with the user U2, and the animation method EV_ANIM according to the invention used to animate this virtual entity EV2 associated with the user U2, then considered by the animation method EV_ANIM as the first user U2, based on second data captured by a sensor c1 worn by the user U1, then considered by the animation method EV_ANIM as the second sensor c1 worn by the second user U2.

There illustrates a simplified diagram of a virtual reality simulation method according to the invention.

The VR_PSIM virtual reality simulation process includes:
- generate RV_GN a virtual reality VR (see ) capable of being reproduced simultaneously by a first portable virtual reality reproduction device DR1 worn by a first user U1 and a second portable virtual reality reproduction device DR2 worn by a second user U2, the virtual reality VR comprising at least one virtual entity EV1; and
- animate EV_ANIM the virtual entity EV1 (see ) associated with a first user U1 in:

+ determining VP_DT a current virtual position of a first mobile part of a virtual entity associated with the first user U1 based on at least the second data relating to the first part of the body of the first user captured by means of a second sensor c ₂ worn by a second user U2 of the virtual reality system: .

In particular, the EV_ANIM animation process includes:

- estimate RP_EST a current real position of at least a first part of the body of a first user of the SRV virtual reality system based on at least second data relating to the first part of the body of the first user captured by means of a second sensor c ₂ worn by a second user U2 of the virtual reality system: ;
the VP_DT determination of the current virtual position of the first moving part of the virtual entity associated with the first user U1 being carried out according to the current estimated actual position from the first part of the first user's body: .

In particular, the VR_PSIM virtual reality simulation process includes:
- generating RV_GN a virtual reality VR capable of being reproduced simultaneously by a first portable virtual reality reproduction device DR1 worn by a first user U1 and a second portable virtual reality reproduction device DR2 worn by a second user U2, the virtual reality VR comprising at least one virtual entity EV1; and
- animate EV_ANIM the virtual entity EV1 associated with a first user U1 by:
+ estimating RP_EST a current real position of at least a first part of the body of a first user of the SRV virtual reality system based on at least second data relating to the first part of the body of the first user captured by means of a second sensor c ₂ worn by a second user U2 of the virtual reality system: ;
+ determining VP_DT a current virtual position of a first mobile part of a virtual entity associated with the first user U1 based on the estimated current actual position from the first part of the first user's body: .

In particular, the VR_GN virtual reality generation includes:
- a generation of virtual entity EV_GN whose virtual entity is associated with the first user U1.

In particular, the VR_GN virtual reality generation includes:
- a generation of ZV_GN virtual environment.

Thus, the virtual reality generated is an immersive virtual reality or an augmented virtuality.

In particular, the virtual reality generation RV_GN provides the virtual entity animation EV_ANIM with data relating to the virtual entity associated with the first user ev1, in particular reproduction data of the virtual entity associated with the first user.

In particular, the EV_ANIM animation of the RV_PSIM virtual reality simulation method comprises one or more of the steps of the EV_ANIM animation method described above in association with the .

In particular, the EV_ANIM animation includes:

- move EV_MV a first mobile part of the virtual entity EV1 associated with the first user U1 in the current virtual position determined .

In particular, the simulation process includes:
- modify RV_MDF this generated data relating to the virtual entity associated with the first user ev1 according to the current virtual position determined :

In particular, the VR_MDF virtual reality modification includes:
- replace EV_RPL (not shown) this generated data relating to the virtual entity associated with the first user ev1 with data relating to the moved virtual entity provided by the virtual entity animation EV_ANIM, in particular by the movement of the virtual entity EV_MV.

In particular, the virtual reality modification RV_MDF performs this replacement in the generated virtual reality data rv provided by the virtual reality generation RV_GN comprising the generated data of the virtual entity associated with the first user ev1 and possibly generated data of other virtual entity(ies). and/or data generated from a virtual environment zv.

So either the virtual entity animation EV_ANIM provides data relating to the virtual entity modified according to the current virtual position determined to a modification of the virtual reality RV_MDF which replaces in the generated data relating to the virtual reality rv provided by the virtual reality generation RV_GN the generated data relating to the generated virtual entity

Alternatively, the VR_MDF virtual reality modification includes:
- an update of the reproduction data relating to either the first moving part of the associated virtual entity EV1, or reproduction data ev ₁ of the associated virtual entity EV1 according to a command from a movement of the associated virtual entity EV1 coming from the displacement of the virtual entity EV_MV.

Thus, the movement of the user U1 triggers a movement of the associated virtual entity EV1. In particular, when the associated virtual entity EV1 is an avatar or a digital twin of the user U1, the movement of the virtual entity eV1 is then a twin movement of the real movement performed by the user U1, that is to say a virtual movement replicating the real movement performed by the user U1.

Thus, the virtual reality simulation method RV_PSIM provides virtual reality data rv comprising data relating to the virtual entity modified according to the current virtual position determined : .

A particular embodiment of the animation method and/or the virtual reality simulation method is a program comprising program code instructions for executing the steps respectively of the animation method and/or the virtual reality simulation method when said program is executed by a processor.

There illustrates a simplified diagram of a virtual reality simulator according to the invention.

Virtual Reality Simulator 1.61 features:
- a generator 11 of virtual reality rv capable of being reproduced simultaneously by a first portable device 2 ₁ , 62 ₁ for reproducing virtual reality worn by a first user 4 ₁ of the virtual reality simulator and a second device 2 _2, , 2 _2, portable virtual reality reproduction worn by a second user 4 ₂ , of the virtual reality simulator, the generated virtual reality rv comprising at least one virtual entity ev1;
- a controller 12 for animating the virtual entity 5 ₁ associated with a first user 4 ₁ during the reproduction of the virtual entity, the virtual entity 5 ₁ being composed of at least a first mobile part , the animation controller 12 being able to:
+ determine a current virtual position of a first mobile part of a virtual entity associated with the first user 4 ₁ based on at least second data relating to the first part of the body of the first user 4 ₁ captured by means of a second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , worn by a second user 4 ₂ of the virtual reality simulator 1, 61: .

In particular, the virtual reality generator 11 comprises:
- a generator 112 of virtual entity ev1 whose virtual entity 5 ₁ is associated with the first user 4 ₁ .

In particular, the virtual reality generator 11 comprises:
- a 111 virtual environment generator zv.

So, the virtual reality generated rv is an immersive virtual reality or an augmented virtuality.

Note, rv, ev, zv represent the reproduction signals generated respectively from a virtual reality 5, from a virtual entity such as that associated 5 ₁ with the first user 4 ₁ , a virtual environment (not illustrated).

In particular, the virtual reality generator 11 provides the virtual entity animation controller 12 with data relating to the virtual entity associated with the first user ev1 whose mobile part is in a predetermined virtual position. : , including reproduction data of the virtual entity associated with the first user. Note that the predetermined virtual position is in particular a default virtual position , or the previous virtual position

In particular, the animation controller 12 includes a current virtual position calculator 125. of a first mobile part of a virtual entity associated with the first user 4 ₁ able to determine the current virtual position based on at least second data relating to the first part of the body of the first user 4 ₁ captured by means of a second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , worn by a second user 4 ₂ of the virtual reality simulator 1, 61: .

In particular, the animation controller 12 is capable of:
- estimate a current real position of at least a first part of the body of a first user of the virtual reality simulator 1, 61 based on at least second data relating to the first part of the first user's body captured by means of a second sensor 3 ₂ , , 23 ₂ , worn by a second user 4 ₂ of the virtual reality simulator 1;
determining the current virtual position of the first moving part of the virtual entity associated with the first user 4 ₁ being carried out according to the estimated current actual position from the first part of the first user's body: .

In particular, the animation controller 12 comprises:
- a 123 estimator of current real position of at least a first part of the body of a first user of the virtual reality simulator 1, 61 capable of estimating a current real position based on at least second data relating to the first part of the first user's body captured by means of a second sensor 3 ₂ , , 23 ₂ , worn by a second user 4 ₂ of the virtual reality simulator 1.

Thus, the calculator 125 is able to determine the current virtual position. based on the current estimated actual position from the first part of the first user's body: provided by estimator 123.

Advantageously, the animation controller 12 comprises a tracer 123 capable of estimating the current real position. of the first part of the body of the first user.

Advantageously, the virtual reality simulator 1, 61 comprises a signal transmitter 10 connected to the first portable virtual reality reproduction device 2 ₁ , 62 ₁ , to the second portable virtual reality reproduction device 2 _2, , 62 _2, , to a first sensor 3 ₁ , 23 ₁ , 63 ₁ , 623 ₁ worn by the first user 4 ₁ and to the second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , carried by the second user 4 ₂ , . The first sensor 3 ₁ , 23 ₁ , 63 ₁ , 623 ₁ and the second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , are co-located. The transmitter 10 is capable of transmitting virtual reality reproduction signals rv to the first portable virtual reality reproduction device 2 ₁ , 62 ₁ and to the second portable virtual reality reproduction device 2 _2, , 62 _2, , and receiving first data d ₁ captured from the first sensor 3 ₁ , 23 ₁ , 63 ₁ , 623 ₁ and second data d2, captured from the second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , .

Advantageously, the transmitter 10 is connected to the first portable virtual reality reproduction device 2 ₁ , 62 ₁ , to the second portable virtual reality reproduction device 2 _{2 ,} , 62 _2, , to a first sensor 3 ₁ , 23 ₁ , 63 ₁ , 623 ₁ and to the second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , via local network 0.

The first user 4 ₁ and the second user 4 ₂ , are co-located. Thus, a second sensor 3 ₂ , , 23 ₂ , , 63 ₂ , , 623 ₂ , carried by a second user 4 ₂ , is able to capture second data relating to the first user 4 ₁ usable by the real position estimator 123 to estimate the current real position of at least a first mobile part of the first user .

In particular, the virtual reality simulator 1 includes:
- a receiver 103 of the captured data d _k from one or more sensors 2 _k (k=1,2), in particular the second captured data d ₂ , of at least one second sensor 3 ₂ , 23 ₂ , , carried by a second user 4 ₂ , and possibly first data captured d1 from a first sensor 3 ₁ , 23 ₁ worn by the first user 4 ₁ .

In particular, the animation controller 12 comprises:
- an extractor 121 of the captured data d _k , d1, d ₂ , data relating to at least a first mobile part of a user : .

Optionally, the extractor 121 is able to extract captured data d _k , d ₁ , d ₂ , only data relating to the first user 4 ₁ using for example attribution means 124.

In particular, when the animation controller 12 receives several of the second captured data of several separate second sensors carried by multiple separate second users , the animation controller 12 comprises:
- a 122 weighter of the second data captured provided by a second sensor able to weight these second captured data according to a current relative position of the second sensor compared to the first user 4 ₁ : , prior to estimating the current actual position based on all of the second weighted captured data: .

In particular, the current relative position of the second sensor relative to the first user is equal to or a function of the current position of the second sensor relative to the current position of a first sensor, when the first user is wearing a first sensor.

In particular, the weighter 122 is capable of weighting the extracted captured data. .

In particular, the estimator 123 is able to estimate a current real position based on the weighted extracted captured data provided by the weighter 122.

In particular, the animation controller 12 comprises:
- conditional allocation means 124 capable of allocating to the first user the current virtual position determined as a function of the captured data relating to at least a first part of a user's body : j=1, when a first part of a user's body to which the captured data relates is connected to at least a second part of the body of the first user .

The data collected contain at least one piece of data among the first data captured and the second data captured .

In particular, the animation controller 12 comprises:
- allocation means 1241 capable of assigning to the first user the current virtual position determined as a function of the captured data relating to at least a first part of a user's body : j=1, .

- a verifier 1240 capable of verifying whether a first part of a user's body to which the captured data relates is connected to at least a second part of the body of the first user , the verifier 1240 is further capable of triggering the allocation means 1241 when the result of the verification is positive [Y].
The data collected contain at least one piece of data among the first data captured and the second data captured .

In particular, the conditional allocation means 124 includes the allocation means 1241 and the verifier 1240.

In particular, the verifier 1240 verifies whether a first part of a user's body to which the captured data relates is connected to at least a second part of the body of the first user constituting all or part of the skeleton of the first user.

In particular, the calculator 125 is able to determine the current virtual position. based on the current estimated actual position of the first body part assigned to the first user: by means of attribution 124, 1241.

In particular, either the virtual reality generator 11 capable of generating one or more virtual reality elements including the virtual entity 5 ₁ associated with the first user 4 ₁ (not illustrated), or the animation controller 12 comprises:
- an actuator 126 capable of moving a first movable part of the virtual entity 5 ₁ associated with the first user 4 ₁ in the current virtual position determined provided by calculator 125.

Let the virtual entity actuator 126 control to the virtual reality generator 11 a movement of the associated virtual entity 5 ₁ in particular by updating the reproduction data relating to the first mobile part of the associated virtual entity 5 ₁ , or by providing new reproduction data ev ₁ of the associated virtual entity 5 ₁ .

In particular, the virtual reality simulator 1 includes:
- a modifier 116 of the generated data relating to the virtual entity associated with the first user ev1 according to the current virtual position determined :

Optionally, the virtual reality generator 11 includes the modifier 116 of the generated data relating to the virtual entity.

In particular, the virtual reality modifier 116 is capable of:
- replace this generated data relating to the virtual entity associated with the first user ev1 with data relating to the moved virtual entity provided by the virtual entity animation controller 12, in particular by the virtual entity actuator 126.

In particular, the virtual reality modifier 116 is able to perform this replacement in the generated virtual reality data rv provided by the virtual reality generator 11 comprising the generated data of the virtual entity associated with the first user ev1 and possibly data generated from other virtual entity(ies). and/or data generated from a virtual environment zv.

Thus, either the virtual entity animation controller 12 is able to provide data relating to the virtual entity modified according to the current virtual position determined to a virtual reality modifier 116 capable of replacing in the generated data relating to the virtual reality rv provided by the virtual reality generator 11 the generated data relating to the generated virtual entity .

Alternatively, Virtual Reality Modifier 116 is based on:
- update reproduction data relating to either the first moving part of the associated virtual entity 5 ₁ , or reproduction data ev ₁ of the associated virtual entity 5 ₁ according to a command of a movement of the associated virtual entity 5 ₁ coming from the virtual entity actuator 126.

Thus, the virtual reality simulator 1 is able to provide virtual reality data rv comprising data relating to the virtual entity modified by virtual reality modifier 116 based on the current virtual position determined : .

A virtual reality system 6 includes:
- a first portable reproduction device 62 ₁ carried by a first user 4 ₁ ,
- a second portable reproduction device 62 _2, carried by a second user 4 ₂ , ,
- a first sensor 63 ₁ , 623 ₁ worn by the first user 4 ₁ ,
- a second sensor 63 ₂ , , 623 ₂ , carried by the second user 4 ₂ , , the first sensor 63 ₁ , 623 ₁ and the second sensor 63 ₂ , , 623 ₂ , being co-located,
- a virtual reality simulator 61 according to the invention, in particular as described above.

Advantageously, the first sensor 623 ₁ and the second sensor 623 ₂ , are respectively sensors integrated into the first virtual reality reproduction device 62 ₁ and the second virtual reality reproduction device 62 _2, .

Advantageously, the virtual reality system 6 comprises several separate second sensors. carried by multiple separate second users.

There illustrates a simplified diagram of a virtual reality system according to the invention.

In the example of the , the virtual reality simulator in particular as described above is embedded in portable virtual reality devices.

The virtual reality system 6 comprises a first portable virtual reality device 62 ₁ worn by a first real user 4 ₁ , called first user by misuse of language, and a second portable virtual reality device 62 ₂ worn by a second real user 4 ₂ , called second user by misuse of language.

Users then share the same virtual reality space or “Shared Virtual Space” in English, SVS.

The virtual reality system 6 is capable of generating an avatar or virtual entity for each real user. Thus, a first mixed user 45 ₁ comprises the first real user 4 ₁ and a first avatar 5 ₁ associated with this first user, and a second mixed user 45 ₂ comprises the second real user 4 ₂ and a second avatar 5 ₂ associated with this second user.

The virtual reality devices, respectively the first virtual reality device 62 ₁ and the second virtual reality device 62 ₂ comprise in particular a hardware part 62H ₁ , 62H ₂ comprising in particular a virtual reality reproduction device and a sensor, and a virtual reality simulator part 62S ₁ , 62S ₂ in particular in the form of a processor executing a program comprising program code instructions for executing the steps of a virtual reality simulation method.

The virtual reality devices 62 ₁ and 62 ₂ are in particular augmented reality headsets also called “Optical See-Trough – Head-Mounted Display” in English, OST-HMD.

The hardware parts 62H ₁ , 62H ₂ of the first and second virtual reality devices 62 ₁ , 62 ₂ are in particular capable of capturing different parts of the body of the first and second real users 4 ₁ and 4 ₂ : ii1, respectively of the second and first real users 4 ₂ and 4 ₁ : ii2. The captured data ii1 and ii2 specific to the user wearing the device, respectively to the first user and to the second user, allow intuitive interaction of the user 4 ₁ and 4 ₂ with the avatar 5 ₁ and 5 ₂ associated with him. Furthermore, the sensors of the hardware parts 62H ₁ , 62H ₂ are capable of capturing data relating to the environment, including data relating to other users, respectively second user and first user d1, d2.

In particular, the first part 62S ₁ of the virtual reality simulator (i.e. the one embedded in the first virtual reality device 62 ₁ ) follows t1 the parts of the body of the first user 4 ₁ and the parts of the body of the second user 4 ₂ according to the intuitive interaction data ii1 captured, respectively the second part 62S ₂ of the virtual reality simulator (i.e. the one embedded in the second virtual reality device 62 ₂ ) follows t2 the parts of the body of the first user 4 ₁ and the parts of the body of the second user 4 ₂ according to the intuitive interaction data ii2 captured.

Not only does the virtual reality simulator, in particular the first part 62S ₁ of the virtual reality simulator, respectively the second part 62S ₂ of the virtual reality simulator, control an avatar, in particular the first avatar 5 ₁ , respectively the second avatar 5 ₂ , based on a self-estimation of the position of the part of the associated user (head, eyes, hands, fingers, etc.), in particular of the first user 4 ₁ , respectively of the second user 4 ₂ : , but also the virtual reality simulator, in particular the first part 62S ₁ of the virtual reality simulator, respectively the second part 62S ₂ of the virtual reality simulator, controls an avatar, in particular the second avatar 5 ₂ , respectively the first avatar 5 ₁ , as a function of a reciprocal estimation or remote estimation of the position of the part of the associated user (head, eyes, hands, fingers, etc.), in particular of the second user 4 ₂ , respectively of the first user 4 ₁ : .

Remote estimation means the estimation of a position of a user other than the user wearing the virtual reality device implementing this remote estimation. The other user is nevertheless co-located with the user wearing the virtual reality device implementing this remote estimation.

Thus, tracking a moving part of a user, particularly the user's hand, benefits from the fusion of the two hand tracking areas and the extension of tracking to other parts of the body.

The advantage of using self-estimation and reciprocal or remote estimation is to gain tracking precision and to continue tracking even if part of the user is hidden from the view of the sensor worn by this user.

There illustrates a simplified diagram of users of a virtual reality system according to the invention, when the virtual entity is a virtual representation of at least one of the users.

The first user 4 ₁ comprises several first parts connected to each other and to a skeleton (not illustrated) constituting a second part of the first user 4 ₁ . The first parts of the first user 4 ₁ are in particular the head , shoulders: right shoulder and left shoulder , arms: right arm and left arm , elbows: right elbow and left elbow , forearms: right forearm and left forearm , hands: right hand and left hand .

The second user 4 ₂ comprises several first parts connected to each other and to a skeleton (not illustrated) constituting a second part of the second user 4 ₂ . The second parts of the second user 4 ₂ are in particular the head , shoulders: right shoulder and left shoulder , arms: right arm and left arm , elbows: right elbow and left elbow , forearms: right forearm and left forearm , hands: right hand and left hand .

Another advantage of the invention is that it can operate on a local network consisting solely of portable virtual reality devices, such as mixed reality headsets. This avoids having to go through a “directory” server to connect the headsets with fixed sensors instrumenting an environment. Thus, the fusion/disambiguation processing of the tracking data of a user’s parts can be performed solely by the headsets without knowledge of the user’s location.

An estimation of the topology of mixed reality headsets allows to prioritize the validity of the tracking data (e.g. estimated real position) of each of the headsets. In particular, the headset most in front of a first user will have the most relevant data to disambiguate the tracking of the hands of this first user.

Not only does the animation controller 12, in particular the tracer 123, track the hands (first moving part) of a first user detected in the field of vision of its own sensor, but it also reconstructs the skeleton (second part) of the other users (second users). This reconstruction of the skeleton of the other users makes it possible to remove the ambiguity, that is to say to determine that the tracked hands are the hands of the first user and not of second users because they are not connected to the reconstructed skeletons of these second users.

Another interest of the invention is to allow the feet of the first user to be tracked by means of the second sensor of the second user. Tracking the feet makes it possible to improve the realism of the animation of the first avatar associated with the first user both in an immersive virtual reality (metaverse), as well as mixed realities.

• Les utilisateurs du premier groupe de personnes colocalisées équipées en réalité augmentée n’ont pas besoin de percevoir de l’autre utilisateur colocalisé mais ont besoin d’un bon suivi sans ambiguïtés des parties réelles du corps animant des parties virtuelles de leur avatar interagissant avec l’espace virtuel partagé pour limiter les erreurs d’interaction ;
• Les utilisateurs du deuxième groupe de personnes distantes ont quant à eux besoin de voir les avatars des utilisateurs du premier groupe, y compris leurs pieds.

This is particularly useful in the case of two groups of collaborators working remotely in a shared virtual space:

• Users of the first group of co-located people equipped with augmented reality do not need to perceive the other co-located user but need good unambiguous tracking of the real body parts animating virtual parts of their avatar interacting with the shared virtual space to limit interaction errors;
• Users in the second group of remote people need to see the avatars of users in the first group, including their feet.

As shown in FIG. 4, some parts of the body of the first user 4 ₁ may be outside the visual field 25 ₁ (hatched dashed area) of the first sensor 23 ₁ worn by the first user 4 ₁ . In this case, the first sensor 23 ₁ allows the animation controller 12 according to the invention to estimate the position of only the right hand of the first user.

The animation controller 12 according to the invention further uses the data captured by the second sensor 23 ₂ worn by the second user 4 ₂ to track the parts of the body of the first user 4 ₁ , the visual field 25 ₂ (wavelet-hatched area) of the second sensor 23 ₂ covering the entire body of the first user can thus track all parts of the body of the first user 4 ₁ , and not just his right hand. .

Conversely, some parts of the body of the second user 4 ₂ may be outside the visual field 25 ₂ (wavelet-hatched area) of the second sensor 23 ₂ worn by the second user 4 ₂ . In this case, the second sensor 23 ₂ allows the animation controller 12 according to the invention to estimate the position of only the right hand of the second user.

The animation controller 12 according to the invention further uses the data captured by the first sensor 23 ₁ worn by the first user 4 ₁ to track the parts of the body of the second user 4 ₂ , the visual field 25 ₁ (zone hatched in dashes) of the first sensor 23 ₁ covering the entire body of the second user can thus track all parts of the body of the second user 4 ₂ , and not just his right hand. .

Zone 25 ₁₊₂ constitutes a shared monitoring zone for users’ bodies.

Thus, our invention allows tracking of all parts of the bodies of both users by means of the two sensors worn by both users while the sensor of one user does not allow tracking of all parts of his own body.

The invention therefore makes it possible to benefit from the pooling of information captured by the headsets and glasses of co-located users collaborating in a shared virtual reality space. Indeed, by combining this captured information, for example the depth map, of each headset or glasses present, the detection and tracking of a user's hands is improved.

In addition, each helmet or goggle captures a restricted field of vision or visual field (cf. ) located in front of the user. Thus, when the user's arms are outside the sensor's field of view, for example the camera, their virtual reality headset or their augmented reality glasses, in particular along their body or behind their back, the user's sensor, also called the first user, does not allow tracking of their arms. This tracking is made possible by a sensor worn by another user, also called the second user, who is co-located and whose visual field includes the first user's arms.

Pooling this data to reconstruct users' hands not only improves tracking of users' fingers when the hands are the field of vision of the first user's camera, but also extends this tracking to areas not covered by this single camera.

Therefore, the higher the number of co-located users, the more efficient this mutualization is.

The invention also relates to a medium. The information medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM, for example a CD ROM or a microelectronic circuit ROM or a magnetic recording means, for example a floppy disk or a hard disk.

On the other hand, the information medium may be a transmissible medium such as an electrical or optical signal which can be conveyed via an electrical or optical cable, by radio or by other means. The program according to the invention may in particular be downloaded from a network, in particular of the Internet type.

Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.

In another implementation, the invention is implemented by means of software and/or hardware components. In this regard, the term module can correspond to either a software component or a hardware component. A software component corresponds to one or more computer programs, one or more subprograms of a program, or more generally to any element of a program or software capable of implementing a function or a set of functions according to the description above. A hardware component corresponds to any element of a hardware assembly capable of implementing a function or a set of functions.

Claims (15)

Method for animating a virtual entity associated with a first user of a virtual reality system during reproduction of the virtual entity, the virtual entity being composed of at least a first mobile part, the animation comprising:
- determining a current virtual position of a first mobile part of a virtual entity associated with the first user based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system. Animation method according to the preceding claim, the animation method comprising:
- estimating a current real position of at least a first part of the body of a first user of the virtual reality system based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system;
the determination of the current virtual position of the first mobile part of the virtual entity associated with the first user being carried out as a function of the estimated current real position of the first part of the body of the first user. Animation method according to one of the preceding claims, in which the determination of the current virtual position is furthermore a function of first data relating to the first part of the body of the first user captured by means of a first sensor worn by the first user, the first sensor and the second sensor being co-located. Animation method according to one of the preceding claims, the animation method comprising:
- if a first part of a user's body to which the captured data relates is connected to at least a second part of the first user's body, assigning to the first user the current virtual position determined as a function of the captured data relating to at least a first part of a user's body, the captured data comprising at least one piece of data among the first captured data and the second captured data. Animation method according to one of claims 2 to 4, in which, when the estimation of the current real position receives several of the second data captured from several distinct second sensors worn by several distinct second users, the estimation of the current real position comprises:
- weighting the second captured data provided by a second sensor according to a current relative position of the second sensor with respect to the first user prior to estimating the current actual position according to all of the weighted second captured data. Animation method according to one of the preceding claims, in which the captured data is one or more of the following data:
- a captured image;
- an infrared image captured;
- a captured video;
- a captured movement;
- a captured depth. Virtual reality simulation method comprising:
- generating a virtual reality capable of being reproduced simultaneously by a first portable virtual reality reproduction device worn by a first user and a second portable virtual reality reproduction device worn by a second user, the virtual reality comprising at least one virtual entity; and
- animate the virtual entity associated with a first user by:
+ determining a current virtual position of a first mobile part of a virtual entity associated with the first user based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality system. Program comprising program code instructions for executing the steps of the animation method according to one of claims 1 to 6 and/or of the virtual reality simulation method according to the preceding claim when said program is executed by a processor. Virtual reality simulator including:
- a virtual reality generator capable of being reproduced simultaneously by a first portable virtual reality reproduction device worn by a first user of the virtual reality simulator and a second portable virtual reality reproduction device worn by a second user of the virtual reality simulator, the virtual reality comprising at least one virtual entity;
- an animation controller of the virtual entity associated with a first user during the reproduction of the virtual entity, the virtual entity being composed of at least a first mobile part, the animation controller being able to:
+ determine a current virtual position of a first mobile part of a virtual entity associated with the first user based on at least second data relating to the first part of the body of the first user captured by means of a second sensor worn by a second user of the virtual reality simulator. Virtual reality simulator according to the preceding claim, in which the animation controller comprises a tracer capable of estimating the current real position of the first part of the body of the first user. Virtual reality simulator according to one of claims 9 or 10, the virtual reality system comprising a signal transmitter connected to the first portable virtual reality reproduction device, to the second portable virtual reality reproduction device, to a first sensor worn by the first user and to the second sensor, the first sensor and the second sensor being co-located, the transmitter being able to emit virtual reality reproduction signals to the first portable virtual reality reproduction device and to the second portable virtual reality reproduction device, and to receive first data captured from the first sensor and second data captured from the second sensor. Virtual reality simulator according to the preceding claim, wherein the transmitter is connected to the first portable virtual reality reproduction device, to the second portable virtual reality reproduction device, to the first sensor and to the second sensor via a local network. Virtual reality system comprising:
- a first portable reproduction device carried by a first user,
- a second portable reproduction device carried by a second user,
- a first sensor worn by the first user,
- a second sensor worn by the second user, the first sensor and the second sensor being co-located,
- a virtual reality simulator according to one of claims 9 to 12. Virtual reality system according to the preceding claim, wherein the first sensor and the second sensor are respectively sensors integrated into the first virtual reality reproduction device and the second virtual reality reproduction device. Virtual reality system according to one of claims 13 or 14, the virtual reality system comprising several distinct second sensors worn by several distinct second users.