WO2022061858A1

WO2022061858A1 - Method for assisting a first user provided with a terminal at a scene

Info

Publication number: WO2022061858A1
Application number: PCT/CN2020/118296
Authority: WO
Inventors: Kun QIAN; Zhihong Guo
Original assignee: Orange
Priority date: 2020-09-28
Filing date: 2020-09-28
Publication date: 2022-03-31
Also published as: WO2022064278A1

Abstract

The present invention relates to a method for assisting a first user provided with a terminal at a scene, characterized it comprises performing, by a processing unit of a virtual reality system and connected to the first terminal, the steps of: Obtaining (a) a tridimensional model of the scene; Displaying (b) to the second user said tridimensional model of the scene in virtual reality; obtaining (c) annotations of said tridimensional model of the scene inputted by a second user in order to assist the first user; and Sending (d) to the terminal data describing said annotations, said data enabling enriching a view of the scene acquired by a camera of the device with said annotations.

Description

METHOD FOR ASSISTING A FIRST USER PROVIDED WITH A TERMINAL AT A SCENE

TECHNICAL FIELD

The field of this invention is that of customer remote assistance.

More particularly, the invention relates to a method for remotely assisting a user provided with a terminal at a scene.

BACKGROUND

The maintenance of complex electronic appliances or machines is generally difficult for non-professional customers, and should preferably be performed by professional technicians on the spot. However, this is time and cost consuming.

Alternatively, customers can call an assistance service, which provides for a technician to orally guide the customers. However, the communication in such a situation is not easy and, when the maintenance operation is really complex, the customer may not be successful even with this guide of assistance.

In order to improve the situation, AR customer remote guidance systems have been developed. In these systems, the customer captures a photo/video in the field, and this photo/video is either directly enriched by an AI tool, or transmitted to a remote technician. The technician can then edit the photo/video with some visual aids (e.g. notations or even virtual objects) then send it back to the customer, so that the customer can execute himself some maintenance operations with the help of the edited photo/video.

While improving the situation, such a technical solution is not really user-friendly for the customers, because they have to watch the edited photos/videos at the same time as they execute the maintenance operations, which may be unpractical especially when the maintenance operations are complex ones. Furthermore, it does not allow a real time interaction with the technician, and several exchanges of photos/videos may be necessary.

There is consequently a need for a remote assistance method allowing a true real-time and more user friendly interaction between a customer and a professional technician, so as to provide any customer with enough guidance to perform even complex maintenance operations.

SUMMARY OF THE INVENTION

For these purposes, the present invention provides a method for assisting a first user provided with a terminal at a scene, characterized it comprises performing, by a processing unit of a virtual reality system connected to the first terminal, the steps of:

obtaining a tridimensional model of the scene;

displaying said tridimensional model of the scene in virtual reality;

obtaining annotations of said tridimensional model of the scene inputted by a second user in order to assist the first user; and

sending to the terminal data describing said annotations, said data enabling enriching a view of the scene acquired by a camera of the device with said annotations.

Preferred but non limiting features of the present invention are as follow:

- The method comprises a previous step of generating, by a processing unit of the terminal, said tridimensional model of the scene;

- Said tridimensional model of the scene is generated from views of the scene acquired by said camera of the terminal;

- The method further comprises a step of generating, by a processing unit of the terminal, from a real view of the scene acquired by the camera, an enriched view of the scene using the data describing said annotations;

- The method further comprises a step of displaying by an interface of the terminal said enriched view of the scene.

- Said terminal is smart or AR glasses wearable by the first user.

- The virtual reality system comprises a virtual reality headset wearable by the second user, displaying said tridimensional model of the scene comprising rendering virtual reality views of the tridimensional model from virtual reality headset position in space;

- The virtual reality system further comprises at least one motion controller, said annotations of said tridimensional model being inputted by the second user at the obtaining step using the motion controller;

- Said data describing said annotations sent at the sending step comprise coordinates of said annotations in the tridimensional model of the scene;

- Integrating the annotations into said real view in the generating step comprises mapping the captured view to the tridimensional model and calculating coordinates of the annotations in the real view from coordinates of the annotations in the tridimensional model.

In a second aspect, the invention provides a virtual reality system connectable to a terminal of a first user at a scene, the system comprising a processing unit configured to implement:

- obtaining a tridimensional model of the scene;

- displaying said tridimensional model of the scene in virtual reality;

- obtaining annotations of said tridimensional model of the scene inputted by a second user in order to assist the first user; and

- sending to the terminal data describing said annotations, said data enabling enriching a view of the scene acquired by a camera of the device with said annotations.

In a third aspect, the invention provides a terminal connectable to a virtual reality system, the terminal comprising a camera and a processing unit configured to implement:

- generating a tridimensional model of a scene acquired by said camera of the terminal;

- sending said tridimensional model to a virtual reality system able to display said tridimensional model in virtual reality;

- receiving, from said virtual reality system, data describing annotations, inputted by a second user, of said tridimensional model of the scene; and

- generating (e) an enriched view of the scene, from a real view of the scene acquired by said camera of the device, using the data describing said annotations.

In a fourth aspect, the invention provides an assembly of a virtual reality system according to the second aspect and a terminal according to the second aspect.

According to a fifth and a sixth aspects, the invention provides a computer program product, comprising code instructions for executing a method according to the first aspect for assisting a first user provided with a terminal at a scene; and a computer-readable medium, on which is stored a computer program product comprising code instructions for executing a method according to the first aspect for assisting a first user provided with a terminal at a scene.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of this invention will be apparent in the following detailed description of an illustrative embodiment thereof, which is to be read in connection with the accompanying drawings wherein:

- figure 1 illustrates an example of architecture in which the method according to the invention is performed; and

- figures 2 and 3 are diagrams representing steps of a preferred embodiment of a method according to the invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Remote assistance architecture

The present invention proposes a method for assisting a first user provided with a terminal 1 at a scene S, as represented by figure 1. As it will be explained later, the first user is to be assisted by a remote second user provided with a virtual reality system 2 connected to the terminal 1, for instance through a communication network 20 such as internet.

By “assisting” the first user, it meant providing him/her with any relevant information that may be of interest for him/her at the scene S. In particular, the first user may be an unexperienced user (such as a customer) while the second user may be an experienced user (such a professional technician) .

The scene S designates the visible environment wherein the first user needs assistance, and typically comprises an appliance or a machine to be maintained, i.e. assistance is required for performing said maintenance. Generally speaking, the scene shall display something about which the first user seeks assistance from the second user

For example, may be visible at the scene S, among others:

- the screen of a computer on which the first user wants to perform a given task;

- the back of a network appliance that the first user wants to use for connection;

- the internal mechanism of a faulty home appliance;

- an electronic circuitry to be wired;

- etc.

The present invention will not be limited to any kind of assistance and only relates to a technical solution for allowing the second user to easily and efficiently provide suitable information to the first user.

The terminal 1 is an electronic device designed for augmented reality (AR) display, in particular a hand-free AR device such as pair of smart glasses or AR glasses, but could also be a smartphone, a tablet, a digital camera, or any other user terminal able to provide AR rendering.

The terminal 1 comprises at least a processing unit 11 (such as one or more processors) , a user interface 13 for displaying images and/or videos (such as a screen or a projector) , a camera 14 for acquiring images and/or videos and possibly a storage unit 12 (a memory, for instance flash) .

The first user is supposed to hold the terminal 1 such that the camera 14 is directed toward the scene S.

The terminal 1 is preferably configured to output on said interface 13 a view of what is visible from the camera 14 (i.e. the scene S) . To this end, the camera 14 is advantageously located on a front face of the terminal 1, while the display 13 is located on a rear face of the terminal 1. In the case of smart or AR glasses, the interface 13 is typically embedded in one lens, while the camera is either in the bridge over the nose or in an arm of the glasses.

Note that the present invention is not limited to any kind or architecture of terminal 1.

The method also involves a virtual reality (VR) system 2, i.e. a system able to generate a virtual reality environment for a second user. The VR system 2 comprises at least a processing unit 21 (typically one or more processors) and comprises advantageously a VR headset 23 wearable by the second user and/or at least one motion controller 24 (typically a pair of VR handles) which can be held by the second user. Such motion controller 24 is able to track the position of a body part of the second user (a hand for instance) in space.

In more details, the processing unit 21 is typically that of a server to which the VR headset 23 and the motion controller 24 are directly connected (for instance by short distance wireless connection such as Bluetooth, or by a wired connection such as USB) . The system/server may also comprise a storage unit 22 (a memory, for instance a hard drive) . In another embodiment, processing unit 21 is part of the VR headset 23 and the VR system 2 consists then of VR headset 23 directly connected with the motion controller 24.

Alternatively to VR headset and/or handles, note that the VR system 2 may simply comprise a 2D display (a screen) with ordinary input means (a keyboard and a mouse) , the second user being able to move and orient a tridimensional model and interact with it, even if it less convenient that with full VR equipment.

Virtual reality systems are known to the skilled person, and any existing virtual reality system may be used for the present invention.

As explained, the VR system 2 is not located at the scene S, i.e. the VR system 2 is remotely connected to the terminal 1 through a communication network 20 such as internet.

Remote assistance method

With reference to figures 2 and 3, the present method first comprises a step (a) of obtaining, by the processing unit 21 of the VR system 2, a tridimensional model of the scene S (also referred as “3D model” in these figures) .

This tridimensional model may already exist (for instance in the case of a maintenance of a known machine at a known place) and may already be stored by the storage unit 22 of the VR system 2.

Alternatively, the tridimensional model is received by VR system 2 from the terminal 1. In other words, the method preferably comprises a previous step of generating (a0) , by the processing unit 11 of the terminal 1, a tridimensional model of the scene S. Once generated, this tridimensional model is transmitted from the terminal 1 to the VR system 2. These two steps may be triggered by the first user needing some assistance on the scene S.

In a known fashion, the tridimensional model of the scene S may be generated from views of the scene S acquired by said camera 14 of the terminal 1. Algorithms for constructing a 3D model from a plurality of various 2D view are known to the skilled person (the first user may be requested to “scan” the scene S with the camera 14 up to enough data is acquired) , and the present invention is not limited to any technique when it comes to generate such a tridimensional model from a captured scene. For instance, such a tridimensional model may be constructed using a 3D reconstruction algorithm such as :

- DynamicFusion ( https: //github. com/mihaibujanca/dynamicfusion) ;

- BundleFusion ( http: //graphics. stanford. edu/projects/bundlefusion/) ;

- Scene Reconstruction ( http: //qianyi. info/scene. html) .

These algorithms may use data captured by a depth-camera (RGB-D channel) such as Kinect or

Realsense ^TM.

In a following step (b) , the tridimensional model of the scene S is displayed in virtual reality by the VR system 2, by means of the processing unit 21 of VR system 2. This enables a second user, provided with the VR system 2, to watch the scene S as if the second user was at the scene S. Thanks to the VR, the rendering of the tridimensional model of the scene S to the second user is preferably function of his/her position in space.

More precisely, if VR system 2 comprises a virtual reality headset 23 worn by the second user, displaying (b) said tridimensional model of the scene S typically comprises rendering virtual reality views of the tridimensional model from virtual reality headset 23 position in space. Note that step (b) typically comprises preprocessing the tridimensional model by the processing unit 21, so as to allow real-time displaying to the second used. As already indicated, it is also possible that the tridimensional model is ordinary displayed by a screen, the second user then moving and orientating the screen for instance with a mouse.

In a further step (c) , which is generally simultaneously performed with step (b) , annotations of the tridimensional model of the scene S, inputted by the second user in order to assist the first user, are obtained by the processing unit 21 of VR system 2.

By “annotations” , it is meant any visual aid that could be added to the tridimensional model, such as, among others:

- locations of key points;

- virtual objects;

- color, shapes;

- notations (words, numbers) ;

- wirings;

- etc.

These annotations may be inputted in any way by the second user, for instance using a keyboard and/or a mouse.

Preferably, when the VR system 2 comprises at least one motion controller 24 which can be held by the second user, these annotations of the tridimensional model are inputted by the second user at step (c) using this motion controller 24. In other words, the second user “designates” a point of the space in the tridimensional model with his/her movement to place an annotation at this point, for instance by pushing a trigger on the motion controller 24 once the point is reached by his/her hand.

In step (d) , the processing unit 21 of the system 2 sends to the terminal 1 data describing the inputted annotations, these data enabling enriching a real view of the scene S acquired by a camera 14 of the device 1 with these annotations.

The data describing the annotations may typically comprise coordinates of the annotations in the tridimensional model of the scene S and possibly further parameters (description of a type of annotation, attribute of the annotation such as a color or a size, etc. ) .

Step (d) is advantageously performed in real-time, i.e. as soon as a new annotation is inputted by the second user (e.g. a new movement is performed by the user) , data describing this annotation is directly sent to the terminal 1. The idea is to use the annotations inputted by the second user for augmented reality (AR) for the first user, and preferably live AR when the second user simultaneously annotates the tridimensional model. A real-time convergent AR/VR guidance solution is therefore presently proposed.

The method preferably comprises the further step (e) of generating, by the processing unit 11 of the terminal 1, from a real view of the scene S acquired by the camera 14 (e.g. an image or a video) , an enriched view of the scene S using the data describing the annotations inputted by the second user. What is meant here by “real view” is the view (e.g. image or video) captured in real time by the camera 14. In other words, using the received data, processing unit 11 integrates the received annotations into the real view, i.e. augmenting the real view (the enriched view may be referred to as “augmented” view) . The annotations may be rendered visually as overlaying the real view captured by camera 14.

The method may comprise a final step (f) of displaying, by the interface 13 of the terminal 1, said enriched view of the scene S, thereby providing the first user with a view of the scene S which is annotated almost in real-time (i.e. taking into account the processing time at both the VR system 2 and terminal 2 as well as the data transmission time on the communication network between them) by the second user.

In the case of a terminal 1 comprising smart or AR glasses, the annotations are thus rendered on a real view in the first user’s glasses, making it easier for the first user to execute the maintenance operations as he gets visual assistance in the “real world” rather than in an edited video.

Note that even with more common terminals such as a smartphone, the first user can still get useful visual assistance in real-time, with the limitation that the first user still has to hold the terminal with one hand in front of the scene S while performing maintenance operations with the other hand, which is less convenient that when the terminal is a hand free terminal such as smart or AR glasses.

Algorithms for augmentation of a real view of the scene S with annotations of a tridimensional model of the scene S are known to the skilled person, and the present invention is not limited to any technique.

As an example, integrating the annotations into the real view in step (e) may comprise mapping the real view to the tridimensional model, and calculating coordinates of the annotation in the real view from coordinates of the coordinates of the annotation in the tridimensional model. Size/orientation of the annotation may also be adapted as a function of their coordinates.

Terminal, system, assembly and computer program

In a second aspect, the present invention proposes a virtual reality system 2 comprising a processing unit 21, possibly a storage unit 22, which may be connected to a terminal 1 of a first user at a scene S, and is adapted for carrying out the method for assisting the first user as previously described.

The virtual reality system 2 advantageously comprises a VR head set 23 and/or at least one motion controller 24.

Said processing unit 21 is configured to implement:

- obtaining a tridimensional model of the scene S;

- displaying this tridimensional model of the scene S in virtual reality;

- obtaining annotations of said tridimensional model of the scene S inputted by the second user to assist the first user;

- sending to the terminal 1 data describing these annotations, these data enabling enriching a view of the scene S acquired by a camera 14 of the device 1 with the inputted annotations.

In a third aspect, the present invention proposes a terminal 1, connectable to the aforementioned virtual reality system 2, terminal 1 comprising a camera 14 and a processing unit 11 configured to implement:

- generating a tridimensional model of a scene S acquired by the camera 14 of the terminal 1;

- sending this tridimensional model to a virtual reality system 2 able to display said tridimensional model in virtual reality;

- receiving, from said virtual reality system 2, data describing annotations, inputted by a second user, of said tridimensional model of the scene S; and

- generating a enriched view of the scene S, from a real view of the scene S acquired by said camera 14 of the device 1, using said data describing annotations.

This terminal 1 preferably comprises an interface 13 for displaying to the first user the enriched view of the scene S.

In a fourth aspect, the present invention proposes an assembly of the virtual reality system 2 and the terminal 1 which may be connected together (for example through a communication network 20) .

The invention further proposes a computer program product, comprising code instructions for executing (in particular with a processing unit 21 of the system 2) a method according to the method for assisting a first user provided with a terminal 1 at a scene S; and a computer-readable medium (in particular a storage unit 22 of the system 2) , on which is stored a computer program product comprising code instructions for executing said method.

Claims

A method for assisting a first user provided with a terminal (1) at a scene (S) , characterized it comprises performing, by a processing unit (21) of a virtual reality system (2) connected to the first terminal (1) , the steps of:

obtaining (a) a tridimensional model of the scene (S) ;

displaying (b) said tridimensional model of the scene (S) in virtual reality;

obtaining (c) annotations of said tridimensional model of the scene (S) inputted by a second user in order to assist the first user; and

sending (d) to the terminal (1) data describing said annotations, said data enabling enriching a view of the scene (S) acquired by a camera (14) of the device (1) with said annotations.
A method according to claim 1, comprising a previous step of generating (a0) , by a processing unit (11) of the terminal (1) , said tridimensional model of the scene (S) .
A method according to claim 2, wherein said tridimensional model of the scene (S) is generated from views of the scene (S) acquired by said camera (14) of the terminal (1) .
A method according to any one of claims 1 to 3, further comprising a step of generating (e) , by a processing unit (11) of the terminal (1) , from a real view of the scene (S) acquired by the camera (14) , an enriched view of the scene using the data describing said annotations.
A method according to claim 4, further comprising a step of displaying (f) by an interface (13) of the terminal (1) said enriched view of the scene (S) .
A method according to any one of claims 1 to 5, wherein said terminal (1) is smart or AR glasses wearable by the first user.
A method according to any one of claims 1 to 6, wherein the virtual reality system (2) comprises a virtual reality headset (23) wearable by the second user, displaying (b) said tridimensional model of the scene (S) comprising rendering virtual reality views of the tridimensional model from virtual reality headset (23) position in space.
A method according to claims 7, wherein the virtual reality system (2) further comprises at least one motion controller (24) , said annotations of said tridimensional model being inputted by the second user at step (c) using the motion controller (24) .
A method according to any one of claims 1 to 8, wherein said data describing said annotations sent at step (d) comprise coordinates of said annotations in the tridimensional model of the scene (S) .
A method according to claims 4 and 9 in combination, wherein integrating the annotations into said real view in step (e) comprises mapping the captured view to the tridimensional model and calculating coordinates of the annotations in the real view from coordinates of the annotations in the tridimensional model.
A virtual reality system (2) connectable to a terminal (1) of a first user at a scene (S) , the system (2) comprising a processing unit (21) configured to implement:

- obtaining a tridimensional model of the scene (S) ;

- displaying said tridimensional model of the scene (S) in virtual reality;

- obtaining annotations of said tridimensional model of the scene (S) inputted by a second user in order to assist the first user; and

- sending to the terminal (1) data describing said annotations, said data enabling enriching a view of the scene (S) acquired by a camera (14) of the device (1) with said annotations.
A terminal (1) connectable to a virtual reality system (2) , the terminal (1) comprising a camera (14) and a processing unit (11) configured to implement:

- generating a tridimensional model of a scene (S) acquired by said camera (14) of the terminal (1) ;

- sending said tridimensional model to a virtual reality system (2) able to display said tridimensional model in virtual reality;

- receiving, from said virtual reality system (2) , data describing annotations, inputted by a second user, of said tridimensional model of the scene (S) ; and

- generating (e) an enriched view of the scene (S) , from a real view of the scene (S) acquired by said camera (14) of the device (1) , using the data describing said annotations.
An assembly of a virtual reality system (2) according to claim 11 and a terminal (1) according to claim 12.
A computer program product, comprising code instructions for executing a method according to any one of claims 1 to 10 for assisting a first user provided with a terminal (1) at a scene (S) , when executed by a processing unit.
A computer-readable medium, on which is stored a computer program product comprising code instructions for executing a method according to any one of claims 1 to 10 for assisting a first user provided with a terminal (1) at a scene (S) .