DE102023105347A1 - Enrichment of classified objects in room scans with real-time data through radio and optical trackers - Google Patents

Abstract

1. A method for scanning a room, the method comprising scanning one or more real objects of the room, wherein scanning an object (84, 85) comprises detecting a number of survey points (89) of the real object (84, 85) by means of a gesture or a hand controller (40, 42), and wherein the real objects are provided with respective radio or optical trackers which are recognized in their position.

Description

TECHNICAL AREA

The present disclosure relates generally to methods and apparatus and systems for virtual reality (VR) and augmented reality (AR) for presentation on a display of an electronic device, and more particularly to the dynamic generation of such VR/AR experiences.

TECHNICAL BACKGROUND

“Virtual reality” (VR for short) is a computer-generated reality with images (3D) and possibly also sound. VR/AR systems enable the user to immerse themselves in a simulated virtual environment to varying degrees. A VR environment refers to a fully or partially simulated environment that a user can perceive and/or interact with via an electronic system. The term “augmented reality” (AR, or “extended reality” (XR) refers, for example, to the computer-aided extension of the perception of reality. A VR/AR environment includes, for example, virtual scenes and virtual objects that a user can perceive via an electronic display and/or interact with, for example by simulating the user’s presence in the computer-generated environment. The VR/AR environment can be part of a video game and can, for example, represent the scene from a video game.

A VR experience is provided to a user, for example, by a head-mounted display worn on the head, which is also known as a headset or VR glasses. An "augmented reality" (AR or XR), for example, is provided by so-called AR glasses, in which an optical projection is displayed on a partially transparent glass through which the natural physical objects are also viewed ("see-through" HMD).

Virtual reality (VR) or augmented reality (AR) can also be conveyed through a smartphone or tablet computer by superimposing virtually generated objects on the images from smartphone or tablet cameras, supplementing them or replacing them in whole or in part.

To provide the user with a VR/AR experience, physical sensory content corresponding to a physical environment close to the user is replaced by virtual content or, in the case of AR environments, supplemented by virtual content.

For example, state-of-the-art automated scanners allow the user to scan physical objects and planes to replace them with virtual content. They align the scanner, and cameras automatically detect physical objects and planes and convert them into data points. The disadvantage of this technology is that it requires specialized equipment such as LIDAR, which is not widely used in consumer devices. In addition, automated scanning is not very reliable when it comes to completeness and precision. The latter is a particularly important factor because haptic feedback plays a crucial role in developing a user-friendly system.

Based on this, the invention is based on the task of improving the user experience in VR/AR environments.

This object is achieved by the method according to claim 1 and the device according to claim 10. Further advantageous embodiments of the invention emerge from the subclaims and the following description of preferred embodiments of the present invention.

The embodiments show a method for scanning a room, wherein the method comprises scanning one or more real objects of the room, wherein the scanning of an object comprises detecting a number of measurement points of the real object by means of a gesture or a hand controller, and wherein the real objects are provided with respective radio or optical trackers which are recognized in their position.

A radio tracker determines its position using radio technologies such as Bluetooth and/or ultrawideband and sends this to a VR/AR device. A tracker can also be an inertial measuring unit (IMU) to determine its position and/or orientation in space and transmit this to the VR/AR device.

The position of an optical tracker can be determined, for example, using the camera or cameras of a smartphone or a headset (e.g. VR/AR glasses). For this purpose, LEDs can be provided on the tracker, the position of which is determined by the headset from the camera data and from whose position the VR/AR device can determine the position of the optical tracker. The position of an optical tracker can also be determined from the camera data of a camera external to the headset, which is in communication with a VR/AR console or with the headset, for example. According to another alternative embodiment, an optical tracker can also be equipped with one or more cameras or depth sensors, by means of which the tracker locates itself in space by looking outside, whereby techniques such as SLAM ("Simultaneous Localization and Mapping") or the like can be used.

The room can, for example, be an interior space of a building or an outdoor space.

The object to be scanned can be, for example, a sofa, a seat, a table, a chair, a shelf, a cupboard, a plant, a bed, or a window.

A survey point can be, for example, a corner of the object (e.g. a corner of a table) or a point on a corner of a wall. With the method described here, it is possible, for example, to provide powerful development tools that allow users to build creatively and effectively on such scans. The room scans created can be used both for the internal development of these tools and for providing an effective package for the users of the development tools. End users of applications (such as games or the like) can also use scanning to design virtual rooms.

The method for scanning a virtual room can, for example, enable manual scanning of objects or walls by touch.

Manual scanning allows the user to identify relevant objects or layer data and provide some form of interaction to map those objects to data as accurately (and ideally as efficiently) as possible. The method has the advantage of being fast and accurate and works well even in low light conditions.

According to one embodiment, the gesture for detecting a measurement point of an object or a wall comprises a pointing gesture performed with one hand to point to a measurement point of the object or the wall and a confirmation gesture performed with the other hand, wherein the position of the hand at the time of executing the confirmation gesture determines the detected position of the measurement point.

When manually scanning a spatial object, the user preferably performs this procedure by touching the object to be scanned. This enables direct physical feedback by touching the real object with the hands.

By scanning a real object based on the recorded survey points, a virtual object is created.

After scanning an object, a virtual box can be created around the object based on the object's captured survey points. In this way, physical objects can be represented using boxes. Rectangular meshes can be created as representative data when the user assigns a physical object.

Preferably, a tracker determines the position of the associated real object in space and the position determined by the tracker is transferred to the linked virtual object.

A tracker can also determine the orientation of the associated real object in space and the orientation determined by the tracker can be transferred to the linked virtual object.

When creating a virtual object, the position of the survey points of a scanned object can also be linked to the position information of the associated tracker in order to calibrate the tracker.

According to the embodiment, the room is scanned using a VR/AR device that detects the respective positions of the trackers assigned to the objects.

The VR/AR device can preferably locate itself in the room using inside-out tracking. Although inside-out tracking is described above as the preferred embodiment, outside-in tracking can also be used, where base stations with cameras are placed in the room to track the headset from the outside. In this case, the headset has infrared markings (like controllers, for example).

In embodiments it is assumed that an object is either static or moving.

The method can also classify the scanned objects and link the classification information to the virtual objects. The classification of the scanned objects can be done manually or automatically, for example. For example, the classification information can be recorded during the process of scanning the room.

The embodiments also show a device for scanning a room, wherein the device is designed to perform a scan of one or more real objects of the room, wherein the scanning of a real object comprises detecting a number of survey points of the real object by means of a gesture or a hand controller, and wherein the device is designed to receive position information from respective radio or optical trackers and to link this to respective virtual objects.

The embodiments also show a system with the above functionality, wherein the system comprises, for example, a VR console and a headset.

The embodiments also relate to a computer program comprising program instructions that implement the methods described here.

The devices and methods provided in the embodiments described herein enable the user to interact with his physical environment and to incorporate real-world elements from the physical environment into the virtual environment, the position of which changes analogously to the position of associated real objects.

• 1 schematisch die Erstellung von klassifizierten Scans durch Extended-Reality-Geräte zeigt, die sich durch Inside-Out-Tracking im Raum verorten, kombiniert mit dem Prinzip von Funk- oder Optik-Trackern, die durch ein Extended-Reality-Gerät in ihrer Position erkannt werden;
• 2 eine schematisierte Darstellung einer beispielhaften Virtual-Reality (VR)-Systemumgebung ist;
• 3a-c eine beispielhafte vordefinierte Handgeste zeigen, mittels welcher der Benutzer die Anzeige eines Menüs bewirken kann;
• 3d zeigt, wie der Benutzer ein virtuelles Auswahlmenü und eine Handgeste zur Auswahl eines Raumobjekts mittels des Auswahlmenüs virtuell erlebt;
• 4a und 4b ein Beispiel für eine Zeigegeste der linken Hand und für eine Bestätigungsgeste der rechten Hand darstellen, um einen Eckpunkt eines Objekts oder einer Wand festzulegen;
• 5 den Ablauf der Selektion eines bereits erstellten Scans aus einer Menüliste zum Starten eines Benutzererlebnisses zeigt;
• 6 zeigt, wie der Benutzer die Eingabe zum Start des Benutzererlebnisses virtuell erlebt.
• 7 ein Beispiel eines VR-Erlebnisses darstellt, wie es dem Benutzer nach Auswahl eines Raumscans, oder auch nach Erstellen eines neuen Raumscans, über das Headset geboten wird;
• 8a-e einen beispielhaften Prozess des Scannens eines Raumes zeigen;
• 9 zeigt, wie der Benutzer das Scannen eines Raumobjekts mittels der oben beschriebenen Handgesten virtuell erlebt;
• 10 zeigt, wie der Benutzer das Scannen einer Wand eines Raumes mittels der oben beschriebenen Handgesten virtuell erlebt;
• 11 eine beispielhafte Datenstruktur einer gescannten virtuellen Umgebung zeigt, wie sie durch den oben dargestellten Scan-Prozess gewonnen wird;
• 12 eine genauere Darstellung der Unterstruktur Raumobjekte_0 der 11 des Raums Raum_0 zeigt;
• 13a-d schematisch zeigen, wie - sobald Funk- oder Optik-Tracker an realen Objekten platziert werden - diese durch das Extended-Reality-Gerät mit virtuellen Objekten verknüpft werden, um die realen Objekte in ihrer relativen Position zum Gerät zu verfolgen;
• 14 schematisch ein virtuelles Objekt, hier in Form eines Würfels, zeigt, das in einem Objektkoordinatensystem definiert ist;
• 15a beispielhaft die Verschiebung eines virtuellen Objekts von einer Ausgangsposition in eine Endposition zeigt; und
• 15b beispielhaft die Rotation eines virtuellen Objekts von einer Ausgangsposition in eine Endposition zeigt.

Embodiments will now be described by way of example and with reference to the accompanying drawings, in which:

• 1 schematically shows the creation of classified scans by extended reality devices that locate themselves in space through inside-out tracking, combined with the principle of radio or optical trackers that are recognized in their position by an extended reality device;
• 2 is a schematic representation of an exemplary virtual reality (VR) system environment;
• 3a -c show an example predefined hand gesture by which the user can cause a menu to be displayed;
• 3d shows how the user virtually experiences a virtual selection menu and a hand gesture for selecting a spatial object using the selection menu;
• 4a and 4b show an example of a left-hand pointing gesture and a right-hand confirmation gesture to specify a corner point of an object or a wall;
• 5 shows the process of selecting an already created scan from a menu list to start a user experience;
• 6 shows how the user virtually experiences the input to start the user experience.
• 7 represents an example of a VR experience as provided to the user after selecting a room scan, or even after creating a new room scan, via the headset;
• 8a -e show an example process of scanning a room;
• 9 shows how the user virtually experiences the scanning of a spatial object using the hand gestures described above;
• 10 shows how the user virtually experiences scanning a wall of a room using the hand gestures described above;
• 11 an example data structure of a scanned virtual environment shows how it is obtained by the scanning process shown above;
• 12 a more detailed representation of the substructure Raumobjekte_0 of the 11 of the room Room_0;
• 13a -d show schematically how - once radio or optical trackers are placed on real objects - these are linked to virtual objects by the extended reality device in order to track the real objects in their relative position to the device;
• 14 schematically shows a virtual object, here in the form of a cube, defined in an object coordinate system;
• 15a shows, by way of example, the movement of a virtual object from a starting position to a final position; and
• 15b shows, as an example, the rotation of a virtual object from a starting position to a final position.

Combining tracking in space with the principle of radio or optical trackers

As in 1 As shown, the embodiments described below combine two technologies that are used in a real indoor or outdoor space 1. The creation of classified scans by extended reality devices 2, which locate themselves in the room using inside-out tracking, is combined with the principle of radio or optical trackers 3, whose position is recognized by the extended reality device. It is assumed that the user is in a real indoor or outdoor space 1 with natural boundaries. On the one hand, the user uses an extended reality device 2, which can orient itself in the room using inside-out tracking and display a virtual scene. Inside-out tracking means that the headset or smartphone locates itself in the room by looking outside (cameras or depth sensors). Techniques such as SLAM (“Simultaneous Localization and Mapping”) are used here, for example. The extended reality device 2 can, for example, be a smartphone or a tablet that the user holds in their hand, or extended reality glasses that they wear on their head. The device must also have the ability to create a scan of the geometry that surrounds it with or without the user's intervention. Furthermore, radio or optical trackers 3 are used, which are attached to objects in the room 1 and are recognized in their position by the extended reality device 2.

VR/AR system environment

2 is a schematic representation of an example virtual reality (VR) system environment. In this example, the VR system environment includes a VR console 30 and a headset 20 connected to the VR console 30. The headset 20 is configured to be worn by the user to immerse the user in a three-dimensional (3D) virtual environment. In 2 a single headset 20 is shown. However, in alternative embodiments, the system environment may also include multiple headsets 20, with each headset 20 communicating with the VR console 30.

The VR console 30 provides the headset 20 with content that is presented to the user as a virtual environment. 2 In the example shown, the VR console 30 comprises a communication interface 31, a gesture recognition unit 32, a VR unit 33, a memory 34, a tracking unit 36 and an input interface 38.

The memory 34 stores one or more applications that, when executed by a processor (eg, the VR unit 33), generate virtual content for presentation to the user, as well as data representing the virtual reality. The content generated by an application may respond to inputs provided by the user by moving the headset 20 or via the gesture recognition unit 32 or the input interface 38. Examples of applications are gaming applications, video playback applications or the like.

The gesture recognition unit 32 performs gesture recognition based on image information provided by a camera 24 of the headset 2. In particular, the gesture recognition unit 32 recognizes, based on the received image information, predefined hand gestures performed by the user, for example a pointing gesture or a confirmation gesture, as described below with respect to 3a -c, 4 and 6a , b are described in more detail. Gestures can, for example, correspond to action requests, such as a request to perform a specific action. An action request can, for example, be starting or stopping an application or performing a specific action within the application.

The input interface 38 is, in addition to the gesture recognition unit 32, another functionality that allows a user to send action requests to the VR console 30. The input interface 38 may, for example, contain one or more input devices. Examples of input devices are VR controllers, a mouse, a keyboard, a joystick, or another suitable device for receiving action requests and transmitting the received action requests to the VR console 30.

The VR unit 33 executes applications within the VR system and determines the virtual content to be made available to the headset 20 for presentation to the user. In particular, the VR unit 33 receives information from the gesture recognition unit 32, the tracking unit 36 and the input interface 38 and, based on the information received, controls the virtual content for display on the headset 20. The VR unit 33 can use a rendering engine such as the Unity SDK for this purpose, for example.

The tracking unit 36 calibrates the VR system using, for example, the information received from the internal measurement unit (IMU) 22 of the headset 20. The tracking unit 36 receives, for example, current position information, acceleration information, velocity information, and/or predicted future positions of the headset 20. Based on the information from the tracking unit 36, the VR unit 33 controls the virtual content for display on the headset 20. For example, if the received information indicates that the user has performed a head movement, the VR unit 33 generates content for the headset 20 that reflects the user's head movement in a virtual environment. In addition, the VR unit 33 performs an action within an application running on the VR console 30 in response to an action request received from the gesture recognition unit 32 or the input interface 38.

The information transfer between the headset 20 and the VR console 30 takes place via communication interfaces 21 (headset) and 31 (VR console). The communication can take place, for example, via a wireless radio connection.

The headset 20 presents virtual content to the user - controlled by the VR console 30. Virtual content may include, for example, images, videos and/or audio content. Audio content may be presented via a separate device (e.g., speakers and/or headphones) outside the headset 20 that receives audio information from the headset 20, the VR console 30, or both. The headset 20 may include a communication interface 21, an inertial measurement unit (IMU) 22, a camera 24, a rendering unit 26, and a display 28.

The rendering unit 26 of the headset 20 receives content for the virtual environment from the VR unit 33 of the console and provides the content for display on the display 28. Furthermore, the rendering unit 26 can adapt the content based on information from the IMU 22, for example.

The inertial measurement unit (IMU) 22 is an electronic component that generates calibration data based on measurement signals received from one or more head tracking sensors that generate one or more measurement signals in response to the movement of the headset 20. Examples of head tracking sensors are accelerometers 23 for measuring translational movements (forward/backward, up/down, left/right movements), gyroscopes 25 for measuring rotational movements (e.g. pitch, yaw and roll movements), magnetometers 27 for determining the orientation with respect to the earth's magnetic field and the like, which are suitable for detecting movements and/or correcting errors associated with the IMU 22. Based on the measurement signals of the head tracking sensors and by means of sensor fusion methods known to those skilled in the art, the IMU 22 Calibration data indicating an estimated position of the headset 20 relative to an initial position. The IMU 22 may, for example, rapidly sample the measurement signals and calculate the estimated position of the headset 20 from the sampled data. For example, the IMU 22 integrates the measurement signals received from the acceleration sensors 23 over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on the headset 20. The reference point is, for example, a point that can be used to describe the position of the headset 20 (e.g., a center point of the IMU 22).

Alternatively or additionally, the headset can also track itself in space using technologies such as SLAM (“Simultaneous Localization and Mapping”). To do this, the optical data from the cameras or depth sensors is evaluated.

In this way, VR/AR experiences delivered by the VR console 30 via the headset 20 may track physical movements of a user and, depending on the tracked physical movements, adjust one or more properties of one or more virtual objects simulated in the VR/AR environment. For example, the VR/AR system may detect that the user's head is turning and subsequently adjust the image data presented to the user to correlate with the user's changed field of view.

The camera 24 takes pictures of the room in front of the user, on which the room and possibly also the user's hands can be seen. According to an exemplary embodiment, the camera 24 can comprise a pair of RGB cameras or grayscale cameras, which take a respective image of a scene intended for a left or a right eye of the user. The camera 24 can further comprise one or more optical lenses, possibly an illumination source and electronic controls. The image data provided by the camera 24 is used, for example, to provide the user with an image of the real environment in front of him via the display 28 of the headset, or it is fed to the gesture recognition unit 32 so that it can perform gesture recognition based on the images, for example to determine whether the user is performing a pointing gesture or a confirmation gesture as described below.

The image data of the camera 24 are processed by a gesture recognition (32 in 2 ) to determine, for example, the position of the user's hands and fingers. In particular, algorithms can be applied to determine a hand skeleton model based on the image data and to recognize predefined gestures based on the hand skeleton model.

The functionality of the console and headset can be continually improved through software updates.

The above example of the 2 shows a VR system environment consisting of a VR console and headset, in which the environment is recorded using cameras integrated in the headset and is fed to the user via displays in the headset. In alternative embodiments, the VR system environment can also be implemented in a smartphone or a tablet that is configured to offer a user a computer-generated reality experience ("VR/AR experience"). In this case, the camera integrated in the smartphone or tablet is used and the virtual reality or an augmented reality is shown to the user on the display of the smartphone or tablet.

2 shows a VR system in which the headset and console are physically separated from each other. In alternative embodiments, such as current standalone glasses and smartphones, however, both components may be located in one device.

Gesture recognition

The VR console described here can use a number of gestures to detect user input.

The 3a -c show an example predefined hand gesture by which the user can cause a menu to be displayed. As can be seen from the 3a -c, a defined hand gesture for displaying a menu can be to move the thumb of the right hand from a spread position (see 3a) into a snug position (see 3c ) with the hand clenched into a fist.

In 3d it is shown how the user virtually experiences a virtual selection menu and a hand gesture for selecting a room object using the selection menu, whereby only the virtual objects displayed to the user can be seen, but not the real impression of the room, which is captured by the headset's cameras and is displayed to the user at the same time as the virtual objects. Using a virtual menu 80, the user can select an object from among several room objects, such as a sofa, a seat, a table, a chair, a shelf, a cupboard, a plant, a bed, a window or the like. The user is not only shown the virtual menu 80, but also a virtual image 81, 82 of his two hands. The selection can be made, for example, as in 3d shown, via a pointing gesture of the left hand 81 and confirmed via a confirmation gesture of the right hand 82.

Through the object selection, as shown in 3d As shown, a categorization of objects can take place. The available object classes can be adapted and extended by the developer.

Another example of a pointing gesture of the left hand and a confirmation gesture of the right hand is shown in the 4a and 4b This gesture can be used, for example, to scan objects and walls of a room, as shown below with reference to the 9 and 10 A spatial object can be scanned by the user by hand by performing such a predetermined hand gesture. For example, a spatial object can be scanned by defining three corner points of the spatial object by repeatedly performing a pointing gesture with the left hand and a confirmation gesture with the right hand.

4a shows, in a first stage, how the user performs a pointing gesture with the left hand 81 while performing a predefined confirmation gesture with the right hand 82 in order to define a corner point of an object. In the first stage, the user points to a specific point with the left hand and an outstretched index finger in order to define a corner point of an object. Preferably, the user touches the real object so that he receives haptic feedback from the real object. The user begins by bringing the thumb of the right hand together with the index finger of the right hand in order to perform a confirmation gesture.

4b shows, in a second stage, how the user performs a pointing gesture with the left hand 81 while performing a confirmation gesture with the right hand 82 to set a corner point of an object. In the second stage, the user has brought the thumb of the right hand together with the index finger of the right hand. This gesture of the right hand is recognized by the gesture recognition unit (32 in 1 ) as confirmation. The position of the fingertip of the index finger of the left hand at the time of executing the confirmation gesture determines the detected position of the survey point (eg the corner) of the object.

The gesture example above uses the index finger of the left hand as a virtual mouse arrow to touch and confirm object vertices. However, the function of the left and right hands can also be switched.

The 4a and 4b The gesture shown is an example of a pointing gesture or a confirmation gesture to specify a corner point of an object. However, both the pointing gesture and the confirmation gesture can also be any other predefined gestures.

To avoid errors in gesture recognition, such gestures can be taught to the user via tutorials.

VR user experience

5 represents the process of selecting an already created scan from a menu list to start a user experience (steps S2 to S8 of 2 ) from the user's perspective.

5 shows in the picture on the left a menu that is shown to the user via the display (28 in 2 ) of the headset. By selecting the menu field "List saved scans on", the user indicates that he would like to see a list of the scans already saved in order to select a scan from it. The input can be made, for example, using a predefined pointing gesture. The next image shows the list of scans already saved from which the user can select. Using the list displayed in the menu, the user can, for example, select from a scan "Scan 0", a scan "Scan 1" and a scan "Scan 2". In addition, the user can select the listed scans can also be deleted by making a corresponding input (see image above). The input can be made again, for example, by a predefined pointing gesture to the menu. After selecting a scan from the list, the user sees a representation of the selected room with the scanned room objects and walls. By making a corresponding input, the user can start the user experience associated with the selected scan. An example of an input to start the user experience is shown in 5 This allows the user to start a user experience and immerse themselves in a virtual world enriched with virtualized representations of the scanned room objects and walls of the selected scan.

Selecting a scan from a list of scans, as shown in 5 shown allows you to manage room profiles and switch between them.

6 shows how the user virtually experiences the input to start the user experience. As with the 4 , 7 and 9 Only the virtual objects displayed to the user are visible, but not the real spatial impression, which is captured by the headset's cameras and displayed to the user at the same time as the virtual objects. As can be seen from 6 As can be seen, the user selects a start button (“Run”) in a menu 80. The selection is made, as is the selection of a spatial object to be scanned in 5 , for example via a pointing gesture of the left hand 81 and a confirmation gesture of the right hand 82.

7 represents an example of a VR experience as offered to the user via the headset after selecting a room scan or after creating a new room scan. Unlike during the process of scanning a room, during the VR experience the user is not given the spatial impression of the real room (which is captured by the headset's cameras and shown to the user during the scanning process via the headset's displays). The VR experience is a purely virtual experience in which reality is replaced by a (virtual) artificial reality. During the VR experience, real objects in the room are replaced by virtual objects, for example by placing any virtual objects in place of the virtual boxes around real objects obtained by scanning. For example, a real table can be replaced by a virtual altar. A wall can be replaced by a virtual fire wall. In this way, the user can immerse themselves in a virtual world that is designed with virtualized objects and walls that are located in places where real objects are located in the space in which the user moves during the VR experience. The correspondence between the virtual objects of the VR experience and real objects in the real space offers the advantage that the user avoids collisions with real objects when moving in real space during the VR experience, or at least receives physical feedback if he collides with a virtual object in the virtual world that has a correspondence in the real world that provides physical feedback. Furthermore, the user receives haptic feedback if he touches a virtual object with his hands that has a physical correspondence in real space.

Scanning the objects in a room

In the 8a -e, 9 and 10 The scanning of a spatial object is described in more detail. A spatial object can be scanned by the user by hand by performing predetermined hand gestures. For example, a spatial object can be scanned by defining three corner points of the spatial object by repeatedly performing a pointing gesture with the left hand and a confirmation gesture with the right hand.

The 8a -e describe schematically the process of creating a new scan of a room.

8a shows a schematic of a not yet scanned room in which the user is located. The room can contain one or more physical objects. In 5c For example, a table 84 and a sofa 85 are shown. The physical objects are captured by cameras (24 in 2 ) of the headset and the image data is used to show the user an image of the real environment in front of him via the display (28 in 2 ) of the headset.

8b schematically shows a stage of the scanning process in which the user has scanned a first spatial object 84, namely the table, so that a virtualized representation 86 of the first spatial object, in the form of a box around the object, is displayed by means of the headset.

8c schematically shows a stage of the scanning process in which the user has scanned a second spatial object 85, namely the sofa, so that a virtualized representation 87 of the second spatial object, in the form of a box around the object, is displayed by means of the headset.

Through this scanning by the user, the physical objects can be integrated into a virtual environment, as described above with reference to 7 This allows the user to navigate around and interact with physical objects while immersed in a virtual environment.

8d schematically shows a stage of the scanning process in which the user, after completing the scanning of the room objects, has scanned the walls of the room so that a virtualized representation 88 of the walls of the room is displayed on the headset. The walls are symbolically represented by dashed lines.

8e shows a schematic of the finished scanned room. The headset displays both virtualized representations of the scanned room objects and virtualized representations of the walls.

By means of a start button that is displayed to the user via the headset display, the user can start the user experience with the newly created scan, as shown in the above 5 , 6 and 7 described.

In 9 is shown how the user virtually experiences the scanning of a spatial object using the hand gestures described above, whereby 9 only the virtual objects displayed to the user are shown, but not the real spatial impression, which is captured by the headset's cameras and displayed to the user at the same time as the virtual objects (which in 9 not shown for reasons of clarity). To scan a spatial object, the user points with the left hand 81 to a first corner, for example the left front upper corner, of the object to be scanned with a pointing gesture and then performs a confirmation gesture with the right hand 82 to set a first corner point. In a second step (not shown), the user points with the left hand 81 to a second corner, for example the left rear upper corner, of the object and then performs the confirmation gesture again with the right hand to set a second corner point of the object to be scanned. In a 9 In the third step shown, the user points with the left hand 81 to a third corner, for example the right front upper corner 83, of the object and then performs the confirmation gesture again with the right hand 82 to specify a third corner point. 9 shows the point in time shortly after confirmation of the third corner point 83 by the confirmation gesture of the right hand 82 (the thumb of the right hand has already been released from the index finger of the hand). After setting these three corner points, a virtual object corresponding to the room object is displayed on the headset in the form of a box that is defined by the three corner points and the predefined level of the floor of the room.

For example, assuming that a side surface is perpendicular to the ground, a first side surface of the box is obtained from a first and a second measurement point. By moving the first side surface parallel to the third measurement point, a first side surface of the box is obtained that is opposite the first side surface. The other side surfaces of the box are obtained from these two side surfaces.

As described above, the lower boundary of the virtual box is automatically chosen to be either the ground plane of the room or the upper edge of an underlying geometry if an object has already been scanned that is located under the new object. The boundary can, for example, be the upper edge of a table while scanning an object on the table.

The three points can be confirmed either clockwise or counterclockwise.

The approach described above for manually scanning a spatial object by touch uses predefined gestures to detect user inputs and enables direct physical feedback by touching real objects with the hands.

For example, assuming that a side surface is perpendicular to the ground, a first side surface of the box results from a first and a second measurement point. Parallel displacement of the first side surface to the third measurement point results in a first side surface of the box that is opposite the first side surface. The other side surfaces of the box result from these two side surfaces.

The scan data (e.g. corner points) representing physical objects may also contain some information about the type of object that goes beyond the location and geometry data and allows a classification or categorization of the respective object. This classification is relevant, for example, for the later use of the scan data when building a new virtual reality.

In a second phase, after scanning the room objects has been completed, the user can scan the walls of the room using a similar procedure in order to display them together with the scanned physical objects in a virtualized representation. As soon as the user has confirmed by an input that he does not want to scan any more objects, he is prompted by a corresponding display to scan the walls of the room in which he is located. The scanning of the walls of the room is carried out in a similar way to that described above for the physical objects.

In 10 is shown how the user virtually experiences the scanning of a wall of a room using the hand gestures described above. As with the 3d and 9 are in 10 only the virtual objects displayed to the user are shown, but not the real spatial impression, which is captured by the headset's cameras and displayed to the user at the same time as the virtual objects (which in 10 not shown for reasons of clarity). To scan a wall, the user, in a first step (not shown), points with the left hand to a first corner point at the left end of the wall and performs the confirmation gesture with the right hand to set a first corner point. In a second step, the user points with the left hand to a second corner point at the right end of the wall and then performs the confirmation gesture again with the right hand to set a second corner point. 9 shows the time at which the left hand 81 points to the second corner point 89 at the right end of the wall and the right hand 82 performs the confirmation gesture. After setting these two corner points, the user is shown a virtual surface corresponding to the wall via the headset, which is defined by the two corner points and the predefined plane of the floor of the room.

When generating the virtual plane that describes the wall, it is assumed that the wall is perpendicular to the floor and runs through the two specified corner points. The virtual wall is limited laterally by the specified corner points. The generated virtual wall is limited downwards by the predefined floor, which corresponds to the real floor of the room in which the user moves during the VR/AR experience. The virtual wall is limited upwards by a predefined room height, whereby this room height (or wall height) can be chosen arbitrarily in a virtual experience and does not necessarily have to correspond to the physical room height of the real room in which the user moves during the VR/AR experience. This is because the user does not normally come into contact with the ceiling of the physical room in which he moves during the VR/AR experience.

In this method, it is therefore irrelevant at what height a corner point of the wall is recorded. When scanning the wall, the user can choose a comfortable height to determine the corner points that define the wall.

The example shown here for scanning one or more walls is particularly efficient because to scan a typical room with four walls the user only needs to specify four corner points of the room, with each corner point located in a respective one of the four corners of the room. Any two of the four scanned corner points define a wall of the room. For example, a first recorded corner point and a second recorded corner point define a first wall of the room, the second recorded corner point and a third recorded corner point define a second wall of the room, the third recorded corner point and a fourth recorded corner point define a third wall of the room that is opposite the first wall, and the fourth recorded corner point and the first recorded corner point define a fourth wall of the room that is opposite the second wall.

The user continues this process until all walls in a room have been measured. When the last wall has been measured, the user performs a predefined gesture (e.g. "thumbs up") so that the measured walls are connected to each other in the above sense.

In the manner described above, any number of rooms can be scanned by the user.

There are different approaches to marking and assigning objects with manual scanners. The approach described above relies on the user's hands (and thus on the gesture recognition of a VR console) to scan objects. However, VR controllers can alternatively be used for pointing and confirming.

11 shows an example data structure of a scanned virtual environment, as obtained by the scanning process shown above. The data structure comprises several stored scans, of which a first stored scan Profile_0 and an nth stored scan Profile_n are shown. The first stored scan Profile_0 comprises a first room Room_0, a second room Room_1 and a third room Room_2. Room_0 comprises a substructure

RoomObjects_0 of scanned objects of the room and a substructure Walls_0 of scanned walls of the room. Room_1 includes a substructure RoomObjects_1 of scanned objects of the room, but no assigned walls. Room_2 has no assigned objects or walls yet. The nth saved scan Profile_n includes an nth room Room_n and an n+1th room Room_n+1 and a third room Room_2. Room_n includes a substructure RoomObjects_n of scanned objects of the room. Room_n+1 includes a substructure Walls_n+1 of scanned walls of the room.

Tracking of classified objects in room scans with real-time data through radio and optical trackers

In order to offer the user of extended reality devices the possibility of moving freely in a structure of real objects in virtual environments, it is helpful, as described above, to work with a digital image of the geometry of reality. The embodiments described below provide a spatial understanding of the extended reality device that can react to changing positions of the real objects that surround it.

In order to move individual virtual objects analogously to the position changes of their real counterparts despite the use of a one-time static room scan, the scan creation is combined with the manual or automatic classification of the scanned objects in real indoor or outdoor spaces by extended reality devices that locate themselves in the room using inside-out tracking, with the principle of radio or optical trackers that are recognized in their position by the extended reality device. In order to assign virtual objects the property of free positioning in space, the radio or optical trackers provided for this purpose in the room are logically linked to the corresponding classified virtual objects.

Since the reality around the user can change in its geometric shape during the use of extended reality devices, an occluding digital image of a moving object should react to this circumstance. If the digital image is based on a static scan that was generated either manually or automatically and whose objects were classified based on their nature and purpose, any parameters that continuously change their position in the real world must be recorded separately. This can be done by dedicated radio and optical trackers that are logically linked to the classified moving objects in the spatial scan.

Inside-out tracking allows extended reality devices such as handhelds and glasses to locate themselves in space without restrictions using SLAM methods. If the real objects are occluded by virtual content, the virtual objects should appropriately map the real geometries and react to their movements. By avoiding deviations between real geometries and virtual objects, collisions between users and real objects are avoided.

Depending on the technical equipment of the extended reality device, a solution can be used to map reality that only measures the geometry once at the beginning of the use of the extended reality device, as is the case in the 8a -e, 9 and 10 It does not matter whether the measurement of the geometries is done manually by the user or automatically by laser or video-based methods. In order to track individual real moving geometries and their positions in space live despite this static room scan, the approach of the following The described embodiments combine the clear classification of all geometries present in the scan and the connection of these with radio or optical trackers.

The radio or optical trackers, such as infrared, Bluetooth or UWB trackers, are logically related to individual classified objects. This means that the virtual counterpart of a moving real object can be tracked live in a statically generated scan for each tracker.

In a real indoor or outdoor space, it can be seen that real objects are usually positioned either in a fixed or a mobile manner due to their properties. By classifying the behavior of the real objects, it can be decided which objects make sense to attach radio and optical trackers to in order to track their behavior in motion.

In the following 12 and 13a -d it is assumed that in real space there are objects that are primarily statically positioned 4 and those that are primarily freely moved 5 .

12 shows a more detailed representation of the substructure Raumobjekte_0 of the 11 of the room Room_0. The substructure Room_Objects_0 includes the scanned objects Object_0, Object_1, Object_2 and Object_3. Object_0 of the Room_Objects_0 is an object that is statically positioned in the room and is therefore not connected to a tracker. Object_1 of the Room_Objects_0 is an object that can be moved freely and is therefore logically connected to a Tracker_0. Object_2 of the Room_Objects_0 is an object that can be moved freely and is therefore logically connected to a Tracker_1. Object_3 of the Room_Objects_0 is an object that is statically positioned in the room and is therefore not connected to a tracker.

The virtual objects are linked to the radio or optical trackers located in the room, for example, by the user selecting the extended property for a classified object in the user interface of the scanning program on the extended reality device and specifying the tracker identification features.

13a shows the room to which the data structure RoomObjects_0 is assigned. The room includes a table 85, which is recorded as object 0 in the data structure RoomObjects_0, which references a corresponding virtual object. The table 85 is considered static. The room further includes an object 86 located on the table 85. The object 89 is considered movable. The room further includes a door 90 arranged in a wall. The door 90 is considered movable. The room further includes a sofa 85. The sofa 85 is considered movable.

13b shows schematically how a virtual image of the real geometry is created by means of the scanning process based on the scanning method. During or after the scanning process, the type of the individual real objects 6 is understood and the virtual geometries lying on them are defined in their shape and classified accordingly. Table 84 is recorded as object_0 in the data structure room objects_0, which references a corresponding virtual object. Sofa 85 is recorded as object_3 in the data structure room objects_0, which references a corresponding virtual object. Object 89 is recorded as object_1 in the data structure room objects_0, which references a corresponding virtual object. Door 90 is recorded as object_2 in the data structure room objects_0, which references a corresponding virtual object.

The aim of the room scan is to create a digital image of the real room that geometrically corresponds to reality. This can be done with the help of the user by measuring manually or by an automatic scanning of the room by the sensors of the extended reality device. During this process or afterwards, the virtual object geometries are classified by the user or automatically by the scanning software based on the real objects on which they are based and their overall shape is understood. This is necessary in order to be able to assign different properties to the virtual objects so that they are independent of a static position.

13c shows schematically how - as soon as radio or optical trackers are placed on real objects - these are linked to virtual objects by the extended reality device in order to track the real objects in their relative position to the device. This takes place independently of the SLAM tracking of the extended reality device 20, which tracks the relationship between device and space. The virtual object Object_1, which corresponds to the object (89 in 13a) on the table (84 in 1a3 ) is logical linked to a tracker Tracker_0. The virtual object Object_2, which is the door (90 in 13a) is logically linked to a tracker Tracker_1. The virtual objects are assigned the property of free positioning in space by logically linking the radio or optical trackers provided for this purpose in the room with the corresponding virtual objects.

In order to be able to use radio and optical trackers such as infrared, Bluetooth or UWB trackers, the extended reality device used in the room in this embodiment is not only able to calculate its relation to the real geometry through inside-out tracking, but it can also track its spatial relation to the tracker in real time through the direct radio or optical connection.

When using different technologies for inside-out tracking of the extended reality device and location of the trackers by it, such as when mixing radio and optical data, care is taken to ensure that there is no delay or offset in the data so that the spatial relationship of the real objects in the room matches the display of the virtual objects.

13d shows schematically how - as soon as radio or optical trackers 7 are placed on real objects - these are tracked by the extended reality device 8 in their relative position to the device. While tracking the movement of the trackers Tracker_0 and Tracker_1, the extended reality device moves the respective logically linked virtual objects Object_1 and Object 2 of the classified room scan analogously. For example, the door (90 in 13a) , which is linked to Tracker_1, is opened. This movement is tracked by Tracker_1, and the extended reality device transfers the movement of the real door to the virtual object Object_2 linked to it. In the example here, for example, a virtual door is opened whose position corresponds to the real door. If the user moves a real geometry to which a radio or optical tracker is attached in real time while using the extended reality device, this movement will be processed in real time by the extended reality device. By directly connecting the extended reality device and the tracker, the tracker will move the logically linked digital object in digital space analogous to the change in position of the tracker in real space. This happens while the remaining virtual objects in the virtual scene are held in their position by the inside-out tracking of the extended reality device.

For example, three-dimensional transformations of a virtual object are performed by transforming each object point (e.g. vertex).

For example, if an object is defined by several vertices, the translation is achieved by moving all vertices to a new position.

Instead of positioning each vertex of the object separately in space, the relative position of the object vertices is usually defined in an object coordinate system whose origin lies, for example, at the center of gravity of the object in the spatial coordinate system, or at another location that defines the position of the object in the spatial coordinate system ("object position").

14 shows schematically a virtual object 93, here in the form of a cube, which is defined in an object coordinate system 91 in which the vertices v1, v2, v3, v4, v5, v6, v7, v8 of the object are defined.

15a shows, as an example, the displacement of a virtual object 93 from a starting position (shown with solid lines) to a final position (shown with dashed lines). In this case, the object 93 is translated by displacing the object coordinate system 91 in the spatial coordinate system 92, i.e. by displacing the object position in space. Since this displaces the entire object coordinate system 91, all vertices of the object are automatically displaced by this displacement of the object position.

dargestellt, wobei t_x, t_y, and t_z die Einträge des Translationsvektors sind.A translation is usually represented in homogeneous coordinates (in which 3D transformations are represented by 4x4 matrices) by a translation matrix $$T=\left(\begin{array}{cccc}1& 0& 0& t_x\\ 0& 1& 0& t_y\\ 0& 0& 1& t_e\\ 0& 0& 0& 1\end{array}\right)$$

, where t _x , t _y , and t _z are the entries of the translation vector.

wobei P^obj(t) die Position des Objekts zur Zeit t nach der Verschiebung ist.The displacement of an object located at the object position P ^obj (t ₀ ) in the spatial coordinate system at time t ₀ is expressed by matrix multiplication as follows: $${\left(P^{Obj}\left(t\right)\mathrm{,1}\right)}^T=T{\left(P^{Obj}\left(t_0\right)\mathrm{,1}\right)}^T$$

where P ^obj (t) is the position of the object at time t after the translation.

$$R_z=\left(\begin{array}{cccc}\text{cos}\alpha & -\text{sin}\alpha & 0& 0\\ \text{sin}\alpha & \text{cos}\alpha & 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right)$$

$$R_y=\left(\begin{array}{cccc}\text{cos}\gamma & 0& \text{sin}\gamma & 0\\ 0& 1& 0& 0\\ -\text{sin}\gamma & 0& \text{cos}\gamma & 0\\ 0& 0& 0& 1\end{array}\right)$$

wobei α, β, und y die Drehwinkel (Euler-Winkel) sind, welche die Änderung der Orientierungsänderung des Objekts bestimmen.Any rotation of an object in 3D space is expressed in homogeneous coordinates by three rotation matrices R _z (α), R _y (γ), and R _x (β): $$R_x=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& \text{cos}\beta & -\text{sin}\beta & 0\\ 0& \text{sin}\beta & \text{cos}\beta & 0\\ 0& 0& 0& 1\end{array}\right)$$

$$R_e=\left(\begin{array}{cccc}\text{cos}\alpha & -\text{sin}\alpha & 0& 0\\ \text{sin}\alpha & \text{cos}\alpha & 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right)$$

$$R_y=\left(\begin{array}{cccc}\text{cos}\gamma & 0& \text{sin}\gamma & 0\\ 0& 1& 0& 0\\ -\text{sin}\gamma & 0& \text{cos}\gamma & 0\\ 0& 0& 0& 1\end{array}\right)$$

where α, β, and y are the rotation angles (Euler angles) that determine the change in orientation of the object.

To change the orientation of an object in three-dimensional space, the vertices are rotated according to the rotation angles α, β, and γ in the object coordinate system.

15b shows, by way of example, the rotation of a virtual object 93 from a starting position (shown with solid lines) to a final position (shown with dashed lines). In this case, rotation of the object 93 is achieved by rotating the object coordinate system in the spatial coordinate system 92, or by rotating the vertices in the object coordinate system. As a result, the object is rotated around its central point, as shown in 15b is shown (the object coordinate system is in 15b not shown for simplicity).

wobei P^vertex(t) die Position des Vertex im Objektkoordinatensystem zur Zeit t nach der Rotation ist.If one works with an object coordinate system as described above, the orientation change of the object can be made directly in the object coordinate system by multiplying the position P ^vertex (t ₀ ) of each vertex before the rotation (time t ₀ ) with the corresponding rotation matrices: $${\left(P^{vertex}\left(t\right)\mathrm{,1}\right)}^T=R_e\left(\alpha \right)R_y\left(\gamma \right)R_x\left(\beta \right){\left(P^{vertex}\left(t_0\right)\mathrm{,1}\right)}^T$$

where P ^vertex (t) is the position of the vertex in the object coordinate system at time t after the rotation.

Let P ^tracker (t ₀ ) be the initial position of the tracker of an object at time t ₀ of the object's creation and P ^tracker (t) be the current position of the tracker at time t, as provided by the tracker.

beschreibt die Positionsveränderung des Trackers zwischen dem Ausgangszeitpunkt t₀ der Erzeugung des Objekts und zum aktuellen Zeitpunkt t.The translation vector $$t=\left(t_x,t_y,t_e\right)=P^{tracker}\left(t\right)-P^{tracker}\left(t_0\right)$$

describes the change in position of the tracker between the initial time t ₀ of the creation of the object and the current time t.

The tracker also provides the rotation angles α, β, and γ (Euler angles), which describe the change in the orientation of the tracker between the initial time t ₀ and the current time t.

The rotation angles α(t), β(t), and y(t) and the translation vector t(t) = (t _x (t), t _y (t), t _z (t)) describe the movement of the tracker in space.

wobei P^obj(t₀) die Ausgangsposition des virtuellen Objekts zum Zeitpunkt t₀ darstellt und T(t) die Translationsmatrix beschreibt, welche der Verschiebung zum Zeitpunkt t entspricht.Assuming that the virtual object assigned to the tracker is to perform the same movement, i.e. translation and change of orientation (rotation), as the tracker attached to the physical object (or the physical object itself), the following applies to the current position P ^obj (t) of the virtual object at time t: $${\left(P^{Obj}\left(t\right)\mathrm{,1}\right)}^T=T\left(t\right){\left(P^{Obj}\left(t_0\right)\mathrm{,1}\right)}^T$$

where P ^obj (t ₀ ) represents the initial position of the virtual object at time t ₀ and T(t) describes the translation matrix, which corresponds to the displacement at time t.

A change in the orientation of the object corresponding to that of the tracker can be achieved in the object coordinate system as follows: $${\left(P_i^{vertex}\left(t\right)\mathrm{,1}\right)}^T=R_e\left(\alpha \left(t\right)\right)R_y\left(\gamma \left(t\right)\right)R_x\left(\beta \left(t\right)\right){\left(P_i^{vertex}\left(t_0\right)\mathrm{,1}\right)}^T$$

Calibration occurs when the scan is created, while the headset knows the tracker's starting position P ^tracker (t ₀ ) through radio data or a direct optical connection. Optical trackers work in the same way as the controllers of the glasses themselves and have infrared features that the glasses (or smartphone/tablet) see.

To simplify calibration when restarting a VR/AR application, the "Spatial Anchor" technology can be used. If the Spatial Anchor correctly places the old scan again when the application is restarted, it should not be a problem to integrate new data from the trackers. If the moving objects in the room have moved since the application was last used, and so have the trackers, the AR/VR device can then adapt the logically linked virtual elements to the new position data of the trackers - in other words, move them.

Through this direct connection of the position and orientation of the virtual object with the position and orientation of the tracker, the virtual object is moved in virtual space analogously to the change in position of the tracker in real space.

Reference symbols

20, 50

Headset

21, 51

Communication interface headset

22, 52

IMU

23, 53

Accelerometer

24, 54

camera

25.55

gyroscope

26, 56

Rendering unit

27, 57

Magnetometer

28, 58

Display

30, 60

VR console

31, 61

Communication interface VR console

32

Gesture recognition unit

33, 63

VR unit

34, 64

memory

36, 66

Tracking unit

38, 68

Input interface

40, 42

Hand controller

80

Display menu

81

left hand

82

right hand

83, 89

Survey point (e.g. corner point)

84

Spatial object (table)

85

Room object (sofa)

86, 87

virtualized representation of a spatial object

88

virtualized representation of a wall

89

Spatial object (object)

90

Room object (door)

91

Object coordinate system

92

Spatial coordinate system

Claims (10)

A method for scanning a room, the method comprising scanning one or more real objects of the room, wherein scanning an object (84, 85) comprises detecting a number of survey points (89) of the real object (84, 85) by means of a gesture or a hand controller (40, 42), and wherein the real objects are provided with respective radio or optical trackers which are recognized in their position. Procedure according to Claim 1 , whereby a virtual object is created by scanning a real object based on the recorded survey points. Procedure according to Claim 1 or 2 , in which a tracker determines the position of the associated real object in space and the position determined by the tracker is transferred to the linked virtual object. Procedure according to Claim 3 , in which a tracker further determines the orientation of the associated real object in space and the orientation determined by the tracker is transferred to the linked virtual object. Method according to one of the preceding claims, wherein when generating the virtual object, the position of the survey points of a scanned object are linked to the position information of the associated tracker in order to calibrate the tracker. Method according to one of the preceding claims, wherein the scanning of the room is carried out by means of a VR/AR device that recognizes the respective positions of the trackers assigned to the objects. Procedure according to Claim 6 , whereby the VR/AR device locates itself in the room through inside-out tracking. Method according to one of the preceding claims, wherein it is assumed that an object is either static or moving. Method according to one of the preceding claims, in which a classification of the scanned objects is carried out and the classification information is linked to the virtual objects. Device for scanning a room, wherein the device is designed to perform a scan of one or more real objects of the room, wherein the scanning of a real object (84, 85) comprises detecting a number of survey points (89) of the real object (84, 85) by means of a gesture or a hand controller (40, 42) and wherein the device is designed to receive position information from respective radio or optical trackers and to link this to respective virtual objects.

Images