DE102016001998A1 - A motor vehicle operating device and method for operating an operating device to effect an interaction between a virtual display plane and a hand - Google Patents

Abstract

The invention relates to a method for operating an operating device (1) in order to effect an interaction (21) between graphical objects (10) and a hand (16) of a user, wherein a detection device (11) detects a hand (16). free hand carried out in the room, predetermined hand gesture (17), by which on at least one or some of the objects (10) to be applied interaction (21) is determined is detected; a selection mode (A) is activated by a control device (5) when the hand gesture (17) is detected; by the detection means (11), when the selection mode (A) is activated, the user receives a user selection (10 ') of at least one of the objects (10) on which the hand gesture (17) is intended to operate and a user selection (10') terminating Confirmation gesture (20) is detected and by the control device (5) the determined by the detected hand gesture (17) interaction (21) is applied to the at least one selected by the user selection (10 ') object (10).

Description

The invention relates to a method for operating an operating device in order to effect an interaction between graphical objects which are displayed in a virtual display plane of a display device and a user's hand located outside this display device. The invention also includes an operating device for carrying out the method according to the invention. The operating device can be provided in particular in a motor vehicle, which is why the invention also includes a motor vehicle with the operating device according to the invention.

In the field of displaying virtual objects on a display device, it is often necessary for a user to be able to interact with such a virtual object, that is, for example, to be able to manipulate the object. For this purpose, for example, a mouse pointer or another representation can be displayed together with the graphical object in the virtual presentation plane, which the user then from outside the virtual representation level by means of a real control element, eg. B. a joystick, can move. In order to directly implement a movement of a user's hand in the virtual display plane, it is known to represent a three-dimensional virtual representation of the hand in a virtual reality (VR) for this purpose. But this is expensive in the implementation.

From the DE 10 2008 048 825 A1 It is known to couple a graphical object presented in a virtual presentation plane, such as a virtual desktop, to the motion of a finger by first performing an activation gesture by approaching the hand with splayed fingers, thereby modifying a modification mode the graphic objects in the virtual plane is activated. A disadvantage of this solution is that only a single Modifiziermodus can be activated by the activation gesture, in which case after activating this Modifiziermodus the user must make additional inputs to indicate what he actually wants to change the objects.

From the EP 2 930 603 A1 It is known to place grouped objects in a different display state in groups when an approach of a user's hand to the display device is detected. The user can then manipulate this group through further operations. Again, after activating the group, it is first necessary to wait and see which command should actually be performed on this group.

From the US 2007 0157089 A1 It is known to start a reconfiguration process as a function of a predetermined user action on a touch screen, in which the user can then move graphical objects on the display device of the touchscreen with his finger. A disadvantage here is that the user has to handle each individual object with his finger, that is to say, for example, to apply a single shift process to each object, and thus no simultaneous movement of several objects is possible.

It would be desirable if a user could freely execute a gesture in the room with one hand, for example, and this would act as if he were gripping the virtual representation plane with his hand and there directly interacting with the graphic objects. Although this is possible with Virtual Reality (VR), it requires a high implementation effort.

The invention is based on the object of effecting an interaction between graphic objects in a virtual display plane of a display device on the one hand and a user's hand located outside the display direction on the other hand.

The object is solved by the subject matters of the independent claims. Advantageous developments of the invention are disclosed by the features of the dependent claims, the following description and the figures.

The invention provides a method for operating an operating device. The operating device can be designed, for example, as an infotainment system (information entertainment system) of a motor vehicle. However, the operating device can also be realized, for example, by a portable mobile terminal, that is, for example, a smartphone or a tablet PC. The method effects an interaction between graphical objects which are displayed in a virtual display plane of a display device and a user's hand located outside this display device. By means of a detection device, a hand gesture that is free of contact with the outside of the display device is detected by the hand in a predetermined manner. The hand gesture determines the interaction to be applied to one or more of the objects. For example, the user can perform a sliding gesture with his hand as if he were pushing objects floating freely in space away from him or pushing them aside. By a control device, a selection mode is activated in the virtual presentation level when detected hand gesture. The control device So changes from a normal operating mode or operating mode in a selection mode. In normal operation mode z. B. by touching a respective object z. B. an associated operating function can be activated, for. B. set an air conditioner or a navigation device can be configured. When the selection mode is activated, the user then receives from the user a user selection of at least one of the objects to which the hand gesture is to act. In other words, the user determines through user selection which of the objects to manipulate from the interaction represented or determined by the hand gesture. For example, it determines by the user selection which of the objects is to be moved by its shift gesture. The user selection can be done for example by tapping or touching the objects on the display device. Unlike the normal operating mode, the associated operating function is not activated here.

The detection device finally acquires a confirmation gesture concluding the user selection. The confirmation gesture finally determines which of the objects as a whole is to be recorded by the hand gesture. By the control device, the interaction determined by the hand gesture is then averted to the at least one object selected by the user selection in the virtual presentation plane. In the example described, the selected objects are then moved. In other words, the user establishes a manipulation function by the hand gesture, determines, by the user selection, the objects to which the manipulation function is to be applied, and finally the control device applies the selected manipulation function to the selected objects.

By the invention, there is the advantage that the user can determine freely in space with a hand gesture, which interaction he wants with the objects, so if they are for example moved or deleted. Since the user can not intervene, for example, with a virtual representation of his hand directly into the virtual representation plane and can directly touch or touch the objects to be manipulated or manipulated, the invention solves this problem by determining the interaction in a selection mode by the user By means of the user selection it is still determined on which of the objects the interaction is to take effect. This saves the complex representation of the user's hand by a virtual hand in the virtual presentation level. Finally, the selected objects can then actually be manipulated according to the selected or specified interaction.

The invention includes optional developments, the characteristics of which provide additional advantages.

According to one development, a gripping movement or catching position is recognized as the hand gesture. The gripping movement may be, for example, a movement of the hand with the fingers open and then closing the fingers to the fist. The catching posture can be, for example, the holding of the flat or curved hand with the palm upwards. As a result of the control device, as the interaction, the sliding out or falling out of the at least one object defined by the user selection is thereby implemented from the virtual plane. This deletion of these objects is realized or implemented by the display device. The user thus accesses objects or holds the hand in the hand gesture as if he wanted to catch objects that fall out of the display with the Auffanghaltung. The collecting position is detected in particular below the display device. The user selection determines which of the objects are then pulled out of the virtual presentation plane or fall out of the virtual presentation plane. The falling out can be visualized by a corresponding animation of the selected objects on the display device.

A further development provides that the user selection is received as an inverse selection, in that all objects except those marked by the user are included in the user selection. So the user first marks some objects. The user selection then includes all unmarked objects. This results in the advantage that the selection of a plurality of objects, namely in particular more than half of the objects, takes place with fewer operating steps.

Upon receiving the user selection, it is possible to apply the interaction directly after selecting one of the objects, ie to delete or move this object, for example. A further development, on the other hand, provides that the user selection is terminated by the recognition device only when the final confirmation gesture is recognized, and only then, by the control device, is the interaction simultaneously applied to each selected object. This has the advantage that the user can still reschedule during the user selection, ie before the termination, and can undo a selection, for example. With immediate deletion of an object this would not be possible.

The method for operating the operating device can be used for different fields of application. A further embodiment provides that hereby a menu structure or a menu content and a content of a desktop view are adjusted. The operating menu or the desktop in this case represents the virtual presentation level. In the further development, the content of an operating menu provided in the virtual presentation level for activation of operating functions of the operating device and / or a collection of icons (image symbols) is adapted. For example, the user can remove menu items of the operating menu or remove icons. Only the menu entries or icons selected by the user remain. As operating functions can be provided by the operating device, for example, controlling an air conditioning device and / or media playback (radio or MP3 player) and / or setting a seating position in a motor vehicle and / or operating a navigation device.

A further development relates to the selection mode. In this development, the objects are displayed differently and graphically and / or dynamically, that is to say in their movement, by the control device when the selection mode is activated than in the case of an inactive selection mode. As a graphic change, for example, a color and / or shape and / or marking can be changed. As a dynamic change, for example, a silent or motionless representation on the one hand and, for example, a pulsating or rotating representation on the other hand be provided.

In order to reliably detect the hand gesture outside the display device, a development provides that the hand gesture is detected by means of a camera by the detection device. The camera here means the actual image capture device as well as an associated image processing device by which a hand is recognized in image data of the image capture device and its position in space is detected. In particular, it is provided that a time-of-flight camera is used. This has the advantage that directly on the basis of the 3D image data of the time-of-flight camera, a relative position of the hand in space with respect to the camera can be detected.

In order to detect the user selection, a development provides that the user selection of the at least one object is detected by the detection device in each case by means of a sensor matrix on a display surface of the display device as a respective contact of the object. The user thus interacts directly with this by touching each object. Such a sensor matrix can be realized, for example, by means of a touchscreen, by means of which both the display surface of the display device and the touch-sensitive sensor matrix can be provided.

In order to carry out the method according to the invention, the invention also provides an operating device which comprises a display device, a detection device and a control device. The display device is set up to display graphic objects in a virtual display plane. The detection device is configured to recognize outside the display device a hand gesture of a user's hand, which it carries out freely in the room. This predetermined hand gesture represents an interaction to be applied to at least one or more of the objects, that is, for example, a deletion function. The detection device is further configured to receive a user selection of those of the objects to which the hand gesture is to act. The controller is configured to activate a selection mode for receiving the user selection and to translate or apply the interaction determined by the detected hand gesture to the at least one object selected by the user selection in the virtual presentation plane. It is provided that the operating device according to the invention performs an embodiment of the method according to the invention by means of its display device, detection device and control device. The display device can be realized, for example, by a screen or a head-up display. The detection device may comprise a camera, in particular a time-of-flight camera, and / or a sensor matrix in the manner described. The control device can be realized on the basis of a processor device, for example a microcontroller or microprocessor. The control device then has a corresponding program module, by means of which, when it is executed by the processor device, those method steps which relate to the control device are carried out.

The operating device can be provided in the manner described in a portable, mobile terminal, for example a smartphone or tablet PC. In particular, it is provided that the operating device is realized in a motor vehicle. Accordingly, the invention also provides a motor vehicle with an embodiment of the operating device according to the invention. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck.

In the following an embodiment of the invention is described. This shows:

1 a schematic representation of an embodiment of the operating device according to the invention in a normal operating mode;

2 a schematic representation of the operating device of 1 upon detecting a hand gesture of a user;

3 a schematic representation of the operating device in a different from the normal operating mode selection mode;

4 a schematic representation of a hand of the user when executing a confirmation gesture; and

5 a schematic representation of the operating device after it has returned from the selection mode in the normal operating mode.

The exemplary embodiment explained below is a preferred embodiment of the invention. In the exemplary embodiment, the described components of the embodiment each represent individual features of the invention that are to be considered independently of one another, which also each independently further develop the invention and thus also individually or in a different combination than the one shown as part of the invention. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

In the figures, functionally identical elements are each provided with the same reference numerals.

1 shows an operating device 1 For example, in a motor vehicle 2 or (not shown) may be provided in a portable, mobile terminal, such as a smartphone or a tablet PC. The operating device 1 can be a display device 3 with a display area 4 exhibit. The display device 3 may be, for example, a screen or a head-up display. A display content on the display area 4 can by a control device 5 controlled, which may be realized for example on the basis of a microcontroller or microprocessor. By the control device 5 can, for example, device components 6 For example, vehicle components of the motor vehicle 2 , by a corresponding control signal 7 to be controlled. The device components 6 are in the 1 represented only by a single element. For each of the device components 6 can by the control device 5 a respective operating function 8th be provided, for example, by a graphical user interface for operating the respective device component or a control function for triggering a device function of the respective device component can be realized. By the control device 5 can by appropriate graphic data 9 on the display surface 4 the display device 3 the operating functions 8th each by a graphic object 10 , for example, an icon or a lettering, or a menu item, be represented.

At the operating device 1 It allows a user to simultaneously display several of the graphical objects 10 selects and thereby, for example, in a menu a selection of the user on the display area 4 to be offered operating functions 8th that is, a selection of graphic objects to be displayed only 10 , meets. It can also be a simultaneous activation of several of the operating functions 8th by a corresponding simultaneous selection of several of the graphic objects 10 be realized. For detecting or detecting the user selection, the operating device 1 a detection device 11 have, for example, a camera 12 , in particular a TOF camera, and / or an approximation-sensitive and / or touch-sensitive sensor matrix 13 may include. The sensor matrix 13 can, for example, on the display area 4 be provided, that is, the display device 3 is in this case a touch screen. A detection area 14 the camera 12 can also be on the display area 4 be directed.

The graphic objects 10 can be, for example, list entries of an operating menu or icons of a desktop. Accordingly forms a representation level 15 on the display area 4 is shown and on which the objects 10 are arranged, a virtual presentation level, such as a control menu or a desktop. The user is now able to select which graphical objects 10 he from the presentation level 15 would like to get presented. In other words, he can customize an operating menu or an arrangement of icons. Through the operating device 1 is it possible for him, by means of gesture operation and on the sensor matrix 13 the display device 3 the deletion / hiding of menu contents / elements, ie the graphic objects 10 to make.

In the 1 displayed display of graphical objects 10 For example, a standard orientation of the menu arrangement of individual menu items in the form of graphic objects 10 represent. Each menu item, for example, one of the operating functions 8th for example for radio, Navigation, climate control, seat adjustment represent.

2 shows how the user can now set some of the graphical objects of the presentation plane 15 are to be removed. Below the display 3 the user guides with his hand 16 an operating gesture 17 from which the desire of a deletion or the desire to hide one of the graphical objects 10 determines, thus an interaction announces, which the user with the hand 16 and the graphic objects 10 would like to effect. The hand gesture 17 For example, an attitude of the hand 16 represent where the hand to catch the out of the virtual presentation plane 15 and thus from the display device 3 falling out graphic objects 10 represents by a flat, turned up palm or a curved, turned up palm. Through the camera 12 becomes the hand 16 in the hand gesture 17 through 3D image data 18 displayed. The recognition device 11 recognize the hand gesture 17 and signals it to the controller 5 , which thereupon for the representation level 15 a selection mode A is activated, which is also in 2 is shown. In selection mode A is a graphical and / or dynamic representation of the objects 10 changed so that the graphic objects 10 for example, compared to in 1 Normal mode shown are displayed animated and / or in a different shape, size, color and / or in a dynamically variable form, for example, pulsating larger and smaller. It can also be a wobble or flashing provided.

3 shows how by the detection device 11 in selection mode A, a selection of the graphic objects to be removed 10 is determined. Here, a direct selection takes place, that is, the user can remove the objects to be removed 10 to mark. For this purpose, for example, the user selection via the sensor matrix 13 on the display surface 4 be recorded. The user can, for example, with a finger 19 the hand 16 on the display surface 4 the objects to be selected 10 tap or touch. These can then also be marked graphically as selected. The selected or selected objects 10 make a user selection 10 ' represents.

Exemplary is in 4 illustrates that to terminate the selection mode A, the user freely in the room a confirmation gesture 20 in which the user, for example, a swipe by hand 16 performs in a predetermined wiping direction. This confirmation gesture 20 in turn, with the camera 12 captured and through 3D image data 18 be represented by the image recognition means of the detection means 11 as the confirmation gesture 20 can be recognized. By signaling the confirmation gesture 20 is through the control device 5 the user selection is completed in such a way that the interaction or manipulation function, by the hand gesture 17 has been set, now to the selected objects 10 is applied. In the illustrated example, it can thus be provided that the selected objects "from the display device 15 fall, "for example, as in 5 presented in an animation 21 on the virtual presentation layer 5 be moved down and then out of the virtual presentation layer 15 that is the display device 3 fall out or disappear. The animation 21 thus represents the falling out of the selected object 10 and thus by the hand gesture 17 fixed interaction. What is left are then the graphic objects desired by the user 10 ,

This provides the user with quick access or direct access to operating functions 8th about the remaining graphical objects 10 possible without the user first having to visually pick the right objects from a variety of otherwise unneeded graphical objects.

Overall, the example shows how can be used by the invention, a gesture operation for deleting a control device.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102008048825 A1 [0003]
• EP 2930603 A1 [0004]
• US 20070157089 A1 [0005]

Claims (10)

Method for operating an operating device ( 1 ), an interaction ( 21 ) between in a virtual representation plane ( 15 ) a display device ( 3 ) displayed graphical objects ( 10 ) and one outside the display device ( 3 ) hand ( 16 ) of a user, wherein - by a detection device ( 11 ) one outside the display device ( 3 ) from the hand ( 16 ) freely in space without contact, predetermined hand gesture ( 17 ), by which on one or more of the objects ( 10 ) interaction ( 21 ) is detected; By a control device ( 5 ) with recognized hand gesture ( 17 ) a selection mode (A) in the virtual representation plane ( 15 ) is activated; By the detection device ( 11 ) when the selection mode (A) is activated by the user, a user selection ( 10 ' ) at least one of the objects ( 10 ), to which the hand gesture ( 17 ), is received and a user selection ( 10 ' ) final confirmation gesture ( 20 ) is detected; and - by the control device ( 5 ) by the recognized hand gesture ( 17 ) fixed interaction ( 21 ) by the user selection ( 10 ' ) selected at least one object ( 10 ) in the virtual presentation plane ( 15 ) is applied. The method of claim 1, wherein as the hand gesture ( 17 ) is detected a gripping movement or Auffanghaltung and thereby by the control device ( 5 ) as the interaction ( 21 ) pushing out or falling out by the user selection ( 10 ' ) defined at least one object ( 10 ) from the virtual level ( 15 ) and thus a deletion of these objects ( 10 ) from the display device ( 3 ) is implemented. Method according to one of the preceding claims, wherein the user selection ( 10 ' ) is received as an inverse selection by all objects ( 10 ) except for those marked by the user from the user selection ( 10 ' ) are included. Method according to one of the preceding claims, wherein the recognition device ( 11 ) the user selection ( 10 ' ) upon recognition of the final confirmation gesture ( 20 ) is terminated and by the control device ( 5 ) only then the interaction ( 21 ) simultaneously on each selected object ( 10 ) is applied. Method according to one of the preceding claims, characterized by the interaction ( 21 ) contents of one in the virtual presentation level ( 15 ) for activation of operating functions ( 8th ) of the operating device ( 3 ) and / or a collection of icons adapted to the user. Method according to one of the preceding claims, wherein the control device ( 5 ) with activated selection mode (A) the objects ( 10 ) are displayed graphically and / or dynamically differently than in inactive selection mode. Method according to one of the preceding claims, wherein the detection device ( 11 ) the hand gesture ( 17 ) by means of a camera ( 12 ), in particular a time-of-flight camera ( 12 ). Method according to one of the preceding claims, wherein the detection device ( 111 ) the user selection ( 10 ' ) of the at least one object ( 10 ) each by means of a sensor matrix ( 13 ) on a display surface ( 4 ) of the display device ( 3 ) as a respective touch of the object ( 10 ) is detected. Operating device ( 1 ), comprising: - a display device ( 3 ) for displaying graphic objects ( 10 ) in a virtual presentation plane ( 15 ); A detection device ( 11 ) for detecting an outside of the display device ( 3 ) from one hand ( 16 ) of a user freely in space without contact, predetermined hand gesture ( 17 ), through which one on one or more of the objects ( 10 ) interaction ( 21 ) and to receive a user selection ( 10 ' ) at least one of the objects ( 10 ), to which the hand gesture ( 17 ) by the user; and a control device ( 5 ) for activating a selection mode (A) for receiving the user selection (A) 10 ' ) and to implement the by the recognized hand gesture ( 17 ) fixed interaction ( 21 ) to the at least one user selection ( 10 ' ) selected object ( 10 ) in the virtual presentation plane ( 15 ); the operating device ( 1 ) is adapted to perform a method according to any one of the preceding claims. Motor vehicle ( 2 ) with an operating device ( 1 ) according to claim 9.

Images