DE102012218837A1 - Method for displaying a representation on head-up display or head-mounted display e.g. LCD in passenger car, involves determining representation that is to-be-superimposed from view of user in environment, and assigning representation - Google Patents

Abstract

The method involves determining the representation (2,3) that is to-be-superimposed from the view of user in environment. The direction in which the representation is seemingly for the user, is determined. The determination is made whether the head-mounted display is superimposed on the display in the direction or not. The determination is made whether the head-up display is superimposed on the display in the direction or not. The representation is assigned either for displaying on the head-mounted display or the head-up display. Independent claims are included for the following: (1) a device for displaying content on head-up display and head-mounted display; and (2) a computer program for displaying content on head-up display and head-mounted display.

Description

The invention relates to a method for displaying a representation on a head-up display and a head-mounted display, a device and a computer program for the same purpose.

Nowadays it is known to install head-up displays in motor vehicles such as cars. Head-up displays are fixedly mounted in a vehicle displays that display representations of a reflection on a disc, such as the windshield, a user, typically the driver of a motor vehicle. The display does not affect the user's view of the environment or is negligible for driving a vehicle.

In addition, considerations have become known to use head-mounted displays in motor vehicles. Head-mounted displays herein include all devices that are worn on the user's head and from there display an image to the user; Also devices with so-called retina projection count for this purpose.

So far, however, head-up displays and head-mounted displays are not used together and resulting potential benefits are not used.

The task of the person skilled in the art is to provide improved display options using head-up and head-mounted displays.

The object is achieved by a method according to claim 1, an apparatus according to claim 11 and a computer program according to claim 12. Advantageous developments are defined in the dependent claims.

In one aspect, a method for displaying a display on a head-up display or a head-mounted display comprises: determining the representation to be superimposed on its environment from the perspective of the user; Assign the display either for display by the head-mounted display or for display by the head-up display. The determination of the representation can also include the determination of the direction in which the representation should appear and the distance in which the representation should appear. The representation may be a contact-analogue representation superimposed on the user's view of the environment according to the position and / or movement of an object, sometimes referred to as augmented reality.

In this way, a representation can be provided either on the head-mounted display or the head-up display. A rigid allocation of a representation to one of the displays, which may be less advantageous depending on the situation, is avoided. The assignment may take into account on which of the two displays the display is more advantageous to the user. Criteria for this may be the content of the representation, the direction and distance of the representation, or the direction in which the user looks as well as the properties of the displays. Properties of the displays can be, for example, the area of the focal plane, covered image angles and the orientation or alignment possibilities of the display. In a typical head-mounted display, the focal plane can be moved in the distance range of 30 cm to 10 m. Often, the image angle that can be displayed by the head-mounted display is greater than with a head-up display. Generally, the head-mounted display moves with the head movement and thus the central line of sight of the user. In contrast, the angle of view of a typical head-up display is smaller and the focal plane fixed. The angle of view thus does not move with the head movement of the user. However, a typical head-up display allows the display of a representation in a focal plane that is greater than 10 m, for example, 15 m away. In general, this results in a better suitability of a head-mounted display for non augmented reality representations, such as a speed, symbols or warnings, which should always be displayed in the driver's field of vision. Also, augmented reality representations, but typically only for focal planes of 30 cm to 10 m, can be displayed by the head-mounted display if additional tracking of the orientation of the head-mounted display is provided, for example by head tracking using a camera becomes.

When using either a head-mounted display or a head-up display in a vehicle, this results in certain advantages and disadvantages depending on the type of display. For example, the contact-analogous marking of a 100 m distant vehicle ahead with a head-mounted display can be realized less well than with a head-up display. This is because the head-mounted display can typically only represent representations with a focal plane of up to 10 m. The representation and the vehicle to be marked are therefore not in the same plane or focal plane. The user may therefore have difficulty associating the contact analog tag with the vehicle. In addition, the user may have difficulty adjusting his view of both elements, the mark and the vehicle, which ultimately contributes to the driver's eye fatigue. Especially with longer trips this effect reduces the traffic safety of the user. For this reason, the should Marking the vehicle ahead advantageously assigned to the head-up display. On the other hand, a permanently installed in the vehicle head-up display is typically not able to display representations in all directions of the driver and limited in his representable field angle range. Representations to be displayed in directions that the head-up display does not cover are advantageously assigned to the head-mounted display.

The determination of the representation may be part of a comprehensive three-dimensional scene generation that determines the representations to be displayed for the entire environment of the driver. Three-dimensional means that each representation is assigned a position in space or a direction (possibly via a solid angle specification) and distance. Each representation is then assigned to either the head-up display or the head-mounted display, in particular according to the characteristic of the representation (direction, distance) and the properties of the displays. In this case, a head-tracking system can be used to determine the area in which the head-mounted display can display representations, which is taken into account in the assignment. The head-mounted display will then only transfer the representations assigned to it. Advantageously, the renderings of the representations are performed directly by the head-mounted display, which reduces the data to be transferred to the head-mounted display and reduces the latencies.

For example, a user's environment also includes a vehicle interior when the user is in a vehicle. Superimposing a representation of the environment means completely covering the environment behind it, or rendering it transparent (showing only an outline) or semi-transparent, so that users can still perceive the environment behind them.

The assignment of the representation may be based on one or more of the following parameters: a monitoring status of the head-tracking system, a monitoring status of the head-mounted display, the vehicle's own speed, the type of road or road being used (city, highway, motorway ), a gear selector lever position (eg reverse gear), the status of the parking brake of the vehicle, the driving situation (eg traffic jam, stopping at a traffic light intersection, parking), a set driving experience mode (eg sport, comfort or ECO), Driving condition (self-driving, semi-autonomous driving or autonomous driving), navigation settings, settings in communication systems, status settings in social media portals (active, inactive, etc.).

In a development, the method further comprises: determining the direction in which the representation is supposed to be located for the user; Determining whether the head-mounted display can superimpose the representation in the direction; Determining whether the head-up display can superimpose the representation in the direction; Assigning the display either for display by the head-mounted display or for display by the head-up display based on determining whether the head-mounted display or the head-up display can superimpose the representation in the direction.

In this way, a display is selected according to whether it can superimpose a representation of the circumference in the concrete direction in which a representation is to be displayed. At the same time a "best effort" is operated by the use of two displays, which can optionally superimpose representations in different directions, so trying as possible to display the representation in the specific direction. Compared to a rigid assignment of a representation to a display, as known from the prior art, so in more cases, the representation can be displayed in the specific direction.

A (also three-dimensional) area to which a head-up display or a head-mounted display can overlay representations is also referred to here as a display area. The range can be characterized by angular ranges and a range of distances and depend on the position and orientation of the head-up display or head-mounted display.

Determining the direction and / or distance and / or range to which the head-up display superimposes representations may be performed using stored characteristics of the head-up display. In a simple case, the direction and / or the distance and / or the area is already stored and only needs to be recalled.

Determining the direction and / or distance and / or range to which the head-mounted display may superimpose representations using a camera that determines the position and / or direction of the head-mounted display or the head supporting the head-mounted display , and optionally stored properties of the head-mounted display, such as the field of view or the so-called field of view, so regardless of the orientation of the head-mounted displays design-related area in which the head-mounted display can overlay representations determined.

In an advantageous development, the presentation is assigned to the head-up display, if only the head-up display can superimpose the presentation, and wherein the representation is assigned to the head-mounted display, if only the head-mounted display can superimpose the presentation; if the presentation can be superimposed on both the head-up display and the head-mounted display, use further criteria for assigning, in particular a priority assignment to the head-up display or the head-mounted display. In the case of the priority assignment, the criterion can be used which display has the higher resolution or which display is to be used by default.

In a further refinement, the determination of the representation in which the distance to be apparent to the user is to be displayed, and wherein the assignment is based at least on the criterion of distance, wherein according to the criterion a representation is assigned to the head-up display, in particular the distance is greater than a predetermined distance, for example about 10 m or 15 m, and is assigned to the head-mounted display if the distance is less than the predetermined distance, for example about 10 m or 15 m. Augmented reality representations can thus be advantageously assigned to the head-up display, which is in fact able to display the representations at the desired distance.

In an advantageous development, the method comprises determining the area to which the head-mounted display or the head-up display can superimpose representations; Determining the direction and / or the distance of the representation taking into account, oriented towards or adjacent to the area to which the head-mounted display or the head-up display can overlay representations; Assign the presentation to the head-up display or the head-mounted display. The display is therefore not assigned to the display for which the area was determined.

In this way, coupling takes place between the location where a representation is displayed and the physical property of a display, namely the area where representations can be superimposed (also referred to herein as the presentation area). For example, a speed indicator can be arranged below the area in which a head-up display can superimpose representations. The area of the head-up display is thus relieved of the representation of the speed (because the head-mounted display takes over the representation of the speed). Consequently, the display area of the head-up display can be optimized so that it covers as much as possible of the driver's field of vision in which displays are frequently to be displayed which have a focal plane, ie distance, which lie outside the focus area in which a head -mounted display can display representations. Consequently, little or no consideration needs to be given to the representation of the speed in the arrangement of the display area of the head-up display. The positioning of certain representations taking into account the display area of the head-up display (or the head-mounted display) thus allows an improved arrangement of the display area of the head-up display (or the head-mounted display).

In another advantageous embodiment, the method comprises determining the area to which the head-up display can superimpose representations; Determining the area to which the head-mounted display can superimpose representations; Determining the direction and / or the distance of the representation taking into account the geometric arrangement of the specific areas to each other, in particular, the overlap of the two areas and the proportion of overlap of the respective area is taken into account with the common area.

Thus, in order to determine the location of the representation (ie the direction and / or the distance), a ranking of rules can be defined, where the representation is to be displayed and under what conditions. A rule that is the highest in the ranking, and therefore should be tried first, may be to display a representation (eg, the speed of a vehicle) in the middle below the area to which the head-up display may superimpose representations. However, this is only so long as the area to which the head-mounted display can superimpose representations is arranged such that a display on the head-mounted display maintains a predetermined distance from the edge of the head-mounted display. A second rule to be met, if the previous requirement is not met, may be that the representation for the user is to be displayed in the middle below in the display area of the head-mounted display. In this way, an advantageous location for the presentation is chosen. This also taking into account the variability of the display area of the head-mounted display, which indeed moves with the head movement of the user.

In another development, the method comprises determining that the representation should appear in a fixed direction with respect to its viewing direction for the user, in particular depending on the content of the presentation; Defining the direction in which the display should remain unchanged for the user with regard to his viewing direction. This type of display is particularly useful for representations that the user should always be aware of, especially warnings or symbols such as speed, no matter where the user is looking.

In a preferred embodiment, the head-up display is configured to display representations at a user-apparent distance of about 10 to 20 meters, and the head-mounted display is adapted to display at a distance apparent to the user of about one m to 10 m and with a particular horizontal angle of view of at least about 20 ° to 60 ° display.

Furthermore, it can be provided that depictions are released for the head-mounted display depending on the driving situation in which the vehicle is located (for example in automated driving, in particular by a traffic jam assistant). These representations may include text messages or entertainment functions such as video telephony, which must be suppressed during normal driving due to legal requirements (driver distraction).

In a preferred development, the method according to one of the preceding claims further comprises: detecting whether the user is wearing the head-mounted display correctly; Detecting if the head-mounted display is working properly; Deactivating a display other than the head-up display or the head-mounted display, in particular an LCD, TFT or OLED display, or a backlight for a mechanical display in response to detecting that the user is wearing the head-mounted display correctly and that the head-mounted display works flawlessly. If a malfunction of the head-mounted display is detected or the user removes the head-mounted display, it can be provided that the other display is reactivated.

The deactivation can be carried out, in particular, when classic representations, which are just displayed on the other display, are displayed by the head-mounted display. This concerns, for example, the speed of the vehicle. If this is displayed in the head-mounted display, the driver carries the head-mounted display correctly and this works perfectly, so a classic instrument cluster or its backlight can be switched off. The instrument cluster could also be folded or retracted. Verifying the correct function and wearing of the head-mounted display before deactivating the classic display may serve to meet legal requirements.

Another case is that the driver has not put on the head-mounted display or it is currently being loaded in a charging cradle. This is detected by at least one camera of the head-tracking system and then switched to another operating mode, which uses limited functionality only on the display surface of the head-up display and other built in the vehicle displays.

The head tracking system thus detects whether the driver is wearing the head-mounted display correctly. At the same time, the correct operation of the head-mounted display can be monitored via a wireless connection in real time. In the event that the driver wears the head-mounted display and this works correctly, a known instrument cluster of a vehicle, which may for example consist of a thin and extendable display, retracted. Alternatively, it may be provided that an instrument that only serves to fulfill the legal requirements, when wearing the head-mounted display and correct function of the same, is turned off, and immediately turned on again, as soon as the driver decreases the head-mounted display or a malfunction detected or reported by the head-mounted display.

In another aspect, a device for displaying content on a head-up display and a head-mounted display comprises: the head-up display; the head-mounted display; electronic processing means, the apparatus being arranged to carry out one of the methods presented above. The electronic processing means can be a microcontroller, dedicated circuits or a miniature computer, which can be set up programmatically if necessary.

In yet another aspect, a computer program for displaying content on a head-up display and a head-mounted display includes instructions for performing one of the methods presented above. The computer program may be executed by a microcontroller, a miniature computer or dedicated circuits executing one of the above-described methods.

The above-described methods, devices and computer programs can additionally also have the following function:

When the driver wearing a head-mounted display reaches his destination and parks his vehicle Activates first a "Safe Exit" function and allows before opening the driver's door an X-ray view through B and C-pillar to z. B. to recognize a covered motorcyclist. Even children who get off on the right side warn if the driver, as this cyclist who drive past the right side of the vehicle can see. A navigation destination is automatically taken over and navigates the customer, also as a pedestrian, to his destination and also safely back to the vehicle. B. on a large supermarket parking lot is very helpful.

BRIEF DESCRIPTION OF THE DRAWINGS

1 schematically shows a vehicle and display areas according to an embodiment.

2 schematically shows a system structure according to an embodiment.

3 to 5 indicate the timing of a presentation to the user, according to one embodiment.

Like reference numerals refer to corresponding elements throughout the figures.

DETAILED DESCRIPTION OF THE EMBODIMENTS

1 schematically shows the view of a driver in a vehicle 1 , here a car, on the interior of the vehicle 1 , The driver of the vehicle is wearing a head-mounted display (not shown) and a head-up display (not shown) is installed in the vehicle. Displayed areas are shown in the driver's view of the road. With the reference number 2 are exemplary display areas referred to, in which the head-mounted display can display representations. A presentation area 2 is displayed within the horizontal viewing angle of 20 ° and within the vertical viewing angle of 15 ° of the head-mounted display. The display areas do not exist at the same time, but arise in succession with a rotation of the head of the driver. The rotation of the head and thus the driver's central line of sight is detected by a head-tracking system.

Furthermore, in 1 also the presentation area 3 a head-up display shown. The presentation area 3 is displayed within the horizontal angle of view of 9 ° and within the vertical angle of view of 4 ° of the head-mounted display. The display area of the head-up display is independent of the driver's line of sight.

2 schematically shows a system structure according to an embodiment. In a 3D scene model generation environment data from measuring systems are received and processed to determine a three-dimensional model of the environment. This model and measurement data of a head-tracking system are forwarded to a presentation generation and allocation unit. This unit determines the representations to be superimposed on the environment and their position in space (in Cartesian coordinates or by direction (angle) and distance, for example in polar coordinates). In the following, the display and allocation unit assigns the representations to be displayed to the head-up display or the head-mounted display.

3 to 5 show the time sequence of a representation of the speed "50 km / h" of a vehicle in which the user is located, according to an embodiment when driving on a highway. 3 shows the presentation area 2 in which a head-mounted display (not shown) carried by the user / driver can display representations. The presentation area 3 exists when the user is looking straight ahead onto the road. Furthermore, in 3 the presentation area 3 showing in which area a vehicle-mounted head-up display (not shown) can display representations. Compared to conventional head-up displays, which are not designed for contact-analogue display, the display is shifted further upwards. The resulting disadvantage that static and less time-sensitive messages (more generally: representations), if they were displayed by the head-up display, would obscure the traffic, is offset by the fact that a head-mounted display takes over the display of such messages. Also, an arrangement of the head-up display is such that a display of contact-analogue representations is meaningfully possible, only by the "delivery" of certain representations and their display by the head-mounted display allows. With the aid of environment sensors, here a camera, a vehicle in front has been recognized and marked 4 contact analog, at a similar distance as the recognized vehicle has been marked for the driver. Optionally, the fact can be exploited that for the human eye two representations in two larger but different distances (eg> 30 m) can hardly or not be distinguished. The mark 4 is indicated by the head-up display. The marked vehicle is located at a distance in which the head-mounted display no more displays can be displayed (not even by taking advantage of the weak discriminating ability for greater distances of the human eye). The superimposition of the presentation surfaces gives the user the impression of a larger display area than this is actually, in particular with regard to the covered image angle. Further, the three-dimensional impression that the driver of the mark 4 and the speed indication is larger than when using only one display. This is because of the marking 4 is displayed in a display distance range of the head-up display, the speed indication but at the same time in a display distance range of the head-mounted display. It is possible that the driver experiences elements that can be very close, such as: B. a virtual assistant who stands on an element of the dashboard ("hologram") or even objects that appear seemingly marked 100 meters away, like a vehicle in front.

Below the presentation area 3 the head-up display shows the current speed of the vehicle ("50 km / h") through the head-mounted display. This corresponds to a rule order to be observed in this example, that the speed should be below the display range, if possible 3 of the head-up display is, if this is the arrangement of the display area 2 the head-mounted display allows. Otherwise, the speed should be displayed centrally in the lower area of the display area of the head-mounted display.

4 shows the shift of the display area 2 the head-mounted display due to a change in the head alignment of the user, which is detected by a head-tracking system. It is determined that the speed "50 km / h" for the driver is stationary under the display area 3 of the head-up display should be displayed, so elsewhere in the display area 2 the head-mounted display. The head-mounted display shows the speed accordingly.

5 shows how the viewport 2 the head-mounted display has been postponed due to another detected change in the user's head alignment. The presentation areas 3 and 4 do not overlap anymore. At the same time is the viewport 2 shifted so that the speed indication is no longer below the display area 3 can be displayed. For this reason, it is determined that the speed is now centered and at the bottom of the viewport 2 displayed according to the rule ranking.

During the head movement of the user so a contact analog representation (here: marking) is retained (so to speak, on the vehicle in front), while a non-contact analog representation moves from a threshold with the head movement and thus the central line of sight of the user and at the same place in the The user's field of view remains, even if he does not look at the road. This can also be used to ensure that the driver perceives the warnings that are displayed, even if he does not look at the road.

Claims (12)

A method for displaying a display on a head-up display or a head-mounted display, comprising: Determining the representation to be superimposed on the environment from the user's point of view; Assign the display either for display by the head-mounted display or for display by the head-up display. The method of claim 1, further comprising: Determining the direction in which the representation is supposed to be for the user; Determining whether the head-mounted display can superimpose the representation in the direction; Determining whether the head-up display can superimpose the representation in the direction; Assigning the display either for display by the head-mounted display or for display by the head-up display based on determining whether the head-mounted display or the head-up display can superimpose the representation in the direction. Method according to claim 2, wherein the representation is assigned to the head-up display, if only the head-up display can superimpose the representation, and wherein the representation is assigned to the head-mounted display, if only the head-mounted display can superimpose the representation; if the presentation can be superimposed on both the head-up display and the head-mounted display, use further criteria for assigning, in particular a priority assignment to the head-up display or the head-mounted display. Method according to one of the preceding claims, wherein determining the representation comprises in which for the user apparent distance the representation is to be displayed, and wherein the assignment is based at least on the criterion of the distance, wherein according to the criterion a representation in particular the head-up display is assigned when the distance is greater than a predetermined distance, for example about 10 m, and the head-mounted display is assigned, if the distance is less than another predetermined distance, for example about 10 m. The method of any one of the preceding claims, further comprising: Determining the area to which the head-mounted display or the head-up display can superimpose representations; Determining the direction and / or the distance of the representation taking into account, oriented towards or adjacent to the area to which the head-mounted display or the head-up display can overlay representations; Assign the presentation to the head-up display or the head-mounted display. The method of any one of the preceding claims, further comprising: Determining the area to which the head-up display can superimpose representations; Determining the area to which the head-mounted display can superimpose representations; Determining the direction and / or the distance of the representation taking into account the geometric arrangement of the specific areas to each other, in particular, the overlap of the two areas and the proportion of overlap of the respective area is taken into account with the common area. The method of any one of the preceding claims, further comprising: Determining that the representation should appear to the user in a fixed direction relative to his viewing direction, in particular depending on the content of the representation; Defining the direction in which the display should remain unchanged for the user with regard to his viewing direction. The method of any one of the preceding claims, wherein the head-up display is adapted to display representations at a user-apparent distance of about 10 meters to 20 meters, and the head-mounted display is arranged to provide representations in one to the user apparent distance of about 1 m to 10 m and with an angle of view of at least about 20 ° to 60 °. The method of any one of the preceding claims, further comprising: Detecting whether the user is wearing the head-mounted display correctly; Detecting if the head-mounted display is working properly; Deactivating a display other than the head-up display or the head-mounted display, in particular an LCD, TFT or OLED display, or a backlight for a mechanical display in response to detecting that the user is wearing the head-mounted display correctly and that the head-mounted display works flawlessly. Method according to one of the preceding claims wherein the assignment of the representation is based on one or more of the following parameters: a monitoring status of a head-tracking system, a monitoring status of the head-mounted display, the vehicle's own speed, the type of road or road traveled , a gear selector lever position, a parking brake status of the vehicle, the driving situation, a set driving experience mode, the driving state, navigation settings, settings in communication systems and status settings in social media portals. Device for displaying content on a head-up display and a head-mounted display, comprising: The head-up display; The head-mounted display; Electronic processing means, the device being adapted to carry out a method according to any one of the preceding claims. Computer program for displaying content on a head-up display and a head-mounted display, which contains instructions for carrying out a method according to one of claims 1 to 10.

Images