WO2024181153A1

WO2024181153A1 - Virtual image display device, virtual image display method, program, and mobile object

Info

Publication number: WO2024181153A1
Application number: PCT/JP2024/005318
Authority: WO
Inventors: 和正松田; 薫草深
Original assignee: 京セラ株式会社
Priority date: 2023-02-28
Filing date: 2024-02-15
Publication date: 2024-09-06

Abstract

A virtual image display device according to the present disclosure includes a display unit, an optical element, an optical system, a camera, and a controller. The display unit displays a disparity image. The optical element defines the propagation direction of image light of the disparity image. The optical system propagates the image light, the propagation direction of which is defined by the optical element, toward the eyes of a user, and displays a virtual image of the disparity image in the field of vision of the user. The camera captures images of the eyes of the user. The controller detects the positions of the eyes of the user and the position of a gaze point of the user on the basis of captured image data outputted from the camera, and controls the disparity image on the basis of the positions of the eyes and the position of the gaze point.

Description

Virtual image display device, virtual image display method, program, and mobile object

This disclosure relates to a virtual image display device, a virtual image display method, a program, and a moving object.

　A virtual image display device, such as that described in Patent Document 1, is known.

International Publication No. 2016/166887

A virtual image display device according to an embodiment of the present disclosure is a virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image,
a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image;
an optical element configured to define a propagation direction of image light of the parallax image;
an optical system configured to propagate image light whose propagation direction is regulated by the optical element toward an eye of the user and display a virtual image of the parallax image in a field of view of the user;
a camera configured to image the eye of the user; and
A controller,
The controller:
Detecting the position of the user's eyes and the position of the user's gaze point based on the imaging data output from the camera;
The parallax image is controlled based on the eye position and the position of the gaze point.

A virtual image display method according to an embodiment of the present disclosure is a virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, the virtual image display method being executed by the virtual image display device including: a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image; an optical element configured to define a propagation direction of image light of the parallax image; an optical system configured to propagate the image light, the propagation direction of which is defined by the optical element, toward an eye of the user and display a virtual image of the parallax image in a field of view of the user; a camera configured to capture an image of the eye of the user; and a controller,
taking an image of the user's eye;
Detecting the position of the user's eye and the position of the user's gaze point based on imaging data of the user's eye;
The parallax image is controlled based on the eye position and the position of the gaze point.

A program according to an embodiment of the present disclosure is a virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, the program being executed by a virtual image display device including: a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image; an optical element configured to define a propagation direction of image light of the parallax image; an optical system configured to propagate the image light, the propagation direction of which is defined by the optical element, toward an eye of the user and display a virtual image of the parallax image in the field of view of the user; a camera configured to capture an image of the eye of the user; and a controller,
The controller is a program for causing the camera to capture an image of the user's eyes, detecting the position of the user's eyes and the position of the user's point of gaze based on the image data of the user's eyes, and controlling the parallax image based on the position of the eyes and the position of the point of gaze.

A moving object according to one embodiment of the present disclosure includes the above-described virtual image display device.

The objects, features and advantages of the present disclosure will become more apparent from the following detailed description and drawings.
FIG. 1 is a diagram illustrating an example of a virtual image display device mounted on a moving object. FIG. 2 is a diagram showing a schematic configuration of the virtual image display device shown in FIG. 1 . 3 is a diagram showing an example of the display panel shown in FIG. 2 as viewed from the depth direction. FIG. 3 is a diagram showing an example of the optical element shown in FIG. 2 as viewed from the depth direction. FIG. 2 is a diagram for explaining the relationship between the virtual image shown in FIG. 1 and the user's eyes; FIG. FIG. 2 is a diagram showing an example of sub-pixels observed by the left and right eyes of a user. FIG. 13 is a diagram showing another example of sub-pixels observed by the left and right eyes of a user. 11A and 11B are diagrams for explaining the relationship between the display position of a virtual image and the convergence angle and the amount of parallax. 4A to 4C are diagrams for explaining an example of control of a virtual image in the virtual image display device shown in FIG. 1 . 10A to 10C are diagrams for explaining another example of the control of a virtual image in the virtual image display device shown in FIG. 10 is a flowchart illustrating an operation of the virtual image display device. 10 is a flowchart illustrating an operation of the virtual image display device. 10 is a flowchart illustrating an operation of the virtual image display device.

　Conventionally, there is known a head-up display type virtual image display device mounted on a vehicle that displays a virtual image by superimposing it on the background of the user's field of vision. With such a virtual image display device, when an obstacle is present in front of the vehicle, the virtual image may be displayed as if it is penetrating the obstacle, which may cause the user to feel uncomfortable viewing the virtual image. Patent Document 1 describes a virtual image display device that, when an obstacle is present in front of the vehicle, changes the display form of the virtual image depending on the distance between the vehicle and the obstacle, thereby preventing the virtual image from being displayed as if it is penetrating the obstacle.

In the conventional virtual image display device described in Patent Document 1, if an obstacle suddenly appears between the user's eyes and the position of the virtual image while gazing at the virtual image, the gaze distance can change significantly, causing the user to suffer from visually-induced motion sickness or being unable to intuitively understand the situation around them.

Below, an embodiment of the present disclosure will be described in detail with reference to the drawings. Note that the drawings used in the following description are schematic. The dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones. In this specification, for the sake of convenience, a Cartesian coordinate system xyz is defined in some of the drawings.

As shown in FIG. 1, a virtual image display device 1 according to an embodiment of the present disclosure is mounted on a moving body 10, and allows a user (driver or operator) 12 of the moving body 10 to visually recognize a virtual image V including various information related to the moving body 10. The virtual image display device 1 is also called a head-up display. A head-up display is also called a HUD (Head Up Display). When the moving body 10 is a vehicle, the various information included in the virtual image V may include, for example, the speed of the moving body 10, the engine rotation speed, the remaining fuel such as gasoline, the total travel distance of the moving body 10, the total travel distance of the moving body 10, and the water temperature of the engine cooling water system. Part of the configuration of the virtual image display device 1 may be shared with other devices and parts provided in the moving body 10.

"Mobile object" in this disclosure may include, for example, vehicles, ships, and aircraft. Vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, residential vehicles, and fixed-wing aircraft that run on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, and trolley buses. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. Vehicles may include those that run by human power. Vehicle classification is not limited to the above examples. For example, automobiles may include industrial vehicles that can run on roads. The same vehicle may be included in multiple classifications. Ships may include, for example, marine jets, boats, and tankers. The aircraft may include, for example, fixed-wing aircraft and rotary-wing aircraft. In the following, a case will be described in which the moving body 10 mounting the virtual image display device 1 is a vehicle, but the moving body 10 is not limited to a vehicle and may be any of the moving bodies described above.

As shown in Figures 1 and 2, the virtual image display device 1 includes a display unit 2, an optical element 5, an optical system 6, a detection unit 7, and a controller 8.

The detection unit 7 is also referred to as a camera or an in-car camera. The detection unit 7 is configured to capture an image of the eye 13 of the user 12 of the virtual image display device 1. The user 12 may be the driver of the moving body 10. The detection unit 7 may capture an image of the left eye 13L and the right eye 13R of the user 12. The detection unit 7 may or may not capture an image of the entire face of the user 12. In this specification, when there is no particular distinction between the left eye 13L and the right eye 13R of the user 12, the left eye 13L and the right eye 13R may be collectively referred to as the eye 13.

The detection unit 7 may be attached to the rearview mirror of the vehicle 10. The detection unit 7 may be attached to, for example, a cluster in the instrument panel. The detection unit 7 may be attached to a center panel. The detection unit 7 may be attached to a support portion of the steering wheel disposed at the center of the steering wheel. The detection unit 7 may be attached on the dashboard. The detection unit 7 does not have to be disposed in a position that directly captures an image of the eye 13 of the user 12. For example, the detection unit 7 may be disposed inside the dashboard and detect the eye 13 of the user 12 reflected on the windshield. The detection unit 7 may be disposed inside the dashboard and detect an image reflected on the windshield.

The detection unit 7 is configured to obtain an image of the subject and generate an image of the subject. The detection unit 7 includes an imaging element. The imaging element may include, for example, a CCD (Charged Coupled Device) imaging element or a CMOS (Complementary Metal Oxide Semiconductor) imaging element. The detection unit 7 is positioned so that the face of the user 12 is located on the subject side. The detection unit 7 may be configured to capture images of the first eye 13L and the second eye 13R using two or more imaging devices. The detection unit 7 may be configured to detect the positions of the first eye 13L and the second eye 13R as coordinates in three-dimensional space.

The detection unit 7 may not include a camera, but may be connected to an external camera. The detection unit 7 may include an input terminal that inputs a signal from the external camera. The external camera may be directly connected to the input terminal. The external camera may be indirectly connected to the input terminal via a shared network. The detection unit 7 that does not include a camera may include an input terminal configured to receive a video signal from the camera. The detection unit 7 that does not include a camera may be configured to detect the left eye and right eye from the video signal input to the input terminal.

The detection unit 7 is configured to output imaging data generated by imaging the eye 13 to the controller 8. The detection unit 7 may be configured to output the imaging data to the controller 8 via a communication network such as a wired or wireless communication network or a CAN (Controller Area Network).

The display unit 2 includes a display panel 3. The display panel 3 may be a transmissive display panel. When the display panel 3 is a transmissive display panel, the display unit 2 may include an illuminator 4.

The transmissive display panel may include a liquid crystal panel. The transmissive display panel may have a known liquid crystal panel configuration. As known liquid crystal panels, various liquid crystal panels such as IPS (In-Plane Switching) type, FFS (Fringe Field Switching) type, VA (Vertical Alignment) type, and ECB (Electrically Controlled Birefringence) type may be used. In addition to liquid crystal panels, the transmissive display panel includes MEMS (Micro Electro Mechanical Systems) shutter type display panels.

The illuminator 4 is disposed on the side of the display panel 3 opposite the display surface 3a, and illuminates the display panel 3 in a planar manner. The illuminator 4 may include a light source, a light guide plate, a diffusion plate, a diffusion sheet, etc. The illuminator 4 emits illumination light from the light source, and homogenizes the illumination light in the planar direction of the display panel 3 by the light guide plate, the diffusion plate, the diffusion sheet, etc. The illuminator 4 emits the homogenized light toward the display panel 3.

The following describes an example in which the display unit 2 includes a transmissive display panel 3 and an illuminator 4, but the display unit 2 is not limited to a transmissive display panel and may include a self-luminous display panel. If the display panel 3 is a self-luminous display panel, the display unit 2 does not need to include the illuminator 4. A self-luminous display panel may include a plurality of self-luminous elements. As the self-luminous elements, various self-luminous elements such as LEDs (Light Emitting Diodes), organic EL (Electro Luminescence), and inorganic EL may be used.

The display panel 3 has a plurality of partitioned regions on an active area 31 formed in a planar shape. The active area 31 is configured to display a parallax image. The parallax image includes a left eye image (also called a first image) and a right eye image (also called a second image) having parallax with respect to the left eye image. Each of the plurality of partitioned regions is an area partitioned in a first direction and a second direction perpendicular to the first direction. The direction perpendicular to the first direction and the second direction is called a third direction. The first direction may be called a horizontal direction. The second direction may be called a vertical direction. The third direction may be called a depth direction. The first direction, the second direction, and the third direction are not limited to these. In FIG. 3, the first direction is represented as the x-axis direction. The second direction is represented as the y-axis direction. The third direction is represented as the z-axis direction.

Each of the multiple partitioned regions corresponds to one subpixel. The active area 31 includes multiple subpixels arranged in a grid pattern along the first and second directions.

Each subpixel corresponds to one of the colors R (Red), G (Green), or B (Blue), and a set of three subpixels R, G, and B can form one pixel. One pixel can be called one picture element. The horizontal direction is, for example, the direction in which multiple subpixels that form one pixel are lined up. The vertical direction is, for example, the direction in which multiple subpixels of the same color are lined up.

The subpixels arranged in the active area 31 constitute a plurality of subpixel groups Pg under the control of the controller 8. The subpixel groups Pg are arranged repeatedly in the first direction. The subpixel groups Pg can be arranged at the same position in the second direction, or can be arranged shifted in the second direction. For example, the subpixel groups Pg can be arranged repeatedly adjacent to each other in the second direction at positions shifted by one subpixel in the first direction. The subpixel groups Pg include a plurality of subpixels in a predetermined row and column. The subpixel groups Pg include b subpixels (b rows) in the second direction and 2×n subpixels (2×n columns) in the first direction, (2×n×b) subpixels P1 to PN (N=2×n×b) arranged consecutively. In the example shown in FIG. 3, n=6, b=1. The active area 31 is provided with a plurality of subpixel groups Pg including 12 subpixels P1 to P12 arranged consecutively, 1 in the second direction and 12 in the first direction. In the example shown in FIG. 3, some subpixel groups Pg are labeled with symbols.

The multiple subpixel groups Pg are the smallest units that the controller 8 controls to display an image. Each subpixel included in the multiple subpixel groups Pg is identified by identification information P1 to PN (N = 2 x n x b). The multiple subpixels P1 to PN (N = 2 x n x b) having the same identification information in all subpixel groups Pg are configured to be controlled simultaneously by the controller 8. For example, when switching the image displayed in subpixel P1 from a left eye image to a right eye image, the controller 8 can simultaneously switch the image displayed in the multiple subpixels P1 in all subpixel groups Pg from a left eye image to a right eye image.

The optical element 5 may be composed of a parallax barrier or a lenticular lens. Below, an example will be described in which the optical element 5 is a parallax barrier. As shown in FIG. 5, the parallax barrier 5 is formed by a plane that follows the active area 31, and is separated from the active area 31 by a predetermined distance (gap) g. The parallax barrier 5 may be located on the opposite side of the illuminator 4 with respect to the display panel 3. The parallax barrier 5 may be located on the illuminator 4 side of the display panel 3.

As shown in FIG. 4, the parallax barrier 5 determines the propagation direction of image light emitted from the subpixels for each of the light-transmitting regions 51, which are strip-shaped regions extending in a predetermined direction in the plane. The parallax barrier 5 has a plurality of attenuation regions 52 that attenuate the image light. The attenuation regions 52 define a light-transmitting region 51 between two adjacent attenuation regions 52. The light-transmitting regions 51 have a higher light transmittance than the light-transmitting regions 52. The light transmittance of the light-transmitting regions 51 may be 10 times or more, 100 times or more, or 1000 times or more, that of the light-transmitting regions 52. The light-transmitting regions 52 have a lower light transmittance than the light-transmitting regions 51. The attenuation regions 52 may block the image light.

The parallax barrier 5 may be made of a film or a plate-like member. In this case, the multiple light-attenuating regions 52 are made of the film or plate-like member. The multiple light-transmitting regions 51 may be openings provided in the film or plate-like member. The film may be made of resin or other materials. The plate-like member may be made of resin, metal, or other materials. The parallax barrier 5 is not limited to a film or plate-like member and may be made of other types of members. The parallax barrier 5 may be made of a base material having light-blocking properties. The parallax barrier 5 may be made of a base material containing a light-blocking additive. The parallax barrier 5 may be of a fixed type or an active barrier type.

The parallax barrier 5 as an active barrier type can also be configured with a liquid crystal shutter. The liquid crystal shutter can control the light transmittance according to the applied voltage. The liquid crystal shutter is configured with multiple pixels, and can control the light transmittance of each pixel. The multiple light-transmitting regions 51 and multiple light-attenuating regions 52 of the liquid crystal shutter correspond to the multiple pixels of the liquid crystal shutter. When the parallax barrier 5 is configured with a liquid crystal shutter, the boundary between the multiple light-transmitting regions 51 and multiple light-attenuating regions 52 can be stepped to correspond to the shape of the multiple pixels.

The parallax barrier 5 determines the light direction of the image light emitted from multiple sub-pixels, thereby determining the area on the active area 31 that is visible to the user's eyes. The area in the active area 31 that emits the image light that propagates to the user's left eye is called the left visible area (first visible area) 31aL. The area in the active area 31 that emits the image light that propagates to the user's right eye is called the right visible area (second visible area) 31aR.

The barrier pitch Bp, which is the arrangement interval of the light-transmitting regions 51 in the first direction, and the gap g between the active area 31 and the parallax barrier are defined so that the following equations (1) and (2) hold, using the optimum viewing distance d and interocular distance E.
E:d=(n×Hp/b):g…(1)
d:Bp=(d+g):(2×n×Hp/b)…(2)

The optimal viewing distance d is the distance between the left and right eyes of the user and the parallax barrier 5. The direction of the line passing through the left and right eyes (interocular direction) is horizontal. The interocular distance E is the distance between the left and right eyes of the user. The interocular distance E may be, for example, 61.1 mm to 64.4 mm, a value calculated in research by the National Institute of Advanced Industrial Science and Technology. Hp is the horizontal length of a subpixel, as shown in Figure 3.

The optical system 6 may include a first optical member 6a and a second optical member 6b. The first optical member 6a is configured to reflect (a part of) the image light emitted from the active area 31 toward the second optical member 6b. The second optical member 6b is configured to make a part of the image light reflected by the first optical member 6a reach (enter) the user's eye. The optical system 6 may function as a magnifying optical system that magnifies the image displayed on the active area 31. At least one of the first optical member 6a and the second optical member 6b may have optical power. The first optical member 6a may be a concave mirror having optical power. The windshield of the moving body 10 may also serve as the second optical member 6b. The number of optical members constituting the optical system 6 is not limited to two, and may be one, or may be three or more. The first optical member 6a and the second optical member 6b may include an optically functional surface that is a spherical shape, an aspherical shape, or a free-form shape.

A portion of the image light emitted from the active area 31 of the display panel 3 passes through the multiple light-transmitting regions 51 and reaches the second optical member 6b via the first optical member 6a. A portion of the image light that reaches the second optical member 6b is reflected by the second optical member 6b and reaches the user's eye. This allows the user's eye to view a virtual image V of the image displayed in the active area 31 in front of the second optical member 6b. The surface on which the virtual image V is projected is called the virtual image surface S. In this specification, the forward direction is the direction of the second optical member 6b as seen by the user. The forward direction is the direction in which the moving body 10 normally moves.

The left visible area (first visible area) 31aL shown in FIG. 5 is an area of the virtual image surface S viewed by the left eye 13L of the user 12 as a result of image light that has passed through multiple light-transmitting areas of the parallax barrier 5 reaching the left eye 13L of the user 12, as described above. The right visible area (second visible area) 31aR is an area of the virtual image surface S viewed by the right eye 13R of the user 12 as a result of image light that has passed through multiple light-transmitting areas of the parallax barrier 5 reaching the right eye 13R of the user 12, as described above.

For example, when the aperture ratio of the parallax barrier 5 is 50%, the arrangement of multiple subpixels of the virtual image V as seen by the left eye 13L and right eye 13R of the user 12 is shown in FIG. 6. When the aperture ratio is 50%, the multiple light-transmitting regions and the multiple light-attenuating regions of the parallax barrier 5 have the same width in the interocular direction (x-axis direction). In FIG. 6, the dashed line indicates the virtual image of the boundary between the multiple light-transmitting regions and the multiple light-attenuating regions of the parallax barrier 5. The left visible region 31aL visible from the left eye 13L and the right visible region 31aR visible from the right eye 13R are regions extending diagonally in the x and y directions located between the two-dot dashed lines. The right visible region 31aR is not visible from the left eye 13L. The left visible region 31aL is not visible from the right eye 13R.

In the example of FIG. 7, the left visible region 31aL includes the entirety of the multiple subpixels P2 to P5 arranged in the active area and virtual images of most of the multiple subpixels P1 and P6. The left eye 13L of the user 12 has difficulty viewing the virtual images of the multiple subpixels P7 to P12 arranged in the active area. The right visible region 31aR includes the entirety of the multiple subpixels P8 to P11 arranged in the active area and virtual images of most of the multiple subpixels P7 and P12. The right eye 13R of the user 12 has difficulty viewing the virtual images of the multiple subpixels P1 to P6 arranged in the active area. The controller 8 can display a left eye image in the multiple subpixels P1 to P6. The controller 8 can display a right eye image in the multiple subpixels P7 to P12. By doing this, the left eye 13L of the user 12 mainly sees the virtual image of the left eye image VL in the left visible region 31aL, and the right eye 13R mainly sees the virtual image of the right eye image VR in the right visible region 31aR. Because the left eye image VL and the right eye image VR are parallax images having parallax with respect to each other, the user 12 sees the left eye image VL and the right eye image VR as a stereoscopic image.

As shown in the center of Figure 8, when the virtual image V is located at the appropriate viewing distance d, the virtual image V is located so as to intersect with the virtual image surface S. The left eye image VL and the right eye image VR are displayed at approximately the same position as the virtual image V viewed by the user 12. The convergence angle at which the left eye 13L and the right eye 13R view a point on the virtual image V located at the appropriate viewing distance d is represented by the convergence angle Θ0.

As shown on the right side of FIG. 8, when the virtual image V is located on the far side of the appropriate viewing distance d, the left eye image VL and the right eye image VR are displayed at positions that differ by the amount of parallax D1 on the virtual image surface S. The left eye image VL is an image of the virtual image V viewed from the left side at a smaller angle than when viewed from the appropriate viewing distance d. The right eye image VR is an image of the virtual image V viewed from the right side at a smaller angle than when viewed from the appropriate viewing distance d. The user 12 perceives the virtual image V as being present at the position where the direction in which the left eye 13L views the left eye image VL intersects with the direction in which the right eye 13R views the right eye image VR. The convergence angle at which the user 12 views a point on the virtual image V is represented by the convergence angle Θ1. The convergence angle Θ1 is smaller than the convergence angle Θ0 at which a point located at the appropriate viewing distance d is viewed.

As shown on the left side of FIG. 8, when the virtual image V is located on the near side of the appropriate viewing distance d, the left eye image VL and the right eye image VR are displayed at positions that differ by the amount of parallax D2 on the virtual image surface S. The left eye image VL is an image of the virtual image V viewed from the left side at a larger angle than when viewed from the appropriate viewing distance d. The right eye image VR is an image of the virtual image V viewed from the right side at a larger angle than when viewed from the appropriate viewing distance d. The user 12 perceives the virtual image V as being present at the position where the direction in which the left eye 13L views the left eye image VL intersects with the direction in which the right eye 13R views the right eye image VR. The convergence angle at which the user 12 views a point on the virtual image V is represented by the convergence angle Θ2. The convergence angle Θ2 is greater than the convergence angle Θ0 at which a point located at the appropriate viewing distance d is viewed.

As shown in FIG. 8, when the parallax amount D between the left eye image VL and the right eye image VR on the virtual image surface S is increased, the display position of the virtual image V moves away from the eye 13 (i.e., the display distance of the virtual image V increases). When the parallax amount D is decreased, the display position of the virtual image V moves closer to the eye 13 (i.e., the display distance of the virtual image V decreases). The parallax amount D can take positive or negative values. For example, when the display position (display position in the horizontal direction) of the left eye image VL on the virtual image surface S is DL and the display position (display position in the horizontal direction) of the right eye image VR is DR, the parallax amount D may be defined as D=DR-DL. The display position may be defined with the right direction in FIG. 8 as the positive direction. The origin R of the display position may be set arbitrarily, but may be, for example, within the virtual image surface S of the virtual image V located at the suitable viewing distance d.

The controller 8 is connected to each component of the virtual image display device 1 and controls each component. The controller 8 is configured as a processor, for example. The controller 8 may include one or more processors. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC: Application Specific Integrated Circuit). The processor may include a programmable logic device (PLD: Programmable Logic Device). The PLD may include an FPGA (Field-Programmable Gate Array). The controller 8 may be either a SoC (System-on-a-Chip) in which one or more processors work together, or a SiP (System In a Package). The controller 8 includes a memory unit 81, and may store various information, programs for operating each component of the virtual image display device 1, etc. in the memory unit 81. The memory unit 81 may be configured of, for example, a semiconductor memory, etc. The memory unit 81 may function as a work memory for the controller 8.

The controller 8 detects the position of the eye 13 of the user 12 and the position of the gaze point of the user 12 (hereinafter referred to as gaze point P) based on the imaging data output from the in-vehicle camera 7. The position of the eye 13 may be the midpoint of the line segment connecting the positions of the left eye 13L and the right eye 13R, or may be the position of the dominant eye of the user 12. The dominant eye of the user 12 may be set in advance by the user 12, or may be input to the controller 8 from outside. The gaze point P is the position of the intersection of the gaze points of the left eye 13L and the right eye 13R in a virtual plane including the gaze points of the left eye 13L and the right eye 13R. The gaze point is also called the visual axis. The gaze point may be a virtual straight line passing through the center of each lens that opens from the iris and faces outward, and the central fossa, which has the best visual acuity, in each of the left eye 13L and the right eye 13R.

The controller 8 controls the parallax image to be displayed on the display panel 3 based on the position of the eye 13 and the position of the gaze point P. This allows the gaze point P of the user 12 to be approximately aligned with the horizontal and vertical display positions of the virtual image V. As a result, the risk of the user 12 experiencing motion sickness or being unable to intuitively understand the surrounding situation can be reduced. The controller 8 may control the display form of the parallax image based on the position of the eye 13 and the position of the gaze point P. The display form of the parallax image may include the parallax amount, brightness, color, transparency, size, shape, etc. of the parallax image. In addition, the controller 8 may hide the virtual image V or display the virtual image V as a frame based on the position of the eye 13 and the position of the gaze point P. Displaying the virtual image V as a frame is also referred to as displaying the virtual image V as a frame only, and means that information represented by letters or numbers is hidden in the virtual image V consisting of letters, numbers, and figures.

The controller 8 may calculate a gaze distance (hereinafter also referred to as a first distance) L1 between the position of the eye 13 and the position of the gaze point P. The controller 8 may control the parallax image based on the first distance L1. In this case, it is possible to adjust the display position of the virtual image V according to the gaze distance L1. As a result, it is possible to further reduce the risk that the user 12 will suffer from visually-induced motion sickness or that the user 12 will be unable to intuitively understand the surrounding situation.

The controller 8 may calculate a display distance (hereinafter also referred to as the second distance) L2 between the position of the eye 13 and the display position of the virtual image V. The controller 8 may compare the first distance L1 with the second distance L2, and if the second distance L2 differs from the first distance L1, control the amount of parallax of the parallax image so that the second distance L2 matches the first distance L1, as shown in FIG. 9. In this case, it is possible to adjust the display distance L2 of the virtual image V according to the gaze distance L1. As a result, the risk that the user 12 will experience visually-induced motion sickness or will be unable to intuitively understand the surrounding situation can be further reduced.

When changing the parallax amount D of the parallax image from the first parallax amount to the second parallax amount, the controller 8 may change it from the first parallax amount to the second parallax amount all at once, or may change it stepwise from the first parallax amount to the second parallax amount. When changing the parallax amount D all at once, the time during which the gaze distance L1 and the display distance L2 do not match can be shortened. When changing the parallax amount D stepwise, the discomfort felt by the user 12 due to the change in the display distance L2 can be reduced.

The controller 8 may determine whether the eye 13 of the user 12 is included in the eye box 14 (see FIG. 1). The eye box 14 is the range of eye positions from which the user 12 observes a virtual image, and is an area in real space in which the eye 13 of the user 12 is assumed to be present, taking into account, for example, the physique, posture, and changes in posture of the user 12. The shape of the eye box 14 is arbitrary. The eye box 14 may be a planar area or a three-dimensional area. If the eye 13 of the user 12 is not included in the eye box 14, the controller 8 does not need to change the display form of the parallax image. In this case, the processing load on the controller 8 and the display unit 2 can be reduced.

As shown in FIG. 2, the virtual image display device 1 includes a sensor device 11. The sensor device 11 is configured to acquire at least one of information (situation) of the moving body 10 and environmental information (ambient environment) surrounding the moving body 10. The sensor device 11 may include an exterior camera 11a, a GPS (Global Positioning System) receiver, communication equipment, a laser radar, a millimeter wave radar, sensor equipment, a car navigation device, a drive recorder, etc.

The exterior camera 11a can capture images of the scenery around the moving body 10, particularly the environment ahead of the moving body 10, and obtain information ahead of the moving body 10 (e.g., information about other moving bodies, pedestrians, animals, roadside installations such as guardrails, buildings, etc., that are in front of the moving body 10). The exterior camera 11a may be configured to include, for example, a CCD imaging element or a CMOS imaging element. The exterior camera 11a may be located at the front end of the moving body 10, inside the passenger compartment of the moving body 10, or in the engine room. The exterior camera 11a may have a lens with a wide angle of view, for example, a wide-angle lens, a fisheye lens, etc.

The GPS receiver can obtain information regarding the position (latitude and longitude) and speed of the moving body 10. The communication device may include an external communication device capable of communicating with an external communication network. The communication device can obtain information regarding the position and speed of the moving body 10, the weather around the moving body 10, etc. from the external communication network via the external communication device. The laser radar and millimeter wave radar can obtain information ahead of the moving body 10. The sensor device may include a luminance meter, an ultrasonic sensor, etc. The sensor device may include an ECU (Electronic Control Unit) of the moving body 10, etc. The sensor device can obtain information ahead of the moving body 10, information regarding the luminance around the moving body 10, information regarding whether the headlights equipped on the moving body 10 are turned on, etc. The car navigation device can obtain information regarding the position and speed of the moving body 10. The drive recorder can obtain information ahead of the moving body 10 and information regarding the speed of the moving body 10.

As shown in FIG. 2, the controller 8 includes an information acquisition unit 8a, an information analysis unit 8b, a display determination unit 8c, and a display instruction unit 8d.

The information acquisition unit 8a acquires information on the moving body 10 and information on the surroundings of the moving body 10 from the sensor device 11, and outputs the information on the moving body 10 and information on the surroundings of the moving body 10 to the information analysis unit 8b. The information acquisition unit 8a may acquire information on the moving body 10 and information on the surroundings of the moving body 10 at predetermined time intervals (for example, 0.008 seconds to 1 second) and output the information to the information analysis unit 8b. The information on the moving body 10 may include information on the position and speed of the moving body 10, and information on whether the headlights are on or not. The information on the moving body 10 may include information on the user 12, such as the position of the eye 13 of the user 12, the gaze point P, the viewpoint, the line of sight, and the gaze distance L1. The information on the surroundings of the moving body 10 may include information ahead of the moving body 10, information on the brightness around the moving body 10, and information on the weather around the moving body 10.

The information analysis unit 8b analyzes information about the moving body 10 and information about the surroundings of the moving body 10, and judges the status of the moving body 10 and the status around the moving body 10. The information analysis unit 8b outputs the status of the moving body 10 and the status around the moving body 10 to the display determination unit 8c. The information analysis unit 8b may judge the status of the moving body 10 and the status around the moving body 10 at a predetermined time interval (e.g., 0.008 seconds to 1 second), and output it to the display determination unit 8c. The information analysis unit 8b may be configured to store the previously judged status of the moving body 10 and the status around the moving body 10. When the information analysis unit 8b newly judges the status of the moving body 10 and the status around the moving body 10, it may be configured to compare the newly judged status of the moving body 10 and the status around the moving body 10 with the previously judged status of the moving body 10 and the status around the moving body 10. The display determination unit 8c sets and stores the display form (parallax amount D, size, shape, brightness, etc. of the parallax image) to be displayed on the display unit 2 based on the status of the moving body 10 and the status around the moving body 10, and outputs it to the display instruction unit 8d. When the display determination unit 8c acquires the display form of the parallax image from the information analysis unit 8b, the display determination unit 8c may discard the display form of the parallax image acquired previously and stored in the display determination unit 8c. The display instruction unit 8d instructs the display unit 2 to display the parallax image in the display form set by the display determination unit 8c.

The situation around the moving body 10 may include the presence or absence of an obstacle O. The obstacle O may be an object that is present in front of the moving body 10 and whose display distance calculated from the parallax amount D is longer than the distance to the object. The obstacle O may include, for example, other moving bodies that are present around the moving body 10, particularly in front of the moving body 10, roadside installations such as guardrails, buildings, etc. If an obstacle O is present, the information analysis unit 8b may detect the relative position and relative speed of the obstacle O with respect to the moving body 10. The situation around the moving body 10 may include the background luminance in the field of view of the user 12, particularly the background luminance in the line of sight of the user 12.

The following describes how the controller 8 controls the parallax image (i.e., the virtual image V).

The controller 8 detects the position of the eye 13 of the user 12 and the position of the gaze point P at every predetermined time interval Δt, and calculates the gaze distance L1. The time interval Δt may be, for example, 0.008 seconds to 1 second. The controller 8 compares the gaze distance L1 at the current time with the gaze distance L1 at a time Δt before the current time, and calculates the amount of change in the gaze distance L1. The controller 8 compares the amount of change in the gaze distance L1 with a threshold (also called the first threshold) T1, and if the amount of change in the gaze distance L1 is equal to or greater than the first threshold T1, the controller 8 may change the parallax amount D of the parallax image so that the display distance L2 matches the gaze distance L1, as shown in FIG. 9. If the amount of change in the gaze distance L1 is less than the first threshold T1, the controller 8 does not need to change the parallax amount of the parallax image. This makes it possible to reduce the processing load on the controller 8 and the display unit 2 while suppressing visually induced motion sickness of the user 12. The first threshold T1 may be, for example, a distance in the range of 5 m or more and less than 10 m, 10 m or more and less than 15 m, 15 m or more and less than 20 m, or 20 m or more, or may be some other distance.

The controller 8 may change the parallax amount D of the parallax image all at once so that the display distance L2 matches the viewing distance L1, or may change the parallax amount D of the parallax image in stages. When the parallax amount D of the parallax image is changed all at once, the time during which the viewing distance L1 and the display distance L2 do not match can be shortened. When the parallax amount D of the parallax image is changed in stages, the discomfort felt by the user 12 due to the change in the display distance L2 can be reduced.

When the controller 8 changes the parallax amount D in stages, it may change the parallax amount D in stages so that the display distance L2 matches the gaze distance L1 over about 1 to 2 seconds, or it may change the parallax amount D in stages so that the display distance L2 matches the gaze distance L1 over about 3 seconds. The controller 8 may change the parallax amount D so that the display distance L2 matches the gaze distance L1 in a shorter time the faster the speed of the moving body 10 is. Furthermore, when the controller 8 changes the parallax amount D in stages, it may change the size, color, shape, etc. of the virtual image V viewed by the user 12 in stages.

The controller 8 may control the display unit 2 not to display the parallax image when the change in the gaze distance L1 is equal to or greater than the threshold (also referred to as the second threshold) T2 and the speed of the moving body 10 is equal to or greater than the threshold (also referred to as the third threshold) T3. In other words, the virtual image display device 1 may turn off the virtual image V when the change in the gaze distance L1 is equal to or greater than the second threshold T2 and the speed of the moving body 10 is equal to or greater than the third threshold T3. In this case, as shown in FIG. 9, when an obstacle O appears in front of the moving body 10, the attention of the user 12 can be directed to the obstacle O, thereby improving the safety of driving. The second threshold T2 may be, for example, 30 m, but is not limited thereto. The second threshold T2 may be a distance in the range of 5 m or more and less than 10 m, 10 m or more and less than 15 m, 15 m or more and less than 20 m, or 20 m or more, or may be another distance. The second threshold T2 may be the same as the first threshold T1 or may be different from the first threshold T1. The third threshold T3 may be a speed of 80 km/h or more, but is not limited to this. The second threshold T2 may be, for example, a speed of 60 km/h or more and less than 80 km/h, 50 km/h or more and less than 60 km/h, 40 km/h or more and less than 50 km/h, 30 km/h or more and less than 40 km/h, 10 km/h or more and less than 30 km/h, or less than 10 km/h.

The controller 8 may control the display unit 2 to display the virtual image V as a frame when the change in the gaze distance L1 is equal to or greater than the second threshold T2 and the speed of the moving body 10 is equal to or greater than the third threshold T3. In this case, too, for example, when an obstacle O appears ahead of the moving body 10, the attention of the user 12 can be directed to the obstacle O, thereby improving driving safety.

When the amount of change in the gaze distance L1 is equal to or greater than the second threshold T2 and the speed of the moving body 10 is equal to or greater than the third threshold T3, the controller 8 may control the display unit 2 to not display the virtual image V or to display it in a frame, but to reduce the brightness of the virtual image V or to increase the transparency of the virtual image V. In this case, it is possible to increase the safety of driving and to continue to provide the user 12 with various information related to the moving body 10.

When the change amount of the gaze distance L1 is equal to or greater than a threshold (also referred to as a fourth threshold) T4 and the decrease amount of the background luminance in the line of sight of the user 12 is equal to or greater than a threshold (also referred to as a fifth threshold) T5, the controller 8 may change the parallax amount so that the display distance L2 coincides with the gaze distance L1 and increase the luminance of the virtual image V. This can suppress visually induced motion sickness of the user 12. Also, as shown in FIG. 10, when the gaze point P of the user 12 moves from a bright place BP to a dark place DP, the visibility of the virtual image V can be increased, and as a result, the time during which the user 12 has difficulty viewing the virtual image V can be shortened. The controller 8 may determine the increase amount of the luminance of the virtual image V based on the speed of the moving object 10. The controller 8 may increase the increase amount of the luminance of the virtual image V as the speed of the moving object 10 increases. The fourth threshold T4 may be, for example, a distance in the range of 5 m or more and less than 10 m, 10 m or more and less than 15 m, 15 m or more and less than 20 m, or 20 m or more, or may be another distance. The fourth threshold T4 may be the same as the first threshold T1 or the second threshold T2, or may be different from the first threshold T1 and the second threshold T2. The fifth threshold T5 may be, for example, a luminance in the range of 1000 cd/ ^m2 or more and less than 2000 cd/ ^m2 , 2000 cd/ ^m2 or more and less than 3000 cd/ ^m2 , or 3000 cd/ ^m2 or more, or may be another luminance.

When the change amount of the gaze distance L1 is equal to or greater than a threshold (also referred to as a sixth threshold) T6 and the increase amount of the background luminance in the line of sight of the user 12 is equal to or greater than a threshold (also referred to as a seventh threshold) T7, the controller 8 may change the parallax amount so that the display distance L2 coincides with the gaze distance L1 and may reduce the luminance of the virtual image V. In this case, it is possible to suppress visually induced motion sickness of the user 12. In addition, when the gaze point P of the user 12 moves from a dark place DP to a bright place BP (see FIG. 10), it is possible to suppress the virtual image V from dazzling the user 12, and as a result, it is possible to shorten the time during which the user 12 has difficulty viewing the virtual image V. The controller 8 may determine the amount of reduction in the luminance of the virtual image V based on the speed of the moving object 10. The controller 8 may increase the amount of reduction in the luminance of the virtual image V as the speed of the moving object 10 increases. The sixth threshold T6 may be, for example, a distance in the range of 5 m or more and less than 10 m, 10 m or more and less than 15 m, 15 m or more and less than 20 m, or 20 m or more, or may be another distance. The sixth threshold T6 may be the same as the fourth threshold T4, or may be different from the fourth threshold T4. The seventh threshold T7 may be, for example, a luminance in the range of 1000 cd/ ^m2 or more and less than 2000 cd/ ^m2 , 2000 cd/ ^m2 or more and less than 3000 cd/ ^m2 , or 3000 cd/ ^m2 or more, or may be another luminance. The seventh threshold T7 may be the same as the fifth threshold T5, or may be different from the fifth threshold T5.

The memory unit 81 of the controller 8 may store a table showing the correspondence between the amount of change (amount of decrease and amount of increase) in background luminance in the speed of the moving body 10 and the line of sight direction of the user 12, and the amount of change (amount of increase and amount of decrease) in the luminance of the virtual image V. The controller 8 may change the luminance of the virtual image V based on the table stored in the memory unit 81.

The controller 8 may control the virtual image V based on the lighting state of the headlights of the moving body 10. The lighting state of the headlights includes whether the headlights are on or off and the light distribution state of the headlights (high beam or low beam). The controller 8 may control the brightness of the virtual image V based on the lighting state of the headlights. For example, when the headlights are on, the controller 8 may increase the brightness of the virtual image V, in which case the visibility of the virtual image V can be improved. The controller 8 may control the brightness of the virtual image V based on the lighting state of the headlights and the background brightness in the line of sight of the user 12. In this case, it is possible to further improve the visibility of the virtual image V.

In the above, a case has been described in which the controller 8 hides or frames the virtual image V, or changes the display position, size, color, brightness, and transparency of the virtual image V, based on the gaze distance L1, the speed of the moving body 10, and the background brightness in the line of sight of the user 12, but the control of the virtual image V by the controller 8 is not limited to this. For example, the controller 8 may turn the virtual image V, which is a stereoscopic image (three-dimensional image), into a planar image (two-dimensional image), based on information about the moving body 10 and information about the surroundings of the moving body 10.

The control of the display unit 2 by the controller 8 will be described below. Figures 11 to 13 are flowcharts showing the control of the display unit 2 by the controller 8. In the flowcharts, "step" is abbreviated as [S], and within the charts, "positive" in the judgment control is represented by [YES] and "negative" is represented by [NO].

The flowchart in FIG. 11 [starts] when the moving body 10 starts. In [S1], information on the moving body 10 and information on the surroundings of the moving body 10 are acquired from the sensor device 11. Next, in [S2], the information acquired from the sensor device 11 is analyzed to determine the status of the moving body 10 and the status of the surroundings of the moving body 10.

Next, in [S3], it is determined whether or not the position of the eye 13 is within the eye box 14. If the position of the eye 13 is within the eye box 14 [Yes], the process proceeds to [S4], and if the position of the eye 13 is not within the eye box 14 [No], the flow chart ends.

In [S4], it is determined whether or not there has been a change in the line of sight of the user 12. If there has been a change in the line of sight of the user 12 [Yes], proceed to [S5], and if there has been no change in the line of sight of the user 12 [No], end the flowchart.

In [S5], it is determined whether the gaze distance L1 of the user 12 has been determined. If the gaze distance L1 has been determined [Yes], proceed to [S6]. If the gaze distance L1 has not been determined [No], proceed to [S7], and after determining the gaze distance L1 in [S7], proceed to [S6].

In [S6], it is determined whether there has been a change in the gaze distance L1 of the user 12. If there has been a change in the gaze distance L1 [Yes], proceed to [S8], and if there has been no change in the gaze distance L1 [No], proceed to [S9].

In [S8], it is determined whether the change in gaze distance L1 is greater than or equal to a threshold value. The threshold value may be the smallest of the threshold values (e.g., the first threshold value T1, the second threshold value T2, and the fourth threshold value T4) that are set for the change in gaze distance L1. If the change in gaze distance L1 is greater than or equal to the threshold value [Yes], proceed to [S10], and if the change in gaze distance L1 is not greater than or equal to the threshold value [No], end the flowchart.

In [S10], the change in gaze distance L1 is compared with thresholds (e.g., first threshold T1, second threshold T2, and fourth threshold T4) set for the change in gaze distance L1, and a change pattern for the parallax image (i.e., virtual image V) is selected based on the comparison result, and the process proceeds to [S11] of the flowchart shown in FIG. 12. The change pattern may be either a change pattern according to information about the moving body 10 or a change pattern according to information ahead of the moving body 10.

If there is no change in the gaze distance L1 in [S6] [No], then in [S9] it is determined whether there is a change in the situation at the gaze point of the user 12, and a change pattern for the parallax image (i.e., virtual image V) is selected. The change in the situation at the gaze point may be, for example, a change in the background luminance in the line of sight of the user 12. If there is a change in the situation at the gaze point [Yes], proceed to [S11] in the flowchart shown in Figure 12, and if there is no change in the situation at the gaze point [No], end the flowchart.

Next, in [S11], it is determined whether the change pattern performs processing to change the parallax image according to the speed of the moving body 10. If the result is [Yes] to perform processing to change the parallax image according to the speed of the moving body 10, the process proceeds to [S12], and if the result is [No] to not perform processing to change the parallax image according to the speed of the moving body 10, the process proceeds to [S13].

In [S12], the speed of the moving body 10 is compared with a threshold value (e.g., third threshold value T3) set for the speed of the moving body 10, and a change pattern for the parallax image is selected based on the comparison result, and the process proceeds to [S14].

If the change pattern in [S11] does not change the parallax image in accordance with the speed of the moving body 10 [No], then in [S13] it is determined whether or not to change the parallax image in accordance with the situation ahead of the moving body 10. If the change process in accordance with the situation ahead of the moving body 10 is to be performed [Yes], proceed to [S13], and if the change process in accordance with the situation ahead of the moving body 10 is not to be performed [No], proceed to [S14].

In [S14], the parameters of the parallax image (virtual image V) to be changed and the amount of change of the parameters are determined based on the change patterns selected in [S10], [S12], and [S15], and the process proceeds to [S15] of the flowchart shown in FIG. 13. The parameters of the parallax image (virtual image V) may include the amount of parallax D, the time for changing the amount of parallax D, and parameters related to the brightness, color, transparency, size, and shape of the virtual image V. The time for changing the amount of parallax D is the time from the start to the end of changing the amount of parallax D. The parameters of the parallax image (virtual image V) may include a flag for hiding the virtual image V, and a flag for displaying the virtual image V in a frame.

In [S15], it is determined whether the parameters of the parallax image to be changed include the parallax amount D. If the parameters of the parallax image to be changed include the parallax amount D [Yes], proceed to [S16], and if the parameters of the parallax image to be changed do not include the parallax amount D [No], proceed to [S19].

In [S16], one or more objects (e.g., a speedometer, a tachometer, a direction indicator, etc.) included in the parallax image are selected for which the parallax amount D is to be changed. Then, in [S17], the amount of change in the parallax amount D is calculated based on the gaze distance L1 of the user 12. In [S18], the parallax amount D stored in the display determination unit 8c is rewritten.

In [S19], the display unit 2 is instructed to change the parallax image displayed on the display panel 3 based on the parallax image parameters and the amount of change of the parameters determined in [S14]. Next, if the parallax amount D is to be changed in stages, in [S20] a process (delay process) is performed to change the parallax amount D in stages rather than all at once. In [S21], it is determined whether or not all changes to the parallax image parameters have been completed. If all changes to the parallax image parameters have been completed [Yes], the flow chart is terminated, and if all changes to the parallax image parameters have not been completed [No], the process returns to [S15].

The above-described embodiment is not limited to implementation as the virtual image display device 1. For example, the above-described embodiment may be implemented as a virtual image display method using the virtual image display device 1, or may be implemented as a program for controlling the virtual image display device 1.

According to the embodiments of the present disclosure, it is possible to prevent the user from experiencing motion sickness caused by video. Furthermore, according to the embodiments of the present disclosure, it is possible to allow the user to intuitively understand the surrounding situation.

The configuration of the present disclosure is not limited to the embodiments described above, but can be modified or changed in many ways. For example, the functions contained in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined into one or divided.

The figures illustrating the configurations related to this disclosure are schematic. The dimensional ratios and other details in the drawings do not necessarily correspond to the actual ones.

In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure may have their numbers exchanged. For example, the first optical member may exchange identifiers "first" and "second" with the second optical member. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. Descriptions of identifiers such as "first" and "second" in this disclosure alone should not be used to interpret the order of the configurations or to justify the existence of identifiers with smaller numbers.

In this disclosure, the x-axis, y-axis, and z-axis are provided for convenience of explanation and may be interchanged. The configurations according to this disclosure have been described using an orthogonal coordinate system consisting of the x-axis, y-axis, and z-axis. The positional relationship of each configuration according to this disclosure is not limited to being orthogonal.

This disclosure can be implemented in the following configurations (1) to (13).

(1) A virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, comprising:
a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image;
an optical element configured to define a propagation direction of image light of the parallax image;
an optical system configured to propagate image light whose propagation direction is regulated by the optical element toward an eye of the user and display a virtual image of the parallax image in a field of view of the user;
a camera configured to image the eye of the user; and
A controller,
The controller:
Detecting the position of the user's eyes and the position of the user's gaze point based on the imaging data output from the camera;
The virtual image display device is configured to control the parallax image based on the position of the eye and the position of the gaze point.

(2) The virtual image display device according to the above configuration (1), wherein the controller is configured to control the parallax image based on a first distance, which is the distance between the eye position and the gaze point position.

(3) The virtual image display device according to the above configuration (2), wherein the controller is configured to control the amount of parallax of the parallax image when a second distance, which is the distance from the eye position to the display position of the virtual image, differs from the first distance, so that the second distance coincides with the first distance.

(4) The virtual image display device according to the above configuration (3), wherein the controller is configured to control the amount of parallax in stages.

(5) A sensor device configured to acquire a status of the moving object,
The virtual image display device according to any one of the above configurations (1) to (4), wherein the controller is configured to control the parallax image based on a state of the moving object acquired by the sensor device.

(6) The sensor device is configured to acquire a speed of the moving object,
The virtual image display device according to the above-mentioned configuration (5), wherein the controller is configured to control the parallax images based on a speed of the moving object.

(7) The sensor device is configured to acquire a lighting state of a headlight of the moving object,
The virtual image display device according to the above configuration (5) or (6), wherein the controller is configured to control the parallax images based on a lighting state of the headlights.

(8) A sensor device configured to acquire an ambient environment of the moving object,
The virtual image display device according to any one of the above configurations (1) to (4), wherein the controller is configured to control the parallax image based on the surrounding environment of the moving object acquired by the sensor device.

(9) The sensor device is configured to acquire a background luminance of the user's field of view,
The virtual image display device according to the above-mentioned configuration (8), wherein the controller is configured to control the parallax images based on a background luminance in the user's field of vision.

(10) A virtual image display device according to any one of the above configurations (1) to (9), wherein the controller is configured not to change the parallax image if the eye position is not included in an eye box.

(11) A virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, the virtual image display method being executed by the virtual image display device including: a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image; an optical element configured to define a propagation direction of image light of the parallax image; an optical system configured to propagate the image light, the propagation direction of which is defined by the optical element, toward the user's eye and display a virtual image of the parallax image in the user's field of vision; a camera configured to capture an image of the user's eye; and a controller,
Taking an image of the user's eye;
Detecting the position of the user's eye and the position of the user's gaze point based on imaging data of the user's eye;
A virtual image display method, comprising: controlling the parallax image based on the position of the eye and the position of the gaze point.

(12) A virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, the virtual image display device including: a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image; an optical element configured to determine a propagation direction of image light of the parallax image; an optical system configured to propagate the image light, the propagation direction of which is determined by the optical element, toward the user's eye and display a virtual image of the parallax image in the user's field of vision; a camera configured to capture an image of the user's eye; and a controller, the program being executed by the virtual image display device,
A program for causing the controller to cause the camera to capture an image of the user's eye, detecting the position of the user's eye and the position of the user's point of gaze based on the image data of the user's eye, and controlling the parallax image based on the position of the eye and the position of the point of gaze.

(13) A moving object including a virtual image display device according to any one of the above configurations (1) to (10).

1 Virtual image display device 2 Display unit 3 Display panel 3a Display surface 4 Illuminator 5 Optical element (parallax barrier)
6 Optical system 6a First optical member 6b Second optical member 7 Detection unit (in-vehicle camera)
8 controller 8a information acquisition section 8b information analysis section 8c display judgment section 8d display instruction section 10 moving object 11 sensor device 12 user 13 eye 13L left eye (first eye)
13R Right eye (second eye)
14 Eye box 31 Active area 31aL Left visible area 31aR Right visible area 51 Light-transmitting area 52 Light-reducing area O Obstacle P Point of gaze S Virtual image surface V Virtual image VL Left eye image VR Right eye image d Suitable viewing distance g Gap

Claims

A virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image,
a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image;
an optical element configured to define a propagation direction of image light of the parallax image;
an optical system configured to propagate image light whose propagation direction is regulated by the optical element toward an eye of the user and display a virtual image of the parallax image in a field of view of the user;
a camera configured to image the eye of the user; and
A controller,
The controller:
Detecting the position of the user's eyes and the position of the user's gaze point based on the imaging data output from the camera;
The virtual image display device is configured to control the parallax image based on the position of the eye and the position of the gaze point.
The virtual image display device according to claim 1, wherein the controller is configured to control the parallax image based on a first distance that is a distance between the eye position and the gaze point position.
The virtual image display device according to claim 2, wherein the controller is configured to control the amount of parallax of the parallax image when a second distance, which is a distance from the eye position to a display position of the virtual image, differs from the first distance so that the second distance coincides with the first distance.
The virtual image display device according to claim 3, wherein the controller is configured to control the amount of parallax in stages.
a sensor device configured to acquire a status of the moving object;
5. The virtual image display device according to claim 1, wherein the controller is configured to control the parallax images based on a state of the moving object acquired by the sensor device.
The sensor device is configured to acquire a speed of the moving object;
The virtual image display device according to claim 5 , wherein the controller is configured to control the parallax images based on a speed of the moving object.
The sensor device is configured to acquire a lighting state of a headlamp of the moving object,
The virtual image display device according to claim 5 , wherein the controller is configured to control the parallax images based on a lighting state of the headlights.
A sensor device configured to acquire an ambient environment of the moving object,
5. The virtual image display device according to claim 1, wherein the controller is configured to control the parallax image based on the surrounding environment of the moving object acquired by the sensor device.
The sensor device is configured to acquire a background luminance of the user's field of view;
The virtual image display device according to claim 8 , wherein the controller is configured to control the parallax images based on a background luminance in the user's field of view.
The virtual image display device according to any one of claims 1 to 9, wherein the controller is configured not to change the parallax image if the eye position is not included in an eye box.
A virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, the virtual image display device including: a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image; an optical element configured to define a propagation direction of image light of the parallax image; an optical system configured to propagate the image light, the propagation direction of which is defined by the optical element, toward an eye of the user and display a virtual image of the parallax image in a field of view of the user; a camera configured to capture an image of the eye of the user; and a controller, the virtual image display method being executed by the virtual image display device, the virtual image display device including:
taking an image of the user's eye;
Detecting the position of the user's eye and the position of the user's gaze point based on imaging data of the user's eye;
A virtual image display method, comprising: controlling the parallax image based on the position of the eye and the position of the gaze point.
A virtual image display device that allows a user of a moving body to visually recognize a stereoscopic image, the virtual image display device including: a display unit configured to display a parallax image including a first image and a second image having parallax with respect to the first image; an optical element configured to define a propagation direction of image light of the parallax image; an optical system configured to propagate the image light, the propagation direction of which is defined by the optical element, toward an eye of the user and display a virtual image of the parallax image in the field of view of the user; a camera configured to capture an image of the eye of the user; and a controller, the virtual image display device including: a program executed by the virtual image display device, the program including:
A program for causing the controller to cause the camera to capture an image of the user's eye, detecting the position of the user's eye and the position of the user's point of gaze based on the image data of the user's eye, and controlling the parallax image based on the position of the eye and the position of the point of gaze.
A moving object including a virtual image display device according to any one of claims 1 to 10.