JP7596039B2 - Display System - Google Patents

Description

The present invention relates to a display system mounted on a moving object such as an automobile.

Patent Document 1 discloses a display control system for generating a virtual environment inside a vehicle, which displays traffic and weather conditions on the vehicle windows while the vehicle is moving along the current travel route. Patent Document 2 discloses a display system for identifying the driving conditions of other vehicles around the vehicle and making the driver aware of them. For example, an image of the other vehicle is displayed so as to be superimposed on the view seen through the windshield from the driver's seat. Furthermore, Patent Document 3 discloses a communication vehicle display device that, when the source of the transmission is another vehicle visible from the vehicle window, displays the source vehicle in the driver's field of view through the vehicle window.

JP 2020-515886 A JP 2017-37634 A JP 2010-218568 A

With advances in autonomous driving technology, the interior of a vehicle is changing from a space for travel to a space for comfort. When we move to fully autonomous driving, it is predicted that passengers will spend time in the vehicle as if they were in their living room or workplace.

When fully automated driving becomes possible, cars are expected to run at legal speeds, which means that there will likely be more situations where cars will run side by side. For example, as shown in Figure 1, when there is another car 20 running side by side with one's own car 10, the interiors of each car will remain clearly visible, and passengers 14A and 14B will be able to see the other car 20 from the car window 12. Even if the passengers 22 of the other car 20 are not actually looking at them, they will still be bothered by their gazes. In particular, if the passenger is spending the day feeling as if they are in their living room at home, they will feel uncomfortable as if someone is peering into their house through the window 12.

On the other hand, it is possible to cover the windows 12 with curtains or film so that the inside cannot be seen from the outside, but this would make it difficult for passengers 14A and 14B to see outside, reducing the enjoyment of traveling. Furthermore, even in fully automated driving, it is important for the driver to be aware of the external situation.

The present invention aims to provide a display system that can solve these conventional problems and eliminate the discomfort caused by external gazes.

The display system according to the present invention is mounted on a vehicle such as a moving body, and has a detection means for detecting parallel running vehicles, a display means for displaying an image on a window of the vehicle, and a display control means for controlling the display means, and when a parallel running vehicle is detected by the detection means, the display control means displays an image on the window in a position that overlaps with the parallel running vehicle.

In one aspect, the display control means includes a calculation means for calculating the vehicle size of the parallel running vehicle, and displays an image corresponding to the vehicle size calculated by the calculation means. In one aspect, the display control means includes a detection means for detecting a relative position of the parallel running vehicle with respect to the vehicle, and a detection means for detecting an eye point of a passenger, and the display control means displays the image based on the relationship between the relative position and the eye point. In one aspect, the display control means includes an identification means for identifying a passenger, and a storage means for storing images corresponding to the preferences of multiple passengers, and the display control means acquires images corresponding to the preferences of the identified passenger from the storage means and displays the acquired images. In one aspect, the display means displays the images in an AR format on an external view seen through a window of the vehicle. In one aspect, the display system further includes an imaging means for capturing an image of the periphery of the vehicle, and the display means displays an image of the periphery of the vehicle captured by the imaging means on the window, and the display control means combines the image with the periphery image displayed on the window.

According to the present invention, when a parallel running vehicle is detected, an image is displayed in a position above the window that overlaps with the parallel running vehicle, providing an interior space where passengers do not feel gazed upon by the parallel running vehicle, and eliminating discomfort felt by passengers due to gazes from outside.

FIG. 1 is a diagram for explaining a problem to be solved by the invention. FIG. 1 is a diagram showing a configuration of a display system according to an embodiment of the present invention. 13 is an example of a management table showing a relationship between user identification and animation data. 1 is a block diagram showing a configuration of a line of sight removal function of a display system according to an embodiment of the present invention. 11 is a diagram showing an example of calculation of the distance and direction between the host vehicle and a vehicle traveling parallel to the host vehicle; FIG. 11 is a flow diagram illustrating the operation of a line of sight removal function of a display system according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of AR display of a display system according to an embodiment of the present invention. 10A to 10C are diagrams illustrating an example of AR display of a display system according to a modified example of the present invention;

Next, an embodiment of the present invention will be described. The display system of the present invention is mounted on a vehicle such as an automobile, and has a function of displaying images on the vehicle windows, etc. The display system of the present invention uses AR (Augmented Reality) technology to overlay images onto surrounding vehicles (vehicles traveling parallel to the vehicle), thereby providing an interior space in which the presence of people is erased and the driver does not feel the gaze of vehicles traveling parallel to the vehicle. AR technology generates a virtual field of view by overlaying virtual visual information on real scenery. In addition, by synthesizing images only of vehicles traveling parallel to the vehicle, a field of view that does not impede understanding of the situation outside the vehicle is ensured.

Next, an embodiment of the present invention will be described in detail. FIG. 2 is a block diagram showing the configuration of a display system according to an embodiment of the present invention. The display system 100 includes a plurality of image capturing cameras 110, a sensor 120, a vehicle information acquisition unit 130, a communication unit 140, a display unit 150, a storage unit 160, and a control unit 170. Although not shown here, the display system 100 can also include an input unit that receives input from a user, an audio output unit that outputs audio related to the image displayed by the display unit 150, and a media playback unit that plays media such as music and images.

The imaging camera 110 captures images of the surroundings of the vehicle and provides the captured image data to the control unit 170. This image data can be displayed by the display unit 150, or can be used to detect surrounding vehicles and calculate the size of surrounding vehicles. In addition to capturing images of the outside of the vehicle, the imaging camera 110 also includes an in-vehicle imaging camera that captures images of the interior of the vehicle. The image data captured by the in-vehicle imaging camera is used to detect the presence or absence of passengers in the vehicle and the passengers' viewpoints (eyepoints).

The sensor 120 includes, for example, an infrared sensor, a radar, etc., and detects objects around the vehicle and the distance to the objects by transmitting electromagnetic waves such as radio waves and receiving the reflected waves. The detection results are provided to the control unit 170. The sensor 120 can also include an illuminance sensor for detecting day or night. The detection results of the illuminance sensor are used to adjust the brightness of the display unit 150, etc.

The vehicle information acquisition unit 130 acquires information related to the operation of the vehicle. The vehicle information can include, for example, information identifying the level of autonomous driving, as well as other information such as vehicle speed information (vehicle speed pulse), gear position information, parking information, turn signal information, and vehicle position information, and this information is acquired via an in-vehicle bus or the like. Note that the vehicle position information can include, for example, position information using a GPS satellite, and position information using an acceleration sensor or an angular velocity sensor. The information acquired by the vehicle information acquisition unit 130 is provided to the control unit 170.

The communication unit 140 enables wired or wireless communication with electronic devices mounted on the vehicle, or wireless communication with a server or the like via an external network, or vehicle-to-vehicle communication with other vehicles. In one aspect, the communication unit 140 is capable of downloading images to be displayed by the display unit 150 from a server or the like.

The display unit 150 has a function of displaying an image on the vehicle window and displaying an image in the vehicle interior space other than the window. As shown in FIG. 1, the vehicle is provided with a glass window 12 attached to the side of the seat, as well as a front windshield and a rear windshield. In this embodiment, the display unit 150 displays an image at least on a window (window 12 in the example of FIG. 1) where the driver feels the line of sight of a parallel vehicle when the vehicle is traveling parallel. The configuration of the display unit 150 is not particularly limited, but in one embodiment, the display unit 150 includes a projector that projects an image, and projects an image on the glass window 12 to superimpose the projected image on the scenery seen through the window 12. In another embodiment, the display unit 150 is mounted on the window 12 with a display panel such as a liquid crystal display that corresponds to the shape of the window 12. For example, the liquid crystal display has a transparent state that allows the scenery outside the vehicle to be seen through when the image is not displayed, and in this state, the image is displayed in a specific area to superimpose the image on the scenery seen through the window.

The storage unit 160 stores data and application software necessary for the display system 100. The storage unit 160 also stores video data to be displayed by the display unit 150, and in one embodiment, stores animation data that matches the user's preferences. FIG. 3 is an example of a management table that stores the relationship between user identification and animation data. For example, when the display unit 150 displays an image that matches the passenger's preferences, the management table is referenced. The animation data stored in the storage unit 160 may be either a video or a still image, and the animation data that matches the user's preferences may be multiple, or may include multiple images with different image sizes. When images with different image sizes are stored, attribute information that identifies the size is also stored at the same time. Such animation data may be stored in the storage unit 160 by user input, or may be downloaded from a server or the like via the communication unit 140 and stored in the storage unit 160. The data in the management table is updated when the animation data is stored.

The control unit 170 controls the overall operation of the display system 100. This control is implemented by hardware and/or software. In one aspect, the control unit 170 includes a microcontroller including a ROM/RAM, and the like, and the ROM/RAM stores a program for controlling the operation of the display system 100. In this embodiment, the control unit 170 has, as part of its functions, a line of sight removal function for removing the line of sight of a parallel running vehicle when one is present. In one aspect, the control unit 170 may operate the line of sight removal function in accordance with the autonomous driving level from the vehicle information acquisition unit 130. This line of sight removal function is described below.

Figure 4 is a block diagram showing the configuration of the line of sight removal function 200 in this embodiment. The line of sight removal function 200 is configured to include a parallel vehicle detection unit 210, a vehicle size calculation unit 220, a distance/direction detection unit 230, a passenger identification unit 240, an animation data acquisition unit 250, an animation video generation unit 260, an eye point detection unit 270, a display coordinate calculation unit 280, and an animation video output unit 290.

The parallel vehicle detection unit 210 detects vehicles running parallel to the own vehicle. A parallel vehicle is another vehicle that is in such a position that the own vehicle feels a line of sight from the outside through the window, for example, another vehicle 20 running parallel to the own vehicle 10 as shown in FIG. 1. There are no particular limitations on the method of detecting parallel vehicles, but the parallel vehicle detection unit 210 detects other vehicles running parallel to the own vehicle by analyzing image data of the surroundings of the own vehicle captured by the imaging camera 110, or detects other vehicles running parallel to the own vehicle using the radar of the sensor 120, etc.

When a parallel running vehicle is detected by the parallel running vehicle detection unit 210, the vehicle size calculation unit 220 calculates the size of the parallel running vehicle. For example, the vehicle size calculation unit 220 extracts the vehicle outline by analyzing image data captured by the imaging camera 110, or extracts the vehicle outline from the detection results of the radar of the sensor 110, and calculates a rectangular size that approximates the extracted outline. Alternatively, the vehicle size calculation unit 220 calculates the maximum horizontal and vertical values of the extracted outline. The vehicle size calculated by the vehicle size calculation unit 220 is used to determine the image size when displaying an image on the window 12.

When the parallel vehicle detection unit 210 detects a parallel vehicle, the distance/direction detection unit 230 detects the distance and direction (angle) to the parallel vehicle in order to know the relative positional relationship of the parallel vehicle to the vehicle. The distance/direction detection unit 230 uses the analysis of image data from the imaging camera 110 and the detection results of the sensor 120. In one aspect, the distance/direction detection unit 230 calculates the shortest distance L between the vehicle and the parallel vehicle, and the direction from the reference point P1 of the vehicle to the reference point P2 of the parallel vehicle. The reference point P2 of the parallel vehicle can be calculated from the vehicle size calculated by the vehicle size calculation unit 220. For example, when the vehicle size is expressed by a rectangle, the intersection of the diagonal lines of the rectangle can be used as the reference point, or when the vehicle size is expressed by the maximum horizontal and maximum vertical values, the intersection of the horizontal and vertical lines can be used as the reference point P2. The reference point P1 of the vehicle is not particularly limited, but can be, for example, the mounting position of the imaging camera 110 or the sensor 120, or the point where the diagonals intersect when the outline of the side of the vehicle is approximated to a rectangle, or the center of the window 12.

In addition, since the vehicle and the parallel vehicle run on the same road surface, it may be assumed that the heights of both reference points are equal if the difference in height between the reference point P1 of the vehicle and the reference point P2 of the parallel vehicle is within a certain range. In that case, the distance and direction detected by the distance/direction detection unit 230 can be expressed in a horizontal XY plane including the reference point P1 of the vehicle 10 and the reference point P2 of the parallel vehicle 20, as shown in FIG. 5. FIG. 5(A) shows a case where the parallel vehicle 20 is present directly to the side of the vehicle 10. In this case, the shortest distance between the vehicle 10 and the parallel vehicle 20 is L, and the angle between the line connecting the reference point P1 and the reference point P2 and the shortest distance L is 0 degrees. FIG. 5(B) shows a case where the parallel vehicle 20 is present ahead of the vehicle 10. The shortest distance between the vehicle 10 and the parallel vehicle 20 is L, but the angle between the line connecting the reference point P1 and the reference point P2 and the line of the shortest distance L is α. FIG. 5(C) shows a case where a parallel vehicle 20 is present behind the vehicle 10. The shortest distance between the vehicle 10 and the parallel vehicle 20 is L, but the angle between the line connecting the reference points P1 and P2 and the line of the shortest distance L is β.

Of course, if the parallel vehicle is a vehicle that is higher than the host vehicle, such as a large vehicle or truck, the heights of reference points P1 and P2 will be different, and in that case, the distance/direction detection unit 230 may detect the relative distance and direction of the parallel vehicle to the host vehicle in three-dimensional space or three-dimensional coordinates.

The passenger identification unit 240 identifies passengers aboard the vehicle. The passenger identification unit 240 performs facial recognition based on image data from the imaging camera 110 that captures the interior space of the vehicle, and identifies the passengers. In this case, facial feature data of the passengers (users) is pre-stored in the memory unit 160, and the passengers are identified by comparing this feature data with the facial image. If there are multiple passengers, the passenger identification unit 240 identifies each passenger, and provides the identification results to the animation data acquisition unit 250.

The animation data acquisition unit 250 refers to the management table shown in FIG. 3 and acquires animation data corresponding to the preferences of the identified passenger. For example, if the passengers are identified as user A and user B, animation video A and animation video B are acquired. In addition, if multiple animation data of different sizes are stored, the animation data acquisition unit 250 may select animation data corresponding to the vehicle size calculated by the vehicle size calculation unit 220.

The animation image generation unit 260 processes the size and orientation of the animation image acquired by the animation data acquisition unit 250 to match the vehicle size of the parallel running vehicle and the distance and direction of the parallel running vehicle. For example, when the parallel running vehicles are in the positional relationship shown in FIG. 5(A), the size of the animation image is determined according to the vehicle size and distance L, and when the parallel running vehicles are in the positional relationship shown in FIG. 5(B) or FIG. 5(C), the size of the animation image is determined taking into account the angles α and β.

The eye point detection unit 270 calculates the position of the passenger's viewpoint based on image data from the imaging camera 110 that captures the interior space of the vehicle. The eye point calculation may be performed simultaneously when the passenger identification unit 240 recognizes the passenger's face.

The display coordinate calculation unit 280 calculates the display coordinates on the window 12 when the animation image generated by the animation image generation unit 260 is displayed on the window 12, based on the relative positional relationship between the eye point coordinates calculated by the eye point detection unit 270 and the distance and direction detected by the distance and direction detection unit 230. In other words, the display coordinates are calculated so that the animation image displayed on the window 12 overlaps with the vehicle traveling alongside.

The animation video output unit 290 outputs the animation video generated by the animation video generation unit 260 and the coordinate position calculated by the display coordinate calculation unit 280 to the display unit 150, causing the display unit 150 to display the animation video on the window 12.

Next, the operation of the display system 100 of this embodiment will be described with reference to the flow of FIG. 6. First, the parallel vehicle detection unit 210 detects the presence or absence of a parallel vehicle (S100), and if there is no parallel vehicle, the flow is terminated. If a parallel vehicle is detected, the vehicle size calculation unit 220 calculates the vehicle size of the parallel vehicle (S110), and the distance/direction calculation unit 230 calculates the distance and direction to the parallel vehicle. Next, the animation data acquisition unit 250 acquires animation data according to the vehicle size and the passenger's preferences from the storage unit 160 (S120). Next, the animation image generation unit 260 generates an animation image according to the distance and direction to the parallel vehicle (S130), the eye point calculation unit 270 calculates the passenger's eye point, and the animation image output unit 290 displays the animation image at a coordinate position on the window based on the positional relationship between the parallel vehicle and the eye point (S140).

Figure 7 shows an example of an AR display when the line of sight removal function of the display system 100 of this embodiment is implemented. When a parallel vehicle 20 is present to the side of the vehicle 10, an animation image 300 according to the preferences of the passenger 14A is displayed on the window 12 of the vehicle 10 so as to be superimposed on the field of view of the passenger 14A, and an animation image 310 according to the preferences of the passenger 14B is displayed so as to be superimposed on the field of view of the passenger 14B.

In this way, according to this embodiment, by displaying animation images in the passenger's field of vision through the window 12, it is possible to eliminate the discomfort caused by the gaze of vehicles traveling alongside and provide an interior space where passengers do not feel the gaze of others. In addition, by displaying animations that suit the passenger's preferences, it is possible to increase the entertainment value of the interior of the vehicle. Furthermore, by displaying animation images in AR, it is possible to grasp the surrounding situation, such as the level of traffic congestion outside the vehicle, and enjoy the scenery and landscapes outside the vehicle.

Next, a modified example of the display system of this embodiment will be described. The display system of this modified example projects image data captured by the imaging camera 110 onto the window 12, and combines this image data with an animation image. Figure 8 shows an example in which, when a parallel vehicle 20 is present directly beside the vehicle 10, image data 400 captured by the imaging camera 110 is displayed on the window 12, and an animation image 300 is combined and displayed in the center.

When a parallel vehicle is detected, the vehicle size is calculated, the distance and direction of the parallel vehicle is detected, and the passenger is identified, as in the above embodiment. The animation data acquisition unit 250 acquires animation data that matches the preferences of either passenger 14A or passenger 14B, the animation image generation unit 260 processes the animation image according to the vehicle size and the distance and direction of the parallel vehicle, the eye point calculation unit 270 calculates the eye point of the passenger, and the display coordinate calculation unit 280 calculates the display coordinates on the window 12 that correspond to the field of view of passengers 14A and 14B. In the example of FIG. 8, since the parallel vehicle 20 is located directly to the side of the vehicle 10, the display coordinates of the animation image 300 are determined to be approximately the center of the window 12. In this way, the animation image 300 is synthesized with the image data 400 that shows the scenery outside the vehicle that is displayed on the window 12.

Since the window 12 is a display screen that displays the image captured by the imaging camera 110, the passenger 22 of the parallel vehicle 20 cannot actually see inside the vehicle 10, but this can eliminate the discomfort of having a stranger constantly appear on the display screen of the window 12.

In the above embodiment, an example was shown in which the display unit 150 displayed an animated image, but the image is not limited to an animated image and may be other images.

The above describes in detail the preferred embodiment of the present invention, but the present invention is not limited to a specific embodiment, and various modifications and variations are possible within the scope of the gist of the invention described in the claims.

10: Vehicle 12: Window 14, 14A, 14B: Passenger 20: Parallel vehicle 22: Passenger 100: Display system 110: Imaging camera 120: Sensor 130: Vehicle information acquisition unit 140: Communication unit 150: Display unit 160: Memory unit 170: Control unit 200: Line-of-sight removal function

Claims (6)

A display system mounted on a vehicle including a moving body,
A detection means for detecting a parallel running vehicle;
A display means for displaying an image on a window of the vehicle;
A calculation means for calculating a vehicle size of a parallel running vehicle;
a position detection means for detecting a relative position of a parallel running vehicle with respect to the vehicle, the position detection means detecting a shortest distance to the parallel running vehicle and a direction from a reference point P1 of the vehicle to a reference point P2 of the parallel running vehicle calculated based on the vehicle size of the parallel running vehicle calculated by the calculation means;
A detection means for detecting an eye point of a passenger;
A display control means for controlling the display means,
When a parallel running vehicle is detected by the detection means, the display control means generates an image according to the distance and direction of the parallel running vehicle detected by the position detection means and according to the vehicle size, and displays the generated image based on the relationship between the position of the parallel running vehicle detected by the position detection means and the eye point position detected by the eye point detection means, at a position on the window overlapping with the parallel running vehicle. 2. The display system according to claim 1, wherein the calculation means calculates a maximum value in a horizontal direction and a maximum value in a vertical direction of the vehicle contour, and the position detection means detects an intersection point of the maximum value in the horizontal direction and the maximum value in the vertical direction as the reference point P2. 2. The display system according to claim 1, wherein the calculation means calculates a rectangular size that approximates an outline of the vehicle, and the position detection means detects an intersection of diagonals of the rectangular size as the reference point P2. The display control means includes an identification means for identifying a passenger and a storage means for storing images according to the preferences of a plurality of passengers,
4. The display system according to claim 1, wherein the display control means acquires an image corresponding to the identified passenger's preference from the storage means and displays the acquired image. The display system according to claim 1, wherein the display means displays the image in an AR manner on the external scenery visible through the vehicle window. The display system further includes an imaging means for imaging an area around the vehicle;
The display means displays an image of the surroundings of the vehicle captured by the imaging means on the window,
6. The display system according to claim 1, wherein the display control means combines the image with a surrounding image displayed on the window.

Images