JP7638259B2 - Display device, visibility assistance system, industrial vehicle, control method and program - Google Patents

Description

The present disclosure relates to a display device, a visibility assistance system, an industrial vehicle, a control method, and a program.

Patent Document 1 discloses a loading and unloading support device that installs a camera on the tip of the forks of a forklift and displays the image captured by the camera on a monitor installed in the driver's seat. With this support device, loading and unloading work can be performed while checking the image in front of the raised forks. In such a support device, if it is considered that the worker works while looking at the work target position and the monitor at the same time, it is recommended to install the monitor above the driver's seat, but in that case, the worker needs to look upward even when only looking at the monitor, which is inconvenient when it is necessary to visually check other directions. However, if a monitor is installed in front of the driver's seat, part of the field of view is blocked by the monitor when driving. In addition, the focal length of the work target position as seen from the driver's seat when the forks are raised and the image displayed on the monitor are different, so if you try to check both, you will have to switch the focus.

Patent No. 4666154

There is a demand for technology that allows workers to work while checking images of the target location, such as the area in front of the raised forks.

This disclosure provides a display device, a visibility assistance system, an industrial vehicle, a control method, and a program that can solve the above problems.

The display device of the present disclosure is a display device worn on a person's head, and includes a transparent display unit capable of displaying image data, an acquisition unit that acquires images captured by a camera attached to a work device equipped on an industrial vehicle, and a control unit that has the function of being able to display the image data, including a virtual monitor on which the image is displayed, at any position in virtual space so that the image data appears to be at a position in the real world corresponding to that position, and which places the image data at a position in the virtual space that corresponds to the vicinity of a position that is a focal length away in the direction of the person's line of sight, as seen through the display unit.

The visibility assistance system of the present disclosure comprises a camera mounted on a work device equipped on an industrial vehicle and the above-mentioned display device.

The industrial vehicle of the present disclosure includes a camera provided on a work device provided on the industrial vehicle and the display device described above.

The control method disclosed herein is a control method for a display device worn on a person's head, having a transparent display unit capable of displaying image data, and has a function of acquiring an image captured by a camera attached to a work device equipped on an industrial vehicle, and displaying the image data, including a virtual monitor on which the image is displayed, at any position in a virtual space so that the image data appears to be present at a position in the real world corresponding to that position, and the image data is placed at a position in the virtual space corresponding to the periphery of a position that is a focal length away in the direction of the person's line of sight, as seen through the display unit .

In addition, the program disclosed herein causes a computer of a display device worn on a person's head, having a transparent display unit capable of displaying image data, to function as a means for acquiring images captured by a camera attached to a work device equipped on an industrial vehicle, a means for placing the image data, including a virtual monitor on which the image is displayed, at any position in virtual space and displaying the image data on the display unit as if it were located at a position in the real world corresponding to that position, and a means for placing the image data at a position in the virtual space corresponding to the periphery of a position that is a focal length away in the direction of the person's line of sight, which is visible through the display unit .

The above-mentioned display device, visibility assistance system, industrial vehicle, control method, and program allow a user to check an image of a target work position while viewing the target work position using a work device provided on the industrial vehicle.

FIG. 1 is a schematic diagram illustrating an example of a forklift according to an embodiment. 1 is a block diagram showing an example of a display device according to an embodiment; 1A and 1B are diagrams illustrating a focal length and a size of a virtual object according to an embodiment. FIG. 2 is a diagram showing a forklift during loading and unloading operations according to the embodiment; FIG. 2 is a diagram showing a display example of an image captured by a camera according to an embodiment. 10 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device according to an embodiment.

<Embodiment>
Hereinafter, the display device of the present disclosure will be described with reference to the drawings.
(composition)
FIG. 1 is a schematic diagram illustrating an example of a forklift according to each embodiment.
The forklift 1 includes a vehicle body 10, a loading device 11, a control device 18, and a display device 20. The vehicle body 10 includes a drive device and a steering device (not shown) and drives the forklift 1. The loading device 11 includes a fork 12, a mast 13 for raising and lowering the fork 12, and a backrest 14 for preventing the load from falling backward (toward the mast 13), and performs a loading operation by raising and lowering the load loaded on the fork 12. The loading device 11 is an example of a working device. The mast 13 includes an outer mast (not shown) and an inner mast (not shown), and is configured such that the inner mast is raised and lowered relative to the stationary outer mast. The mast 13 is provided with a lifting height sensor 15, which detects the height position of the inner mast. In addition, the outer mast is provided with a proximity sensor 16. The proximity sensor 16 detects when the movable parts such as the backrest 14, the forks 12, the inner mast, etc. leave the proximity sensor 16 provided on the outer mast. A camera 17 is provided at the tip of the forks 12. The camera 17 takes an image of the area in front of the forks 12. The image (which may be a video or a still image) taken by the camera 17 is transmitted to the display device 20. The control device 18 controls the operation of the forklift 1. For example, when the driver 100 operates the lift lever 19, the control device 18 detects the operation and performs an operation to raise and lower the mast 13. The control device 18 is connected to the lift height sensor 15 and the proximity sensor 16, and obtains information detected by these sensors. The control device 18 transmits to the display device 20 information detected by the lift height sensor 15 and the proximity sensor 16, a signal indicating that the driver 100 has operated the lift lever 19, and other information such as the moving direction and vehicle speed of the forklift 1. The display device 20 is an augmented reality (AR) glass or a mixed reality (MR) glass. The driver 100 wears the display device 20 and drives the forklift 1. In this specification, the direction in which the loading device 11 of the forklift 1 is provided as shown in Fig. 1 is referred to as the front, the opposite direction is referred to as the rear, and the left and right directions when facing forward are referred to as the left side and the right side, respectively.

Figure 2 is a block diagram showing an example of a display device according to an embodiment. The display device 20 is AR glasses or MR glasses (hereinafter, both will be referred to as AR glasses) that include a display device and a computer. The display device 20 is communicatively connected to the control device 18 and the camera 17, and acquires information from these devices. The display device 20 displays an image captured by the camera 17. As shown in the figure, the display device 20 includes a sensor unit 21, a display unit 22, a transmission/reception unit 23, a control unit 24, a display determination unit 25, and a memory unit 26.

The sensor unit 21 includes a sensor for head tracking of the driver 100 (e.g., multiple visible light cameras and an IMU (Inertial Measurement Unit)), a sensor for eye tracking of the driver 100 (e.g., an infrared camera), a depth sensor, a microphone, etc.

The display unit 22 constitutes the lens (screen) of the AR glasses, and is composed of, for example, a transparent holographic display (the outside world can be seen through the lens). When no virtual object is projected onto the lens, the driver 100 can see the real world as it is. A virtual object is image data projected onto the display unit 22. When a virtual object is displayed on the display unit 22, the driver 100 sees the virtual object superimposed on the real world.

The transceiver 23 transmits and receives data to and from other devices. For example, the transceiver 23 transmits and receives data to and from other devices via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). For example, the transceiver 23 receives images captured by the camera 17. For example, the transceiver 23 receives from the control device 18 information detected by the lift height sensor 15 and the proximity sensor 16, a signal indicating that the driver 100 has operated the lift lever 19, the direction of movement and the vehicle speed of the forklift 1, etc.

The control unit 24 performs display control of the display unit 22, etc. The control unit 24 recognizes the spatial shape around the driver 100 based on the information detected by the sensor unit 21, creates three-dimensional map information of the surrounding space, and generates a virtual space. The control unit 24 places a virtual object at a desired position in the virtual space based on the created map information. The virtual object is displayed to the driver 100 superimposed on an object existing at that position in the real world. The virtual object can be displayed at (1) a predetermined position in the space (the virtual object exists at a fixed position), (2) a predetermined position based on an object existing in the surrounding space (for example, if the object is the vehicle body 10, the virtual object moves following the running of the vehicle body 10), or (3) a predetermined position based on the driver 100 (for example, the virtual object exists at a position a predetermined distance away in the line of sight of the driver 100). In the case of (1) or (2), the virtual object is recognized by the driver 100 as if it actually exists at that position. In the case of (3), the virtual object can be placed at any position based on the driver 100, and the placement position can be freely changed by moving it closer to or farther away from the driver 100. For example, if the virtual object is placed far away, the driver 100 sees the virtual object as if it exists far away, and if the virtual object is placed close to the driver 100, the driver 100 sees the virtual object as if it exists close to the driver. When the driver 100 is looking 3 m ahead, if a virtual object is placed 3 m ahead in the line of sight, the virtual object appears to exist at that position. For the driver 100 looking 3 m ahead, the virtual object appears to exist at the position where the driver's line of sight is exactly aligned (the position where the driver's eyes are focused/in focus), so there is no need to refocus to view the virtual object. In contrast, when the driver 100 is looking 3 m ahead, if a virtual object is placed 30 cm ahead in the line of sight, the driver 100 looking 3 meters ahead sees the virtual object as if it exists 30 cm away from the driver, so the driver 100 cannot clearly see the virtual object unless he switches his focus from 3 m ahead to 30 cm ahead. In other words, in order to allow the driver 100 to view the virtual object as is without imposing a burden on the driver 100 to switch focus, it is preferable to place the virtual object as close as possible to the focal position of the driver 100. Because the characteristics of AR glasses allow the placement position of the virtual object to be freely controlled, the burden on the driver 100 to focus can be reduced by adjusting the position of the virtual object to match the focal length at that time by following the line of sight of the driver 100.

The AR glasses have the function of freely arranging virtual objects and arranging a virtual object at a certain position in the virtual space so that it appears as if the virtual object exists at that position. The user sets the desired virtual object for the display device 20 (a developer uses an AR development tool to develop a virtual object and implements the developed program on the display device 20). The control unit 24 then places the virtual object at the desired position according to the setting. In this embodiment, the control unit 24 is set to follow the line of sight of the driver 100, place a virtual monitor VM (a virtual monitor is a virtual object displayed by AR) near the focal length at that time (corresponding to (3) above), and display an image captured by the camera 17 on the virtual monitor VM. Next, the relationship between the position at which the virtual object is placed and the size of the virtual object will be described with reference to FIG. 3.

3 shows a virtual object 31a placed a distance L1 ahead of the driver 100, a display range (range of the field of view that fits in the display unit 22) 31b placed a distance L1 ahead of the driver 100, a virtual object 32a placed a distance L2 ahead of the driver 100, and a display range 32b placed a distance L2 ahead. If a real monitor that is not a virtual object is placed on the back side of the mast 13 (for example, a distance L2 ahead of the driver 100), the monitor will be hidden by the mast 13 and will not be visible, but in the case of an AR virtual monitor VM, even if it is placed on the back side of the mast 13, the virtual monitor VM will not be hidden by the mast because it is displayed on the AR glasses. AR technology also has an occlusion function that displays the virtual monitor VM as if it is hidden by the mast 13, but in this embodiment, the occlusion function is not used to prioritize the information displayed on the virtual monitor VM. As shown in FIG. 3, when two virtual objects 31a and 32a of different sizes (virtual object 32a is larger in the virtual space) are placed at positions of distances L1 and L2, respectively, the driver 100 can see both of them as the same size due to the nature that the farther an object is, the smaller it appears. In other words, when a virtual object is placed farther away, the size of the virtual object is made larger, and when it is placed closer, the size of the virtual object is made smaller. The specific size of the virtual object can be calculated based on the placement position (distance from the driver 100) and the size at which it is displayed on the display unit 22, and by changing the size of the virtual object according to the distance from the driver 100, it is possible to make the virtual object always appear at a constant size to the driver 100. Assuming that the virtual object is not hidden by the mast 13 and is not visible, the difference between the two virtual objects 31a and 32a displayed at the same size to the driver 100 is only the placement position, in other words, the focal length. By controlling the placement position and size of the virtual object placed in the virtual space in this way, it is possible to freely change only the focal length without changing the display size of the virtual object as seen by the driver 100.

By utilizing this property, if a virtual monitor VM with an appropriate size control is placed near the tip of the fork 12, the focus when looking at the tip of the real fork 12 and when looking at the virtual monitor VM will be almost the same, and the driver 100 will be able to see both at the same time. This is shown in Figure 4. Figure 4 shows a scene in which the fork 12 is being inserted into a hole 43 in a pallet 42 loaded with luggage 41 placed on a shelf in a warehouse or the like. The camera 17 captures an image of the front at the same height as the fork 12, and the driver 100 grasps the position of the hole 43 from the image captured by the camera 17 and positions the fork 12. In this case, if a real monitor that displays the image captured by the camera 17 is installed in the driver's seat as in the conventional technology, the driver will have to look alternately at the direction of the fork 12 and the monitor, which is inefficient. In addition, a burden of switching the focus will be incurred. Even if the real monitor is installed in a position and direction such that the display screen can be seen when looking up in the direction of the forks 12, the focal length when viewing the luggage 41 and pallet 42 near the tips of the forks 12 is different from the focal length of the image displayed on the monitor, so even if both the real object and the monitor are included in the field of view, the driver 100 cannot check both at the same time, and actually checks one by one while switching the focus. In contrast, in this embodiment, the virtual monitor VM is placed near the tips of the forks 12 (near the pallet 42) using AR technology, for example, as shown in the figure. Then, the image captured by the camera 17 is displayed on the virtual monitor VM. Then, the driver 100 sees the image captured by the camera 17 displayed on the monitor located near the pallet 42. The focal length of the image displayed on the virtual monitor VM is approximately the same as the focal length when viewing the real object such as the forks 12. In addition, by placing the virtual monitor VM near the pallet 42, the image captured by the camera 17 can be displayed within the field of view when the driver 100 views the real object near the tips of the forks 12. Then, the driver 100 can check the image displayed on the virtual monitor VM without switching the focal length while looking at the actual object such as the fork 12. This allows the driver 100 to perform loading and unloading work while simultaneously checking both the work target position near the tip of the fork 12 and the image captured by the camera 17 without the burden of focusing. For example, when the fork 12 is raised to a higher position to perform loading and unloading work, the control unit 24 places the virtual monitor VM at a higher position. In this case, since the distance from the driver 100 becomes farther, as described with reference to FIG. 3, the control unit 24 increases the size of the virtual monitor VM. Even if the size of the virtual monitor VM is increased, the virtual monitor VM (the image captured by the camera 17) appears to the driver 100 to be the same size due to the principle of perspective. In this way, by changing the placement position and size of the virtual monitor VM according to the work target position viewed by the driver 100, the driver 100 can perform work while checking the work target position in the real world and the image captured by the camera 17 without switching the focal length. Regarding the target position of the driver's 100 line of sight, for example, when the forks 12 are being raised or lowered, the vicinity of the tips of the forks 12 may be set as the target of the driver's 100 line of sight (for example, the focal length is calculated using the height detected by the lifting height sensor 15 and the horizontal length from the driver's seat to the tips of the forks 12). Alternatively, the target position of the driver's 100 line of sight may be estimated based on the value detected by an eye tracking sensor provided in the sensor unit 21. Alternatively, the focal length may be fixed and set at about several meters. By setting the focal length at about several meters, the virtual monitor VM can be effectively used not only in situations where the forks 12 are raised to perform loading and unloading work.

Figure 5 shows an example of a virtual monitor VM displayed on the display unit 22. In the example of Figure 5, an image captured by the camera 17 with the forks 12 lowered is displayed on the virtual monitor VM. For example, when driving the forklift 1 and moving forward, the image captured by the camera 17 is displayed on the virtual monitor VM, allowing the driver 100 to visually check the area around the vehicle and at the same time check the condition around the feet of the forklift 1, which is prone to blind spots. In this case, too, by positioning the virtual monitor VM according to the focal length when visually checking the area around the vehicle, the driver 100 can easily check the real world around the vehicle and the virtual monitor VM.

The position and range of the virtual monitor VM displayed on the display unit 22 can be set arbitrarily so as not to obstruct the field of vision. In addition, the virtual monitor VM can be displayed or hidden automatically or by the actions or operations of the driver 100. For example, in situations where the virtual monitor VM is obstructive, the virtual monitor VM can be hidden.

The display determination unit 25 determines whether to display or hide the virtual monitor VM. When the display determination unit 25 determines that the virtual monitor VM should be displayed, the control unit 24 displays the virtual monitor VM (places the virtual monitor VM in the virtual space), and when the display determination unit 25 determines that the virtual monitor VM should be hidden, the control unit 24 hides the virtual monitor VM (does not place the virtual monitor VM in the virtual space). For example, the display determination unit 25 determines that the virtual monitor VM should be displayed when loading and unloading work is in progress, and determines that the virtual monitor VM should be hidden when this is not the case. An example of the determination process of the display determination unit 25 is described below.

(A) The display determination unit 25 acquires information such as a signal indicating that the lift lever 19 in the driver's seat has been operated and the vehicle speed of the forklift 1 from the control device 18 through the transmission/reception unit 23. When the operation of the lift lever 19 is detected, the display determination unit 25 determines that the virtual monitor VM is to be displayed. In addition, when the operation of the lift lever 19 is detected and then no longer detected, the display determination unit 25 determines that the virtual monitor VM is to be displayed for a certain period of time (e.g., about 10 seconds) from that time. Alternatively, when the operation of the lift lever 19 continues for a predetermined period of time (e.g., about 1 to 2 seconds) or more, the display determination unit 25 determines that the virtual monitor VM is to be displayed for a certain period of time (e.g., about 10 seconds) from that time. During that time, if there is even a slight lift or tilt operation by the lift lever 19, the 10-second count is reset each time. In other words, the display determination unit 25 determines that the virtual monitor VM is to be displayed for a certain period of time (e.g., about 10 seconds) from the detection of the lift or tilt operation. Furthermore, when the vehicle speed reaches or exceeds a certain level (for example, 4 km/h or more), the display determination unit 25 determines that the loading and unloading work has ended, forcibly ends the 10-second count, and determines that the virtual monitor VM should not be displayed.

(B) The display determination unit 25 acquires information detected by the height sensor 15 (height of the forks 12 and inner mast) from the control device 18 via the transmission/reception unit 23. When the height detected by the height sensor 15 is equal to or greater than a predetermined threshold, the display determination unit 25 determines to display the virtual monitor VM, and when the height is less than the threshold, determines to hide the virtual monitor VM.

(C) The display determination unit 25 obtains information detected by the proximity sensor 16 (whether the outer mast on which the proximity sensor 16 is provided is in close proximity to the backrest 14, the forks 12, the inner mast, etc.) from the control device 18 via the transmission/reception unit 23. If the proximity sensor 16 detects that the outer mast and the inner mast, etc. are not in close proximity, the display determination unit 25 determines to display the virtual monitor VM, and if it detects that the outer mast and the inner mast, etc. are in close proximity, it determines to hide the virtual monitor VM.

(D) The display determination unit 25 uses the visible light camera provided in the sensor unit 21 to determine whether the inner mast is raised, using landmarks such as the position of the inner mast's connecting material and whether the lift cylinder rod (not shown) is visible (the lift cylinder rod is hidden from the driver's seat when the mast 13 is not raised, but becomes visible from the driver's seat when the mast 13 is raised). The display determination unit 25 calculates the height position of the inner mast based on how the landmarks are visible. The display determination unit 25 determines to display the virtual monitor VM when the inner mast is raised by a predetermined threshold or more, and determines to hide the virtual monitor VM when the distance the inner mast has been raised is less than the threshold.

(E) The display determination unit 25 may determine that loading and unloading work is to be performed when a predetermined time has elapsed since the loading of cargo onto the forks 12, and may determine to display the virtual monitor VM, and may determine to hide the virtual monitor VM when the cargo is unloaded from the forks 12. The loading of cargo onto the forks 12 may be detected by a load sensor (not shown) provided on the mast 13 or the like, or the presence or absence of cargo may be detected by analyzing an image captured by a visible light camera provided in the sensor unit 21 above the forks 12.

(F) The display determination unit 25 may determine to display the virtual monitor VM when the driver's 100 gaze direction is a predetermined angle or more and continues to face upward for a predetermined period of time or more, and may determine to hide the virtual monitor VM when the driver's 100 gaze direction is a predetermined angle or less. The gaze direction of the driver 100 can be estimated from information detected by an eye tracking sensor or a head tracking IMU provided in the sensor unit 21.

The memory unit 26 stores information detected by the sensor unit 21, information received by the transceiver unit 23, virtual objects, CAD information of the forklift 1 or various types of forklifts, etc.

(Operation)
Next, control of the display device 20 will be described with reference to FIG.
FIG. 6 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment.
When the driver 100 wearing the display device 20 sits in the driver's seat of the forklift 1, the driver starts up the display device 20 and performs calibration (step S1). In the calibration, the positional relationship between the driver 100 and the forklift 1 is calculated. For example, the control unit 24 recognizes the handle and other structures of the forklift 1 using the visible light camera of the sensor unit 21, and estimates the type of vehicle of the forklift 1 from the shape and appearance of the photographed handle, etc. The control unit 24 reads out CAD information corresponding to the estimated type of vehicle from the CAD information recorded in the storage unit 26, and grasps the overall shape of the forklift 1. In addition, for example, the control unit 24 analyzes an image of the handle captured by the visible light camera of the sensor unit 21, or measures the distance to the handle by the depth sensor of the sensor unit 21, thereby calculating the positional relationship between the driver 100 and the handle, that is, where the driver 100 is sitting in the driver's seat, how high the driver 100's head is, or the positional relationship between the driver 100 and each part of the forklift 1 (horizontal distance, vertical distance, etc.) from the read CAD information of the forklift 1. This makes it possible to grasp the exact distance between the driver 100 and the tip of the fork 12, and to place the virtual monitor VM near the tip of the raised fork 12, for example. In addition, for example, a two-dimensional marker may be provided at a predetermined position of the forklift 1, the two-dimensional marker may be recognized by the sensor unit 21, and the positional relationship between the driver 100 and each part of the forklift 1 may be calculated in the same manner as above from the positional relationship with the two-dimensional marker, the model of the forklift 1 indicated by the two-dimensional marker, and the CAD information of the model.

When the calibration is completed, the driver 100 performs a predetermined operation on the display device 20 to activate the function of displaying the virtual monitor VM. When the function is activated, the control unit 24 acquires an image captured by the camera 17 from the camera 17 through the transmission/reception unit 23 (step S2). Next, the display determination unit 25 determines whether to display or hide the virtual monitor VM using the determination method described in (A) to (F) above (step S3). If the display determination unit 25 determines to display it (step S4; Yes), the control unit 24 places the virtual monitor VM at a predetermined position in the virtual space (step S5). The predetermined position is, for example, a position (gaze target position) that is a focal distance away in the line of sight of the driver 100. The control unit 24 places the virtual monitor VM at a position that is, for example, around the line of sight target position and is displayed in the peripheral part of the field of view on the display unit 22 (for example, the four corners). The control unit 24 also sets the size of the virtual monitor VM so that it is within a predetermined range of the field of view of the driver 100 (a size that does not get in the way). The display size of the virtual monitor VM (the size seen by the driver 100, the size it occupies on the display unit 22) and where on the display unit 22 the virtual monitor VM is displayed may be configured so that the driver 100 can arbitrarily set them. For example, an input interface of a virtual object that allows the driver 100 to input these settings may be displayed in the virtual space, and the driver 100 may be able to input the display position and display size of the virtual monitor VM from this input interface. The technology that allows the driver 100 to operate the input interface displayed in the virtual space in this way can be realized by MR. Also, as described above, the control unit 24 changes the size of the virtual monitor VM (the size at the arrangement position) according to the focal length of the driver 100. For example, a table associating focal length with the display size of the virtual monitor VM may be registered in the memory unit 26, and the control unit 24 may calculate the focal length based on the detection value of the height sensor 15 or the detection value of the eye tracking sensor of the sensor unit 21 (the focal length may not be calculated but may be set to a fixed value (e.g., several meters)), and may calculate the size of the virtual monitor VM based on the focal length, the display size set by the driver 100, and the table registered in the memory unit 26. If the display determination unit 25 determines in step S3 to hide the display (step S4; No), the process proceeds to step S6.

Next, the control unit 24 determines whether or not to terminate the function of displaying the virtual monitor VM (step S6). For example, if the driver 100 performs an operation to terminate the function of displaying the virtual monitor VM, the control unit 24 determines that the function of displaying the virtual monitor VM is to be terminated, and otherwise determines that the function is not to be terminated. If it is determined that the function is not to be terminated (step S6; No), the processing from step S2 onwards is repeated. If it is determined that the function is to be terminated (step S6; Yes), the processing flow of FIG. 6 is terminated.

In the above embodiment, the visibility assistance system including the camera 17 and the display device 20 is applied to a forklift 1, but the visibility assistance system can also be applied to industrial vehicles other than forklifts, such as excavators, bulldozers, crane trucks, and reach stackers. Although the camera 17 is set up to capture an image in front of the forks 12, the camera 17 may capture an image in another direction, and is not limited to the forward direction. In the case of other industrial vehicles, the camera can be set up to capture an image in any position on the work equipment (e.g., the boom or bucket of an excavator, the bucket or arm of a bulldozer, the boom, jib, or hook of a crane, etc.) in any direction.

(effect)
Conventionally, a system has been provided in which a real monitor is provided in front of the driver's seat and an image of the front of the fork 12 is displayed on the monitor. In the case of this system, the monitor obstructs the view during driving, and the focal distance between the real work target position and the monitor is different, so that a burden of switching the focus occurs when comparing the two. In contrast, in this embodiment, the driver is made to wear AR glasses (display device 20), a virtual monitor VM is displayed in a virtual space, and an image of the front of the fork 12 is displayed on the virtual monitor VM. As a result, for example, in a loading and unloading operation, the driver 100 can stably align the fork 12 and the pallet 42 without looking up even if the height of the shelf is high. Even if the driver 100 looks up, the driver 100 can check the work target position and the image without switching the focal distance by matching the distance to the actual object of the line of sight with the distance to the position where the virtual monitor VM is placed. In addition, not only can the virtual monitor VM be placed anywhere in the virtual space, but it can also be prevented from obstructing the view by erasing the virtual monitor VM when it is not needed. Furthermore, by positioning the virtual monitor VM so that it follows the line of sight of the driver 100, the driver 100 can check the image ahead of the forks 12 while looking at other positions, such as the periphery of the vehicle.

FIG. 7 is a diagram illustrating an example of a hardware configuration of the display device according to the embodiment.
The computer 900 includes a CPU 901 , a main memory device 902 , an auxiliary memory device 903 , an input/output interface 904 , and a communication interface 905 .
The above-mentioned display device 20 is implemented in a computer 900. The above-mentioned functions are stored in the auxiliary storage device 903 in the form of a program. The CPU 901 reads the program from the auxiliary storage device 903, loads it in the main storage device 902, and executes the above-mentioned processing in accordance with the program. The CPU 901 also reserves a storage area in the main storage device 902 in accordance with the program. The CPU 901 also reserves a storage area in the auxiliary storage device 903 for storing data being processed in accordance with the program.

A program for implementing all or part of the functions of the display device 20 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing by each functional unit. The term "computer system" here includes hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes a homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to portable media such as CDs, DVDs, and USBs, and storage devices such as hard disks built into a computer system. In addition, if the program is distributed to the computer 900 via a communication line, the computer 900 that receives the program may expand the program into the main storage device 902 and execute the above processing. In addition, the program may be for implementing part of the functions described above, and may further be capable of implementing the functions described above in combination with a program already recorded in the computer system.

As described above, several embodiments of the present disclosure have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

<Additional Notes>
The display device, the visibility assistance system, the industrial vehicle, the control method, and the program described in each embodiment can be understood, for example, as follows.

(1) The display device according to the first aspect is a display device 20 worn on the head of a person (driver 100 or passenger), and includes a transparent display unit 22 capable of displaying image data, an acquisition unit (transmitter/receiver 23) that acquires an image captured by a camera 17 provided on a work implement (fork 12) equipped on an industrial vehicle (forklift 1), and a control unit 24 that displays the image data, including a virtual monitor on which the image is displayed, in the vicinity of a position a focal length away in the direction of the person's line of sight, which is visible through the display unit.
With this, by using AR technology to install a virtual monitor VM in the vicinity of a position that is a focal length away in the line of sight of the driver 100, an image captured by the camera 17 can be displayed in the field of view of the driver 100 at approximately the same focal length as the line of sight of the driver 100. This allows the driver 100 to perform work (e.g., loading and unloading work) while checking the surroundings (e.g., forward) of the work device (e.g., raised fork 12) equipped on the industrial vehicle (forklift 1).

(2) A second aspect of the present invention is the display device according to (1), in which the control unit changes the size of the virtual monitor in accordance with the focal length.
This allows the size of the virtual monitor seen by the driver 100 to remain unchanged.

(3) A display device according to a third aspect is the display device of (1) to (2), in which the control unit displays the image data on a periphery of the display unit.
This makes it possible to ensure visibility in the central part of the display unit by displaying on the peripheral part of the display unit.

(4) A display device according to a fourth aspect is a display device according to any one of (1) to (3), wherein the control unit sets the focal length based on the state (height, position) of the working device (fork 12).
For example, by setting the focal length, i.e., the position of the object, based on the detection value of a height sensor, the driver 100 can check the area around the work implement without the burden of switching his or her line of sight.

(5) A display device according to a fourth aspect is a display device according to any one of (1) to (4), further comprising a display determination unit that determines whether to display or hide the image data, and when the display determination unit determines that the image data should be displayed, the control unit displays the image data, and when the display determination unit determines that the image data should be hidden, the control unit does not display the image data.
This allows the virtual monitor VM to be displayed only when necessary. For example, in a situation where the virtual monitor VM blocks the view and becomes an obstacle, the virtual monitor VM can be hidden.

(6) A display device according to a sixth aspect is the display device of (5), in which the display determination unit determines to display the image data when it detects an operation of the working device.
This makes it possible to display the virtual monitor VM only when the maintenance device is in operation.

(7) A display device according to a seventh aspect is the display device of (5) to (6), in which the display determination unit determines whether to display or not display the image data based on a position of the working device.
This makes it possible to display the virtual monitor VM only when the operating device is present at a predetermined position.

(8) A display device according to an eighth aspect is the display device according to any one of (5) to (7), wherein the display determination unit determines whether to display or not display the image data based on a line of sight of the person.
This makes it possible to display the virtual monitor VM only when the driver's line of sight is directed in a certain direction.

(9) The visibility assistance system according to the ninth aspect includes a camera provided on a work device of an industrial vehicle and a display device as described in (1) to (8).

(10) The industrial vehicle according to the tenth aspect includes a camera provided on a work device provided on the industrial vehicle and a display device according to (1) to (8).

(11) The control method according to the eleventh aspect is a control method for a display device worn on a person's head, having a transparent display unit capable of displaying image data, which acquires an image captured by a camera attached to a work device equipped in an industrial vehicle, and displays the image data, including a virtual monitor on which the image is displayed, in the vicinity of a position that is a focal length away in the line of sight of the person, as seen through the display unit.

(12) The program according to the twelfth aspect causes a computer of a display device worn on a person's head, having a transparent display unit capable of displaying image data, to function as a means for acquiring an image captured by a camera attached to a work device equipped on an industrial vehicle, and a means for displaying the image data, including a virtual monitor on which the image is displayed, in the vicinity of a position that is a focal length away in the line of sight of the person, as seen through the display unit.

Reference Signs List 1: Forklift 10: Body 11: Cargo handling device 12: Fork 13: Mast 14: Backrest 15: Lifting height sensor 16: Proximity sensor 17: Camera 18: Control device 19: Lift lever 20: Display device 21: Sensor section 22: Display section 23: Transmitter/receiver section 24: Control section 25: Display determination section 26: Memory section 100: Driver VM: Virtual monitor 900: Computer 901: CPU
902: Main memory device 903: Auxiliary memory device 904: Input/output interface 905: Communication interface

Claims (12)

A display device that is worn on a person's head,
a transparent display capable of displaying image data;
an acquisition unit that acquires images captured by a camera provided on a work device of the industrial vehicle;
a control unit that has a function of displaying the image data, including a virtual monitor on which the image is displayed, at any position in a virtual space so that the image data is displayed on the display unit as if the image data were located at a position in the real world corresponding to the position, and that positions the image data at a position in the virtual space corresponding to the vicinity of a position that is a focal length away in the line of sight of the person and that is visible through the display unit;
A display device comprising: The control unit changes the size of the virtual monitor according to the focal length.
The display device according to claim 1 . The control unit displays the image data on a peripheral portion of the display unit.
The display device according to claim 1 or 2. The control unit sets the focal length based on a state of the working device.
The display device according to claim 1 or 2. a display determination unit that determines whether the image data is to be displayed or not;
Further equipped with
The control unit displays the image data when the display determination unit determines that the image data is to be displayed, and does not display the image data when the display determination unit determines that the image data is to be hidden.
The display device according to claim 1 or 2. The display determination unit determines that the image data is to be displayed when the operation of the working device is detected.
The display device according to claim 5 . The display determination unit determines whether to display or not display the image data based on a position of the working device.
The display device according to claim 5 . The display determination unit determines whether to display or not display the image data based on a line of sight direction of the person.
The display device according to claim 5 . A camera provided on a work device of the industrial vehicle;
The display device according to claim 1 or 2;
A visual aid system. A camera provided on a work device provided on an industrial vehicle; the display device according to claim 1 or 2;
An industrial vehicle equipped with A method for controlling a display device to be worn on a person's head, the display device having a transparent display unit capable of displaying image data, comprising the steps of:
Acquire images captured by a camera installed on a work device of the industrial vehicle,
By using a function that allows the image data, including a virtual monitor on which the image is displayed, to be displayed on the display unit as if the image data were present at a real-world position corresponding to the position by placing the image data at an arbitrary position in a virtual space possessed by the display device, the image data is placed at a position in the virtual space corresponding to the focal length in the line of sight of the person, which is visible through the display unit;
Control methods. A computer of a display device mounted on a person's head, the computer having a transparent display capable of displaying image data,
A means for acquiring images captured by a camera provided on a work device of the industrial vehicle;
a means for arranging the image data, including a virtual monitor on which the image is displayed, at any position in a virtual space possessed by the display device, and displaying the image data on the display unit as if the image data were present at a position in the real world corresponding to the position; and a means for arranging the image data at a position in the virtual space corresponding to a focal length in the line of sight of the person, which is visible through the display unit;
A program to function as a

Images