JP2006146778A - Head mount display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a head mount display capable of displaying video information which enables a user to recognize surrounding situations precisely and clearly in a short time. <P>SOLUTION: The head mount display capable of displaying a video acquired from a small-sized camera is provided with an infrared sensor. When the infrared sensor detects an obstruction, such as a level difference or the like around an observer, video processing of outline emphasis or the like is performed by a video processing means and risk forecasting video emphasizing the level difference or the like is displayed. Thus, even those who have visual impairment such as weak eyesight can early recognize the level difference or the like. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a head-mounted display device, and more particularly to a head-mounted display provided with a means for detecting a step or an obstacle and issuing a warning to a viewer.

  2. Description of the Related Art Conventionally, a head mounted display (hereinafter simply referred to as “HMD”) that includes a small display on a head suspension device and displays an electronic image in front of the eyes is known. Various HMDs for industrial use, medical use, movie watching, etc. have been developed, and their uses are diverse. Among them, the type in which a small display is mounted on a suspension device having a shape of glasses has a see-through property that displays an electronic image superimposed on the outside scene, and is visually impaired such as low vision. For those who have, there is an increasing need as a second eye to supplement normal vision.

Some HMDs that function as visual assistance include a small camera that captures the viewing direction of the viewer. An image picked up from a small camera is displayed on a display in front of the eyes. The viewer can obtain visual information while properly using the field of view by the external light and the image on the display according to the external environment.
In addition, in the HMD having a function as a visual aid, not only the scenery in the traveling direction is displayed but also a detection function for various obstacles is provided to provide information to the viewer more actively. Types are also being developed. As an invention from the viewpoint of such danger prediction, the HMD provided in Patent Document 1 is an HMD provided with an infrared sensor that detects infrared rays from a plurality of infrared irradiation markers previously arranged on a passage or a ceiling. Thus, the present location of the viewer is measured by signals from a plurality of markers at different positions, and an image corresponding to the measurement result is displayed.
Patent Documents 2 and 3 provide devices that detect the position of a user from various position detection means and notify information on the surrounding situation by voice based on the detection results.
Japanese Patent No. 3372926 JP-A-8-66441 JP 2001-319288 A

However, the invention provided in Patent Document 1 depends on the infrared rays emitted from the infrared irradiation marker, and can provide information on the surrounding situation only in a limited space due to restrictions on marker installation. Can not. In addition, since the video to be displayed is prepared in advance, it cannot cope with a sudden change in the situation, and it can be said that the risk is high as a risk prediction means.
The invention provided in Patent Document 2 or 3 guides the surrounding situation by voice. Although the user can recognize that there is a danger in the vicinity from the voice information, it is difficult to instantly determine where and in what form it specifically exists. In particular, for those who are difficult to detect external light such as amblyopia, information through vision cannot be obtained, and such a tendency becomes strong.

  An object of the present invention is to provide auxiliary information that allows a user to accurately and clearly recognize a surrounding situation in a short time.

  In order to solve the above-described problem, the invention according to claim 1 is directed to an image pickup unit that picks up an image of a subject, a state detection unit that detects a state of the subject and outputs a signal in the head mounted display device, Video processing means for generating a risk prediction video signal from the video signal acquired from the imaging means based on the detection signal, and video display means for displaying a video from the risk prediction video signal converted by the video processing means It is characterized by providing.

  According to a second aspect of the present invention, in the head-mounted display device according to the first aspect, the signal generation processing performed by the video processing means includes edge enhancement correction, tone correction, and contrast of a video signal captured by the imaging device. It is any one of correction | amendment or these combination, It is characterized by the above-mentioned.

  According to a third aspect of the present invention, in the head-mounted display device according to the first or second aspect, the situation measuring means is any one of an optical sensor and a sound wave detector.

  According to a fourth aspect of the present invention, in the head mounted display device according to any one of the first to third aspects, the head mounted display emits risk prediction information based on a detection signal of the situation detecting means. An auxiliary warning means is further provided.

  According to a fifth aspect of the present invention, in the head mounted display device according to the fourth aspect, the auxiliary warning means is any one of a vibration generating device, a sound generating device, and an LED lamp.

  According to a sixth aspect of the present invention, in the head mounted display device according to any one of the first to fifth aspects, the mounting angle of the situation detecting means is 15 degrees above and 60 degrees below the horizontal direction of the field of view. It is the angle of this.

  According to a seventh aspect of the present invention, in the head-mounted display device, the first signal generation for performing pattern matching processing on an image pickup unit that picks up a subject and a video signal acquired from the image pickup unit using a predetermined video signal set in advance Video processing means including processing and second signal generation processing for generating a danger prediction video signal from the video signal based on a detection signal detected by pattern matching processing by the first signal generation processing, and the video processing means And a video display means for displaying the danger prediction video signal converted by the above.

  According to an eighth aspect of the present invention, in the head-mounted display device according to the seventh aspect, the signal generation processing performed in the second signal generation processing includes edge enhancement correction and tone correction of a video signal imaged by the imaging device. , Contrast correction, or a combination thereof.

  According to a ninth aspect of the present invention, in the head-mounted display device according to the seventh or eighth aspect, the head-mounted display is an auxiliary warning unit that issues danger prediction information based on a detection signal in the first signal generation process. Is further provided.

  According to a tenth aspect of the present invention, in the head mounted display device according to the ninth aspect, the auxiliary warning means is any one of a vibration generating device, a sound generating device, and an LED lamp.

  According to the first aspect of the present invention, the head-mounted display device includes the situation detection unit, so that it is possible to determine a change in the surrounding situation of the viewer. Further, since the danger prediction video signal that makes an obstacle such as a step conspicuous in the video signal acquired by the imaging unit is generated based on the detection signal of the situation detection unit, the viewer can detect the obstacle on the video display unit. It becomes possible to instantly recognize the position, size and mode. As a result, the viewer can determine the traveling direction in consideration of specific information such as obstacles, and the viewer can walk safely and with a sense of security.

  According to the second aspect of the present invention, since the signal generation performed by the video processing means is any one of contour enhancement correction, color correction, contrast correction, or a combination thereof, the visibility of obstacles such as steps is visible. Can be improved. As a result, an obstacle can be recognized instantly. At the same time, the direction of travel can be determined in consideration of the presence of an obstacle, and the viewer can walk safely and with a sense of security.

  According to the third aspect of the present invention, since the measurable area of the situation measuring means is a relatively wide optical sensor or sound wave detector, it is possible to measure a place at a certain distance from the viewer. it can. For this reason, the viewer can walk safely and with a sense of security while recognizing obstacles such as steps earlier.

  According to the invention described in claim 4, in addition to enabling the risk prediction by the video processing means generating the video signal, the viewer is further provided with the auxiliary warning means for further issuing the risk prediction information. It becomes possible to recognize obstacles such as steps more reliably. For this reason, it becomes possible for the viewer to walk safely and with a sense of security while more surely recognizing a step or the like.

  According to the fifth aspect of the present invention, since the auxiliary warning means is a vibration generating device, the viewer can recognize the presence of a dangerous place by touch. Moreover, since it is an audio | voice generator, presence of a dangerous location can be recognized by hearing. Furthermore, since it is an LED lamp, it is possible to visually recognize the presence of a dangerous spot from other than the video display means. Thus, since the prediction of danger is recognized through a plurality of senses, the viewer can walk more reliably and safely.

  According to the sixth aspect of the present invention, since the angle of attachment of the situation detection means is an angle from 15 degrees above to 60 degrees above the horizontal direction of the field of view, an obstacle present in the direction of travel of the viewer Can be reliably detected.

  According to the invention described in claim 7, it is possible for the head mounted display device to discriminate a change in the surrounding situation by performing pattern matching processing on the video acquired by the imaging means with a pre-stored prescribed video signal. It becomes. Further, since the process of generating the video signal into the danger prediction video signal is performed in the second signal generation process based on the detection signal of the first signal generation process, the viewer can detect the obstacle on the video display means. It becomes possible to instantly recognize the position, size and mode. As a result, the viewer can determine the traveling direction in consideration of specific information such as obstacles, and the viewer can walk safely and with a sense of security.

  According to the invention described in claim 8, since the signal generation performed in the second signal generation process is any one of contour emphasis correction, color correction, contrast correction, or a combination of these, Visibility can be improved. As a result, an obstacle can be recognized instantly. At the same time, the direction of travel can be determined with specific information on the obstacles in mind, and the viewer can walk safely and with a sense of security.

  According to the ninth aspect of the invention, in addition to the generation of the danger prediction video signal by the second signal generation process, the auxiliary warning means for emitting the risk prediction information based on the detection signal of the first signal generation process is further provided. Thus, the viewer can more reliably recognize obstacles such as steps. For this reason, it becomes possible for the viewer to walk safely and with a sense of security while more surely recognizing a step or the like.

  According to the tenth aspect of the present invention, since the auxiliary warning means is a vibration generating device, the viewer can recognize the presence of a dangerous place by touch. Moreover, since it is an audio | voice generator, presence of a dangerous location can be recognized by hearing. Furthermore, since it is an LED lamp, it is possible to visually recognize the presence of a dangerous spot from other than the video display means. Thus, since the prediction of danger is recognized through a plurality of senses, the viewer can walk more reliably and safely.

Next, the best mode for carrying out the present invention will be described with reference to the drawings. The HMD in the first embodiment detects an ambient situation using an infrared sensor as a situation detection means, and displays specific danger prediction information on the display of the HMD based on the detection result.
In the following description, the situation detected by the infrared sensor refers to a level difference on the ground, a staircase or an obstacle existing in an area extending around the user's facing direction, or a height near the height of the user from the ground. It shall mean an existing wall, beam, or signboard.

FIG. 1 is a schematic diagram showing an external configuration of an HMD to which the present invention is applied. In the following description, the direction opposite to the eyepiece direction is the front.
The HMD 1 includes a frame 4 and a lens 5 provided with a nose pad 2 and a temple 3, as in normal glasses. Above the lens 5 of the frame 4, an image display device 6, an eyepiece optical system 7, and an imaging device 8 are integrally installed. In addition, an infrared light source 9 that irradiates infrared rays in a substantially front direction is provided on the side surface (in the vicinity of the hinge of the glasses) in the opposite direction to where the video display device 6 and the like are installed. The infrared light emitted from the infrared light source 9 is reflected by the subject and is incident on an imaging device 8 having an infrared light receiving function to be described later, thereby functioning as a situation detecting means.
The infrared light source 9 is connected to the control unit 20 which is a control box through the cable 10 together with the video display device 6 and the imaging device 8.

  The lens 5 and the eyepiece optical system 7 are fitted so that the surface directions coincide with each other without a step. Note that the HMD 1 in the first embodiment forms a virtual image on the eyeball according to the principle of a hologram that displays video by causing the video light emitted from the video display device 6 to interfere. With such a configuration, a hologram image can be displayed in a superimposed manner on a field of view by external light, and auxiliary information can be provided without hindering the field of view by external light.

The imaging device 8 is one of video information input means, and is provided near the hinge connecting the frame 4 and the temple 3 above the video display device 6. Further, in order to capture the field of view of the user's line of sight, it is installed in a direction that allows photographing of the field of view mainly centering on the front of the viewer. The imaging device 8 converts the reflected light from the subject into an electronic signal by the photoelectric conversion action of the light receiving element, and transmits it to the control unit 20 connected via the cable 10. From the control unit 20, a signal subjected to a certain video process is transmitted to the video display device 6 so that a video as a visual assistance means can be displayed on the viewer's field of view.
As the imaging device 8, an imaging device using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) image sensor or the like can be applied. In the first embodiment, the imaging device 8 is irradiated from the infrared light source 9 and reflected by the subject. In order to capture infrared rays, a small CCD camera having an infrared light receiving function is applied.

  Here, the infrared light receiving region of the imaging device 8 is set to a region that can be displayed by the video display device 6, but as shown in FIG. 2, the upward direction when the horizontal direction of the field of view is 0 degrees. If it is in the range of −90 degrees from 90 degrees downward, the risk prediction effect of the HMD 1 in the first embodiment can be sufficiently secured. A range of 15 degrees upward and 60 degrees downward is more preferable. This is because obstacles, such as steps, existing in the direction of travel of the user are relatively concentrated on the feet.

  The video display device 6 converts the video signal transmitted from the control unit 20 and irradiates the eyepiece optical system 7 with the video. FIG. 3 is a side sectional view showing a section from the left side of the image display device 6 and the eyepiece optical system 7. Inside a housing 30 that is a main body of the video display device 6, a transmissive liquid crystal display 31 that generates a video from a video signal transmitted from the control unit 20, a light emitting diode (LED) 32 that is a backlight light source, and An illumination optical system 33 for guiding the irradiation light from the light emitting diode 32 to the entire surface of the liquid crystal display 31 is provided.

The eyepiece optical system 7 further includes a prism 34 and a hologram element 35. The prism 34 collects the image light flux spectrum-converted by the liquid crystal display 31 and guides it to the hologram element 35 while reflecting it inside. The hologram element 35 is configured to display a hologram image upon receiving the irradiation of the guided image light beam.
Note that the upper end of the prism 34 has a wedge shape in which a daylighting portion 36 is formed thick in the front direction so that an image from the liquid crystal display 31 is easily lit. The light collected from the daylighting unit 36 located at the upper end of the prism 34 is guided to the hologram element 35 provided at the lower end while being totally reflected inside the prism 34. The guided image light beam is irradiated onto the hologram element 35 and a virtual image is formed on the eyeball E by light interference.

Next, the control unit 20 of the HMD 1 will be described.
FIG. 4 is a block diagram showing a functional configuration of the control unit 20. The control unit 20 includes an arithmetic processing unit 21, an A / D converter 22, an infrared measurement circuit 23, a display driving device 24, and a power supply unit 25. The arithmetic processing unit 21 further includes a CPU 26, a ROM 29, a DSP 27, and a graphic memory 28. Consists of

  Below, the outline | summary of the function of each apparatus is demonstrated. A CPU (Central Processing Unit) 26 performs overall control of the HMD 1 based on a pre-recorded program. A ROM (Read Only Memory) 29 stores programs related to the overall control of the HMD 1 and various programs related to video processing performed by the DSP 27.

  The A / D (Analog / Digital) converter 22 converts an analog video signal obtained by photoelectric conversion of the imaging device 8 into a digital video signal and transmits it to the CPU 26. The infrared measurement circuit 23 measures the infrared level output from the imaging device 8 with an infrared light receiving function, and outputs a detection signal to the CPU 26 when it is below (or above) a predetermined input value. It is like that.

  The graphic memory 28 temporarily stores a digital signal acquired from the A / D converter 22 and a digital signal subjected to video processing by the DSP 27 or the like. Further, a program area for expanding a program executed by the DSP 27 is expanded.

  The power supply unit 25 supplies power for operating the HMD 1 via a power switch (not shown), and various external power sources, portable batteries, or rechargeable batteries can also be applied. In this embodiment, the HMD 1 is made portable. In view of this, a portable battery is applied.

  A DSP (Digital Signal Processor) 27 reads a digital video signal recorded in the graphic memory 28 and not yet processed in accordance with an instruction from the CPU 26, and sends it to the video display device 6 based on the infrared signal acquired by the imaging device 8. This is to perform partial tone correction of the video signal to be displayed, various filter processing or synthesis processing.

  Here, as the tone correction, partial luminance conversion of the video signal, contrast adjustment, gradation inversion, or the like can be applied. As the filter processing, partial contour emphasis processing, emboss processing, sharpening processing, unsharpening processing, mosaic processing, or the like can be applied. Further, as the synthesis process, means such as displaying a predetermined symbol or specific text data indicating danger on the video screen can be applied.

Next, the processing procedure of the control unit 20 will be described with reference to FIG.
When infrared rays are emitted from the infrared light source 9 (step S1), infrared rays reflected from the subject enter the imaging device 8 having an infrared light receiving function, and a detection signal is output to the CPU 26 via the A / D converter 22. (Step S2). At this time, when the value of the electronic signal input to the CPU 26 is different from a preset value (for example, the value of the amount of received infrared light in a smooth state without a step) (step S3: YES), the CPU 26 emphasizes the contour of the DSP 27. A processing instruction signal is output (step S4). The DSP 27 to which the contour enhancement processing signal is input from the CPU 26 reads an unprocessed video signal (hereinafter referred to as “RAW signal”) from the graphic memory 28 (Step 5), and based on the predetermined filter data, Outline enhancement processing is performed (step S6). As a result, a danger prediction video signal with a conspicuous step is generated (step S7).

  More specifically, a group of dots having a value averaged in a certain direction from the gradation value for each dot corresponding to the resolution of the liquid crystal display 31 is grouped. This process is performed for each of the R, G, and B channels. In addition, a predetermined gradation value (for example, each of R, G, and B is set to a ratio at which red is displayed) is generated from the gradation of the dot group located at the boundary between these groups. As a result, it is possible to distinguish and display the boundary between the region where the gradation is abruptly different, such as a step, and the planar region where the viewer is located, with a red line.

  Thereafter, the video signal subjected to the contour enhancement process is output to the video display device 6 through the display driving device (step S8).

As described above, according to the present embodiment, the viewer can accurately determine the position and mode of an obstacle such as a step. Moreover, by acquiring such specific information, a sense of security for danger increases and natural walking becomes possible. In particular, the visibility of steps such as stairs is greatly improved by performing image processing such as edge enhancement, contrast adjustment, and gradation inversion.
Further, by using infrared detection, the processing time in the DSP 27 can be suppressed, and the apparatus can be simplified and the power consumption can be reduced.

In addition to the identification of the level difference or the like by the video processing, it is possible to further include a device that gives a warning by hearing or touch (see FIG. 4). For example, as shown to Fig.6 (a), it is good also as a structure provided with the speaker unit 41 connected to the control unit 20 in the temple 3. FIG. At the same time that the CPU 26 outputs the contour emphasis processing instruction signal to the DSP 27, it is also possible to output a warning sound generation instruction signal to the speaker unit 41 so that a peep sound, a warning sound or the like is generated. Moreover, as shown in FIG.6 (b), it is good also as a structure which provides the small vibration motor 40 in the temple 3, and warns by vibration. An LED lamp 42 may be provided in the vicinity of the frame 4 at a position visible from the eyeball.
Further, the ROM 29 may be configured such that a plurality of patterns (continuous or intermittent) are stored in advance in such a warning unit, and the level of warning is stepped according to the level of a step or the like.

  Further, the outline emphasis in the DSP 27 is configured to display the line by specifying the gradation of dots belonging to the boundary of the group having an average gradation value. However, the color of a specific group is displayed in a single color. May be converted to For example, an obstacle such as a step generally tends to have high saturation. By using this tendency, a gradation that exceeds (does not exceed) a certain threshold is converted into a specific color (for example, red). . As a result, the position, size, mode, etc. of the step and the like are more conspicuous, and there is an effect that it is easy to predict the danger.

In this embodiment, infrared rays are used as the obstacle detection means, but pattern matching processing by the DSP 27 can be applied as other detection means.
That is, template data indicating a contour such as a step is stored in advance in the ROM 29, and the presence of an obstacle such as a step is determined by matching the contour data and the template data obtained by the contour emphasis processing in the DSP 27. It is also possible to realize the various warning means by the CPU 26 or the DSP 27. In this case, the infrared light receiving function of the infrared light source 9 and the imaging device 8 can be omitted.

  As mentioned above, although the best form for implementing this invention was demonstrated, this invention is not limited to these.

It is the schematic which showed the external appearance structure in embodiment to which this invention is applied. It is the schematic diagram which showed the light trace of the infrared light source in embodiment to which this invention is applied. It is side sectional drawing of the video display apparatus in embodiment to which this invention is applied. It is the block diagram which showed the functional structure in embodiment to which this invention is applied. It is the flowchart which showed the process sequence of the video signal conversion in embodiment to which this invention is applied. It is the schematic which showed the other warning means in embodiment to which this invention is applied.

Explanation of symbols

1 HMD
2 Nose pad 3 Temple 4 Frame 5 Lens 6 Video display device 7 Eyepiece optical system 8 Imaging device 9 Infrared light source 10 Cable 20 Control unit 21 Arithmetic processing unit 22 A / D converter 23 Infrared measurement circuit 24 Display drive device 25 Power supply unit 26 CPU
27 DSP
28 Graphic memory 29 ROM
30 Housing 31 Liquid crystal display 32 Light emitting diode 33 Illumination optical system 34 Prism 35 Hologram element 36 Daylighting unit 40 Vibration motor 41 Speaker unit 42 LED lamp E Eyeball

Claims (10)

Imaging means for imaging the subject, situation detection means for detecting the situation of the subject and outputting a signal,
Video processing means for generating a danger prediction video signal from the video signal acquired from the imaging means based on the detection signal of the situation detection means;
A head mounted display device comprising: video display means for displaying a video from the danger prediction video signal generated by the video processing means. The head mounted display device according to claim 1,
The signal generation processing performed by the video processing means is any one of contour enhancement correction, color tone correction, contrast correction, or a combination thereof of a video signal imaged by the imaging device. . In the head mounted display device according to claim 1 or 2,
The head mount display device characterized in that the situation measuring means is one of an optical sensor and a sound wave detector. In the head mounted display device according to any one of claims 1 to 3,
The head mounted display further comprises auxiliary warning means for issuing danger prediction information based on a detection signal of the situation detecting means. The head mounted display device according to claim 4,
The head mounted display device, wherein the auxiliary warning means is any one of a vibration generating device, a sound generating device, and an LED lamp. In the head mounted display device according to any one of claims 1 to 5,
The head-mounted display device is characterized in that the mounting angle of the situation detecting means is an angle from 15 degrees above to 60 degrees below the horizontal direction of the field of view. Imaging means for imaging a subject;
Based on a first signal generation process for performing a pattern matching process on a video signal acquired from the imaging unit with a predetermined video signal set in advance, and a detection signal detected by a pattern matching process by the first signal generation process Video processing means including a second signal generation process for generating a danger prediction video signal from the video signal;
A head-mounted display device comprising: video display means for displaying a video from the danger prediction video signal generated by the video processing means. The head mounted display device according to claim 7,
The signal generation processing performed in the second signal generation processing is any one of contour enhancement correction, color correction, contrast correction, or a combination thereof of a video signal imaged by the imaging device. Display device. The head mounted display device according to claim 7 or 8,
The head mounted display further comprises auxiliary warning means for emitting danger prediction information based on a detection signal in the first signal generation process. The head mounted display device according to claim 9, wherein
The head mounted display device, wherein the auxiliary warning means is any one of a vibration generating device, a sound generating device, and an LED lamp.

Images