JP5482712B2 - Image display device - Google Patents

Description

  The present invention relates to an image display apparatus having a head mounted display (HMD) that displays an image corresponding to an image signal to a viewer as a display image that is a virtual image, and in particular, a technique for reproducing a continuous image input from the outside without loss. About.

  An image display device having an HMD that is mounted on the observer's head in order to project image light representing an image to be displayed onto the observer's eyes is already known (see, for example, Patent Document 1). According to this image display device, an observer can directly observe an image without using a screen on which the image is projected.

  When the HMD is classified by the image light forming system, for example, a spatial modulation type disclosed in Patent Document 1, that is, a spatial modulation element (for example, a liquid crystal or an organic EL (electroluminescence)) that operates according to an image signal. And a method of forming the image light from the light emitted from the light source, and forming the image light by scanning a scanning type, that is, light emitted from the light source and having an intensity corresponding to the image signal. It is classified into the method to do.

  Further, when the HMDs are classified according to the display image observation method, for example, the see-through type disclosed in Patent Document 1, that is, the observer can observe the scenery of the outside world superimposed on the display image by the HMD. A certain method and a sealed type, that is, a method that blocks the incidence of external light representing the actual outside world and enables the observer to observe only the display image by the HMD.

  Regarding HMDs, there is a desire to eliminate waste of power consumption by HMDs regardless of whether they are see-through type or sealed type. In order to satisfy this demand, Patent Document 1 determines whether or not the observer is in a sleeping state (the observer is closing his heel). If the observer is in a sleeping state, the HMD is automatically turned on. A technique for cutting is disclosed. According to this technology, it is possible to prevent image light from being generated and projected onto the eyes of the observer even though the observer is in a sleep state, thereby eliminating waste of power consumption. .

Japanese Patent Laid-Open No. 11-249064

  The present inventors consider the state in which the observer is sleeping with the eyelid closed as an example of an image light non-incident state in which the image light does not enter the observer's eye even if the HMD generates image light. Furthermore, as another example of the image light non-incidence state, for example, a state in which the observer's pupil position is out of the exit pupil of the HMD, so that even when the image is reproduced, the image is not observed by the observer. Thought.

  Furthermore, the present inventors have described that the image light non-incidence state as described above is not a continuity as strong as the observer's sleep state described above, and is a temporary event. It was noticed that even after the state was started, it was normal for the image light to return to the state of being incident on the observer's eyes after a short time.

  Based on such knowledge, when the temporary image light non-incident state is started in the HMD, the present inventors stop the reproduction of the image without turning off the power of the HMD, and temporarily A technique has been proposed in which image reproduction is resumed when the image light non-incident state ends.

  However, according to the proposed technique, when the image input from the outside to the HMD for image display, ie, image reproduction, is not a still image but a continuous image (an example of a moving image or streaming content), the image light When the non-incident state occurs, the reproduction of the moving image is automatically stopped, and thereafter, when the image light non-incident state ends, the reproduction of the moving image is automatically resumed. However, the portion of the moving image that would have been reproduced if the image light non-incident state did not occur is not reproduced after the end of the image light non-incident state. Therefore, in this proposed technique, there is a problem that the observer cannot observe a part of the moving image due to the stop of the image display accompanying the occurrence of the image light non-incident state.

  Measures to connect an external device that transmits a moving image to be played back to the HMD and that allows the HMD user to perform operations such as transmission, pause, and recording on the external device. If taken, the observer can record the video to be played back in the external device when the image light is not incident, and send the recorded video to the HMD at any time after the image light is not incident. Since it can be reproduced, if this measure is taken, there will be no problem that a part of the moving image cannot be observed due to the suspension of the image display in the non-incident state of the image light. Unlike the case of operating a personal computer at home, the observer is troublesome to perform operations such as pausing and recording on the above-mentioned external devices when the image light is not incident. Is A problem that may occur.

  Against the background described above, the present invention reproduces a continuous image input from the outside without omission in an image display apparatus having an HMD that allows an observer to observe a display image corresponding to an image signal. Is made as an issue.

In order to solve the problem, according to the first aspect of the present invention, there is provided an image display device having a head mounted display (HMD) that displays an image corresponding to an image signal to a viewer as a display image that is a virtual image. ,
An input unit to which the image signal is input;
A normal reproduction unit for forming the display image by normal reproduction for reproducing an input image represented by an image signal output from the input unit;
Recording and playback memory,
A recording unit for recording the input image in the recording / playback memory;
A recording / playback unit for forming the display image by recording / playback for reading out and playing back the input image from the recording / playback memory;
A determination unit that determines whether or not image light representing the display image is not incident on an observer's eye;
Including
The normal reproduction unit stops the normal reproduction at least for a period from the start time to the end time of the determination that the image light is not incident,
The recording unit records the input image output from the input unit in the recording / playback memory at least from the start of the determination that the image light is not incident,
The recording / playback unit reads the input images recorded in the recording / playback memory in order from the oldest, and performs the recording / playback from the end of the determination that the image light is not incident.
The image display device further includes:
A delay memory having a storable time T2 shorter than a storable time T1 capable of storing an image in the recording / playback memory;
A temporary storage unit that stores the input image output from the input unit in the delay memory in real time;
A delay memory erasure unit for erasing an image read from the delay memory and reproduced from the delay memory;
Including
Thereby, in the delay memory, the input image output from the input unit from the time point when returning from the present time to the past by the same length as the storable time T2 is stored.
The recording / playback unit, from the end of the determination that the image light is not incident, sequentially reads the images from the delay memory in order from the oldest and performs the first recording / playback, and the delay memory becomes empty Later, an image display device is provided in which images are read from the recording / playback memory in order from the oldest and the second recording / playback is performed continuously with the first recording / playback.
Moreover, according to the second aspect of the present invention, there is provided an image display device having a head mounted display (HMD) that displays an image corresponding to an image signal as a virtual image to a viewer.
An input unit to which the image signal is input;
A normal reproduction unit for forming the display image by normal reproduction for reproducing an input image represented by an image signal output from the input unit;
Recording and playback memory,
A recording unit for recording the input image in the recording / playback memory;
A recording / playback unit for forming the display image by recording / playback for reading out and playing back the input image from the recording / playback memory;
A determination unit that determines whether or not image light representing the display image is not incident on an observer's eye;
Including
The normal reproduction unit stops the normal reproduction at least for a period from the start time to the end time of the determination that the image light is not incident,
The recording unit records the input image output from the input unit in the recording / playback memory at least from the start of the determination that the image light is not incident,
The recording / playback unit reads the input images recorded in the recording / playback memory in order from the oldest, and performs the recording / playback from the end of the determination that the image light is not incident.
The determination unit
The first condition that the HMD normally emits the image light but the observer's eye is not in a state of receiving the image light;
A second condition that the HMD itself is normal but cannot emit the image light;
A third condition that the HMD is normal in itself, but the HMD stops emitting the image light because it is inappropriate for the observer to receive the image light;
A fourth condition that the HMD cannot emit the image light because the HMD itself is abnormal;
When at least one of these is established, an image display device that determines that the image light is not incident is provided.
The following aspects are obtained by the present invention. Each aspect is divided into sections, each section is numbered, and is described in the form of quoting the section numbers of other sections as necessary. Thus, each section is referred to the section number of the other section. By describing in the format, the technical features described in each section can be appropriately made independent according to the properties.

(1) An image display device having a head mounted display (HMD) that displays an image corresponding to an image signal to a viewer as a display image that is a virtual image,
An input unit to which the image signal is input;
A normal reproduction unit for forming the display image by normal reproduction for reproducing an input image represented by an image signal output from the input unit;
Recording and playback memory,
A recording unit for recording the input image in the recording / playback memory;
A recording / playback unit for forming the display image by recording / playback for reading out and playing back the input image from the recording / playback memory;
A determination unit that determines whether or not image light representing the display image is not incident on an observer's eye, and
The normal reproduction unit stops the normal reproduction at least for a period from the start time to the end time of the determination that the image light is not incident,
The recording unit records the input image output from the input unit in the recording / playback memory at least from the start of the determination that the image light is not incident,
The image display device that performs the recording / playback by reading the input images recorded in the recording / playback memory in order from the oldest time when the recording / playback unit finishes determining that the image light is not incident.

(2) Furthermore,
A delay memory having a storable time T2 shorter than a storable time T1 capable of storing an image in the recording / playback memory;
A temporary storage unit that stores the input image output from the input unit in the delay memory in real time;
An erasure unit for delay memory that erases an image read and reproduced from the delay memory from the delay memory;
Thereby, in the delay memory, the input image output from the input unit from the time point when returning from the present time to the past by the same length as the storable time T2 is stored.
The recording / playback unit, from the end of the determination that the image light is not incident, sequentially reads the images from the delay memory in order from the oldest and performs the first recording / playback, and the delay memory becomes empty The image display device according to item (1), in which the images are read from the recording / playback memory in order from the oldest and the second recording / playback is performed continuously with the first recording / playback.

(3) The normal playback unit performs delayed playback in which the input image is read from the delay memory and played back at a timing delayed by the same time as the storable time T2 from the time of storage in the delay playback memory. The image display device according to item (2).

(4) Further, when an image is reproduced by the recording / reproducing unit, a recording / reproducing memory erasing unit for erasing the reproduced image from the recording / reproducing memory in order from the oldest (1) to (3) The image display device according to any one of items 1 to 3.

(5) The determination unit includes:
The first condition that the HMD normally emits the image light but the observer's eye is not in a state of receiving the image light;
A second condition that the HMD itself is normal but cannot emit the image light;
A third condition that the HMD is normal in itself, but the HMD stops emitting the image light because it is inappropriate for the observer to receive the image light;
If at least one of the fourth condition that the image light cannot be emitted because the HMD itself is abnormal is determined, it is determined that the image light is not incident (1). Or the image display device according to any one of (4).

(6) Furthermore, the camera which images an observer's eyes is included,
The determination unit estimates an observer's line-of-sight direction based on an image captured by the camera, and based on the estimated line-of-sight direction, whether the observer's eye is in a state of receiving the image light. The image display device according to any one of (1) to (5).

(7) Furthermore, the camera which images an observer's eyes is included,
The determination unit estimates an observer's pupil position based on an image captured by the camera, and based on the estimated pupil position, whether the observer's eye is in a state of receiving the image light. The image display device according to any one of (1) to (5).

  According to the present invention, when the determination that the image light is not incident on the observer's eyes (hereinafter referred to as “non-incident determination”) is started, the input is input to the input unit. The normal reproduction for reproducing the input image represented by the image signal output from the input unit is automatically stopped. In parallel, the input image output from the input unit is automatically recorded. When the non-incidence determination is completed, recording / reproduction is automatically performed in which recorded images are reproduced in order from the oldest.

  Therefore, according to the present invention, when the image reproduction is automatically resumed after the non-incident determination is completed, the image is input to the input unit during the period between the non-incident determination start time and the end time. The image is automatically played back without requiring the viewer to perform additional operations. As a result, according to the present invention, when the image light enters a non-incident period in which the light does not enter the eye, the non-incident period ends and the image reproduction is ended even though the image reproduction is automatically temporarily stopped. When is restarted, it becomes possible to reproduce the image without partial omission of the image, thereby improving the usability of the image display device.

FIG. 2 is a front view of the image display device according to the first embodiment of the present invention, together with a control unit, in a use state where the HMD is mounted on the observer's head. It is a functional block diagram which represents notionally the structure of the image display apparatus shown in FIG. FIG. 3 is a functional block diagram conceptually showing the configuration of a control unit shown in FIG. 2. 4 is a time chart for explaining how the control unit shown in FIG. 3 operates under an exemplary scenario. It is a time chart for demonstrating a mode that the control unit shown in FIG. 3 operate | moves under another example scenario. 3 is a flowchart conceptually showing a program stored in a program ROM shown in FIG. It is a functional block diagram which represents notionally the structure of the image display apparatus according to 2nd Embodiment of this invention. It is a functional block diagram which represents notionally the structure of the control unit shown in FIG. It is a time chart for demonstrating a mode that the control unit shown in FIG. 8 operate | moves under the same example scenario as FIG. 8 is a flowchart conceptually showing a program stored in a program ROM shown in FIG. It is a time chart for demonstrating a control unit in the image display apparatus according to 3rd Embodiment of this invention operating under the same example scenario as FIG. It is a flowchart which represents notionally the program memorize | stored in program ROM in the image display apparatus according to the said 3rd Embodiment.

  Hereinafter, some of the more specific embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a plan view showing an image display device 10 according to the first embodiment of the present invention in use. The image display device 10 includes a see-through type HMD 12. The HMD 12 is configured to project image light representing an image onto the eyes of an observer. The HMD 12 is attached to the observer's head by the head mount device 14.

  The head mount device 14 includes a frame 16 and an attachment device 18. Regardless of whether the frame 16 is a frame of eyeglasses (not shown) worn by the observer or a dedicated frame, the frame 16 is attached to the observer's head while being put on both ears of the observer. . The HMD 12 is mounted on a part of the frame 16 via the attachment device 18.

  Next, referring to FIGS. 1 and 2, the HMD 12 and the control unit 20 for controlling the HMD 12 will be described which are mainly composed of a computer.

  In addition, in the present embodiment, the HMD 12 and the control unit 20 which are both independent components are combined with each other to form the image display device 10. However, by incorporating the control unit 20 in the HMD 12, The image display device 10 may be configured.

  First, the outline of the HMD 12 will be described. FIG. 1 shows the HMD 12 in a plan view. The HMD 12 is monocular, and projects image light representing an image onto one eye of the observer from the front of the eye to display the image to the observer.

  As shown in FIG. 2, the HMD 12 is a spatial modulation type, and planar light from a light source (for example, an LED (light emitting diode)) 30 is converted into an LCD (liquid crystal display) 32 as a spatial modulation element. Each pixel is spatially modulated and the modulated light is projected onto the retina 38 through the pupil 36 of the observer's eye 34, thereby allowing the observer to observe the image as a virtual image. To do.

  As shown in FIG. 2, the light source 30 is driven by a light source driving circuit 40, and the LCD 32 is driven by an LCD driving circuit 42. The HMD 12 further includes an illumination optical system 44 that performs necessary optical processing on the light from the light source 30 and enters the LCD 32, and an eyepiece optical system 46 that guides image light emitted from the LCD 32 to the pupil 36. Yes.

  As shown in FIG. 1, since the HMD 12 is a see-through type as described above, it has a half mirror 48 as a reflecting member that bends the image light emitted from the HMD 12 and guides it to the pupil 36. Not only the image light reflected from the half mirror 48 but also light (external light) representing the scenery of the outside world passes through the half mirror 48 and enters the observer's eye 34. As a result, the observer can observe the scenery of the outside world superimposed on the image (display image) displayed by the image light.

  In addition, in this embodiment, the HMD 12 is a monocular type, a spatial modulation type, and a see-through type. However, the HMD 12 can be changed to a binocular type, or can be changed to a retina scanning type, that is, an image signal from a light source such as a laser. By scanning a light beam having a corresponding intensity with a scanner and projecting the scanned light beam onto the observer's retina 38, the image can be changed to a type that allows the observer to observe the image as a virtual image, or displayed. In parallel with image observation, it is possible to change to a sealed type in which the actual outside world cannot be observed.

  As shown in FIG. 2, the HMD 12 further includes an audio signal output circuit 50. The audio signal output circuit 50 is an audio signal to an earphone 54 (an example of a device that outputs audio using a speaker close to the ear, and another example is a headphone) attached to the ear 52 of the observer. Output the signal by wire or wireless. The HMD 12 can reproduce video and audio together and output them to an observer (user).

  Next, the control unit 20 will be described. The control unit 20 is connected to the HMD 12 via a cable 60 as shown in FIG. The cable 60 includes a control line that supplies a control signal, a power line that supplies electric power, and a video line that supplies a video signal, although none of them is shown. While the HMD 12 is mounted on the observer's head, the control unit 20 is mounted on a portion other than the head (for example, the waist) of the observer.

  The control unit 20 is configured mainly with the computer as conceptually shown in the functional block diagram of FIG. Specifically, the control unit 20 includes a CPU (Central Processing Unit) 70 as a processor, a program ROM (Read Only Memory) 72 and a recording / playback memory 74 as nonvolatile memories, and a RAM ( Random Access Memory) 76. The recording / playback memory 74 is, for example, an HD (hard disk) mounted on a flash ROM or HDD (hard disk drive). The RAM 76 has a video RAM. The CPU 70, program ROM 72, recording / playback memory 74, and RAM 76 are connected to each other via a bus 78, whereby the computer is configured.

  The bus 78 further requests that an observer, who is a user, reproduce various requests (for example, continuous images (moving images, video composed of a series of frames) transmitted as content transmitted from the outside). An operation panel (for example, a key, a button, a touch screen) 90 as an operation unit operated by a user in order to input information to the control unit 20. ing. Further, a timer 92 for measuring time and a camera 94 (for example, a CCD (Charge-Coupled Device) camera) as an image sensor are connected to the bus 78.

  The camera 94 is attached to the HMD 12 or the frame 14 in order to capture an observation eye that should observe a display image by the HMD 12 among the eyes of the observer. Specifically, the camera 94 is an example of an eye state detection unit that detects the state of the eye 34 in order to detect the position of the pupil 36 (pupil position) or the line-of-sight direction of the eye 34 based on an image obtained by imaging the eye. It is provided as. However, the eye state detection unit is not limited to the camera 94, and may be configured mainly by a light receiving sensor that receives reflected light from the surface of the eye 34 and detects the intensity thereof, for example. Is possible.

  The bus 78 is further connected to an image signal input circuit 100 to which an image signal is input from the outside, an audio signal input circuit 102 to which an audio signal is input from the outside, and an image processing circuit 104. Data representing a continuous image (hereinafter simply referred to as “image”) to be reproduced is streamed to the image signal input circuit 100 from an external device (for example, a personal computer or a content server) connected wirelessly or by wire. Sent continuously. The image signal input circuit 100 includes an AD converter (not shown), thereby converting the input analog image signal into a digital image signal (image data) and outputting the digital image signal to the bus 78. In synchronization with the reception of the image signal, the audio signal input circuit 102 continuously receives data representing the audio to be reproduced from the same external device by streaming. Note that the image signal input to the image signal input circuit 100 may be a digital signal. In this case, an AD converter is not necessary.

  The image processing circuit 104 converts the image signal output from the image signal input circuit 100 into a light source drive signal and an LCD drive signal. The light source drive signal is sent to the light source drive circuit 40 and the LCD drive signal is sent to the LCD drive circuit 42, respectively. Supply. An audio signal output circuit 50 in the HMD 12 is also connected to the bus 78, and the audio signal output from the audio signal input circuit 102 is output to the earphone 54 via the audio signal output circuit 50.

  FIG. 3 conceptually shows a part related to image reproduction control in the control unit 20 in a functional block diagram. The control unit 20 includes a normal reproduction unit 110 that performs normal reproduction for reproducing an input image represented by the image signal output from the image signal input circuit 100, thereby forming a display image. The normal reproduction unit 110 performs real-time reproduction for reproducing the input image in real time. The control unit 20 further performs a recording / reproducing for recording an input image in the recording / reproducing memory 74 and a recording / reproducing for reading out and reproducing the image data from the recording / reproducing memory 74, thereby forming a display image. And a recording / playback unit 114. The normal playback unit 110 and the recording / playback unit 114 supply respective image signals (image data) to the HMD 12 via the image processing circuit 104.

  As shown in FIG. 3, the control unit 20 further determines whether the image light representing the display image is not incident on the observer's eye 34 based on the image of the eye 34 captured by the camera 94. A determination unit 118 is provided for determining whether or not.

  Specifically, the determination unit 118 can be designed to determine that the image light is not incident when at least one of the following four conditions is satisfied.

(A) First condition: the HMD 12 normally emits the image light, but the observer's eye 34 receives the image light (for example, the position of the pupil 36 matches the position of the exit pupil of the HMD 12) Or the line-of-sight direction of the eye 34 coincides with the traveling direction of the image light, or the eyelid of the observer's eye 34 is not open)

(B) Second condition: the HMD 12 itself is normal but cannot emit the image light (for example, the HMD 12 is calibrated such as brightness adjustment to compensate for a light source change with time immediately after power-on. The image light cannot be emitted because it is in the stage of

(C) Third condition: the HMD 12 is normal in itself, but it is inappropriate for the observer to receive the image light (for example, the observer observes the display image for a longer period of time). If this continues, there is a possibility that the eye 34 will be overworked), so that the HMD 12 automatically stops emitting the image light (stops the light source).

(D) Fourth condition: a condition that the image light cannot be emitted because the HMD 12 itself is abnormal (for example, the light source has failed).

  However, in the present embodiment, the determination unit 118 determines the position of the pupil 36 of the observer with respect to the image light (that is, the position of the exit pupil of the HMD 12) based on the image of the eye 34 captured by the camera 94. Relative estimation is based on the estimated pupil position, and the observer's eye does not receive the image light (that is, the position of the pupil 36 matches the position of the exit pupil of the HMD 12). Designed to determine whether or not. As a method for estimating the pupil position from the image of the eye 34, for example, as disclosed in Japanese Patent Application Laid-Open No. 2008-6149, the image of the eye 34 is binarized and a circular region in the low luminance region is converted into the pupil. It is possible to adopt a method of extracting as 36 images and estimating the pupil position from the extracted image of the pupil 36.

  Instead, the determination unit 118 estimates the line-of-sight direction of the observer's eye 34 relative to the image light based on the image of the eye 34 captured by the camera 94, and the estimated line-of-sight direction Based on the above, it is designed to determine whether or not the observer's eye 34 is in a state of receiving the image light (a state in which the line-of-sight direction of the eye 34 matches the traveling direction of the image light). Is possible. As a method for estimating the line of sight from the image of the eye 34, for example, as disclosed in Japanese Patent Application Laid-Open No. 2007-136000, the reflected light generated from the corneal surface of the eye 34 when the eye 34 is irradiated with infrared rays is used. It is possible to employ a method of estimating the line of sight from the distance between the position of the represented image and the position of the pupil 36.

  The success or failure of the second condition is performed based on a signal from a detection unit that detects the operating state of the HMD 12 (particularly, the image light generation unit). The success or failure of the third condition is determined by an observer. This is performed based on a signal from a unit that detects a state in which the HMD 12 is used (for example, continuous use time). The success or failure of the fourth condition is determined from a detection unit that detects the operating state of the HMD 12 (particularly, the light source). It is possible to carry out based on the signal.

  Based on the determination result of the determination unit 118 described above, the operation states of the normal reproduction unit 110, the recording unit 112, and the recording / reproduction unit 114 change.

  As shown in FIG. 3, the control unit 20 further includes a first erasing unit 120. The first erasing unit 120 erases the reproduced image from the recording / reproducing memory 74 when the recording / reproducing unit 110 reads out and reproduces the image from the recording / reproducing memory 74.

  In FIG. 4, during the period from the reproduction start time t1 to the reproduction end time t5, the determination unit 118 determines that the image light is not incident (hereinafter simply referred to as “non-incident determination”) only once. Under the hypothetical scenario that gives priority to the convenience of explanation, the time of the operating state of each of the normal playback unit 110, the recording unit 112, the recording / playback unit 114, the determination unit 118, and the first erasing unit 120 The change is illustrated in the time chart.

  As shown in FIG. 4, the normal reproduction unit 110 performs normal reproduction (in this embodiment, the input image is reproduced in real time) from the reproduction start time t1. The normal playback unit 110 stops the normal playback from the start time t2 of the non-incidence determination by the determination unit 118. The recording unit 112 records the input image output from the image signal input circuit 100 in the recording / reproducing memory 74 from the non-incident determination start time t2 (equivalent to the time when the normal reproduction is stopped). The recording unit 112 seamlessly continues to record the input image output from the image signal input circuit 100 in the recording / playback memory 74 even after the non-incident determination end time t3. The recording / playback unit 114 reads the recorded image from the recording / playback memory 74 and performs recording / playback from the end point t3 of the non-incident determination. When the recording / playback unit 110 reads an image from the recording / playback memory 74 and plays it back, the first erasing unit 120 immediately erases the played image from the recording / playback memory 74.

  Therefore, according to the present embodiment, since the image light is in the period in which the light does not enter the eye 34 (hereinafter referred to as “non-incidence period”), the image is not observed by the observer even if the image is reproduced. When the reproduction of the image is stopped and the non-incidence period ends, the reproduced image is reproduced in an environment where the image is reproduced by the observer if the image is reproduced. When the reproduction of the stopped image is finished, the image recorded in the recording / reproducing memory 74 from the non-incident determination end time t3 is seamlessly reproduced with respect to the preceding image reproduction.

  The storage capacity of the recording / playback memory 74 is ideally infinite, but in reality is finite. Therefore, as long as the length of the time interval between the start time t2 and the end time t3 of the non-incident determination does not exceed the storable time T1 of the recording / playback memory 74, the recording / playback is performed without omission of the input image. It is possible.

  In contrast to FIG. 4, FIG. 5 shows a hypothetical scenario that is more realistic than FIG. 4, in which the non-incidence determination is performed twice from the playback start time t 1 to the playback end time t 7. The temporal changes in the operating states of the normal playback unit 110, the recording unit 112, the recording / playback unit 114, the determination unit 118, and the first erasing unit 120 are illustrated in the time chart.

  FIG. 6 conceptually shows a program executed by the CPU 70 of the control unit 20 in order to execute the above-described image reproduction control.

  When the user turns on the power (not shown) of the image display device 10 in step S1, the execution of this program is started. In this program, first, a predetermined standby process is performed in step S2, and then in step S3, the user waits for a reproduction request to be input by operating the operation panel 90. When the user inputs a reproduction request, the determination in step S3 is YES, and in step S4, the observer's eye 34 is imaged by the camera 94, and imaging data is generated.

  Subsequently, in step S5, based on the generated imaging data, it is determined whether the image light is not incident as described above.

  If it is assumed that the image light is not incident at this time, the determination in step S5 is NO. In step S6, whether or not image data to be reproduced, that is, recorded data, is present in the recording / reproducing memory 74. Is determined. Assuming that such recording data does not exist in the recording / reproducing memory 74 this time, normal reproduction is performed in step S7, and the input image is reproduced in real time. Thereafter, the process proceeds to step S8.

  In this step S8, it is determined whether or not the reproduction request from the user has ended, that is, whether or not the user has requested the end of reproduction. If it is assumed that the reproduction request has not been completed this time, there is no image data to be reproduced in step S9 (that is, a new input image and unreproduced image data recorded in the recording / reproducing memory 74 are not present). Is not present). If it is assumed that there is image data to be reproduced this time, the process returns to step S4.

  Thereafter, when the non-incident period is entered, the determination in step S5 is YES during execution of this program, and it is determined in step S10 whether an image is being input. If the image is not being input, the process returns to step S4. If the image is being input, in step S11, the input image output from the image signal input circuit 100 is recorded in the recording / playback memory 74 in real time. Recording is performed. Subsequently, in step S12, the reproduction of the image is stopped regardless of whether the reproduction is normal reproduction or recording reproduction. Thereafter, the process proceeds to step S8.

  Thereafter, when the current non-incidence period ends, during the execution of this program, the determination in step S5 becomes NO, and in step S6, it is determined whether or not the recording data exists in the recording / playback memory 74. This time, since recorded data exists, the determination in step S6 is YES. Subsequently, in step S13, recording / playback is performed, and recorded images are read out from the recording / playback memory 74 in order from the oldest and played back. Thereafter, the reproduced image data is erased from the recording / reproducing memory 74 in step S14. In step S15, a new input image output from the image signal input circuit 100 is recorded in the recording / playback memory 74 in real time. In FIG. 6, for the sake of convenience of representation in the flowchart, this step S15 is represented to be performed after execution of step S13. However, the recording in step S15 is substantially the same as the recording and reproduction in step S13. In parallel. Thereafter, the process proceeds to step S8.

  In step S16, if the reproduction request is completed, the determination in step S8 is YES, or if the reproduction request is not completed but there is no image data to be reproduced, the determination in step S9 is YES. It is determined whether or not the power of the image display device 10 is OFF (that is, the user has turned off the power of the image display device 10). If it is assumed that the power supply is not OFF this time, the process returns to step S3. However, if it is assumed that the power supply is OFF this time, a predetermined end process is performed for each component of the image display device 10 in step S17. Done. Thereafter, in step S18, the execution of this program ends.

  As is clear from the above description, in the present embodiment, the part of the control unit 20 that executes steps S4 and S5 shown in FIG. 6 constitutes the determination unit 118 shown in FIG. Among them, the part that executes steps S6, S7, S10, and S12 constitutes the normal playback unit 110, and the part that constitutes steps S6, S15, S10, and S11 of the control unit 20 constitutes the recording unit 112. The part of the control unit 20 that executes steps S6 and S13 constitutes the recording / reproducing unit 114, and the part of the control unit 20 that executes step S14 is the first erasing unit 118 (the “recording / reproducing unit”). An example of a “memory erasing unit” is configured. Further, the image signal input circuit 100 constitutes an example of the “input unit” in the item (1).

  As described above, in the present embodiment, in an environment where images to be played are continuously input to the control unit 20 by streaming, real-time playback of the input image is stopped while the non-incident determination is continued, and real-time playback is stopped during that time. Although it has been described that the recorded input image is recorded and the recorded image is reproduced after the non-incidence determination is completed, such reproduction processing is performed not only for the image but also for the sound. As a result, in the present embodiment, the reproduction of the image resumed after the end of the non-incident determination is performed in synchronization with the sound.

  Next, an image display apparatus 200 according to the second embodiment of the present invention will be described. However, since this embodiment has many elements in common with the first embodiment, the common elements are referred to with the same reference numerals or names, thereby omitting redundant descriptions and only different elements. This will be described in detail.

  In the first embodiment, as shown in FIG. 6, in step S <b> 4, the observer's eye 34 is imaged by the camera 94 which is an example of an eye state detection unit, and then in step S <b> 5 based on the imaging result. It is determined whether or not the image light is not incident (for example, whether or not the position of the pupil 36 does not coincide with the position of the exit pupil of the HMD 12). In order to ensure the accuracy of the determination, from the actual start of the non-incidence period to the start of the determination that the image light is not incident in step S5, FIG. 9 illustrates a time chart. As you can see, there is a time lag. For this reason, during the time lag period, even if the non-incident period is actually started, the reproduction of the image is not stopped.

On the other hand, in the image display device 200 according to the present embodiment, as shown in FIG. 7, a delay memory 202 (temporarily storing image data in real time with a length longer than the normal time length of the time lag is shown. Is added to the image display apparatus 10 shown in FIG. Furthermore, in this embodiment, from the end time t3 of the non-incidence determination, first, image data is read from the delay memory 202 and first recording / playback is performed. When the delay memory 202 becomes empty, this time, second recording and reproducing from the recording reproduction memory 74 reads the image data is carried out in seamless so as to be continuous to the first recording reproduction.

  Therefore, according to the present embodiment, the same image reproduced during the time lag is reproduced again after the non-incident determination is completed, that is, after the non-incident period is completely completed. Is done. Therefore, even if the observer cannot normally observe the image reproduced during the time lag period, the observer can normally observe the same image after the non-incident period ends.

  In FIG. 7, the configuration of the image display device 200 according to the present embodiment is represented by a functional block diagram in the same manner as the image display device 10 illustrated in FIG. 2. As described above, the image display device 200 additionally includes the delay memory 202 with respect to the image display device 10, and further includes the temporary storage unit 212 and the second erasing unit 216 as additional components. ing.

  The delay memory 202 is used so as to function as a FIFO-type image data buffer, and has a capacity for storing the latest image data continuous for a storable time (maximum storage time) T2. As a result, the delay memory 202 temporarily stores the image data input in the past time zone by the same time as the length of the storable time T2 from the current time while sequentially updating the image data. Usually, since the length of the time lag is shorter than the length of time from the start time t2 to the end time t3 of the non-incident determination, the storable time T2 of the delay memory 202 is the storable time T1 of the recording / playback memory 74. It can be shorter.

  The temporary storage unit 212 stores the input image output from the image signal input circuit 100 in the delay memory 202 in real time. The second erasing unit 216 erases the image read from the delay memory 202 and reproduced from the delay memory 202 in real time.

  8, the configuration of the control unit 20 in the image display apparatus 200 is represented by a functional block diagram as in FIG. The image display apparatus 200 includes a recording / playback unit 220 that replaces the recording / playback unit 114. The recording / playback unit 220 performs the first and second recording / playback operations described above.

  FIG. 9 is a time chart showing a state in which it is assumed that the image reproduction control according to the present embodiment is performed under the same scenario as in FIG.

  As shown in FIG. 9, the normal playback unit 110 reads out an externally input image from the RAM 76 and plays back it in the same manner as in FIG. On the other hand, the recording / playback unit 220 performs the first recording / playback from the end time t3 of the non-incident determination, and then performs the second recording / playback. In the first recording / playback, the recording / playback unit 200 is an image temporarily stored in the delay memory 202, that is, a first time point that has returned to the past by the same time as the storable time T2 from the non-incident determination start time t2. To the second time point that is equal to the non-incidence determination start time point t2 are temporarily stored in the delay memory 202 (all images stored in the delay memory 202) from the delay memory 202 in order from the oldest. Read and play. On the other hand, in the second recording / playback, the recording / playback unit 220 reads and plays back images from the recording / playback memory 74 in order from the oldest.

  FIG. 10 conceptually shows a program executed by the CPU 70 of the control unit 20 in order to execute the above-described image reproduction control.

  In this program, steps S101 to S106 are executed in the same manner as steps S1 to S6 shown in FIG. 6, and steps S111, S112 and S123 to S125 are executed in steps S8, S9 and S16 to S18 shown in FIG. It is executed in the same way.

  In the program shown in FIG. 10, if it is assumed that the non-incident period has not started, the determination in step S105 is NO, and then, in step S106, whether or not recording data exists in the recording / playback memory 74 is determined. Determined.

  This time, assuming that such recording data does not exist in the recording / playback memory 74, the input image output from the image signal input circuit 100 is temporarily stored in the delay memory 202 in real time in step S107. In step S108, if there is image data overflowing from the delay memory 202 due to the temporary storage, the image data is deleted. By cooperating these steps S107 and S108, the delay memory 202 temporarily stores image data for a time not exceeding the storable time T2. Thereafter, in step S109, the input image output from the image signal output circuit 100 is read from the RAM 76 in real time and is normally reproduced.

  Thereafter, when the non-incident period is started, the determination in step S105 is YES during execution of this program, and in step S113, it is determined whether an image is being input. If the image is not being input, the process returns to step S104. If the image is being input, the input image output from the image signal output circuit 100 is temporarily stored in the delay memory 202 in step S114. In step S115, the input image output from the image signal output circuit 100 is recorded in the recording / reproducing memory 74 and recorded. Thereafter, in step S116, the reproduction of the image is stopped regardless of whether the reproduction is normal reproduction or recording reproduction. Then, it progresses to step S111.

  Thereafter, when the current non-incidence period ends, the determination in step S105 is NO during execution of this program, and the determination in step S106 is YES because the recording data exists in the recording / playback memory 74 this time. Subsequently, in step S117, it is determined whether or not image data temporarily stored in the delay memory 202, that is, delay data exists in the delay memory 202. Since there is delay data this time, in step S118, images are read from the delay memory 202 in order from the oldest and recorded and reproduced. The first recording / playback is performed. Thereafter, the reproduced image is deleted from the delay memory 202 in step S119. As a result of repeating the execution of steps S118 and S119, if the delay memory 202 becomes empty, the determination in step S117 is NO.

  When the delay memory 202 becomes empty, in step S120, images are read out from the recording / reproducing memory 74 in order from the oldest and recorded / reproduced. The second recording / playback is performed. Thereafter, in step S121, the reproduced image is erased from the recording / reproducing memory 7. In step S122, the new input image output from the image signal input circuit 100 is recorded in the recording / playback memory 74 in real time. In FIG. 10, for the convenience of expression by a flowchart, this step S122 is expressed so as to be performed after the execution of step S120. In parallel. Then, it progresses to step S111.

  As is clear from the above description, in the present embodiment, the part of the control unit 20 that executes steps S104 and S105 shown in FIG. 10 constitutes the determination unit 118 shown in FIG. Of these, the part that executes steps S106, S109, S117, S120, S113, and S116 constitutes the normal playback unit 110, and the part that executes steps S117, S122, S113, and S115 of the control unit 20 is the recording unit. 112, the portion constituting steps S106 and S117-S120 in the control unit 20 constitutes the recording / playback unit 220, and the portion executing step S122 in the control unit 20 is the first erasure unit 118. (An example of the “recording / playback memory erasure unit”) and a control unit 0, the part that executes steps S108 and S119 constitutes a second erasure unit 216 (an example of the “delay unit for delay memory”), and among the control unit 20, steps S106, S107, S113, and S114 are executed. The part to be executed constitutes the temporary storage unit 212.

  In addition, in this embodiment, in the present embodiment, the delay memory 202 and the second erasing unit 216 (step S108 shown in FIG. 10) configured by software have been received from the outside within a certain period of time from the present time. An image data buffer for temporarily storing the updated image data while being continuously updated is configured. However, the image data buffer is not limited thereto, and may be configured as, for example, a shift register configured by hardware. Is possible. In this case, it is easy to simplify the software configuration without complicating the hardware configuration of the control unit 20.

  Next, an image display apparatus 200 according to the third embodiment of the present invention will be described. However, since this embodiment has many elements in common with the second embodiment, the common elements are referred to with the same reference numerals or names, so that duplicate descriptions are omitted and only different elements are cited. This will be described in detail.

  In the second embodiment, as shown by the time chart in FIG. 9, the input image is read from the RAM 76 and reproduced in real time from the reproduction start time t1. On the other hand, in the present embodiment, as shown in the time chart of FIG. 11 in the same manner as in FIG. 9, the image is reproduced from the reproduction start time t1 to the time t1 ′ when the delay memory 202 is full of image data. Start is delayed. When the time point t1 ′ arrives, the image data is read out from the delay memory 202 in order from the oldest one and played back, so that not the real-time playback but the delayed playback that plays back the input image at a timing later than the timing at which the image is input. Is done.

  FIG. 12 conceptually shows a program executed by the CPU 70 of the control unit 20 for image reproduction control in the image display device 200 according to the present embodiment in the same manner as in FIG. . Hereinafter, this program will be described, but only the steps different from the program shown in FIG. 10 will be described in detail, and redundant description of common steps will be omitted.

When the determination in step S106 is NO, the input image output from the image signal input circuit 100 is temporarily stored in the delay memory 202 in real time in step S207. Subsequently, in step S208, it is awaited that the same time as the storable time T2 of the delay memory 202 elapses from the reproduction start time t1. It waits for the delay memory 202 to be filled with image data. When the time T2 has elapsed and the delay memory 202 is full of image data, the oldest image data is read from the delay memory 202 and reproduced in step S209. At the timing when an image data is delayed by the time T2 from the time stored in the delay memory 202, the image data is reproduced, whereby it is the delay reproduction. Thereafter, the image data reproduced this time is erased from the delay memory 202 in step S210. Then, it progresses to step S111.

  As is clear from the above description, in the present embodiment, the portion of the control unit 20 that executes steps S106, S208, S209, S117, S120, S113, and S116 constitutes the normal playback unit 110, and is controlled. A portion of the unit 20 that executes steps S210 and S119 constitutes a second erasing unit 216 (an example of the “delayed part for delay memory”), and among the control unit 20, steps S106, S207, S113, and S114. The temporary storage unit 212 is configured to execute the above.

  As described above, some of the embodiments of the present invention have been described in detail with reference to the drawings. However, these are merely examples, and the knowledge described by those skilled in the art including the aspects described in the above [Summary of Invention] section. The present invention can be implemented in other forms based on various modifications and improvements.

Claims (6)

An image display device having a head-mounted display (HMD) that displays an image corresponding to an image signal to a viewer as a display image that is a virtual image,
An input unit to which the image signal is input;
A normal reproduction unit for forming the display image by normal reproduction for reproducing an input image represented by an image signal output from the input unit;
Recording and playback memory,
A recording unit for recording the input image in the recording / playback memory;
A recording / playback unit for forming the display image by recording / playback for reading out and playing back the input image from the recording / playback memory;
A determination unit that determines whether or not image light representing the display image is not incident on an observer's eye, and
The normal reproduction unit stops the normal reproduction at least for a period from the start time to the end time of the determination that the image light is not incident,
The recording unit records the input image output from the input unit in the recording / playback memory at least from the start of the determination that the image light is not incident,
The recording and reproducing unit, have row the recording and reproduction from the end, the recorded the input image on the recording reproduction memory from oldest reading in the order of determination of the a time of image light not incident,
The image display device further includes:
A delay memory having a storable time T2 shorter than a storable time T1 capable of storing an image in the recording / playback memory;
A temporary storage unit that stores the input image output from the input unit in the delay memory in real time;
A delay memory erasure unit for erasing an image read from the delay memory and reproduced from the delay memory;
Including
Thereby, in the delay memory, the input image output from the input unit from the time point when returning from the present time to the past by the same length as the storable time T2 is stored.
The recording / playback unit, from the end of the determination that the image light is not incident, sequentially reads the images from the delay memory in order from the oldest and performs the first recording / playback, and the delay memory becomes empty An image display device that reads out images from the recording / playback memory in order from the oldest and performs the second recording / playback continuously with the first recording / playback . An image display device having a head-mounted display (HMD) that displays an image corresponding to an image signal to a viewer as a display image that is a virtual image,
An input unit to which the image signal is input;
A normal reproduction unit for forming the display image by normal reproduction for reproducing an input image represented by an image signal output from the input unit;
Recording and playback memory,
A recording unit for recording the input image in the recording / playback memory;
A recording / playback unit for forming the display image by recording / playback for reading out and playing back the input image from the recording / playback memory;
A determination unit that determines whether or not image light representing the display image is not incident on an observer's eye;
Including
The normal reproduction unit stops the normal reproduction at least for a period from the start time to the end time of the determination that the image light is not incident,
The recording unit records the input image output from the input unit in the recording / playback memory at least from the start of the determination that the image light is not incident,
The recording / playback unit reads the input images recorded in the recording / playback memory in order from the oldest, and performs the recording / playback from the end of the determination that the image light is not incident.
The determination unit
The first condition that the HMD normally emits the image light but the observer's eye is not in a state of receiving the image light;
A second condition that the HMD itself is normal but cannot emit the image light;
A third condition that the HMD is normal in itself, but the HMD stops emitting the image light because it is inappropriate for the observer to receive the image light;
A fourth condition that the HMD cannot emit the image light because the HMD itself is abnormal;
An image display device that determines that the image light is not incident when at least one of them is established . The normal reproduction unit, according to claim 1, the input image from the delay memory, from the storage point to the delay reproduction memory, performs delay reproduction of reading out and reproducing at a timing delayed by the same time as the storage available time T2 Or the image display apparatus of 2.   4. The recording / reproducing memory erasure unit further comprising: an erasing unit for recording / reproducing memory for erasing the reproduced image from the recording / reproducing memory in order from the oldest when the image is reproduced by the recording / reproducing unit. The image display device described. In addition, it includes a camera that images the observer's eyes,
The determination unit estimates an observer's line-of-sight direction based on an image captured by the camera, and based on the estimated line-of-sight direction, whether the observer's eye is in a state of receiving the image light. claims 1 determines to image display apparatus according to any one of 4. In addition, it includes a camera that images the observer's eyes,
The determination unit estimates an observer's pupil position based on an image captured by the camera, and based on the estimated pupil position, whether the observer's eye is in a state of receiving the image light. claims 1 determines to image display apparatus according to any one of 4.

Images