JP2018025942A - Head-mounted display device and method for controlling head-mounted display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology to dynamically reduce power consumption according to the state of use of a head-mounted display device.SOLUTION: A head-mounted display device has an image processing part mounted therein including a plurality of image processing circuits, determines whether the position and/or attitude of the head-mounted display devices is changing, and acquires the temperature inside the head-mounted display device. The head-mounted display device sequentially stops the respective operations of the plurality of image processing circuits in order according to a result of the determination until the temperature falls below a specified temperature.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a power consumption control technique in a head-mounted display device.

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology is known as a technology for seamlessly combining the real world and the virtual world in real time. As one of MR techniques, the following technique using a video see-through HMD (Head Mounted Display) is known. That is, a subject that substantially matches the subject observed from the pupil position of the HMD wearer is imaged by a video camera or the like, and an image in which CG (Computer Graphics) is superimposed on the captured image is presented to the HMD wearer.

  Image processing of a captured image or an image in which CG is superimposed and displayed in such a system is performed by mounting a large-scale LSI such as FPGA or ASIC on the HMD. The LSI is equipped with a plurality of image processing functions for the purpose of achieving high image quality and high-precision alignment of CG. In such a case, the power consumption of the LSI becomes large, and there is a concern that the temperature of the HMD increases due to the temperature increase of the LSI.

Japanese Patent Application No. 2006-1955025

  According to the invention described in Patent Document 1, the degree of progress is calculated for each process based on the operation rate of the functional blocks mounted on the LSI and a preset end time for each process. If the end times are the same, the priority is determined to be low in the order of increasing progress, and if the end times are different, the end times are in descending order. The clock supplied from the functional block with low priority is stopped. LSI power consumption is reduced by stopping the clock.

  However, in the MR technique, it is necessary to perform high-precision alignment calculation on a captured image obtained by capturing a subject with a video camera or the like, and to superimpose a CG and display it on the HMD. For this reason, if image processing clock stop is performed according to the priority order based on the operation rate and the end time, there is a concern about image quality deterioration and position accuracy deterioration.

  The present invention has been made in view of such problems, and provides a technique for dynamically reducing power consumption in accordance with the usage state of a head-mounted display device.

  One aspect of the present invention is a head-mounted display device on which an image processing unit having a plurality of image processing circuits is mounted, and whether the position and / or posture of the head-mounted display device is changing. Determination means for determining whether or not, acquisition means for acquiring the temperature in the head-mounted display device, and each operation of the plurality of image processing circuits until the temperature is lower than a specified temperature, the determination Control means for sequentially stopping in order according to the determination result by the means.

  According to the configuration of the present invention, the power consumption is dynamically reduced according to the usage state of the head-mounted display device, so that deterioration of image quality and deterioration of the alignment accuracy of the virtual object can be suppressed as much as possible. .

The block diagram which shows the structural example of a system. FIG. 2 is a block diagram illustrating a configuration example of an image processing unit 110. FIG. 3 is a block diagram illustrating a configuration example of an image processing circuit. The figure which shows the structural example of the clock control part 1100d. The figure which shows the waveform of each signal in the clock control part 1100d. 5 is a flowchart showing the operation of the head-mounted display device 100. The block diagram which shows the structural example of a system. 5 is a flowchart showing the operation of the head-mounted display device 100.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In the present embodiment, an example of a system using a head-mounted display device having the following configuration will be described. That is, this head-mounted display device is a head-mounted display device on which an image processing unit having a plurality of image processing circuits is mounted, and the position and / or posture of the head-mounted display device is changing. And the temperature in the head-mounted display device is acquired. Then, the operation of each of the plurality of image processing circuits is sequentially stopped in the order corresponding to the determination result of the determination until the temperature becomes lower than the specified temperature (control).

  First, a configuration example of a system according to the present embodiment will be described with reference to the block diagram of FIG. The system according to the present embodiment is an example of a so-called MR system. That is, an image of a real space according to the position and orientation of the viewpoint of the user wearing the head-mounted display device on the head, and an image of a virtual space (a space composed of virtual objects) according to the position and orientation of the viewpoint This is an example of an MR system that presents the synthesized video obtained by synthesizing these to the user.

  The marker imaging unit 103 and the marker imaging unit 104 are imaging devices for imaging a marker (hereinafter referred to as an MR marker) arranged at a location corresponding to the virtual object arrangement position in the real space. Mount. The video signal from the marker imaging unit 103 is input to the image processing unit 110a, and the video signal from the marker imaging unit 104 is input to the image processing unit 110d.

  The right-eye imaging unit 101 is an imaging device that captures an image of a real space provided to the right eye of a user wearing the head-mounted display device 100 on the head, and includes a CMOS image sensor. The video signal from the right eye imaging unit 101 is output to the image processing unit 110b.

  The left-eye imaging unit 102 is an imaging device that captures an image of a real space provided to the left eye of a user who wears the head-mounted display device 100 on the head, and includes a CMOS image sensor. The video signal from the left eye imaging unit 102 is output to the image processing unit 110c.

  Here, each of the marker imaging unit 103, the marker imaging unit 104, the right eye imaging unit 101, and the left eye imaging unit 102 may use the same image sensor or different image sensors. For example, as an image sensor used in the marker imaging unit 103 and the marker imaging unit 104, an image sensor having a lower resolution than that used in the right eye imaging unit 101 and the left eye imaging unit 102 may be used. In addition, each of the marker imaging unit 103, the marker imaging unit 104, the right eye imaging unit 101, and the left eye imaging unit 102 may perform imaging at the same frame rate, or may perform imaging at different frame rates. Also good. For example, the marker imaging unit 103 and the marker imaging unit 104 may perform imaging at a higher frame rate than the right eye imaging unit 101 and the left eye imaging unit 102.

  The head position detection unit 105 is a sensor that detects its own position and orientation, and data representing the detected position and orientation is output to the CPU 111. Various sensors such as a gyro sensor, an acceleration sensor, and a geomagnetic sensor can be used as sensors for detecting their own position and orientation.

  The right-eye display unit 106 is provided in the head-mounted display device 100 so as to be positioned in front of the right eye of the user wearing the head-mounted display device 100 on his / her head. The video based on the video signal for the right eye output from the mold display device 100 is displayed.

  The left-eye display unit 107 is provided on the head-mounted display device 100 so as to be positioned in front of the left eye of the user wearing the head-mounted display device 100 on his / her head. The video based on the video signal for the left eye output from the mold display device 100 is displayed. The right eye display unit 106 and the left eye display unit 107 are configured by, for example, an organic EL panel or a liquid crystal panel.

  In FIG. 1, the marker imaging unit 103, the marker imaging unit 104, the right eye imaging unit 101, the left eye imaging unit 102, the head position detection unit 105, the right eye display unit 106, and the left eye display unit 107 are each mounted on the head. This is an external device of the mold display device 100. However, some or all of these may be housed in the head-mounted display device 100.

  Next, the head-mounted display device 100 will be described. Each of the image processing units 110a to 100d includes a plurality of image processing circuits, performs various kinds of image processing on the input video signal, and outputs an image-processed video signal. Since all of the image processing units 110a to 100d differ only in the input source of the video signal and the operation for the video signal is the same, in the following description common to the image processing units 110a to 100d, these are collectively displayed. This is referred to as a processing unit 110. That is, the description related to the image processing unit 110 can be similarly applied to the image processing units 110a to 100d. A configuration example of the image processing unit 110 is shown in FIG.

  The pixel defect correction circuit 1100 uses a pixel of the same type as the defective pixel around the defective pixel (for example, assuming that the defective pixel is R) as a defective pixel in the image represented by the input video signal. Is corrected using the pixel value of (R pixel). The density conversion circuit 1101 performs density conversion processing for adjusting the offset and gain on the video signal output from the pixel defect correction circuit 1100.

  A shading correction circuit 1102 performs shading correction on the video signal output from the density conversion circuit 1101. In this shading correction, for example, the following processing is performed. In other words, the shading in which the peripheral light amount decreases and the peripheral portion of the image becomes dark due to the influence of the lens mounted on the imaging device (marker imaging unit 103, marker imaging unit 104, right eye imaging unit 101, left eye imaging unit 102). On the other hand, the pixel value is corrected with a coefficient corresponding to the pixel position.

  The AE / WB correction circuit 1103 performs AE (automatic exposure) correction and WB (white balance) correction on the video signal from the shading correction circuit 1102. The noise reduction circuit 1104 performs noise reduction processing on the video signal from the AE / WB correction circuit 1103 to reduce noise in an image represented by the video signal by digital processing.

  The Bayer conversion circuit 1105 is a video signal output from the noise reduction circuit 1104, that is, a video signal representing an image of a Bayer structure (a structure in which four color filters (R / Gr / Gb / B) are arranged in a checkered pattern). To generate a video signal representing an RGB image by a known interpolation process. An RGB image is an image in which the pixel value of each pixel constituting the image is composed of a pixel value of an R (red) component, a pixel value of a G (green) component, and a pixel value of a B (blue) component.

  The color correction circuit 1106 performs hue correction on the video signal from the Bayer conversion circuit 1105 by performing known matrix processing. The contour emphasizing circuit 1107 performs processing for detecting an edge from the image represented by the video signal and enhancing the detected edge with respect to the video signal from the color correction circuit 1106. The γ correction circuit 1108 performs γ conversion based on an arbitrarily settable γ table for the video signal from the contour enhancement circuit 1107.

  In FIG. 2, the number and types of image processing circuits constituting the image processing unit 110 are not limited to those shown in FIG. Further, two or more image processing circuits shown in FIG. 2 may be combined into one image processing circuit. One or more of the functional units shown as the image processing circuit may be realized by software. In this case, this software is executed by the processor in the image processing unit 110 or the CPU 111 to realize a corresponding function.

  The multiplexing unit 112a time-division-multiplexes the video signal output from the image processing unit 110a, the video signal output from the image processing unit 110b, and the communication data output from the CPU 111 as a single transfer signal, and It outputs to 113a. Regarding the configuration of how the video signal output from the image processing unit 110a, the video signal output from the image processing unit 110b, and the communication data output from the CPU 111 are output to the transmission unit 113a. It is not limited to a specific configuration. The transmission unit 113a transfers the transfer signal received from the multiplexing unit 112a to the video composition unit 300 via the transmission unit 200.

  The multiplexing unit 112b time-division-multiplexes the video signal output from the image processing unit 110c, the video signal output from the image processing unit 110d, and the communication data output from the CPU 111 as one transfer signal, and To 113b. Regarding the configuration of how the video signal output from the image processing unit 110c, the video signal output from the image processing unit 110d, and the communication data output from the CPU 111 are output to the transmission unit 113b. It is not limited to a specific configuration. The transmission unit 113b transfers the transfer signal received from the multiplexing unit 112b to the video composition unit 300 via the transmission unit 200.

  Note that the video signals from each of the image processing units 110 a to 100 d may be multiplexed with the communication data output from the CPU 111 and transferred to the transmission unit 200.

  The temperature detection unit 116 measures the temperature inside the head-mounted display device 100 and notifies the CPU 111 of temperature data indicating the measured temperature. The CPU 111 executes processing using a computer program and data stored in its own memory. As a result, the CPU 111 controls the operation of the entire head-mounted display device 100 and executes or controls each process described later as performed by the head-mounted display device 100. For example, if the temperature represented by the temperature data from the temperature detection unit 116 is equal to or higher than a specified temperature, the CPU 111 determines whether the head-mounted display device 100 based on the position and orientation from the head position detection unit 105 is used. Stop control of a plurality of image processing circuits included in the image processing unit 110 is performed. In addition, the CPU 111 sends data representing the position and orientation measured by the head position detection unit 105 to the multiplexing units 112a and 112b as communication data. The communication data may include other data. For example, data such as voice data, GPS information, and temperature / humidity information may be included in the communication data.

  The receiving unit 114 receives the right-eye video sent from the video synthesis unit 300 and sends the received right-eye video to the right-eye display unit 106. The receiving unit 115 receives the left-eye video sent from the video synthesizing unit 300 and sends the received left-eye video to the left-eye display unit 107.

  The transmission units 113a and 113b, the reception unit 114, and the reception unit 115 function as communication interfaces for performing data communication with the video composition unit 300 via the transmission unit 200. For example, an interface corresponding to communication such as PCI-express, LAN, and other high-speed serial communication is applicable to the transmission units 113a and 113b, the reception unit 114, and the reception unit 115. Further, the transmission units 113a and 113b, the reception unit 114, and the reception unit 115 may be combined into one communication interface.

  Next, the video composition unit 300 will be described. Based on the “position and orientation measured by the head position detection unit 105” included in the communication data received via the transmission unit 200, the video composition unit 300 performs the operations of the right eye imaging unit 101 and the left eye imaging unit 102. Each position and orientation is calculated. For example, by adding information representing the positional relationship between the head position detection unit 105 and the right eye imaging unit 101 to the position and orientation measured by the head position detection unit 105, the position and orientation of the right eye imaging unit 101 can be changed. Can be calculated. Similarly, the position and orientation of the left eye imaging unit 102 is added by adding information representing the positional relationship between the head position detection unit 105 and the left eye imaging unit 102 to the position and orientation measured by the head position detection unit 105. Can be calculated. Of course, there are various known techniques for calculating (or acquiring) the respective positions and orientations of the left eye imaging unit 102 and the right eye imaging unit 101, and the configuration is not limited to a specific configuration. The video synthesis unit 300 then synthesizes a video in the virtual space that can be seen from the viewpoint having the position and orientation of the right eye imaging unit 101 with the video in the real space captured by the right eye imaging unit 101 (synthesized video for the right eye). Is generated. Similarly, the video composition unit 300 synthesizes a virtual space image that can be seen from the viewpoint having the position and orientation of the left eye image capturing unit 102 with the real space image captured by the left eye image capturing unit 102 (the composite image for the left eye). ) Is generated. It is assumed that the virtual object is arranged at the position of the MR marker. That is, the video composition unit 300 recognizes the identification information represented by the MR marker in the video imaged by the marker imaging unit 103 (marker imaging unit 104), and sets the virtual object corresponding to the recognized identification information to the position of the MR marker. A virtual space is constructed by placing them in

  Then, the video composition unit 300 transmits the right-eye composite video and the left-eye composite video to the head-mounted display device 100 via the transmission unit 200. Note that the composite image for the right eye is received by the reception unit 114, and the composite image for the left eye is received by the reception unit 115.

  Next, a configuration example of each image processing circuit included in the image processing unit 110 will be described with reference to FIG. As shown in FIG. 3, each of the image processing circuits included in the image processing unit 110 has five functional units: a register unit 1100e, a clock control unit 1100d, an input unit 1100a, a processing unit 1100b, and an output unit 1100c. The operation of each functional unit differs for each image processing circuit. Although FIG. 3 shows the configuration of the pixel defect correction circuit 1100, the same description applies to image processing circuits other than the pixel defect correction circuit 1100 in the image processing unit 110.

  The register unit 1100e is for temporarily storing data exchanged between the CPU 111 and the pixel defect correction circuit 1100. For example, data sent from the CPU 111 to the clock control unit 1100d and the processing unit 1100b is temporarily stored in the register unit 1100e. Data sent from the processing unit 1100b or the clock control unit 1100d to the CPU 111 is temporarily stored in the register unit 1100e.

  The clock control unit 1100d controls clock output and clock output stop. Specifically, the clock control unit 1100d receives Ex_clk, and controls the output of the clock to the processing unit 1100b and the stop of the clock output in synchronization with clk_en from the register unit 1100e. The clock control unit 1100d outputs a control signal sel to the input unit 1100a and the output unit 1100c. Details of the clock control unit 1100d will be described later with reference to FIGS.

  The input unit 1100a performs control to switch the output destination of the input video signal. Specifically, the input unit 1100a outputs the input video signal to the output unit 1100c when the sel signal from the clock control unit 1100d is High (1), and the sel signal is Low (0). Outputs the input video signal to the processing unit 1100b.

  The output unit 1100c performs control to switch the acquisition source of the output video signal. Specifically, the output unit 1100c receives and outputs the video signal output from the input unit 1100a when the sel signal from the clock control unit 1100d is High (1). On the other hand, when the sel signal is Low (0), the output unit 1100c receives and outputs the video signal output from the processing unit 1100b.

  The processing unit 1100b performs image processing on the video signal input by the input unit 1100a. In the case of FIG. 3, the processing unit 1100 b corrects the defective pixel in the image represented by the video signal using the pixel values of the same type of pixels as the defective pixel around the defective pixel in synchronization with the clock. Note that when the clock stops, the processing unit 1100b stops its operation.

  Next, FIG. 4 shows a configuration example of the clock control unit 1100d. FIG. 5 shows the waveform of each signal in the clock controller 1100d. The operation of the clock control unit 1100d will be described with reference to FIGS.

  As shown in FIG. 4, the clock control unit 1100d includes a phase locked loop (PLL) 1100d1, an AND gate 1100d2, and an inverter gate 1100d3.

  A clock input from EX_clk is caused to oscillate at a frequency multiplied by PLL 1100d and input to AND gate 1100d2 as Ex_clk_m. The clko and control signal sel output from the clock control unit 1100d will be described with reference to FIG. As shown in FIG. 5, when clk_en from the register unit 1100e is HIGH, EX_clk_m is output to clko and LOW is output to the control signal sel. When clk_en is LOW, clko outputs a LOW signal, and control signal sel outputs HIGH.

  The configuration illustrated in FIGS. 3 and 4 is an example of a configuration of an image processing circuit that starts an operation by supplying a clock and stops the operation by stopping the supply of the clock. However, in the present embodiment, any configuration may be adopted as long as it is an image processing circuit that starts operation by supplying a clock and stops operation by stopping the supply of the clock. Further, the image processing circuit may have any configuration as long as the CPU 111 can control the start / stop of the operation.

  Next, the operation of the head-mounted display device 100 will be described using the flowchart of FIG. In step S11, the CPU 111 acquires temperature data representing the temperature inside the head-mounted display device 100 measured by the temperature detection unit 116, and the temperature represented by the acquired temperature data is equal to or higher than a specified temperature (threshold). Judge whether there is. As a result of this determination, if the temperature represented by the temperature data is equal to or higher than the threshold value, the process proceeds to step S12. If the temperature is less than the threshold value, the process waits in step S11.

  In step S12, the CPU 111 determines whether or not the head of the user wearing the head-mounted display device 100 on his / her head is moving based on the position and orientation measured by the head position detection unit 105. to decide. Various methods can be considered for this determination. For example, the CPU 111 determines that the difference between the position and orientation recently acquired from the head position detection unit 105 and the position and orientation acquired last time (a difference between only position components and / or a difference only between posture components) is equal to or greater than a specified amount. Judge that the head has moved. On the other hand, if this difference is less than the prescribed amount, it is determined that the head is not moving. As a result of the determination in step S12, if it is determined that the head is moving, the process proceeds to step S13. If it is determined that the head is not moving, the process proceeds to step S14.

  In step S <b> 13, the CPU 111 identifies an image processing circuit in which the clock control unit 1100 d stops supplying the clock among the plurality of image processing circuits included in the image processing unit 110. Then, the CPU 111 causes the clock control unit 1100d of the identified image processing circuit to start supplying a clock. Then, when the clock control unit 1100d starts supplying the clock in all the image processing circuits of the image processing units 110a to 100d, the process proceeds to step S15.

  In step S15, the CPU 111 identifies an image processing circuit that stops the clock supply by the current clock control unit 1100d among the image processing circuits included in the image processing unit 110. Of the image processing circuits included in the image processing unit 110, which image processing circuit becomes the “image processing circuit for stopping the clock supply” in the first step S15, and which image processing circuit becomes the “clock signal” in the next step S15. The order of image processing circuits to be “image processing circuits to stop supply of clocks”, such as “image processing circuit to stop supply”, is defined in advance. The usage status of the head-mounted display device 100 that is “moving” differs from the usage status of the head-mounted display device 100 that “the user's head is not moving”. However, in step S15, the CPU 111 determines that “the image processing circuit that stops the clock supply this time” in the order corresponding to the usage status of the head-mounted display device 100 that “the user's head is moving”. ”Is specified. Then, the CPU 111 stops the clock supply to the clock control unit 1100d in the specified image processing circuit.

  In step S <b> 17, the CPU 111 acquires temperature data representing the temperature inside the head-mounted display device 100 measured by the temperature detection unit 116, and the temperature represented by the acquired temperature data is equal to or higher than a specified temperature (threshold). Judge whether there is. The determination in step S17 may be performed after a lapse of a specified time from the completion of the process in step S15. If the result of this determination is that the temperature represented by the temperature data is equal to or greater than the threshold, it is determined that there is still an image processing circuit whose operation should be stopped, and the process proceeds to step S15. On the other hand, if the temperature represented by the temperature data is less than the threshold, the process proceeds to step S11.

  In step S14, as in step S13, the CPU 111 identifies an image processing circuit for which the clock control unit 1100d has stopped supplying the clock among the plurality of image processing circuits included in the image processing unit 110. Then, the CPU 111 causes the clock control unit 1100d of the identified image processing circuit to start supplying a clock. When the clock control unit 1100d starts supplying the clock in all the image processing circuits of the image processing units 110a to 100d, the process proceeds to step S16.

  In step S <b> 16, the CPU 111 identifies an image processing circuit that stops the clock supply by the current clock control unit 1100 d among the image processing circuits included in the image processing unit 110. In step S <b> 16, the CPU 111 determines that “the image processing circuit that stops supplying the clock” this time in the order corresponding to the usage state of the head-mounted display device 100 that “the user's head is not moving”. The image processing circuit to be specified is specified. Then, the CPU 111 stops the clock supply to the clock control unit 1100d in the specified image processing circuit.

  In step S18, the CPU 111 acquires temperature data representing the temperature inside the head-mounted display device 100 measured by the temperature detection unit 116, and the temperature represented by the acquired temperature data is defined, as in step S17. It is determined whether the temperature is equal to or higher than the temperature (threshold value). The determination in step S18 may be performed after a lapse of a specified time from the completion of the process in step S16. If the result of this determination is that the temperature represented by the temperature data is equal to or greater than the threshold value, it is determined that there is still an image processing circuit whose operation should be stopped, and the process proceeds to step S16. On the other hand, if the temperature represented by the temperature data is less than the threshold, the process proceeds to step S11.

  Here, the process in step S15 will be described. While the user's head is moving, the image in the real space is blurred. In such an image, it is difficult to detect the MR marker, and it is difficult to position the virtual object. However, since the user's eyes are sensitive to color, even for such a video, it is desirable to avoid image processing related to the color as much as possible for the video presented to the user's eyes. However, in such a case, the supply of the clock to the image processing circuit is stopped in the following order.

(Stop 1) In the image processing units 110b and 110c that process the video in the real space presented to the right and left eyes of the user, stop the supply of the clock to the image processing circuit that performs image processing relating to the resolution (Stop 2) In the image processing units 110a and 110d that process video by the marker imaging unit 103 and the marker imaging unit 104, supply of a clock to the image processing circuit that performs image processing relating to color is stopped (stop 3). In the image processing units 110b and 110c that process the image of the real space presented to the image, the supply of the clock to the image processing circuit that performs the image processing relating to the color is stopped (stop 4). The image by the marker imaging unit 103 and the marker imaging unit 104 In the image processing units 110a and 110d that process the image, the supply of the clock to the image processing circuit that performs the image processing related to the resolution is stopped. Stop 1), for example, to stop the image processing unit 110b, edge enhancement circuit 1107 in 110c, the noise reduction circuit 1104, the supply of the clock in the order of the pixel defect correction circuit 1100. In (Stop 2), for example, the shading correction circuit 1102, density conversion circuit 1101, AE / WB correction circuit 1103, color correction circuit 1106, and γ correction circuit 1108 in the image processing units 110a and 110d are stopped in this order. In (Stop 3), for example, the supply of clocks is stopped in the order of the density conversion circuit 1101, the AE / WB correction circuit 1103, the color correction circuit 1106, the γ correction circuit 1108, and the shading correction circuit 1102 in the image processing units 110b and 110c. In (Stop 4), for example, the supply of clocks is stopped in the order of the pixel defect correction circuit 1100, the noise reduction circuit 1104, and the contour enhancement circuit 1107 in the image processing units 110a and 110d.

  Each time the supply of the clock to the image processing circuit is stopped, the measured temperature by the temperature detection unit 116 is acquired again after a specified time, and if the measured temperature is equal to or higher than the threshold, the clock signal to the next image processing circuit is acquired. Stop supplying.

  Next, the process in step S16 will be described. In a state in which the user's head is not moving, the image in the real space is hardly shaken. In such an image, the MR marker can be easily detected, and the positioning of the virtual object is facilitated. Even when the color of the MR marker on the image does not match the original color, the alignment accuracy does not decrease. Therefore, for the image processing circuit for the image by the marker imaging unit 103 and the marker imaging unit 104 that captures the MR marker, the clock is in the order of the image processing circuit that performs image processing related to color and the image processing circuit that performs image processing related to resolution. Stop supplying. Further, when the user observes such an image, the user pays attention to the color rather than the resolution. Therefore, for the image processing circuit that processes the real space video to be presented to the right and left eyes of the user, the image processing circuit that performs the image processing related to the resolution and the image processing circuit that performs the image processing related to the color are clocked in this order. Stop supplying. More specifically, the supply of the clock to the image processing circuit is stopped in the following order.

(Stop 1) In the image processing units 110a and 110d that process video by the marker imaging unit 103 and the marker imaging unit 104, supply of a clock to the image processing circuit that performs image processing relating to color is stopped. In the image processing units 110b and 110c that process the image of the real space presented to the eyes and the left eye, supply of the clock to the image processing circuit that performs image processing relating to the resolution is stopped (stop 3). Marker imaging unit 103 and marker imaging In the image processing units 110a and 110d that process the video by the unit 104, the supply of the clock to the image processing circuit that performs the image processing relating to the resolution is stopped (stop 4). Real space video to be presented to the right and left eyes of the user In the image processing units 110b and 110c that process the image, the supply of the clock to the image processing circuit that performs the image processing relating to the color is stopped. Stop 1), for example, a shading correction circuit 1102 image processing unit 110a, in 110d, the density conversion circuit 1101, AE / WB correction circuit 1103, a color correction circuit 1106 to stop the supply of the clock in the order of γ correction circuit 1108. In (stop 2), for example, the clock supply is stopped in the order of the contour emphasis circuit 1107, the noise reduction circuit 1104, and the pixel defect correction circuit 1100 in the image processing units 110b and 110c. In (stop 3), for example, the supply of clocks is stopped in the order of the pixel defect correction circuit 1100, the noise reduction circuit 1104, and the edge enhancement circuit 1107 in the image processing units 110a and 110d. In (stop 4), for example, the shading correction circuit 1102, the density conversion circuit 1101, the AE / WB correction circuit 1103, the color correction circuit 1106, and the γ correction circuit 1108 in the image processing units 110b and 110c are stopped in this order. .

  Each time the supply of the clock to the image processing circuit is stopped, the measured temperature by the temperature detection unit 116 is acquired again after a specified time, and if the measured temperature is equal to or higher than the threshold, the clock signal to the next image processing circuit is acquired. Stop supplying.

  As described above, in the present embodiment, marker imaging is performed both when it is determined that the user's head is moving (step S15) and when it is determined that the user's head is not moving (step S16). The image processing processing circuits related to the units 103 and 104 are stopped in the following order. That is, in detecting the MR marker, it is important that the image is not blurred. Therefore, the clock is supplied in the order of the processing circuit that performs image processing relating to color and the processing circuit that performs image processing relating to resolution. Stopped.

  On the other hand, regarding the stop order of the image processing processing circuits related to the images displayed on the user's eyes, the user's head is moving even when it is determined that the user's head is moving (step S15). Also when it is determined that there is no (step S16), the process is stopped in the following order. That is, since the user's eyes are sensitive to color, the supply of clocks is stopped in the order of the processing circuit that performs image processing related to resolution and the processing circuit that performs image processing related to color.

  In addition, when it is determined that the user's head is moving (step S15), since it is difficult to detect the MR marker, image processing relating to the color of the video is performed by the marker imaging units 103 and 104. Priority is given to stopping the supply of the clock to the image processing circuit that performs image processing related to the resolution of the video in the real space presented to the right and left eyes of the user rather than stopping the supply of the clock to the image processing circuit To do.

  Further, when it is determined that the user's head is not moving (step S16), the MR marker detection accuracy does not greatly decrease, so the resolution of the real-space video presented to the user's right eye and left eye Priority is given to stopping the supply of the clock to the image processing circuit that performs the image processing related to the color of the video by the marker imaging units 103 and 104 rather than stopping the supply of the clock to the image processing circuit that performs the image processing related to To do.

  As described above, in this embodiment, when the temperature inside the head-mounted display device becomes equal to or higher than the specified temperature, the position and / or posture of the head-mounted display device is moving. The operation of the image processing circuit is stopped in an order according to whether or not. As a result, it is possible to reduce the image quality of the video presented to the user and the accuracy of the alignment of the virtual object as much as possible, and to suppress the power consumption and suppress the heat in the head-mounted display device. I can do it.

[Second Embodiment]
In the first embodiment, when the temperature inside the head-mounted display device becomes equal to or higher than the specified temperature, whether the position and / or posture of the head-mounted display device is moving. The operation of the image processing circuit is stopped in the order according to the above. In the present embodiment, when the internal temperature of the head-mounted display device becomes equal to or higher than the specified temperature, the image processing circuit operates in the order corresponding to the image of the virtual space displayed on the head-mounted display device. Stop.

  In the present embodiment, an example of a system using a head-mounted display device having the following configuration will be described. That is, this head-mounted display device is a head-mounted display device on which an image processing unit having a plurality of image processing circuits is mounted, and acquires the temperature in the head-mounted display device. Then, the operation of each of the plurality of image processing circuits is sequentially stopped in the order corresponding to the image of the virtual space displayed on the head-mounted display device until the temperature becomes lower than the specified temperature (control). In the following, differences from the first embodiment will be mainly described, and unless otherwise noted, the same as the first embodiment.

  First, a configuration example of a system according to the present embodiment will be described with reference to the block diagram of FIG. In FIG. 7, the same functional units as those shown in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted. In the configuration shown in FIG. 7, the video composition unit 300 transmits a signal 118 representing the area ratio of the virtual space image to the real space image on the composite image of the real space image and the virtual space image to the transmission unit 200. Via the head-mounted display device 100. The signal 118 is notified to the CPU 111. Instead of the area ratio of the virtual space image to the real space image on the combined image of the real space image and the virtual space image, the area ratio of the real space image to the virtual space image on the combined image May be used. In this case, it is necessary to reverse the magnitude relationship with the threshold value in the following description.

  The operation of the head-mounted display device 100 according to the present embodiment will be described using the flowchart of FIG. In FIG. 8, the same processing steps as those shown in FIG. 6 are denoted by the same step numbers, and description thereof will be omitted.

  In step S22, the CPU 111 determines whether or not the area ratio represented by the signal 118 is equal to or greater than a threshold value. If the result of this determination is that the area ratio is equal to or greater than the threshold (the area occupied by the video in the virtual space on the synthesized video is greater than or equal to the area of the real space video), the process proceeds to step S13. On the other hand, if the area ratio is less than the threshold value (the area occupied by the video in the virtual space on the synthesized video is less than the area of the video in the real space), the process proceeds to step S14.

  In step S <b> 25, the CPU 111 specifies an image processing circuit that will be the “image processing circuit that stops supplying the clock” this time in the order corresponding to the situation “the area ratio is equal to or greater than the threshold”. Then, the CPU 111 stops the clock supply to the clock control unit 1100d in the specified image processing circuit.

  In step S <b> 26, the CPU 111 identifies the image processing circuit that will be the “image processing circuit that stops supplying the clock” this time in the order corresponding to the situation where the “area ratio is less than the threshold”. Then, the CPU 111 stops the clock supply to the clock control unit 1100d in the specified image processing circuit.

  Here, the process in step S25 will be described. When the area ratio is equal to or greater than the threshold value, the area of the real space image in the combined image is smaller than that of the virtual space image. In such a case, the supply of the clock to the image processing unit that performs image processing of the video presented to the user is stopped, and then the supply of the clock to the image processing unit that performs image processing of the marker video is stopped. In such a case, however, the supply of the clock to the image processing circuit is stopped in the same order as in step S15.

  Next, the process in step S26 will be described. When the area ratio is less than the threshold value, the area of the real space image in the composite image is larger than that of the virtual space image. In such a case, since the image quality of the video in the real space is important, the supply of the clock to the image processing unit that performs the image processing of the marker video is stopped, and then the image processing that performs the image processing of the video presented to the user Stop supplying the clock to the unit. In such a case, however, the supply of the clock to the image processing circuit is stopped in the same order as in step S16.

  As described above, in this embodiment, when the internal temperature of the head-mounted display device is equal to or higher than the specified temperature, the image processing circuit is arranged in the order according to the area ratio of the virtual space image in the composite image. Stop operation. As a result, it is possible to reduce the image quality of the video presented to the user and the accuracy of the alignment of the virtual object as much as possible, and to suppress the power consumption and suppress the heat in the head-mounted display device. I can do it.

<Modification>
A part or all of the configurations described in the first and second embodiments may be appropriately combined or may be selectively used.

  In the second embodiment, the clock is supplied to the image processing circuit in the order corresponding to the area ratio of the virtual space video to the real space video on the composite video of the real space video and the virtual space video. A stop was made. However, the supply of the clock to the image processing circuit may be stopped in an order according to an index other than the area ratio. For example, when a virtual space image is synthesized at a location (near position) close to the center position on the real space image, even if the size of the virtual space image is small, this is more difficult for the user than the real space image. The image in the virtual space is easier to see. In such a case, even if the area ratio of the video in the virtual space is small, the process may proceed from step S22 to step S13. Conversely, when the virtual space image is synthesized at a location far from the center position on the real space image, even if the size of the virtual space image is large, for the user, the real space image is larger than the virtual space image. The video is easier to see. In such a case, even if the area ratio of the video in the virtual space is large, the process may proceed from step S22 to step S14. It should be noted that instead of the center position of the real space image, another specified location on the real space image may be used as a reference to determine whether or not the virtual space image is located at the specified location. . Further, considering both the area ratio and the position of the image in the virtual space, it may be determined whether to proceed from step S22 to step S13 or to step S14. In this way, it is determined whether to proceed from step S22 to step S13 or to step S14 with an index corresponding to a video (real space video, virtual space video) that is easily noticeable on the composite image. Also good.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  110a: Image processing unit 110b: Image processing unit 110c: Image processing unit 110d: Image processing unit 116: Temperature detection unit 111: CPU 100: Head-mounted display device

Claims (16)

A head-mounted display device equipped with an image processing unit having a plurality of image processing circuits,
Determining means for determining whether the position and / or posture of the head-mounted display device is changing;
Obtaining means for obtaining the temperature in the head-mounted display device;
A head-mounted type comprising: control means for sequentially stopping the operation of each of the plurality of image processing circuits in an order according to a determination result by the determination means until the temperature becomes lower than a specified temperature. Display device.   The control means operates an image processing circuit that is not operating among the plurality of image processing circuits, and determines each operation of the plurality of image processing circuits by the determination means until the temperature becomes less than a specified temperature. The head-mounted display device according to claim 1, wherein the head-mounted display device is sequentially stopped in an order according to the determination result. The image processing unit images a first image processing unit that performs image processing on a real space image displayed on the head-mounted display device, and a marker arranged in the real space in order to place a virtual object A second image processing unit that performs image processing on the captured video,
The control means includes
For the first image processing unit, the temperature is defined in the order of an image processing circuit that performs image processing related to resolution and an image processing circuit that performs image processing related to color, regardless of the determination result by the determination unit. Stop operation until the temperature is below
For the second image processing unit, the temperature is defined in the order of an image processing circuit that performs image processing related to color and an image processing circuit that performs image processing related to resolution, regardless of the determination result by the determination unit. The head-mounted display device according to claim 1, wherein the operation is stopped until the temperature becomes lower than the temperature. The control means includes
When the position and / or orientation of the head-mounted display device is changing, an image processing circuit that performs image processing related to resolution in the first image processing unit, and color processing in the second image processing unit The operation is stopped until the temperature becomes lower than the specified temperature in the order of the image processing circuit that performs image processing,
When the position and / or orientation of the head-mounted display device is not changing, an image processing circuit that performs color-related image processing in the second image processing unit, and a resolution in the first image processing unit The head-mounted display device according to claim 3, wherein the operation is stopped until the temperature falls below a specified temperature in the order of image processing circuits that perform image processing. The control means includes
When the position and / or orientation of the head-mounted display device is changing, an image processing circuit that performs image processing related to resolution in the first image processing unit, and color processing in the second image processing unit The temperature in the order of an image processing circuit that performs image processing, an image processing circuit that performs image processing relating to color in the first image processing unit, and an image processing circuit that performs image processing related to resolution in the second image processing unit Stop until the temperature is below the specified
When the position and / or orientation of the head-mounted display device is not changing, an image processing circuit that performs color-related image processing in the second image processing unit, and a resolution in the first image processing unit The temperature in the order of an image processing circuit that performs image processing, an image processing circuit that performs image processing related to resolution in the second image processing unit, and an image processing circuit that performs image processing related to color in the first image processing unit The head-mounted display device according to claim 4, wherein the operation is stopped until the temperature becomes lower than a specified temperature. Furthermore,
A means for acquiring real-space images;
Means for acquiring the position and orientation of the head-mounted display device;
Means for acquiring a composite image of a virtual space image based on the position and orientation and the real space image;
The head-mounted display device according to claim 1, wherein the head-mounted display device displays the composite image. A head-mounted display device equipped with an image processing unit having a plurality of image processing circuits,
Obtaining means for obtaining the temperature in the head-mounted display device;
Control means for sequentially stopping the operation of each of the plurality of image processing circuits in an order corresponding to the image of the virtual space displayed on the head-mounted display device until the temperature becomes lower than a specified temperature. A head-mounted display device characterized by that.   The control means operates an image processing circuit that is not operating among the plurality of image processing circuits, and performs each operation of the plurality of image processing circuits until the temperature becomes lower than a specified temperature. The head-mounted display device according to claim 7, wherein the head-mounted display device is sequentially stopped in an order corresponding to an image of a virtual space displayed on the type display device. The image processing unit images a first image processing unit that performs image processing on a real space image displayed on the head-mounted display device, and a marker arranged in the real space in order to place a virtual object A second image processing unit that performs image processing on the captured video,
The control means includes
For the first image processing unit, an image processing circuit that performs image processing relating to the resolution and an image that performs image processing relating to the color, regardless of the video of the virtual space displayed on the head-mounted display device Stop the operation until the temperature falls below the specified temperature in the order of the processing circuit,
For the second image processing unit, an image processing circuit that performs image processing relating to color, and an image that performs image processing relating to resolution, regardless of the video of the virtual space displayed on the head-mounted display device The head-mounted display device according to claim 7 or 8, wherein the operation is stopped until the temperature becomes lower than a specified temperature in the order of processing circuits. The control means includes
When the area ratio of the video in the virtual space to the video in the real space is greater than or equal to a threshold value, an image processing circuit that performs image processing related to resolution in the first image processing unit, and a color in the second image processing unit The operation is stopped until the temperature becomes lower than the specified temperature in the order of the image processing circuit that performs image processing,
When the area ratio of the video in the virtual space to the video in the real space is less than a threshold value, an image processing circuit that performs image processing relating to color in the second image processing unit, and a resolution in the first image processing unit The head-mounted display device according to claim 9, wherein the operation is stopped until the temperature becomes lower than a specified temperature in the order of image processing circuits that perform image processing. The control means includes
When the area ratio of the video in the virtual space to the video in the real space is greater than or equal to a threshold value, an image processing circuit that performs image processing related to resolution in the first image processing unit, and a color in the second image processing unit The temperature in the order of an image processing circuit that performs image processing, an image processing circuit that performs image processing relating to color in the first image processing unit, and an image processing circuit that performs image processing related to resolution in the second image processing unit Stop until the temperature falls below the specified temperature,
When the area ratio of the video in the virtual space to the video in the real space is less than a threshold value, an image processing circuit that performs image processing relating to color in the second image processing unit, and a resolution in the first image processing unit The temperature in the order of an image processing circuit that performs image processing, an image processing circuit that performs image processing related to resolution in the second image processing unit, and an image processing circuit that performs image processing related to color in the first image processing unit The head-mounted display device according to claim 10, wherein the operation is stopped until the temperature becomes lower than a specified temperature. The control means includes
When the position of the image in the virtual space on the image in the real space is a specified position, an image processing circuit that performs image processing relating to the resolution in the first image processing unit, and a color in the second image processing unit The operation is stopped until the temperature is lower than the specified temperature in the order of the image processing circuit that performs image processing according to
When the position of the image in the virtual space on the image in the real space is not a specified position, an image processing circuit that performs color image processing in the second image processing unit, and a resolution in the first image processing unit The head-mounted display device according to claim 9, wherein the operation is stopped until the temperature becomes lower than a specified temperature in the order of image processing circuits that perform image processing according to claim 10. The control means includes
When the position of the image in the virtual space on the image in the real space is a specified position, an image processing circuit that performs image processing relating to the resolution in the first image processing unit, and a color in the second image processing unit An image processing circuit that performs image processing according to the above, an image processing circuit that performs image processing related to color in the first image processing unit, and an image processing circuit that performs image processing related to resolution in the second image processing unit. Stop operation until the temperature is below the specified temperature,
When the position of the image in the virtual space on the image in the real space is not a specified position, an image processing circuit that performs color image processing in the second image processing unit, and a resolution in the first image processing unit An image processing circuit that performs image processing according to the above, an image processing circuit that performs image processing according to resolution in the second image processing unit, and an image processing circuit that performs image processing related to color in the first image processing unit. The head-mounted display device according to claim 12, wherein the operation is stopped until the temperature becomes lower than a specified temperature.   The head-mounted display device according to claim 13, wherein the specified position is a position in the vicinity of a center position of the image in the real space. A method for controlling a head-mounted display device equipped with an image processing unit having a plurality of image processing circuits,
A determination step of determining whether the position and / or posture of the head-mounted display device is changing;
An acquisition step of acquiring the temperature in the head-mounted display device;
A head-mounted type comprising: a control step of sequentially stopping each operation of the plurality of image processing circuits in an order according to a determination result by the determination step until the temperature becomes lower than a specified temperature. Display device control method. A method for controlling a head-mounted display device equipped with an image processing unit having a plurality of image processing circuits,
An acquisition step of acquiring the temperature in the head-mounted display device;
A control step of sequentially stopping the operation of each of the plurality of image processing circuits in an order corresponding to an image of a virtual space displayed on the head-mounted display device until the temperature becomes lower than a specified temperature. A control method for a head-mounted display device.

Images