JP6725999B2 - Information processing apparatus, information processing apparatus control method, and program - Google Patents

Description

The present invention relates to an information processing device, a control method of the information processing device, and a program, and more particularly to an image processing technique using an HMD (Head Mounted Display).

In recent years, mixed reality, so-called MR (Mixed Reality) technology has been known as a technology for seamlessly fusing the real world and the virtual world in real time. As one of the MR technologies, a video see-through HMD (Head Mounted Display) is used to image a subject that substantially matches the subject observed from the pupil position of the HMD user with a video camera or the like, and the captured image is CG. A technique is known in which an HMD user can observe an image in which (Computer Graphics) is superimposed and displayed.

In the video see-through HMD, a subject is imaged by a charge-coupled device such as a CCD to obtain digital image data of the subject, and an MR image (mixed reality image) on which a CG image is superimposed is displayed on a display device such as a liquid crystal or an organic EL. It is configured to be displayed to the wearer via the.

In addition, as an image display device used for an HMD or the like, it is desired to present an image with a wide angle of view in order to realize a natural observation and increase a sense of reality. As a technique to achieve wide-angle image presentation, by displaying images with different angles of view on the left and right eyes and overlapping only part of the angles of view on the left and right eyes, the same angle of view on the left and right eyes can be displayed. 2. Description of the Related Art There is known an image display device capable of observing an image having a wider angle of view than the case of displaying the image.

When images with different angles of view are displayed on the left and right eyes so that only part of the angles of view overlaps between the left and right eyes, the boundary between the monocular region and the binocular region is conspicuous, and natural observation cannot be performed. This phenomenon is caused by the fact that an image is displayed on one eye, but no image is displayed on the other eye, and the black part (see FIG. 15) such as the frame of the panel is seen, resulting in a binocular rivalry between the left and right eyes. Caused by.

On the other hand, the HMD described in Patent Document 1 discloses a technique of making a black portion inconspicuous by reducing the brightness of an image from a predetermined position with a predetermined decrease curve.

JP, 2003-337299, A

However, in the technique disclosed in Patent Document 1, the black portion is made inconspicuous, but when an organic EL panel or the like is used for the HMD, the brightness decreasing curve is always applied to the same place, The display was easy to burn.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for making a display unit difficult to burn.

An information processing apparatus according to the present invention which achieves the above object,
When the user observes the first image displayed on the first display section and the second image displayed on the second display section partially overlapping each other, the first image and the second image are displayed. Processing means for performing gradation processing on the first image and the second image by switching a plurality of display modes including a first display mode and a second display mode in which target areas for gradation processing are different for each image. When,
First display time information indicating the first image and the second image that have been gradation-processed in the first display mode by the first display unit and the second display unit, and the first display. Section and a second display section obtain a plurality of display time information including the first display image and the second display time information displaying the second image subjected to gradation processing in the second display mode. Acquisition means,
Among the plurality of display time information, and determining means for determining a display mode corresponding to the shortest display time, a display mode for displaying the first image and the second image,
It is characterized by including.

According to the present invention, it is possible to make the display unit less likely to burn.

1 is a diagram showing a configuration example of an image display system according to a first embodiment of the present invention. 1 is a schematic block diagram of an image display system according to Embodiment 1 of the present invention. 3 is a detailed block diagram of an image processing unit according to the first embodiment of the present invention. FIG. 3 is a pattern diagram of gradations in each display mode according to the first embodiment of the present invention. FIG. 3 is a timing pattern table for determining execution of gradation processing according to the first embodiment of the present invention. 6 is a table showing an example of display time for each display mode according to the first embodiment of the present invention. 6 is a flowchart showing a processing procedure when changing the display mode according to the first embodiment of the present invention. 3 is a flowchart showing a processing procedure when the display unit is turned off according to the first embodiment of the present invention. FIG. 6 is a detailed block diagram of an image processing unit according to the second embodiment of the present invention. FIG. 9 is a diagram showing an example of a region for calculating a brightness average value according to the second embodiment of the present invention. 9 is a table showing an example of performing weighting for each range of screen luminance average values according to the second embodiment of the present invention. 9 is a table in which weighting of screen display time and luminance average for each gradation area according to the second embodiment of the present invention is added. 9 is a flowchart showing a processing procedure when changing a display mode according to the second embodiment of the present invention. 9 is a flowchart showing a processing procedure when the display unit is turned off according to the second embodiment of the present invention. The figure which shows the image synthesize|combined in the head when observing HMD.

Hereinafter, embodiments will be described with reference to the drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(Embodiment 1)
<System configuration>
FIG. 1 is a diagram showing a configuration example of an image display system according to the first embodiment of the present invention. In FIG. 1, the image display system includes a head-mounted display device (hereinafter, HMD) 1101 (information processing device), a controller 1102, and an image processing device 1104 having a display unit 1103.

The HMD 1101 includes an image display unit, a communication unit that communicates with the controller 1102, and a control unit that controls them, and is mounted on the head of the user. The image display unit acquires and displays the image generated by the image processing device 1104. The image display unit is attached in front of each eye of the user in a configuration including an optical system.

The HMD 1101 communicates with a wirelessly connected controller 1102 that constitutes a small-scale network, such as a WLAN (Wireless Local Area Network) and a WPAN (Wireless Personal Area Network). A wired communication method may be used between the HMD 1101 and the controller 1102 without being limited to wireless communication.

The image processing apparatus 1104, which is connected to the controller 1102 by wire, has a reproduction unit that reproduces an image and a storage unit that stores the image to be reproduced. The image processing apparatus 1104 communicates with the HMD 1101 via the controller 1102. The image processing apparatus 1104 also includes a keyboard and the like, and inputs data, commands, and the like. The display of the input data, the result of the command, and the like are displayed on the display unit 1103.

In FIG. 1, the image processing apparatus 1104 and the controller 1102 have different hardware configurations. However, all the functions of the controller 1102 are implemented in the image processing apparatus 1104 and integrated, or the image processing apparatus 1104 and the image processing apparatus 1104 are integrated. It is also possible to collect the functions of the controller 1102 and configure a dedicated image processing apparatus.

<Functional configuration>
FIG. 2 is a functional block diagram of an image processing device and an HMD that form the image display system shown in FIG. The HMD 1201 corresponds to the HMD 1101 of FIG. 1, and includes an image capturing unit L1202/R1203 that captures the outside world and a display unit L1204/R1205 that displays an image. Also, an image processing unit 1206 that performs image processing on an image to be displayed, an I/F communication unit 1207 that transmits and receives images and control signals, a control unit 1208 that controls the HMD 1201, and a detection unit 1209 that detects attachment or detachment of the HMD 1201. , And other functional units (not shown).

The image processing apparatus 1210 corresponds to the image processing apparatus 1104 of FIG. 1 and is configured by an external device different from the HMD 1201 such as a PC (Personal Computer) and a WS (Workstation). The image processing apparatus 1210 includes an I/F communication unit 1211, a position/orientation acquisition unit 1212, a CG drawing unit 1213, a content DB 1214, and other functional units (not shown).

The I/F communication unit 1211 sends and receives images, control signals, and the like. The position/orientation acquisition unit 1212 calculates and acquires the position/orientation of the HMD 1201 from the captured image received from the HMD 1201. The CG drawing unit 1213 superimposes the CG image stored in the content DB 1214 on the captured images captured by the image capturing units L1202 and R1203. The content DB 1214 stores CG content of virtual images.

In the present embodiment, the HMD 1201 sends the captured image to the image processing device 1210, the image processing device 1210 calculates the position and orientation, and superimposes the CG of the virtual image on the captured image. However, the present invention is not limited to this configuration. Absent. For example, the position and orientation acquisition unit 1212 is mounted on the HMD 1201, the position and orientation information of the HMD 1201 is transmitted to the image processing apparatus 1210, only the CG image of the virtual image is sent from the image processing apparatus 1210 to the HMD 1201, and the captured image in the HMD 1201. And a CG image may be superimposed.

In the above-described configuration, the image of the outside world captured by the image capturing units L1202 and R1203 of the HMD 1201 is sent to the image processing apparatus 1210 via the I/F communication unit 1207.

The position/orientation acquisition unit 1212 of the image processing device 1210 calculates the position/orientation of the HMD 1201 from the received captured image. The CG drawing unit 1213 draws a CG image based on the position and orientation of the HMD 1201 calculated by the position and orientation acquisition unit 1212, and superimposes the CG image on the imaged images captured by the image capturing units L1202 and R1203. Each display image is generated. After that, the generated image is transmitted to the HMD 1201 via the I/F communication unit 1211.

The HMD 1201 sends each image for the left eye and the image for the right eye received from the image processing device 1210 to the image processing unit 1206. After the image processing unit 1206 performs image processing such as gradation processing on the display images for the left and right eyes, the processed images are displayed by the display units L1204 and R1205. With the above configuration, by mounting the HMD 1201, it is possible to observe the superimposed image of the CG image drawn by the image processing device 1210 and the captured image.

<Structure of image processing unit>
FIG. 3 is a detailed block diagram of the image processing unit 1206 of the HMD 1201 and specifically describes functional blocks that perform gradation processing on a display image in the present embodiment. The gradation correction units L1301 and R1302 perform gradation processing on the display images for the left eye and the right eye received from the I/F communication unit 1207 in the first display mode or the second display mode, and the processing result To the display unit L1204 (first display unit) and R1205 (second display unit). In this manner, the display unit L1204 and the display unit R1205 are configured such that the same image is displayed in a part of each region and the regions are partially overlapped by the left and right eyes and are observed by the user. Has been done.

The gradation determination unit 1303 determines in which of the first display mode and the second display mode the gradation processing performed by the gradation correction units L1301 and R1302 is performed. The data storage unit 1304 stores the cumulative time when gradation processing is performed in each display mode. The timer unit 1305 counts the time when the gradation process is performed.

<Gradation processing>
FIG. 4A and FIG. 4B are diagrams showing an example of a region where gradation processing is performed in each display mode according to the present embodiment. FIG. 4A is a diagram in which an image subjected to gradation processing is displayed on the display units L1204 and R1205 in the first display mode. FIG. 4B is a diagram in which an image subjected to gradation processing is displayed on the display units L1204 and R1205 in the second display mode.

Whether the display is performed in the first display mode as shown in FIG. 4A or the second display mode as shown in FIG. 4B, the HMD user can display the display portions L1204 and R1205 with both eyes. When you see, you can see Kurobe in a non-conspicuous state.

In the present embodiment, in both the first display mode and the second display mode, there is no change in the target region to be gradation-processed in the images viewed by the left and right eyes. However, in the target area to be gradation-processed, the gradation processing is performed on the image on the display unit L1204 for the left eye in the first display mode in the right-side area. On the other hand, in the second display mode, gradation processing is performed on the image on the display unit R1204 for the right eye.

Further, the gradation processing is performed on the image on the display unit R1204 for the right eye in the first display mode for the area on the left side of the target area to be gradation-processed. On the other hand, in the second display mode, gradation processing is performed on the image on the display unit L1204 for the left eye.

That is, the target area that is subjected to gradation processing on the image of the display section L1204 for the left eye in the first display mode, and the gradation processing for the image of the display section R1204 for the right eye in the second display mode. The target area is the same area in the images viewed by both eyes. In addition, a target region that is subjected to gradation processing for the image of the display unit R1204 for the right eye in the first display mode, and a gradation process for the image of the display unit L1204 for the left eye in the second display mode. The target area is the same area in the images viewed by both eyes.

As described above, according to the present embodiment, in each of the display unit L1204 for the left eye and the display unit R1204 for the right eye, if the target region for gradation processing is different between the first display mode and the second display mode. It is something that can be made. As a result, it is possible to prevent or reduce the burn-in of the display units L1204 and R1205 because the image in which the gradation process is performed on the same area is not always displayed. Although the first display mode and the second display mode are illustrated here as an example, gradation processing may be performed in a plurality of display modes including other display modes.

Next, a method for switching the display mode and changing the gradation processing area in each display unit will be described with reference to FIGS. 5 and 6. First, FIG. 5 is a table showing an example of the timing of changing the region in which the gradation processing as shown in FIGS. 4A and 4B is changed. Item number 1 indicates the time when the user wears the HMD 1201 and is the timing when the detection unit 1209 mounted on the HMD 1201 detects the wearing of the HMD 1201 on the user. Item number 2 is the timing when the display units L1204 and R1205 change from OFF to ON.

The timing to change may be all the timings shown in the table of FIG. 5, or may be any one timing. In each of the timing examples shown in FIG. 5, the user does not switch the gradation processing area while using the HMD 1201, but starts the processing of displaying an image before use or on the display units L1204 and R1205. It sometimes indicates that the gradation processing area is switched. The gradation processing area may be switched while the HMD 1201 is in use, but it is preferable to switch at a timing other than the time when the HMD 1201 is not in use as in the present embodiment so as not to cause discomfort during use. The timing of switching the gradation processing area may be switched at a timing other than the timing shown in FIG.

FIG. 6 shows an example of data stored in the data storage unit 1304, which stores the time when an image is displayed in each display mode. Note that FIG. 6 illustrates an example of the first display mode and the second display mode illustrated in FIGS. 4A and 4B. According to the example of FIG. 6, the time for displaying the image in the first display mode of FIG. 4A is 10 minutes, and the time for displaying the image in the second display mode of FIG. 4B is 20 minutes. ..

When changing the display mode at the timing described with reference to FIG. 5, the table of FIG. 6 is first referenced. Next, the next display mode is determined so that the time displayed in each display mode becomes the same display time. In the example of FIG. 6, the display time of the first display mode of FIG. 4A is shorter than the display time of the second display mode of FIG. The first display mode of a) is determined.

<Process>
FIG. 7 is a flowchart showing a processing procedure when the image processing unit 1206 according to the present embodiment changes the display mode. In the present embodiment, the timing described in FIG. 5 is used as the timing for changing the display mode.

In step S1701, the gradation determination unit 1303 acquires, from the data storage unit 1304, a plurality of display times in which the gradation process is performed in each display mode as display time information.

In step S1702, the gradation determination unit 1303 determines the display mode corresponding to the shorter display time based on the plurality of pieces of display time information acquired in step S1701, and notifies the gradation correction units L1301 and R1302 of the determination result. In step S1703, the gradation correction units L1301 and R1302 perform gradation processing on the display image based on the display mode determined by the gradation determination unit 1303.

In step S1704, the gradation correction units L1301 and R1302 instruct the timer unit 1305 to start the timer and start measuring the display time of the image on which the gradation process has been performed. In step S1705, the gradation correction units L1301 and R1302 output images to be displayed on the display units L1204 and R1205, and the images are displayed on the display units L1204 and R1205.

FIG. 8 is a flowchart showing a processing procedure when the display units L1204, R1205 according to the present embodiment finish displaying an image. The end timing is the opposite case of FIG. For example, when the user finishes the MR experience and removes the HMD 1201. The removal of the HMD 1201 is detected by the detection unit 1209. When the HMD 1201 is detached, it is not necessary to display an image on the display units L1204 and R1205, so the display of the display units L1204 and R1205 is terminated (turned off). Also, when the power of the HMD 1201 is turned off, the display of the display units L1204 and R1205 is finished.

In step S1801, the gradation correction units L1301 and R1302 acquire display time information from the timer unit 1305 and stop the timer. After that, the gradation correction units L1301 and R1302 notify the gradation determination unit 1303 of the acquired time.

In step S1802, the gradation determination unit 1303 acquires information on the display time corresponding to the display mode executed this time from the data storage unit 1304, and adds the time acquired in step S1801. After that, the gradation determination unit 1303 outputs the added display time information to the data storage unit 1304.

In step S1803, the data storage unit 1304 updates the management table as described in FIG. 6 based on the information received in step S1802. Then, in step S1804, the display units L1204 and R1205 end the image display.

As described above, in the present embodiment, in each of the plurality of display units, the target region of gradation processing is different depending on the display mode. Then, at a predetermined timing, the display modes are switched so that the accumulated display time in each display mode is the same as much as possible. As a result, in the present embodiment, it is possible to prevent or reduce burn-in of the display unit.

(Embodiment 2)
In the first embodiment, an example has been described in which the display modes are switched so that the time in each display mode is the same. On the other hand, in the present embodiment, a process of switching the display mode based on the brightness information when the image is displayed on the display unit in addition to the display time of each display mode will be described.

FIG. 9 is a configuration example of the image processing unit 1206 according to this embodiment. Compared with the configuration of FIG. 3, the difference is that luminance calculation units L1901 and R1902 are added. The brightness calculation units L1901 and R1902 calculate the brightness average value of the display image received from the gradation correction units L1301 and R1302 at any time, and when the image display on the display units L1204 and R1205 is finished, the brightness calculation unit determines the gradation. Send to section 1303.

Here, FIG. 10A and FIG. 10B are diagrams illustrating the area in the display image in which the average brightness value is calculated. FIG. 10A is an example in which the average brightness value is calculated only for the peripheral region in which the gradation process is performed. On the other hand, FIG. 10B is an example of calculating the average luminance value for the entire image. Either of the examples of FIG. 10A and FIG. 10B may be used, and any other method may be used as long as it includes a part of the target region to be subjected to gradation processing.

The gradation determination unit 1303 determines a weighting value for the received brightness average value, calculates a weighting value for the displayed time, and passes it to the data storage unit 1304. Here, FIG. 11 is an example of a table for weighting the luminance average value. In the case of this table, the weighting value is 1 when the average luminance value is 0 or more and less than 50, and 1.4 when the average luminance value is 200 or more and 255 or less. As shown in FIG. 11, the weighting value is changed by 0.1 for each 50. FIG. 12 is an example of a table stored in the data storage unit 1304 according to this embodiment. Compared to FIG. 6 of the first embodiment, the value to be stored is not the display time but the cumulative value obtained by multiplying the time and the weighted value.

Subsequently, FIG. 13 is a flowchart showing a processing procedure for changing the display mode according to the present embodiment. In step S2301, the gradation determination unit 1303 acquires information of display time for displaying each gradation×weighting value from the data storage unit 1304. In step S2302, the gradation determination unit 1303 determines the smaller display mode based on the weighted display time information acquired in step S2301, and notifies the gradation correction units L1301 and R1302 of the determination result. The other processes are the same as the processes in FIG.

Further, FIG. 14 is a flowchart showing a processing procedure in which the display units L1204 and R1205 according to the present embodiment finish displaying an image. In step S2401, the gradation determination unit 1303 acquires the average brightness value calculated by the brightness calculation units L1901 and R1902. In S2402, the gradation determination unit 1303 calculates the weighted display time based on the weighted value corresponding to the luminance average value acquired in S2401 and the display time acquired in S1801.

In step S2403, the gradation determination unit 1303 acquires the information stored in the data storage unit 1304, and adds the weighted display time calculated in step S2402 to the display time corresponding to the corresponding display mode. The other processes are similar to those of FIG.

As described above, in the present embodiment, since the brightness value of the screen is also taken into consideration when determining the display mode, it is possible to further prevent or reduce burn-in of the display unit.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1201: HMD, L1202, R1203: Imaging unit, L1204, R1205: Display unit, 1206: Image processing unit, 1207: I/F communication unit, 1208: Control unit, 1209: Detection unit, 1210: Image processing device, 1211: I/F communication unit, L1301, R1302: gradation correction unit, 1303: gradation determination unit, 1304: data storage unit, 1305: timer unit, L1901, R1902: brightness calculation unit

Claims (10)

When the user observes the first image displayed on the first display section and the second image displayed on the second display section partially overlapping each other, the first image and the second image are displayed. Processing means for performing gradation processing on the first image and the second image by switching a plurality of display modes including a first display mode and a second display mode in which target areas for gradation processing are different for each image. When,
First display time information indicating the first image and the second image that have been gradation-processed in the first display mode by the first display unit and the second display unit, and the first display. Section and a second display section obtain a plurality of display time information including the first display image and the second display time information displaying the second image subjected to gradation processing in the second display mode. Acquisition means,
Among the plurality of display time information, and determining means for determining a display mode corresponding to the shortest display time, a display mode for displaying the first image and the second image,
An information processing apparatus comprising: Measuring means for measuring a display time during which the first image and the second image subjected to gradation processing by the processing means are displayed on the first display part and the second display part;
Storage means for adding and storing the display time measured by the measuring means to the display time corresponding to the display mode in which the gradation process has been executed;
The information processing apparatus according to claim 1, further comprising: The information processing device is a head-mounted display device,
Further comprising a detection means for detecting attachment of the head-mounted display device to a user,
The acquisition unit, when the mounting has been detected, the information processing apparatus according to claim 1 or 2, characterized in that starting the acquisition of the plurality of display time information. The information processing device is a head-mounted display device,
Further comprising detection means for detecting removal of the head-mounted display device from a user,
The information processing apparatus according to claim 2 , wherein the storage unit stores the display time measured by the measurement unit when the removal is detected. A display mode in which the determining unit performs gradation processing on the first image and the second image based on the plurality of display time information and the brightness values of the first image and the second image. The information processing apparatus according to claim 2 , wherein the information processing apparatus determines. The information processing apparatus according to claim 5 , wherein the brightness value is a brightness value of a region of the entire image. The information processing apparatus according to claim 5 , wherein the brightness value is a brightness value of a partial area of an image including the target area of the gradation processing. The target area that is gradation processed for the first image in the first display mode and the target area that is gradation processed for the second image in the second display mode are the user. The same area in the image observed by
The target area that is gradation-processed for the second image in the first display mode and the target area that is gradation-processed for the first image in the second display mode are the user The information processing apparatus according to any one of claims 1 to 7 , wherein the images are observed in the same area. A method of controlling an information processing device, comprising:
When the processing means causes the user to observe the first image displayed on the first display part and the second image displayed on the second display part partially overlapping each other, the first image The gradation processing is performed on the first image and the second image by switching a plurality of display modes including a first display mode and a second display mode in which target areas for gradation processing are different in each of the first and second images. A processing step for performing
Acquisition means displays first display time information indicating the first image and the second image subjected to gradation processing by the first display unit and the second display unit in the first display mode; A plurality of display times including a first display section and a second display section, and second display time information displaying the first image and the second image that have been gradation-processed in the second display mode. An acquisition process for acquiring information,
Determining means, among the plurality of display time information, a determination step of determining a display mode corresponding to the shortest display time, a display mode for displaying the first image and the second image,
A method for controlling an information processing device, comprising: A program for causing a computer to function as the information processing device according to any one of claims 1 to 8 .