JP2016033611A - Information providing system, display device, and display device control method - Google Patents

Abstract

The present invention provides an information providing system, a display device, and a display device control method capable of performing display in which a user is specified by a display device worn and used by the user. An information providing system (1) includes a head-mounted display device (100) that is used while being worn on a user's body, an information providing device (4) that is communicably connected to the head-mounted display device (100), Is provided. The information providing device 4 identifies the communication unit 401, the identification unit 411 that identifies the head-mounted display device 100, and the head-mounted display device 100 that is the information transmission destination, and transmits information through the communication unit 401. A communication control unit 412. The head-mounted display device 100 includes: a second communication unit that communicates with the information providing device 4; an information display control unit that receives information from the information providing device 4 through the second communication unit and displays the information on the display unit; Is provided. [Selection] Figure 2

Description

  The present invention relates to an information providing system, a display device, and a display device control method.

  2. Description of the Related Art Conventionally, wearable display devices are known that perform display while being superimposed on an external field of view (see, for example, Patent Document 1). The device described in Patent Document 1 has a function of superimposing and displaying an external field of view and characters, and changing display attributes such as font size and character color for some characters or words to be displayed. Thereby, for example, a person in the city or a person on the move can read the text data using the wearable display device.

JP 2014-56217 A

Patent Document 1 discloses a configuration in which a wearable display device is used for reading pre-prepared text data. However, conventionally, there has not been proposed a method in which a display device of this type specifies a user. .
The present invention has been made in view of the above-described circumstances, and provides an information providing system, a display device, and a display capable of performing display specifying a user by a display device worn and used by the user An object is to provide a method for controlling an apparatus.

In order to achieve the above object, an information providing system of the present invention includes a display device that is used while being worn on a user's body, and an information providing device that is communicably connected to the display device, The information providing apparatus includes: a first communication unit that communicates with the display device; an identification unit that identifies the display device; and the identification unit that identifies the display device that is a transmission destination of information, and the first communication unit A communication control unit that transmits information by the second communication unit that communicates with the information providing device, and the second communication unit receives information from the information providing device and receives the information from the information providing device. And an information display control unit to be displayed on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, with respect to the display apparatus with which a user's body is mounted | worn and used, a display apparatus can be identified and information can be transmitted, and information is separately provided to the user who uses a display apparatus it can.

Further, according to the present invention, in the information providing system, the display device allows the user to visually recognize an image and transmits the outside scene, and an object detection unit that detects an object in the visual line direction of the user. The information display control unit displays related information related to the object detected by the target detection unit on the display unit, and receives information from the information providing apparatus by the second communication unit. It is displayed on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, the relevant information relevant to the target object located in a user's gaze direction and the information which an information provision apparatus transmits to a display apparatus can be displayed so that it can be seen with an external scene. For this reason, it is possible to show information to the user by making use of the characteristics of the display unit that transmits the outside scene.

In the information providing system according to the aspect of the invention, the information display control unit may emphasize information received from the information providing apparatus by the second communication unit, together with the related information, over the related information. And displaying on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, the relevant information relevant to the target object located in a user's gaze direction and the information which an information provision apparatus transmits to a display apparatus can be displayed collectively. In addition, information transmitted from the information providing device to the display device can be displayed with emphasis over the related information, and the user can be strongly noticed.

Further, the present invention is characterized in that in the information providing system, the information display control unit displays information received from the information providing apparatus at a display position overlapping the outside scene.
According to the present invention, information can be displayed so as to overlap the outside scene. For this reason, for example, guidance and explanation regarding things that the user sees as an outside scene can be displayed.

Further, in the information providing system according to the present invention, the communication control unit of the information providing device transmits guidance information including an image for guiding the user to the display device by the first communication unit, The information display control unit displays an image for guiding the user on the display unit based on the guidance information received by the second communication unit.
According to the present invention, a user can be smoothly guided by showing an image to the user.

In the information providing system according to the present invention, the identification unit of the information providing device identifies the display device to identify the position of each display device, and the communication control unit uses the identification unit. The guidance information corresponding to the specified position of the display device is transmitted by the first communication unit.
According to the present invention, the information providing device can identify the display device and transmit the guide information corresponding to the position of the display device. Thereby, it is possible to provide information corresponding to the position of the user to the user who uses the display device and guide it more appropriately.

In the information providing system according to the present invention, the communication control unit of the information providing apparatus transmits common information to the plurality of display devices, and the specific display device is assigned to the identification unit. It is characterized by transmitting information different from the common information that is identified.
According to the present invention, common information can be transmitted and displayed on a plurality of display devices, and non-common information can be transmitted to a specific display device. For this reason, information can be transmitted and displayed in various forms to a plurality of display devices.

Further, according to the present invention, in the information providing system, the display device includes an imaging unit, and the target detection unit is visually recognized by the user through the display unit based on a captured image of the imaging unit. Detecting the object to be detected.
According to the present invention, an object in the user's line-of-sight direction can be quickly detected.

In the information providing system according to the present invention, the imaging unit is an IR (infrared) camera, a night vision camera, or a thermography camera.
According to the present invention, an object can be detected even if it is invisible to human eyes or difficult to see.

In the information providing system according to the present invention, the display device includes a distance detection unit that detects a distance between the object detected by the target detection unit and the user, and the information display control unit includes: The information is displayed in a manner corresponding to the distance detected by the distance detection unit.
According to the present invention, since the distance to the object can be reflected in the information display mode, the user can perceive the distance to the object.

In order to achieve the above object, the present invention provides a display device that is used by being worn on a user's body, the display unit allowing the user to visually recognize an image and transmitting an outside scene, and an external device. A communication unit that communicates with the information providing device, a target detection unit that detects a target in the user's line-of-sight direction, related information related to the target detected by the target detection unit, and the communication unit And an information display control unit that displays information received from the information providing device on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, with respect to the display apparatus with which a user's body is mounted | worn and used, a display apparatus can be identified and information can be transmitted, and information is separately provided to the user who uses a display apparatus it can.

In order to achieve the above object, the present invention provides a control method for a display device that is provided with a display unit that allows a user to visually recognize an image and transmits an outside scene, and is used by being worn on the user's body. , Communicating with an external information providing device, receiving information transmitted by the information providing device, detecting an object in the direction of the user's line of sight, related information related to the object, and the information providing device The information received from is displayed on the display unit.
ADVANTAGE OF THE INVENTION According to this invention, with respect to the display apparatus with which a user's body is mounted | worn and used, a display apparatus can be identified and information can be transmitted, and information is separately provided to the user who uses a display apparatus it can.

It is explanatory drawing which shows the external appearance structure of a head mounted type display apparatus. It is a figure which shows schematic structure of an information provision system. It is a block diagram which shows the functional structure of a head mounted display apparatus. It is a flowchart which shows operation | movement of a head mounting type display apparatus. It is a flowchart which shows operation | movement of an information provision system. It is explanatory drawing which shows the typical application example of an information provision system. (A) is explanatory drawing of a use environment, (B)-(E) shows the example of a display of a head mounted display apparatus.

  FIG. 1 is an explanatory diagram showing an external configuration of the head-mounted display device 100. The head-mounted display device 100 is a display device that is mounted on the head, and is also referred to as a head mounted display (HMD). The head-mounted display device 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the head-mounted display device 100. The image display unit 20 is also simply referred to as a “display unit”.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. (Imaging unit) and a microphone 63. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are arranged on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

The display drive units 22 and 24 include liquid crystal displays 241 and 242 (Liquid Crystal Display, hereinafter also referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 3). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 3) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate 20A is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various ones such as one having almost no light transmission, one having near transparency, one having a light amount attenuated to transmit light, one having a specific wavelength attenuated or reflected, and the like are used. Can do. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The light control plate 20A may be detachable with respect to the right optical image display unit 26 and the left optical image display unit 28, and a plurality of types of light control plates 20A may be exchanged and may be omitted.

The camera 61 is disposed at the end ER that is the other end of the right optical image display unit 26. The camera 61 captures an outside scene that is an external scenery in a direction opposite to the user's eye side, and acquires an outside scene image. The camera 61 of this embodiment shown in FIG. 1 is a monocular camera, but may be a stereo camera.
The shooting direction of the camera 61, that is, the angle of view, is taken from the front side direction of the head-mounted display device 100, in other words, at least a part of the outside view in the user's viewing direction when the head-mounted display device 100 is worn Direction. In addition, although the angle of view of the camera 61 can be set as appropriate, the imaging range of the camera 61 includes the outside world (outside scene) that the user visually recognizes through the right optical image display unit 26 and the left optical image display unit 28. A range is preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the light control plate 20A can be captured.
The camera 61 is an IR (infrared) camera that receives infrared light to pick up an image, a night vision camera that receives weak light and light in the visible near infrared region (VNIR: Visible and Near Infrared), or red. It may be a camera that captures light of a wavelength that is not visible to human eyes or difficult to see, such as a thermographic camera that receives external light and generates a captured image showing the heat distribution in the imaging range.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control device 10. The connection unit 40 includes a main body cord 48 connected to the control device 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and is connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  The specific specifications of the microphone 63 are arbitrary, and may be a directional microphone or an omnidirectional microphone. Examples of directional microphones include unidirectionality (Cardioid), narrow directivity (Super cardioid), sharp directivity (Hypercardioid), and superdirectivity (Ultra Cardioid). In the case where the microphone 63 has directivity, the sound from the line of sight of the user wearing the head-mounted display device 100 may be particularly well collected and detected. In this case, in order to ensure the directivity of the microphone 63, the microphone 63 or a part that accommodates the microphone 63 may have a structural feature. For example, in the example of FIG. 1, the microphone 63 and the connecting member 46 are designed so that the sound collection unit of the microphone 63 faces the user's line-of-sight direction with the user wearing the right earphone 32 and the left earphone 34. It only has to be. Alternatively, the microphone 63 may be embedded in the right holding unit 21 or the left holding unit 23. In this case, if a sound collecting hole is formed in the front side of the right holding unit 21 or the left holding unit 23, that is, the surface aligned with the right optical image display unit 26 and the left optical image display unit 28, the direction of the user's line of sight The directivity corresponding to can be given. The line-of-sight direction of the user is, for example, the direction that the right optical image display unit 26 and the left optical image display unit 28 face, and the center of the field of view that the user sees through the right optical image display unit 26 and the left optical image display unit 28. In other words, the shooting direction of the camera 61, and the like. Note that the direction of directivity of the microphone 63 may be variable. In this case, it may be configured to detect the user's line-of-sight direction and adjust the line-of-sight direction of the microphone 63 so as to face the direction.

  It is also possible to combine the right code 42 and the left code 44 into one code. Specifically, the lead wire inside the right cord 42 is drawn into the left holding portion 23 side through the inside of the main body of the image display unit 20, and is covered with a resin together with the lead wire inside the left cord 44 to be combined into one cord. Also good.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. Each of the end of the main body cord 48 opposite to the connecting member 46 and the control device 10 is provided with a connector (not shown) that fits with each other. By fitting / releasing the connector of the main body cord 48 and the connector of the control device 10, the control device 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control device 10 is a device for controlling the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is touched by a user with a finger.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. As the lighting unit 12, for example, an LED (Light Emitting Diode) is used. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content moving image between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

FIG. 2 is a diagram showing a schematic configuration of the information providing system 1 having the head-mounted display device 100.
The information providing system 1 has a configuration in which the head-mounted display device 100 and the information providing device 4 are communicably connected. Communication between the head-mounted display device 100 and the information providing device 4 is performed via a wireless communication line in this embodiment. In addition to the head-mounted display device 100 and the information providing device 4, the information providing system 1 may include a communication device (not shown) such as a wireless access point device or a router device that mediates communication.

The information providing device 4 can communicate with a plurality of head-mounted display devices 100, and the number of head-mounted display devices 100 included in the information providing system 1 is not particularly limited.
The information providing device 4 displays a communication unit 401 that communicates with the head-mounted display device 100, an input unit 402 that receives an input operation of an operator who operates the information providing device 4, and input contents and processing results by the input unit 402. The display unit 403 is provided. The communication unit 401 (first communication unit) executes various types of wireless communication such as a wireless LAN, WiFi (registered trademark), and Bluetooth (registered trademark) that comply with the IEEE 802.11 standard. The input unit 402 may include an input device such as a keyboard, a mouse, and a touch panel, and the display unit 403 may include a display screen such as a liquid crystal display. Further, the information providing apparatus 4 may be operated by being remotely accessed by another computer that can communicate with the information providing apparatus 4.
The information providing apparatus 4 may be a stationary computer or a portable device, and may be realized using, for example, a personal computer, a server computer, a tablet computer, a smartphone, or a portable phone. Moreover, there is no restriction | limiting in particular about the position of the information provision apparatus 4. FIG.

The information providing apparatus 4 includes a control unit 410. The control unit 410 includes a CPU, a ROM, a RAM, and the like (not shown), reads and executes a basic control program stored in the ROM, and controls each unit of the information providing apparatus 4. The control unit 410 functions as an identification unit 411, a communication control unit 412, and a transmission information generation unit 413 by executing the basic control program.
Further, the information providing apparatus 4 includes a storage unit 420 that stores data processed by the control unit 410. The storage unit 420 can use a storage device that stores data and programs in a nonvolatile manner using an electromagnetic storage medium, an optical storage medium, or a semiconductor memory. The storage unit 420 may be realized using a storage area of a ROM or a RAM included in the control unit 410. In the present embodiment, the storage unit 420 stores HMD identification information 421, HMD position information 422, and transmission information 423.

  The identification unit 411 identifies the head-mounted display device 100 included in the information providing system 1. The identification unit 411 only needs to distinguish at least one of the plurality of head-mounted display devices 100 from other head-mounted display devices 100. For example, in the information providing system 1, when any one head-mounted display device 100 is set as a specific target, the identification unit 411 determines whether or not each head-mounted display device 100 is a specific target. It only needs to be able to judge. In a case where more complicated control is possible, the identification unit 411 displays a plurality of head-mounted display devices 100 to be identified among the head-mounted display devices 100 of the information providing system 1 as other head-mounted display devices. It is only necessary to distinguish from the device 100. Furthermore, it is good also as a structure which can mutually identify the several head mounting type display apparatus 100 of specific object. Moreover, it is good also as a structure which the identification part 411 can identify all the head mounted display apparatuses 100 of the information provision system 1 individually.

  In the present embodiment, identification information is assigned to each head-mounted display device 100 so that it can be distinguished from other head-mounted display devices 100 within the information providing system 1. When the information providing system 1 uses a wireless LAN, a MAC (Media Access Control) address or an IP (Internet Protocol) address of the head-mounted display device 100 can be used as identification information. Further, unique identification information may be assigned to each head-mounted display device 100 in advance. For example, the name, number, and the like of the head-mounted display device 100. In this case, the head-mounted display device 100 stores the assigned identification information. The identification unit 411 acquires identification information of the head-mounted display device 100 that can communicate with the communication unit 401 from the head-mounted display device 100 and writes the identification information in the HMD identification information 421. In this configuration, the head-mounted display device 100 of the information providing system 1 can be specified individually.

The HMD identification information 421 includes identification information of the head-mounted display device 100 acquired by the identification unit 411 and call information necessary for calling the head-mounted display device 100 in the information providing system 1 in association with each other. . The call information is, for example, a network address of the head-mounted display device 100. When the identification information of the head-mounted display device 100 uses a network address, the HMD identification information 421 may include only the network address.
The HMD identification information 421 may include information indicating whether the head-mounted display device 100 is communicable with the information providing device 4 in association with the identification information of the head-mounted display device 100. Information that identifies a user who uses the head-mounted display device 100 may be included. The information for identifying the user is a name, a unique number assigned to the user, and the like.

  The identification unit 411 has a function of specifying the position of the head-mounted display device 100 included in the information providing system 1. The identification unit 411 writes the position of each head-mounted display device 100 in the HMD position information 422 in association with the identification information. The HMD position information 422 is information including the identification information and the position of the head-mounted display device 100 in association with each other.

As a method for the identification unit 411 to specify the position of the head-mounted display device 100, the following method may be mentioned.
The first method is a method in which the head-mounted display device 100 detects a position and transmits position information indicating the detected position to the information providing apparatus 4. The identification unit 411 specifies the position of the head-mounted display device 100 by acquiring position information received from the head-mounted display device 100 by the communication unit 401.
A specific method for detecting the position of the head-mounted display device 100 in the first method is, for example, a method using the GPS 115 provided in the head-mounted display device 100. In addition, a plurality of wireless transmitters (not shown) are installed in an area where the head-mounted display device 100 is used, and the head-mounted display device 100 uses the communication unit 117 to transmit wireless signals transmitted by these wireless transmitters. The position can be detected by the receiving method. In this case, the head-mounted display device 100 calculates the current position based on the position of the wireless transmitter and the wireless signal received from each wireless transmitter. Further, a tag indicating the current position is arranged in an area where the head-mounted display device 100 is installed, the head-mounted display device 100 reads the tag, and transmits information read from the tag to the information providing device 4. There is a way. The identification unit 411 has information for associating the tag arranged in the area with the position of the tag, and can identify the position of the head-mounted display device 100 based on this information. As the tag, a non-contact wireless communication tag (for example, an RFID (Radio Frequency IDentifier) tag), a graphic code (including a barcode and a two-dimensional code), or the like can be used.

  The second method is a method of determining an area where the head-mounted display device 100 is used and a specification position in the area. For example, in an application example of the information providing system 1 described later with reference to FIG. 6, the head-mounted display device 100 is used in a theater TH. When each head-mounted display device 100 is associated with a seat number, this seat number can be used as position information of the head-mounted display device 100.

  The third method is a method in which the identification unit 411 performs an operation of detecting the position of the head-mounted display device 100. For example, the identification unit 411 can calculate the position of the head-mounted display device 100 based on a communication delay or the like when the communication unit 401 communicates with the head-mounted display device 100. In the third method, a rough distance between the head-mounted display device 100 and the information providing device 4 is obtained using one communication unit 401, and the position of the head-mounted display device 100 is estimated. it can. If a communication device corresponding to the communication unit 401 is connected to the information providing device 4 and used, the position of the head-mounted display device 100 can be obtained more accurately.

  The identification unit 411 acquires the position of the head-mounted display device 100 by the method described above, and associates the information indicating the acquired position with the identification information of the head-mounted display device 100 in the HMD position information 422. Write.

The communication control unit 412 controls communication with the head-mounted display device 100 using the communication unit 401.
The communication control unit 412 specifies the HMD identification information when a specific head-mounted display device 100 is specified by an operation received by the input unit 402 and communication with the specified head-mounted display device 100 is requested. 421 is referred to. The communication control unit 412 controls the communication unit 401 based on information included in the HMD identification information 421 and executes communication with the head-mounted display device 100.

The transmission information generation unit 413 generates information to be transmitted to the head-mounted display device 100. Information generated by the transmission information generation unit 413 is transmitted to the head-mounted display device 100 by the communication unit 401 and displayed on the head-mounted display device 100. That is, the information generated by the transmission information generation unit 413 is information intended to be shown to the wearer of the head-mounted display device 100.
Information generated by the transmission information generation unit 413 is not particularly limited, but in the present embodiment, a case where text information including only text (characters) and image information including an image is generated as an example. The entity of the image included in this information may be vector data or raster data. A plurality of images may be included.
In the present embodiment, in an application example described later with reference to FIG. 6, information including characters and images for guiding a user wearing the head-mounted display device 100 is referred to as guidance information. The specific mode of the guidance information is arbitrary. For example, the guidance information image is a graphic image of a character, an image including symbols, arrows, guidance lines, etc., and a user's surroundings. An image (for example, a route map in a building, a map, etc.) is mentioned. Further, guidance information including only characters, guidance information including only symbols, figures, or images may be used.

  The information generated by the transmission information generation unit 413 is information that is intended to be transmitted to a specific head-mounted display device 100, and is transmitted in common to a plurality of head-mounted display devices 100. There is target information. When generating information to be transmitted to the plurality of head-mounted display devices 100, the transmission information generation unit 413 can use a fixed character string or image. These fixed character strings and images are stored in the storage unit 420 in advance. The transmission information generation unit 413 can also generate information suitable for a specific head-mounted display device 100. In this case, the transmission information generation unit 413 generates a character string or an image, or a fixed character string or image. In addition, another character string or image may be added.

  The transmission information generation unit 413 generates information to be transmitted to the head-mounted display device 100 under the control of the communication control unit 412. Whether the information generated by the transmission information generation unit 413 is text information or image information is determined by the content input by the input unit 402 or the content set by the communication control unit 412. Similarly, the transmission information generation unit 413 generates information for a specific head-mounted display device 100 or generates information common to the plurality of head-mounted display devices 100. It depends on the content to be input or the content set by the communication control unit 412.

  Information generated by the transmission information generation unit 413 is stored in the storage unit 420 as transmission information 423. The communication control unit 412 reads the transmission information 423 from the storage unit 420 and transmits it by the communication unit 401.

FIG. 3 is a block diagram illustrating a functional configuration of the head-mounted display device 100.
FIG. 3 illustrates an external device OA that can be connected to the head-mounted display device 100. The external device OA is, for example, a personal computer (PC), a mobile phone terminal, a game terminal, or the like. The external device OA is used as an image supply device that supplies an image to the head-mounted display device 100.

The control device 10 of the head-mounted display device 100 includes a control unit 140, an operation unit 135, an input information acquisition unit 110, a storage unit 120, a power supply 130, an interface 180, a transmission unit (Tx) 51, and And a transmission unit (Tx) 52.
The operation unit 135 detects an operation by the user. The operation unit 135 includes the determination key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18 shown in FIG.

The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. As a signal corresponding to the operation input, for example, there is an operation input to the track pad 14, the direction key 16, and the power switch 18.
The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used.

  The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. The storage unit 120 may store image data to be displayed on the image display unit 20 of the head-mounted display device 100. The storage unit 120 may store display data acquired by the data acquisition unit DA, information received from the information providing device 4 by the communication unit 117, and the like.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control device 10. As the interface 180, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface can be used.

  The control unit 140 reads out and executes the computer program stored in the storage unit 120, thereby realizing the function of each unit. That is, the control unit 140 functions as an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, an object detection unit 171, a distance detection unit 173, an information display control unit 178, and a display control unit 190.

  A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can acquire the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.

The communication unit 117 (second communication unit) performs various types of wireless communication such as wireless LAN, WiFi, Bluetooth, and the like that comply with the IEEE 802.11 standard. The communication unit 117 may transmit and receive image data using Miracast (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and performs control for displaying an image on the image display unit 20. On the other hand, when the external device OA is wired to the interface 180, the control unit 140 acquires content data from the interface 180 and performs control for displaying an image on the image display unit 20. Therefore, the communication unit 117 and the interface 180 are hereinafter collectively referred to as a data acquisition unit DA.
The data acquisition unit DA acquires content data to be displayed by the head-mounted display device 100 from the external device OA. The content data includes display data to be described later, and the display data can be various data such as image data and text data.

  Further, the communication unit 117 receives information transmitted from the communication unit 401 of the information providing apparatus 4 (FIG. 2) and outputs the information to the control unit 140. The control unit 140 processes the information received from the information providing apparatus 4 by the communication unit 117 by the function of the information display control unit 178 as will be described later.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 frame by frame as image data (Data in the figure) of the target image. This image data is, for example, RGB data.
Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control device 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a vibration sensor 65 and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor, and is arranged inside the image display unit 20 as shown in FIG. In the example of FIG. 1, the right holding unit 21 is built in the vicinity of the end ER of the right optical image display unit 26. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). The 9-axis sensor 66 is provided in the image display unit 20. For this reason, the 9-axis sensor 66 detects the movement of the user's head while the user wears the image display unit 20 on the head. Since the orientation of the image display unit 20 is known from the detected movement of the user's head, the control unit 140 can estimate the user's line-of-sight direction.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are provided. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 that are input via the reception unit 53. . The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that changes image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts image light emitted from the left LCD 242 to light beams in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

The head-mounted display device 100 displays display data so as to overlap the outside scene when the user views the outside scene through the right optical image display unit 26 and the left optical image display unit 28.
The target detection unit 171 performs control for causing the camera 61 to perform imaging, and acquires a captured image. The captured image is output from the camera 61 as color image data or monochrome image data. The camera 61 outputs an image signal, and the target detection unit 171 generates image data that conforms to a predetermined file format from the image signal. May be.
The target detection unit 171 analyzes the acquired captured image data. The target detection unit 171 searches the captured image for an image that matches a predetermined feature, and detects the matching image as an image of the object. The target object is an object or a person that exists in the imaging direction of the camera 61, that is, the user's line-of-sight direction. The feature of the target used by the target detection unit 171 is stored in the storage unit 120 as feature amount data, for example. The feature amount data includes the feature amount of the image detected from the captured image, and the target object can be specified by searching a portion corresponding to the feature amount data from the captured image data. In the present embodiment, since the imaging direction of the camera 61 is the line-of-sight direction of the user, the target detection unit 171 can detect the target object based on the captured image data. The object is a person or an object that can be distinguished from the surrounding background or another object, and it does not matter whether the object is movable or not.

The distance detection unit 173 obtains the distance to the object detected by the object detection unit 171. For example, the distance detection unit 173 obtains the distance to the object based on the size of the image of the object detected by the object detection unit 171 in the captured image of the camera 61.
The head-mounted display device 100 may include a distance meter that detects the distance to the object using laser light or ultrasonic waves. For example, the distance meter includes a light source of laser light and a light receiving unit that receives reflected light of the laser light emitted from the light source, and detects a distance to an object based on a light receiving state of the laser light. The distance meter may be an ultrasonic distance meter, for example. That is, a distance meter that includes a sound source that emits ultrasonic waves and a detection unit that detects ultrasonic waves reflected on the object, and detects the distance to the object based on the reflected ultrasonic waves may be used. Further, the distance meter can be configured by combining a distance meter using laser light and a distance meter using ultrasonic waves. Such a distance meter is preferably provided in the right holding unit 21 or the right display driving unit 22 of the image display unit 20, and may be installed, for example, facing the front side on the surface aligned with the light control plate 20A. The direction in which the distance meter measures the distance is preferably the user's line-of-sight direction, similar to the imaging direction of the camera 61.
The distance detection unit 173 detects the distance from the camera 61 or the distance meter to the object, and this distance can be regarded as the distance from the user of the head-mounted display device 100 to the object. When the target detected by the target detection unit 171 is not a specific object, the distance detection unit 173 detects the distance to this target, for example, using the center of the captured image as a virtual target.

  The information display control unit 178 displays characters and images on the image display unit 20 based on the display data acquired by the data acquisition unit DA. The information display control unit 178 may perform display based on the acquired display data when the data acquisition unit DA acquires the display data. Further, the information display control unit 178 may extract display target data from the display data acquired by the data acquisition unit DA, and display characters and images based on the extracted data. The information display control unit 178 of the present embodiment extracts display data related to the object detected by the object detection unit 171 from the display data acquired by the data acquisition unit DA.

Further, the information display control unit 178 generates data for displaying characters and images based on information received by the communication unit 117 from the information providing device 4, outputs the data to the display control unit 190, and outputs the data to the image display unit 20. Display text and images.
Here, the information display control unit 178 may set display attributes of characters and images to be displayed based on information received from the information providing device 4. In this case, the characters and images based on the information received from the information providing device 4 may be displayed so as to stand out from the characters and images displayed based on the display data.
The information display control unit 178 controls the display control unit 190 to cause the image display unit 20 to display an image based on the display image data generated by the image generation unit 177.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker in the left earphone 34 (not shown) connected to the connecting member 46. (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing is performed on the audio signal, and different sounds with different frequencies or the like are output from each of the right earphone 32 and the left earphone 34, for example.

  In addition, the sound processing unit 170 acquires sound collected by the microphone 63, converts it into digital sound data, and performs processing related to sound. For example, the speech processing unit 170 performs speaker recognition that identifies a person who speaks for each voice by recognizing voices of a plurality of persons separately by extracting and modeling features from the acquired speech. May be.

A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can detect the detection value of the triaxial sensor 113 and detect the movement and the direction of movement of the control device 10.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Further, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 10.

The interface 180 is an interface for connecting various image supply apparatuses OA that are content supply sources to the control apparatus 10. The content supplied by the image supply device OA includes a still image or a moving image, and may include sound. Examples of the image supply device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.
Here, it is also possible to connect the image supply device OA to the control device 10 by a wireless communication line. In this case, the image supply device OA performs wireless communication with the communication unit 117 and transmits content data using a wireless communication technique such as Miracast.

  FIG. 4 is a flowchart showing the operation of the head-mounted display device 100, and particularly shows a display process using the function of the information display control unit 178. This display process is a process in which characters and images are displayed on the image display unit 20 based on display data related to the outside scene while the user views the outside scene through the right optical image display unit 26 and the left optical image display unit 28. It is.

First, the control unit 140 acquires display data by the data acquisition unit DA (step S11), and stores the acquired data in the storage unit 120 (step S12).
Subsequently, the target detection unit 171 executes a target detection process (step S13). The information display control unit 178 selects display data related to the object detected by the object detection unit 171 from the display data stored in the storage unit 120, and displays characters and images based on the selected display data ( Step S14). Here, the information selected by the information display control unit 178 corresponds to related information. The related information is information related to the object detected by the object detection unit 171, and may be any information that can be displayed by the image display unit 20, such as characters, images, moving images, and graphics. The related information is stored in advance in the storage unit 120, for example, and is associated with feature amount data used when the target detection unit 171 detects the target, for example. In this case, related information related to the object can be acquired from the feature amount data of the object detected by the object detection unit 171.

Thereafter, the control unit 140 determines whether or not all of the data acquired in step S11 has been processed in steps S12 to S14 (step S15). If there is unprocessed data (step S15; NO), the control unit 140 returns to step S13 and performs processing on the unprocessed data. For example, the information display control unit 178 selects display data related to the object detected by the object detection unit 171 in step S14, but the display data not selected here may be processed.
When all the data has been processed (step S15; YES), the control unit 140 determines whether or not to end the display (step S16). When the display is continued (step S16; NO), the control unit 140 returns to step S11. When the display is terminated according to the operation detected by the operation unit 135 (step S16; YES), the control unit 140 stops the display by the display control unit 190 and ends the present process.

FIG. 5 is a flowchart showing the operation of the information providing system 1, (A) shows the operation of the information providing device 4, and (B) shows the operation of the head-mounted display device 100. FIG. 5B illustrates the operation of the head-mounted display device 100 that is the target of information transmission among the plurality of head-mounted display devices 100 included in the information providing system 1.
The operation in FIG. 5 is an operation in which the information providing device 4 generates information and transmits the information to the head-mounted display device 100, and the head-mounted display device 100 performs display. The operation in FIG. 5 is started with an instruction operation on the information providing apparatus 4 as a trigger. For this reason, for example, the operation of FIG. 5 may be started while the head-mounted display device 100 is executing the processing of FIG.

The information providing apparatus 4 waits until an information transmission instruction is given (step S21). When the information transmission instruction is received by the input unit 402 (step S21; YES), the identification unit 411 identifies the position of the head-mounted display device 100 and writes the identified position in the HMD position information 422 (step S22). ). In step S22, the head-mounted display device 100 performs processing such as transmitting position data to the information providing device 4 as necessary (step S31).
In step S <b> 22, the identification unit 411 only needs to specify the position of the head-mounted display device 100 that is the transmission destination of the information specified by the input of the input unit 402. Further, the identification unit 411 may perform an operation of specifying the position of the head-mounted display device 100 at a predetermined cycle and updating the HMD position information 422, and in this case, the operation of step S22 is omitted. May be.

The communication control unit 412 selects the head-mounted display device 100 as the information transmission destination based on the input content received by the input unit 402 (step S23). The number of head-mounted display devices 100 selected by the communication control unit 412 is not limited, but when selecting a plurality of head-mounted display devices 100, the same information is stored in the plurality of head-mounted display devices 100. Is sent. Further, the information providing device 4 may be configured to be able to perform a simultaneous call for transmitting the same information to all the head-mounted display devices 100 by an operation on the input unit 402. In this case, in step S23, all the head-mounted devices are mounted. The mold display device 100 is selected.
Subsequently, the transmission information generation unit 413 generates information to be transmitted to the head-mounted display device 100 (step S24). The content of the information generated here depends on the input content received by the input unit 402 as described above. The transmission information generation unit 413 stores the generated information in the storage unit 420.

The communication control unit 412 reads out the transmission information 423 generated by the transmission information generation unit 413 from the storage unit 420 and transmits it to the head-mounted display device 100 selected in step S23 (step S25).
The head-mounted display device 100 receives the information transmitted from the information providing device 4 by the communication unit 117 (step S32), and the information display control unit 178 generates and displays data for displaying the received information. It outputs to the control part 190 and performs a display (step S33).
The communication control unit 412 determines whether or not the transmission of information to all the head-mounted display devices 100 instructed has been completed (step S26). If the transmission has not been completed (step S26; NO), Returning to step S23, transmission to another head-mounted display device 100 is executed. When the instructed transmission is completed (step S26; YES), this process is terminated.

FIG. 6 is an explanatory diagram illustrating a typical application example of the information providing system 1. (A) is explanatory drawing of a use environment, (B)-(E) shows the example of a display of a head mounted display apparatus.
The theater TH shown in FIG. 6A has a configuration in which a large number of seats SH for the audience including the user of the head-mounted display device 100 are arranged so as to face the stage ST. The user of the head-mounted display device 100 uses the head-mounted display device 100 when sitting on the seat SH and looking at the stage ST. In this embodiment, a plurality of users are in the theater TH and use the head-mounted display device 100.

  The visual field VR in FIG. 6B shows the visual field seen by the user through the right optical image display unit 26 and the left optical image display unit 28 of the image display unit 20. Since the image display unit 20 has a characteristic that it can be seen through the outside scene, the stage ST is visible in the visual field VR. The visual field VR includes a curtain CT disposed above and on the left and right ends of the stage ST, and stage sleeves SS on the left and right sides of the stage ST. The performer A can be seen on the stage ST. In this example, there is a performer A on the stage ST, and the user is watching the performer A.

In this example, the display data acquired by the head-mounted display device 100 in step S11 (FIG. 4) is text data related to the theater program performed at the theater TH, and the dialogue and the theater that the performer A speaks. Contains descriptive text about the program. In step S11, the control unit 140 acquires text data for all or part of the play program. The information display control unit 178 selects and displays the line of the performer A, which is an object located in the user's line-of-sight direction, from the acquired text data. The dialogue displayed here is illustrated as a speech-corresponding character string 301.
The information display control unit 178 acquires or selects display data and displays it in accordance with the progress of the play program.

FIG. 6C shows an example in which information transmitted by the information providing device 4 is displayed. In the example of FIG. 6C, the head-mounted display device 100 displays the calling character string 321 based on the information transmitted by the information providing device 4 in a state where the voice-corresponding character string 301 is displayed. The calling character string 321 is a character string that requests the user wearing the head-mounted display device 100 to come to the reception. The calling character string 321 is visible only to the user who wears the head-mounted display device 100. For this reason, it is not noticed by other people in the theater TH, and the calling character string 321 does not disturb the theater being shown in the theater TH.
As described above, the information providing device 4 identifies and selects the head-mounted display device 100, and transmits information to the selected head-mounted display device 100, whereby the selected head-mounted display device 100 is selected. Information can be conveyed to the user wearing the.

  As shown in FIG. 6C, the head-mounted display device 100 displays the call character string 321 with emphasis over the voice-corresponding character string 301. In the example of FIG. 6C, emphasis is made by increasing the character size, but character decorations such as bold, italic, underline, etc., that make the display color of the character different from the voice-corresponding character string 301 are added. You may emphasize by the method of doing. Further, the calling character string 321 may be emphasized by adding a background color. Further, the call character string 321 and the background of the call character string 321 may be emphasized by reducing the transparency of the background in a range overlapping the call character string 321 by lowering the transparency of the background. Thereby, the user wearing the head-mounted display device 100 notices the calling character string 321 promptly, so that information can be transmitted more quickly.

FIG. 6D shows a display example when information including an image is transmitted from the information providing device 4 to the head-mounted display device 100. In FIG. 6D, the guidance image 325 is displayed by the head-mounted display device 100.
As shown in FIG. 6D, when the head-mounted display device 100 performs display based on the information received from the information providing device 4, the display of the speech-corresponding character string 301 displayed so far is stopped. May be.
The guide image 325 includes a calling character string 326 that guides the user to come to the reception, and an image showing the reception location designated by the calling character string 326. This image is a plan view of the theater TH, and includes symbols indicating the arrangement of the seats SH and the place of reception in the theater TH. The guidance image 325 can convey detailed information to the user along with the message.
The guidance image 325 is information that can be used regardless of the current position of the user and the user's attributes when the user is called for reception. The same applies to the calling character strings 321 and 326. Therefore, the guide image 325 and the call character strings 321 and 326 are stored in the storage unit 420 in advance in the information providing apparatus 4, and the transmission information generation unit 413 stores the guide image 325 and the call character string stored in the storage unit 420. 321 and 326 may be used as transmission information 423 as they are.

FIG. 6E shows another display example when information including an image is transmitted from the information providing device 4 to the head-mounted display device 100, and a guide image 327 is displayed.
The guidance image 327 includes a call character string 326 that guides the user to come to the reception, an image showing the reception location specified by the call character string 326, and the position from the position of the head-mounted display device 100 to the reception. Guidance information 328 for guiding the route of The guidance information 328 is information generated by the transmission information generation unit 413 based on the position of the head-mounted display device 100 specified by the identification unit 411. By using the guide image 327 including the guide information 328, information adapted to the position of the user can be transmitted to the user.

By the way, when the brightness of the outside scene is dark, the display brightness of the guide image 325 displayed by the information display control unit 178 is relatively high with respect to the outside scene, for example, in the theater TH where the illumination is darkened, or in a movie theater. . In this case, the guidance image 325 can be visually recognized well, but the outside scene may be difficult to visually recognize.
Therefore, the information display control unit 178 may adjust the display form of the information including the guidance image according to the state of the outside scene on the assumption that the guidance image is visually recognized while being superimposed on the outside scene. For example, when the brightness of the outside scene seen through the image display unit 20 is not more than a predetermined value, the visibility of the information displayed so as to overlap the outside scene may be reduced. For example, when the display brightness of information is lowered, it becomes easier to see the outside scene through the information, and the outside scene becomes easier to see than the information. In addition, when only an outline is displayed when displaying an image, it is possible to make it easy to visually recognize the outside scene.

  In addition, for example, when an arrow or a straight line is used as the guidance image and a passage through which the user passes is seen as an outside scene, the arrow or the straight line may be displayed so as to overlap the passage of the outside scene. In this case, since the passage looks conspicuous by AR (Augmented Reality) in which an arrow or a straight line is added to the passage, it is easy to recognize. Here, the display brightness of information such as arrows and straight lines may be adjusted according to the outside scene. Further, the information display control unit 178 may display all or a part of the captured image obtained by capturing the outside scene with the camera 61. In this case, it is preferable that the information display control unit 178 can adjust the display position of the captured image according to the outside scene that the user sees through the image display unit 20. As a result, it is possible to assist the user's vision by displaying the same captured image as the outside scene in an easy-to-view (high visibility state) state where the outside scene is dark and difficult to see. Here, the information display control unit 178 may display a composite image in which information is superimposed on the captured image displayed on the image display unit 20. In addition, when the camera 61 is an IR camera, a night vision camera, or a thermography camera as described above, the user can easily view the outside scene that is not visible or difficult to see with the naked eye. Can help people's vision more effectively.

In addition to the example shown in FIG. 6, a specific application of the information providing system 1 will be illustrated.
(1) The information providing system 1 can provide outdoor guidance. Specifically, the head-mounted display device 100 outputs guidance voices and images corresponding to the position of the head-mounted display device 100 outdoors. An example of outputting tourist guide voices and images for famous places, historic sites, temples and shrines, stores, etc. in a limited area of a city or region is given. In this case, not only outputting guidance text and images as shown in FIGS. 6C to 6E, but also displaying the image of the person performing guidance on the image display unit 20 as an avatar image, for example. A mode is conceivable in which an image simulating a person speaking a word is output corresponding to the position of the head-mounted display device 100. Along with the display of this image, the sound processing unit 170 may output sound. In this example, guidance by AR using an avatar can be performed.
(2) In a public transportation facility such as an airport or a station, when a user who wears the head-mounted display device 100 becomes a passenger, an example of getting on or boarding the transportation facility is given. More specifically, guidance and guidance according to the location of the user regarding the series of actions from airplane guidance at the airport, check-in at the airline counter, departure from the airport, baggage inspection, boarding the airplane. An example is given. For example, the present invention can be applied to a business that lends the head-mounted display device 100 to a customer who is on a first class of an airplane and provides guidance and guidance services.
(3) The present invention can also be applied to an example in which guidance and guidance to entrances, seats, toilets, shops, and other necessary places in sports facilities such as baseball fields and Olympic stadiums are performed.
(4) In a large-scale store such as a department store or a mall having a plurality of stores and facilities, the present invention can be applied to an example in which guidance and guidance to stores, facilities, toilets, and other necessary places are performed.
(5) As described above, it is possible to provide guidance regardless of whether it is outdoors or indoors, without being limited to facilities and limited areas managed by a specific operating entity. For example, if there is a possibility of danger to the user based on the captured image of the camera 61, the sound collected by the microphone 63, the movement of the head-mounted display device 100 detected by the 9-axis sensor 66, etc. Outputs images and sounds that contain the information. For example, the head-mounted display device 100 detects an obstacle, the presence of a step, the approach of a person or an object in the direction in which the user passes, and outputs warning information. For approaching people and objects, the degree of danger may be determined from the approach speed and the characteristics of the image, and information may be output. In addition, it is more useful if the camera 61 can detect an object or a person that is difficult to recognize by human vision, such as a camera that receives infrared light, a night vision camera, or a wide-angle camera.
In each of the above examples, as a method of obtaining the position of the head-mounted display device 100, for example, a method of detecting the current position with the GPS 115, a wireless communication between the wireless access point installed at a known location and the communication unit 117 The method of calculating the position of the head-mounted display device 100 by the above, the method of calculating the position by receiving the light beacon using the light such as infrared light or the radio wave beacon by the head-mounted display device 100 can be used. . Further, the information providing device 4 that provides information to the head-mounted display device 100 may be installed at a position where direct wireless communication with the head-mounted display device 100 can be performed, or the head-mounted display device. It may be installed in a remote place away from 100 and connected to the head-mounted display device 100 via a wireless communication line or a wired communication line. Furthermore, the information providing system 1 can of course be configured to include a plurality of information providing devices 4 and a plurality of head-mounted display devices 100.

  As described above, the information providing system 1 according to the embodiment to which the present invention is applied is a head-mounted display device 100 that is used while being worn on a user's body, and communicates with the head-mounted display device 100. And an information providing device 4 that can be connected. The information providing device 4 identifies a communication unit 401 that communicates with the head-mounted display device 100, an identification unit 411 that identifies the head-mounted display device 100, and a head-mounted display device 100 that is an information transmission destination. And a communication control unit 412 that transmits information by the communication unit 401. The head-mounted display device 100 includes a communication unit 117 that communicates with the information providing device 4 and an information display control unit 178 that receives information from the information providing device 4 and displays the information on the image display unit 20. Therefore, the head-mounted display device 100 can be identified and transmitted, and information can be individually provided to the user who uses the head-mounted display device 100.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize an image and transmits an outside scene. The head-mounted display device 100 includes a target detection unit 171 that detects a target in the direction of the user's line of sight, and the information display control unit 178 displays related information related to the target detected by the target detection unit 171 as an image. It is displayed on the display unit 20. Further, the information display control unit 178 causes the image display unit 20 to display information received from the information providing apparatus 4 through the communication unit 117. For this reason, the related information related to the object located in the user's line-of-sight direction and the information transmitted by the information providing device 4 to the head-mounted display device 100 can be displayed so as to be seen together with the outside scene. For this reason, information can be shown to the user by making use of the characteristics of the image display unit 20 that transmits the outside scene.

Here, the information display control unit 178 causes the right optical image display unit 26 and the left optical image display unit 28 to display an image having parallax (angle of convergence) including related information, and allows the user to display a stereoscopic (3D) image. You may make it visually recognize. In this case, the information display control unit 178 may set or change whether the related information is displayed as a stereoscopic image or a planar image as one display mode.
When the distance detection unit 173 detects the distance to the object, the information display control unit 178 can display information in a manner corresponding to the detected distance. For example, when viewing a stereoscopic image, the information display control unit 178 determines the parallax of the image shown to the user in accordance with the distance obtained by the distance detection unit 173. As an example, the information display control unit 178 displays the stereoscopic image data for display from the data acquired by the data acquisition unit DA so that the parallax is smaller as the distance to the object is shorter and the parallax is larger as the distance is longer. Generate. The information display control unit 178 may generate stereoscopic image data corresponding to the type of the object. For example, when the target detected by the target detection unit 171 is a specific human or a human face imitating a character, or is similar to a general human face or a human face, or is a face The parallax may be changed according to whether the face is in front or the like. In this case, when the object is a face that faces the user from the front of the user, the parallax may be reduced so that an image overlapping the face can be seen near the user. Furthermore, the image display control unit 176 may increase or decrease the parallax according to the size of the image of the object in the captured image of the camera 61. In this case, if the data indicating the attribute of the object is included in the feature amount data used by the object detection unit 171, the object detection unit 171 can detect the object and determine the attribute of the object.

Furthermore, the object detection unit 171 is not limited to the one that detects the object by detecting the image of the object from the captured image of the camera 61. For example, the target detection unit 171 may irradiate ultrasonic waves or light in the direction of the user's line of sight and detect the target object based on the reflection.
Alternatively, the target detection unit 171 may use the above IR camera as the camera 61. In this case, the storage unit 120 stores image feature amount data for detecting an image of the target from the infrared captured image. May be stored.

  Furthermore, instead of the captured image data processed by the target detection unit 171 and the distance detection unit 173, captured image data captured and input by a camera externally connected to the head-mounted display device 100 may be used. The externally connected camera may be, for example, a camera connected to the interface 180 via USB or the like, or a camera that has a wireless communication function capable of communicating with the communication unit 117 and wirelessly transmits captured image data. Also good.

Further, the information display control unit 178 may display the information received from the information providing device 4 by the communication unit 117 on the image display unit 20 in a manner that is emphasized over the related information together with the related information. In this case, the related information related to the object located in the user's line of sight and the information transmitted from the information providing device 4 to the head-mounted display device 100 can be displayed together. In addition, the information transmitted from the information providing device 4 to the head-mounted display device 100 can be displayed with emphasis over the related information, and the user can be strongly noticed.
For example, the communication control unit 412 may transmit guidance information including an image for guiding the user to the head-mounted display device 100 through the communication unit 401. In this case, the information display control unit 178 displays the guidance images 325 and 327 for guiding the user on the image display unit 20 based on the guidance information received by the communication unit 117, so that the user can be guided smoothly.

  The identifying unit 411 identifies the head-mounted display device 100 and specifies the position of each head-mounted display device 100. The communication control unit 412 may transmit guidance information corresponding to the position of the head-mounted display device 100 specified by the identification unit 411 using the communication unit 401. In this case, guidance information corresponding to the position of the head-mounted display device 100 is transmitted, and the head-mounted display device 100 displays a guidance image 327, for example. In this case, information corresponding to the position of the user can be provided to the user who uses the head-mounted display device 100 to guide the user more appropriately.

  The communication control unit 412 may transmit common information to the plurality of head-mounted display devices 100. Further, a specific head-mounted display device 100 may be identified by the identification unit 411 and information different from the common information may be transmitted. Thereby, common information can be transmitted and displayed on the plurality of head-mounted display devices 100, and information that is not common to a specific head-mounted display device 100 can be transmitted. For example, the communication control unit 412 transmits information on the guidance image 325 illustrated in FIG. 6D to the plurality of head-mounted display devices 100, and among them, some of the head-mounted display devices 100. Transmits the information related to the guidance information 328 shown in FIG. In this case, some head-mounted display devices 100 display a guidance image 325, and some head-mounted display devices 100 display a guidance image 327 including guidance information 328. In this way, information can be transmitted and displayed in various forms to the plurality of head-mounted display devices 100.

In addition, this invention is not restricted to the structure of the said embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
In the said embodiment, the example which performs the display based on the information received from the information provision apparatus 4 was shown, while the head mounted display apparatus 100 is performing the display based on the display data acquired by the data acquisition part DA. However, the present invention is not limited to this. If the head-mounted display device 100 is operating and information can be received from the information providing device 4, the operation shown in FIG. The mold display device 100 can perform display. Further, the head-mounted display device 100 is not limited to display data (related information) related to the object detected by the object detection unit 171 and can also display all display data acquired by the data acquisition unit DA. is there. Of course, it is possible to perform display based on the information received from the information providing apparatus 4 during the display of the display data.

  Further, as the image display unit, instead of the image display unit 20, another type of image display unit such as an image display unit worn like a hat may be adopted, corresponding to the left eye of the user. What is necessary is just to provide the display part which displays an image, and the display part which displays an image corresponding to a user's right eye. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display incorporated in a body protective device such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile.

Furthermore, although the image display unit 20 and the control device 10 are separated and described as an example in the above embodiment and connected via the connection unit 40, the control device 10 and the image display unit 20 are integrated. It is also possible to be configured to be mounted on the user's head.
In addition, the control device 10 and the image display unit 20 are connected by a longer cable or a wireless communication line. As the control device 10, a notebook computer, a tablet computer or a desktop computer, a game machine, a portable phone, a smartphone, A portable electronic device including a portable media player, other dedicated devices, or the like may be used.

  Further, for example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used, or an LCOS (Liquid crystal) may be provided. On silicon and LCoS are registered trademarks), and digital micromirror devices can also be used. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. A configuration that allows the user to visually recognize an image may be employed. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  As an optical system that guides image light to the user's eyes, an optical member that transmits external light that is incident from the outside toward the apparatus and that enters the user's eyes together with the image light can be employed. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

  The display device of the present invention is not limited to a head-mounted display device, and can be applied to various display devices such as a flat panel display and a projector. The display device of the present invention may be any device that allows an image to be visually recognized by external light and image light. For example, a configuration in which an image by image light is visually recognized by an optical member that transmits external light is exemplified. Specifically, the above-described head-mounted display includes an optical member that transmits external light, and a light-transmitting flat surface or curved surface (glass) that is fixedly or movably installed at a position away from the user. It is also applicable to a display device that projects image light onto a transparent plastic or the like. As an example, there is a configuration of a display device that projects image light onto a window glass of a vehicle so that a user on board or a user outside the vehicle can visually recognize the scenery inside and outside the vehicle together with the image by the image light. . In addition, for example, image light is projected onto a transparent, translucent, or colored transparent display surface that is fixedly installed such as a window glass of a building, and is displayed along with the image by the image light to a user around the display surface. A configuration of a display device that visually recognizes a scene through the surface can be given.

  Further, at least a part of each functional block shown in FIG. 3 may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in FIG. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 140 may be stored in the storage unit 120 or a storage device in the control device 10, or the program stored in an external device is acquired via the communication unit 117 or the interface 180. It is good also as a structure to execute. Further, in the configuration formed in the control device 10, only the operation unit 135 may be formed as a single user interface (UI), or the power source 130 in the above embodiment is formed independently and is replaceable. It may be a configuration. Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 illustrated in FIG. 3 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 formed in the control device 10 and the CPU formed in the image display unit 20. It is good also as a structure by which the function to perform is divided | segmented separately.

  DESCRIPTION OF SYMBOLS 1 ... Information provision system, 4 ... Information provision apparatus, 10 ... Control apparatus, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding part, 24 ... Left display Drive unit, 26 ... right optical image display unit, 28 ... left optical image display unit, 61 ... camera (imaging unit), 63 ... microphone, 100 ... head-mounted display device (display device), 117 ... communication unit (first) 2 communication units), 120 ... storage unit, 140 ... control unit, 150 ... operating system, 160 ... image processing unit, 170 ... audio processing unit, 171 ... target detection unit, 173 ... distance detection unit, 178 ... information display control , 180 ... interface, 190 ... display control unit, 201 ... right backlight control unit, 202 ... left backlight control unit, 211 ... right LCD control unit, 212 ... left LCD control unit, 221 ... right backlight, 222 ... Backlight, 241 ... right LCD, 242 ... left LCD, 251 ... right projection optical system, 252 ... left projection optical system, 261 ... right light guide plate, 262 ... left light guide plate, 401 ... communication unit (first communication unit) , 402 ... input unit, 403 ... display unit, 410 ... control unit, 411 ... identification unit, 412 ... communication control unit, 413 ... transmission information generation unit, 420 ... storage unit, 421 ... HMD identification information, 422 ... HMD position information 423: Information for transmission, DA: Data acquisition unit.

Claims (12)

A display device used by being worn on a user's body, and an information providing device connected to be able to communicate with the display device,
The information providing apparatus includes:
A first communication unit communicating with the display device;
An identification unit for identifying the display device;
A communication control unit that identifies the display device to which information is transmitted by the identification unit, and transmits information by the first communication unit;
The display device
A second communication unit that communicates with the information providing device;
An information display control unit that receives information from the information providing device by the second communication unit and displays the information on a display unit;
An information provision system characterized by The display device
The display unit that allows a user to visually recognize an image and transmits an outside scene;
An object detection unit for detecting an object in the user's line-of-sight direction,
The information display control unit displays related information related to the object detected by the object detection unit on the display unit,
Causing the display unit to display information received from the information providing apparatus by the second communication unit;
The information providing system according to claim 1.   The information display control unit displays information received from the information providing apparatus by the second communication unit on the display unit together with the related information in a manner that is emphasized more than the related information. The information providing system according to claim 2.   4. The information providing system according to claim 2, wherein the information display control unit displays information received from the information providing device at a display position overlapping the outside scene. The communication control unit of the information providing device transmits guidance information including an image for guiding the user to the display device by the first communication unit,
The said information display control part displays the image which guide | induces the said user on the said display part based on the said guidance information received by a said 2nd communication part, The any one of Claim 1 to 4 characterized by the above-mentioned. Information providing system described in 1. The identification unit of the information providing device identifies the display device and identifies the position of each display device;
The information providing system according to claim 5, wherein the communication control unit transmits the guidance information corresponding to the position of the display device specified by the identification unit using the first communication unit.   The communication control unit of the information providing device transmits common information to a plurality of the display devices, and further identifies information specific to the display device by the identification unit and differs from the common information. The information providing system according to claim 1, wherein the information providing system transmits the information. The display device
With an imaging unit,
The target detection unit is configured to detect the target that the user sees through the display unit based on a captured image of the imaging unit;
The information providing system according to claim 2 or 3, characterized by the above.   The information providing system according to claim 8, wherein the imaging unit is an IR camera, a night vision camera, or a thermography camera. The display device includes a distance detection unit that detects a distance between the object detected by the target detection unit and the user,
10. The information providing system according to claim 1, wherein the information display control unit displays information in a manner corresponding to the distance detected by the distance detection unit. A display device used by being worn on a user's body,
A display unit for allowing the user to visually recognize an image and transmitting an outside scene;
A communication unit that communicates with an external information providing device;
An object detection unit for detecting an object in the line-of-sight direction of the user;
Information related to the object detected by the object detection unit, and information received from the information providing device by the communication unit, an information display control unit for displaying on the display unit,
A display device comprising: A method for controlling a display device that is used by being attached to a user's body, including a display unit that allows a user to visually recognize an image and transmit an outside scene.
Communicating with an external information providing device to receive information transmitted by the information providing device;
Detecting an object in the user's gaze direction,
Displaying related information related to the object and information received from the information providing device by the display unit;
A control method of a display device characterized by the above.

Images