JP7287950B2 - COMMUNICATION METHOD, COMMUNICATION DEVICE, AND PROGRAM - Google Patents

Description

The present invention relates to a communication method, a communication device, a transmitter, a program, and the like.

In recent years, in home networks, in addition to coordinating AV home appliances by IP (Internet Protocol) connection via Ethernet (registered trademark) or wireless LAN (Local Area Network), power usage management in response to environmental issues, A home energy management system (HEMS), which has functions such as turning on/off the power of a home appliance, is promoting the introduction of a home appliance linkage function in which various home appliances are connected to a network. However, some home appliances do not have sufficient computing power to have a communication function, and some home appliances cannot be equipped with a communication function due to cost reasons.

In order to solve such a problem, in Patent Document 1, in an optical space transmission device that uses light to transmit information to free space, communication is performed using a plurality of monochromatic light sources of illumination light, thereby limiting transmission. A technique for realizing efficient communication between devices in a device is described.

Japanese Patent Application Laid-Open No. 2002-290335

However, the conventional method is limited to the case where the equipment to which it is applied has a three-color light source such as lighting. Also, a receiver that receives the transmitted information cannot display an image useful to the user.

The present invention solves such problems and provides a communication method and the like that enable communication between various devices.

A communication method according to one aspect of the present invention is a communication method using a terminal equipped with an image sensor, the terminal determines whether or not the terminal can perform visible light communication, and the terminal performs visible light communication. When it is determined that it is possible to perform acquiring identification information, and acquiring a captured image by capturing an image of the subject with the image sensor when it is determined that the terminal is not capable of performing visible light communication in the determination of the visible light communication, At least one contour is extracted by performing edge detection of the captured image, a predetermined specific area is specified from the at least one contour, and a second line pattern transmitted by the subject is detected from the line pattern of the specific area. to obtain the identity of

In addition, these generic or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media. Moreover, a computer program for executing the method according to one embodiment may be stored in a recording medium of a server, and may be implemented in a mode of being distributed from the server to the terminal in response to a request from the terminal.

According to the present invention, communication between various devices can be realized.

1A and 1B are diagrams showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 2A and 2B are diagrams illustrating an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 3A and 3B are diagrams illustrating an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 4A and 4B are diagrams illustrating an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5A is a diagram showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5B is a diagram showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5C is a diagram showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5D is a diagram illustrating an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5E is a diagram showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5F is a diagram showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5G is a diagram showing an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 5H is a diagram illustrating an example of a method for observing luminance of a light emitting unit according to Embodiment 1. FIG. 6A is a flowchart of an information communication method according to Embodiment 1. FIG. 6B is a block diagram of the information communication device according to Embodiment 1. FIG. 7 is a diagram illustrating an example of a photographing operation of a receiver according to Embodiment 2. FIG. 8 is a diagram illustrating another example of the imaging operation of the receiver according to Embodiment 2. FIG. 9 is a diagram illustrating another example of the imaging operation of the receiver according to Embodiment 2. FIG. 10 is a diagram showing an example of the display operation of the receiver according to Embodiment 2. FIG. 11 is a diagram showing an example of display operation of a receiver according to Embodiment 2. FIG. 12 is a diagram illustrating an example of operation of a receiver according to Embodiment 2. FIG. 13 is a diagram showing another example of the operation of the receiver according to Embodiment 2. FIG. 14 is a diagram showing another example of the operation of the receiver according to Embodiment 2. FIG. 15 is a diagram showing another example of the operation of the receiver according to Embodiment 2. FIG. 16 is a diagram showing another example of the operation of the receiver according to Embodiment 2. FIG. 17 is a diagram showing another example of the operation of the receiver according to Embodiment 2. FIG. 18A is a diagram showing an example of operations of a transmitter and a receiver in Embodiment 2. FIG. 18B is a diagram showing an example of operations of a transmitter and a receiver in Embodiment 2. FIG. 18C is a diagram showing an example of operations of a transmitter and a receiver in Embodiment 2. FIG. FIG. 19 is a diagram for explaining an example of application to route guidance according to the second embodiment. FIG. 20 is a diagram for explaining an example of application to usage log accumulation and analysis according to the second embodiment. FIG. 21 is a diagram showing an application example of a transmitter and a receiver according to Embodiment 2. FIG. 22 is a diagram showing an application example of the transmitter and receiver in Embodiment 2. FIG. 23 is a diagram illustrating an example of an application according to Embodiment 3. FIG. 24 is a diagram illustrating an example of an application according to Embodiment 3. FIG. 25A and 25B are diagrams showing an example of a transmission signal and an example of an audio synchronization method according to Embodiment 3. FIG. 26 is a diagram showing an example of a transmission signal in Embodiment 3. FIG. 27 is a diagram depicting an example of a processing flow of a receiver according to Embodiment 3; FIG. 28 is a diagram showing an example of a user interface of a receiver according to Embodiment 3. FIG. 29 is a diagram depicting an example of a processing flow of a receiver according to Embodiment 3; FIG. 30] FIG. 30 is a diagram illustrating another example of the processing flow of the receiver according to Embodiment 3. [FIG. 31A is a diagram for explaining a specific method of synchronous reproduction according to Embodiment 3. FIG. 31B is a block diagram showing a configuration of a playback device (receiver) that performs synchronous playback according to Embodiment 3. FIG. 31C is a flowchart showing processing operations of a playback device (receiver) that performs synchronous playback according to Embodiment 3. FIG. FIG. 32 is a diagram for explaining advance preparation for synchronous playback according to the third embodiment. 33 is a diagram showing an application example of the receiver in Embodiment 3. FIG. 34A is a front view of a receiver held by a holder according to Embodiment 3. FIG. 34B is a rear view of the receiver held by the holder according to Embodiment 3. FIG. 35 is a diagram for explaining a use case of a receiver held by a holder according to Embodiment 3. FIG. 36 is a flowchart showing processing operations of a receiver held in a holder according to Embodiment 3. FIG. 37 is a diagram showing an example of an image displayed by a receiver according to Embodiment 3. FIG. 38 is a diagram showing another example of the holder according to Embodiment 3. FIG. 39A is a diagram showing an example of a visible light signal according to Embodiment 3. FIG. 39B is a diagram showing an example of a visible light signal according to Embodiment 3. FIG. 39C is a diagram showing an example of a visible light signal according to Embodiment 3. FIG. 39D is a diagram showing an example of a visible light signal according to Embodiment 3. FIG. 40 is a diagram showing a structure of a visible light signal according to Embodiment 3. FIG. 41] FIG. 41 is a diagram illustrating an example in which a receiver displays an AR image according to Embodiment 4. [FIG. 42 is a diagram illustrating an example of a display system according to Embodiment 4. FIG. 43 is a diagram showing another example of the display system according to Embodiment 4. FIG. 44 is a diagram showing another example of the display system according to Embodiment 4. FIG. 45 is a flowchart showing an example of processing operation of a receiver according to Embodiment 4. FIG. 46] FIG. 46 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 47] FIG. 47 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 48] FIG. 48 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 49] FIG. 49 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 50] FIG. 50 is a diagram showing another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 51] FIG. 51 is a diagram showing another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 52 is a flowchart showing another example of processing operation of the receiver according to Embodiment 4. FIG. 53] FIG. 53 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. FIG. 54 is a diagram showing a captured display image Ppre and a decoding image Pdec acquired by imaging by the receiver according to the fourth embodiment. 55 is a diagram showing an example of a captured display image Ppre displayed on the receiver according to Embodiment 4. FIG. 56 is a flowchart showing another example of processing operation of the receiver according to Embodiment 4. FIG. 57] FIG. 57 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 58] FIG. 58 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 59] FIG. 59 is a diagram illustrating another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 60] FIG. 60 is a diagram showing another example in which the receiver displays an AR image according to Embodiment 4. [FIG. 61 is a diagram showing an example of recognition information in Embodiment 4. FIG. 62 is a flowchart showing another example of processing operation of the receiver according to Embodiment 4. FIG. 63] FIG. 63 is a diagram showing an example of identification of a bright line pattern region by a receiver according to Embodiment 4. [FIG. 64 is a diagram showing another example of the receiver in Embodiment 4. FIG. 65 is a flowchart showing another example of processing operation of the receiver according to Embodiment 4. FIG. 66 is a diagram showing an example of a transmission system including a plurality of transmitters according to Embodiment 4. FIG. 67 is a diagram showing an example of a transmission system including a plurality of transmitters and receivers according to Embodiment 4. FIG. 68A is a flowchart showing an example of processing operation of a receiver according to Embodiment 4. FIG. 68B is a flowchart showing an example of processing operation of a receiver according to Embodiment 4. FIG. 69A is a flowchart showing a display method according to Embodiment 4. FIG. 69B is a block diagram showing a configuration of a display device according to Embodiment 4. FIG. 70] FIG. 70 is a diagram illustrating an example in which a receiver displays an AR image according to Modification 1 of Embodiment 4. [ FIG. 71] FIG. 71 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 1 of Embodiment 4. [FIG. 72] FIG. 72 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 1 of Embodiment 4. [FIG. 73] FIG. 73 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 1 of Embodiment 4. [FIG. 74] FIG. 74 is a diagram illustrating another example of the receiver in Modification 1 of Embodiment 4. [FIG. 75] FIG. 75 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 1 of Embodiment 4. [ FIG. 76] FIG. 76 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 1 of Embodiment 4. [ FIG. 77] FIG. 77 is a flowchart illustrating an example of processing operation of a receiver according to Modification 1 of Embodiment 4. [ FIG. 78 is a diagram showing an example of a problem when displaying an AR image assumed in the receiver according to Embodiment 4 or Modification 1 thereof. FIG. 79] FIG. 79 is a diagram illustrating an example in which a receiver displays an AR image according to Modification 2 of Embodiment 4. [FIG. 80 is a flowchart showing an example of processing operation of a receiver according to Modification 2 of Embodiment 4. FIG. 81] FIG. 81 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 2 of Embodiment 4. [FIG. 82 is a flowchart showing another example of the processing operation of the receiver in Modification 2 of Embodiment 4. FIG. 83] FIG. 83 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 2 of Embodiment 4. [FIG. 84] FIG. 84 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 2 of Embodiment 4. [FIG. 85] FIG. 85 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 2 of Embodiment 4. [FIG. 86] FIG. 86 is a diagram illustrating another example in which the receiver displays an AR image according to Modification 2 of Embodiment 4. [ FIG. FIG. 87A is a flowchart illustrating a display method according to one aspect of the present invention. FIG. 87B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention. 88 is a diagram illustrating an example of enlarging and moving an AR image in Modification 3 of Embodiment 4. FIG. 89 is a diagram illustrating an example of enlarging an AR image in Modification 3 of Embodiment 4. FIG. 90 is a flowchart showing an example of processing operations for enlarging and moving an AR image by a receiver according to Modification 3 of Embodiment 4. FIG. 91 is a diagram illustrating an example of superimposing an AR image according to Modification 3 of Embodiment 4. FIG. 92 is a diagram illustrating an example of superimposition of an AR image in Modification 3 of Embodiment 4. FIG. 93 is a diagram illustrating an example of superimposing an AR image according to Modification 3 of Embodiment 4. FIG. 94 is a diagram illustrating an example of superimposing an AR image according to Modification 3 of Embodiment 4. FIG. 95A] FIG. 95A is a diagram showing an example of an imaged display image obtained by imaging by a receiver according to Modification 3 of Embodiment 4. [FIG. 95B is a diagram showing an example of a menu screen displayed on the display of the receiver according to Modification 3 of Embodiment 4. FIG. 96 is a flowchart showing an example of processing operations of a receiver and a server according to Modification 3 of Embodiment 4. FIG. 97] FIG. 97 is a diagram for explaining the volume of sound reproduced by a receiver according to Modification 3 of Embodiment 4. [FIG. 98 is a diagram showing the relationship between the distance from the receiver to the transmitter and the sound volume in Modification 3 of Embodiment 4. FIG. 99] FIG. 99 is a diagram illustrating an example of superimposition of an AR image by a receiver according to Modification 3 of Embodiment 4. [ FIG. 100] FIG. 100 is a diagram illustrating an example of AR image superimposition by a receiver according to Modification 3 of Embodiment 4. [FIG. 101 is a diagram for explaining an example of how the receiver determines the line scan time in Modification 3 of Embodiment 4. FIG. 102] FIG. 102 is a diagram for explaining an example of how the receiver determines the line scan time in Modification 3 of Embodiment 4. [FIG. 103 is a flowchart showing an example of how the receiver determines the line scan time according to Modification 3 of Embodiment 4. FIG. 104] FIG. 104 is a diagram illustrating an example of AR image superimposition by a receiver according to Modification 3 of Embodiment 4. [ FIG. 105] FIG. 105 is a diagram illustrating an example of superimposition of an AR image by a receiver according to Modification 3 of Embodiment 4. [FIG. 106] FIG. 106 is a diagram illustrating an example of superimposition of an AR image by a receiver according to Modification 3 of Embodiment 4. [FIG. 107] FIG. 107 is a diagram illustrating an example of a decoding image acquired according to the attitude of the receiver according to Modification 3 of Embodiment 4. [FIG. 108] FIG. 108 is a diagram illustrating another example of a decoding image acquired according to the attitude of the receiver in Modification 3 of Embodiment 4. [FIG. 109 is a flowchart illustrating an example of processing operation of a receiver according to Modification 3 of Embodiment 4. FIG. 110 is a diagram illustrating an example of camera lens switching processing by a receiver according to Modification 3 of Embodiment 4. FIG. 111 is a diagram illustrating an example of camera switching processing by a receiver according to Modification 3 of Embodiment 4. FIG. 112 is a flowchart illustrating an example of processing operations between a receiver and a server according to Modification 3 of Embodiment 4. FIG. 113] FIG. 113 is a diagram illustrating an example of superimposition of an AR image by a receiver according to Modification 3 of Embodiment 4. [FIG. 114 is a sequence diagram showing processing operations of a system including a receiver, a microwave oven, a relay server, and an electronic settlement server according to Modification 3 of Embodiment 4. FIG. 115 is a sequence diagram showing processing operations of a system including a POS terminal, a server, a receiver, and a microwave oven in Modification 3 of Embodiment 4. FIG. 116 is a diagram illustrating an example of indoor use in Modification 3 of Embodiment 4. FIG. 117 is a diagram illustrating an example of display of an augmented reality object according to Modification 3 of Embodiment 4. FIG. 118 is a diagram illustrating a configuration of a display system according to Modification 4 of Embodiment 4. FIG. 119 is a flowchart illustrating processing operations of a display system according to Modification 4 of Embodiment 4. FIG. FIG. 120 is a flowchart illustrating a recognition method according to one aspect of the present invention. 121 is a diagram illustrating an example of an operation mode of a visible light signal according to Embodiment 5. FIG. 122A is a flowchart illustrating a visible light signal generation method according to Embodiment 5. FIG. 122B is a block diagram showing a configuration of a signal generation device according to Embodiment 5. FIG. 123 is a diagram showing the format of an MPM MAC frame according to Embodiment 6. FIG. 124 is a flowchart showing processing operations of an encoding device that generates an MPM MAC frame according to Embodiment 6. FIG. 125 is a flowchart showing processing operations of a decoding device that decodes an MPM MAC frame according to Embodiment 6. FIG. 126 is a diagram showing PIB attributes of MAC in Embodiment 6. FIG. 127 is a diagram for explaining an MPM dimming method according to Embodiment 6. FIG. 128 is a diagram showing attributes of PIB of PHY in Embodiment 6. FIG. 129 is a diagram for explaining the MPM in Embodiment 6. FIG. 130 is a diagram showing PLCP header subfields in Embodiment 6. FIG. 131 is a diagram showing the PLCP center subfield in Embodiment 6. FIG. 132 is a diagram showing a PLCP footer subfield in Embodiment 6. FIG. 133 is a diagram showing waveforms in PWM mode of PHY in MPM according to Embodiment 6. FIG. 134 is a diagram showing waveforms in PPM mode of PHY in MPM according to Embodiment 6. FIG. 135 is a flowchart illustrating an example of a decoding method according to Embodiment 6. FIG. 136 is a flowchart showing an example of an encoding method according to Embodiment 6. FIG. 137] FIG. 137 is a diagram illustrating an example in which a receiver displays an AR image in Embodiment 7. [FIG. 138 is a diagram showing an example of a captured display image on which an AR image is superimposed, according to Embodiment 7. FIG. 139 ] FIG. 139 is a diagram illustrating another example in which the receiver displays an AR image in Embodiment 7. [ FIG. 140 is a flowchart showing operation of a receiver in Embodiment 7. FIG. 141 is a diagram for explaining operation of a transmitter in Embodiment 7. FIG. 142 is a diagram for explaining another operation of the transmitter in Embodiment 7. FIG. 143 is a diagram for explaining another operation of the transmitter in Embodiment 7. FIG. 144 is a diagram showing a comparative example for explaining ease of receiving an optical ID in Embodiment 7. FIG. 145A is a flowchart showing operation of a transmitter in Embodiment 7. FIG. 145B is a block diagram showing a configuration of a transmitter in Embodiment 7. FIG. 146 ] FIG. 146 is a diagram illustrating another example in which the receiver displays an AR image in Embodiment 7. [ FIG. 147 is a diagram for explaining operation of a transmitter in Embodiment 8. FIG. 148A is a flowchart showing a transmission method according to Embodiment 8. FIG. 148B is a block diagram showing a configuration of a transmitter according to Embodiment 8. FIG. 149 is a diagram showing an example of a detailed configuration of a visible light signal in Embodiment 8. FIG. 150] FIG. 150 is a diagram illustrating another example of detailed configuration of a visible light signal according to Embodiment 8. [FIG. 151 ] FIG. 151 is a diagram showing another example of detailed configuration of a visible light signal according to Embodiment 8. [ FIG. 152 ] FIG. 152 is a diagram showing another example of detailed configuration of a visible light signal according to Embodiment 8. [ FIG. 153 is a diagram showing the relationship between the sum of variables y ₀ to y ₃ and the total time length and valid time length in Embodiment 8. FIG. 154A is a flowchart showing a transmission method according to Embodiment 8. FIG. 154B is a block diagram showing a configuration of a transmitter in Embodiment 8. FIG. 155 shows a configuration of a display system according to Embodiment 9. FIG. 156 is a sequence diagram showing processing operations of a receiver and a server according to Embodiment 9. FIG. 157 is a flowchart showing processing operations of a server according to Embodiment 9. FIG. 158 ] FIG. 158 is a diagram illustrating an example of communication when the transmitter and the receiver in Embodiment 9 are mounted in a vehicle. [ FIG. 159 is a flowchart showing processing operations of a vehicle according to Embodiment 9. FIG. 160] FIG. 160 is a diagram illustrating an example in which a receiver displays an AR image in Embodiment 9. [FIG. 161 ] FIG. 161 is a diagram illustrating another example in which the receiver displays an AR image in Embodiment 9. [ FIG. 162 is a diagram illustrating processing operations of a receiver in Embodiment 9. FIG. 163 ] FIG. 163 is a diagram illustrating an example of operation on a receiver in Embodiment 9. [ FIG. 164 is a diagram showing an example of an AR image displayed on a receiver in Embodiment 9. FIG. 165 is a diagram illustrating an example of an AR image superimposed on a captured display image in Embodiment 9. FIG. 166 is a diagram illustrating an example of an AR image superimposed on a captured display image in Embodiment 9. FIG. 167 is a diagram illustrating an example of a transmitter in Embodiment 9. FIG. 168 is a diagram showing another example of the transmitter in Embodiment 9. FIG. 169 is a diagram showing another example of a transmitter in Embodiment 9. FIG. 170 ] FIG. 170 is a diagram illustrating an example of a system using a receiver compatible with optical communication and a receiver not compatible with optical communication in Embodiment 9. [ FIG. 171 is a flowchart showing processing operations of a receiver in Embodiment 9. FIG. 172 is a diagram illustrating an example of AR image display according to Embodiment 9. FIG. FIG. 173A is a flowchart illustrating a display method according to one aspect of the present invention. FIG. 173B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention. 174 is a diagram showing an example of an image drawn on a transmitter in Embodiment 10. FIG. 175 ] FIG. 175 is a diagram illustrating another example of an image drawn on the transmitter in Embodiment 10. [ FIG. 176 ] FIG. 176 is a diagram illustrating an example of a transmitter and a receiver in Embodiment 10. [ FIG. FIG. 177 is a diagram for explaining fundamental frequencies of line patterns according to the tenth embodiment. 178A is a flowchart showing processing operations of the encoding device in Embodiment 10. FIG. 178B is a diagram for explaining the processing operation of the encoding device in Embodiment 10. FIG. 179 is a flowchart showing processing operations of a receiver which is a decoding device in Embodiment 10. FIG. 180 is a flowchart showing processing operations of a receiver according to Embodiment 10. FIG. 181A is a diagram showing an example of a system configuration according to Embodiment 10. FIG. 181B is a diagram showing processing of the camera in Embodiment 10. FIG. 182 is a diagram showing another example of the configuration of the system according to Embodiment 10. FIG. 183 ] FIG. 183 is a diagram showing another example of an image drawn on the transmitter in Embodiment 10. [ FIG. 184 is a diagram showing an example of a format of a MAC frame forming a frame ID according to Embodiment 10. FIG. 185 is a diagram showing an example of the structure of a MAC header in Embodiment 10. FIG. 186 is a diagram showing an example of a table for deriving the number of packet divisions in Embodiment 10. FIG. 187 is a diagram showing PHY encoding in Embodiment 10. FIG. 188 is a diagram showing an example of a transmission image Im3 having PHY symbols according to Embodiment 10. FIG. 189 is a diagram for explaining two PHY versions according to the tenth embodiment. FIG. 190 is a diagram for explaining a Gray code in Embodiment 10. FIG. 191 ] FIG. 191 is a diagram illustrating an example of decoding processing by a receiver in Embodiment 10. [ FIG. 192] FIG. 192 is a diagram for explaining a method of fraud detection of a transmitted image by a receiver in Embodiment 10. [FIG. 193 is a flowchart showing an example of decoding processing including fraud detection of a transmitted image by a receiver in Embodiment 10. FIG. 194A is a flowchart showing a display method according to a modification of Embodiment 10. FIG. 194B is a block diagram showing a configuration of a display device according to a modification of Embodiment 10. FIG. 194C is a flowchart showing a communication method according to the modification of Embodiment 10. FIG. 194D is a block diagram showing a configuration of a communication apparatus according to this modification of Embodiment 10. FIG. 194E is a block diagram showing a configuration of a transmitter according to Embodiment 10 and its modification. FIG. 195 is a diagram showing an example of a configuration of a communication system including servers according to Embodiment 11. FIG. 196 is a flowchart showing a management method by the first server according to Embodiment 11. FIG. 197 is a diagram showing a lighting system according to Embodiment 12. FIG. 198 ] FIG. 198 is a diagram illustrating an example of an arrangement of lighting devices and a decoding image according to Embodiment 12. [ FIG. 199 ] FIG. 199 is a diagram showing another example of the arrangement of lighting devices and decoding images according to Embodiment 12. [ FIG. 200 is a diagram for explaining position estimation using the lighting device in Embodiment 12. FIG. 201 is a flowchart showing processing operations of a receiver according to Embodiment 12. FIG. 202 is a diagram illustrating an example of a communication system according to Embodiment 12. FIG. 203 is a diagram for explaining self-position estimation processing by a receiver in Embodiment 12. FIG. 204 is a flowchart showing self-position estimation processing by a receiver in Embodiment 12. FIG. 205 is a flowchart showing an outline of self-position estimation processing of a receiver according to Embodiment 12. FIG. 206 is a diagram showing the relationship between radio IDs and optical IDs in Embodiment 12. FIG. 207 is a diagram for explaining an example of imaging by a receiver in Embodiment 12. FIG. 208 is a diagram for explaining another example of imaging by the receiver in Embodiment 12. FIG. 209 is a diagram for explaining a camera used by a receiver in Embodiment 12. FIG. 210 is a flowchart illustrating an example of processing by a receiver to change a visible light signal of a transmitter according to Embodiment 12. FIG. 211 is a flowchart illustrating another example of processing for changing the visible light signal of the transmitter by the receiver in Embodiment 12. FIG. 212 is a diagram for explaining navigation by a receiver in Embodiment 13. FIG. 213 ] FIG. 213 is a flowchart illustrating an example of self-position estimation by a receiver in Embodiment 13. [ FIG. 214 is a diagram for explaining visible light signals received by a receiver in Embodiment 13. FIG. 215 is a flowchart showing another example of self-position estimation by the receiver in Embodiment 13. FIG. 216 is a flowchart illustrating an example of determination of reflected light by a receiver according to Embodiment 13. FIG. 217 is a flowchart showing an example of navigation by the receiver according to Embodiment 13. FIG. 218 is a diagram showing an example of transmitter 100 configured as a projector according to Embodiment 13. FIG. 219 is a flowchart showing another example of self-position estimation by the receiver in Embodiment 13. FIG. 220 is a flowchart illustrating an example of processing by a transmitter in Embodiment 13. FIG. 221 is a flowchart showing another example of navigation by the receiver in Embodiment 13. FIG. 222 is a flowchart illustrating an example of processing by a receiver in Embodiment 13. FIG. 223 is a diagram showing an example of a screen displayed on the display of the receiver in Embodiment 13. FIG. 224 ] FIG. 224 is a diagram showing an example of character display by the receiver in Embodiment 13. [ FIG. 225 is a diagram showing another example of a screen displayed on the display of the receiver in Embodiment 13. FIG. 226 is a diagram showing a system configuration for performing navigation to a meeting place in Embodiment 13. FIG. 227 ] FIG. 227 is a diagram showing another example of a screen displayed on the display of the receiver in Embodiment 13. [ FIG. FIG. 228 is a diagram showing the inside of a concert hall. FIG. 229 is a flow chart showing an example of the communication method according to the first aspect of the present invention.

A communication method according to an aspect of the present invention is a communication method using a terminal equipped with an image sensor, the terminal determines whether or not the terminal is capable of performing visible light communication, and the terminal performs visible light communication. When it is determined that it is possible to perform and acquires a captured image by capturing an image of the subject with the image sensor when it is determined that the terminal is not capable of performing visible light communication in the determination of the visible light communication. and extracting at least one contour by performing edge detection on the captured image, identifying a predetermined specific region from the at least one contour, and detecting the first line pattern transmitted by the subject from the line pattern of the specific region. Acquire the identification information of 2.

As a result, for example, as in Embodiment 10, a terminal such as a receiver receives the first identification information or the second identification information from an object such as a transmitter regardless of whether visible light communication is possible. can be obtained. That is, when the terminal can perform visible light communication, the terminal acquires, for example, the light ID from the subject as the first identification information. On the other hand, even if the terminal cannot perform visible light communication, the terminal can acquire, for example, the image ID or the frame ID from the subject as the second identification information. Specifically, for example, the transmission images shown in FIGS. 183 and 188 are picked up as a subject, the area of the transmission image is selected as a specific area (that is, the selection area), and the second identification information is obtained from the line pattern of the transmission image. is obtained. Therefore, even when visible light communication is impossible, the second identification information can be obtained appropriately. The striped pattern is also called a bright line pattern or bright line pattern area.

Further, in specifying the specific area, an area having a rectangular outline of a predetermined size or more, or an area having a rounded square outline of a predetermined size or more may be identified as the specific area.

As a result, for example, as shown in FIG. 179, a square or rounded square area can be appropriately specified as the specified area.

Further, in determining the visible light communication, if the terminal is specified as a terminal capable of changing the exposure time to a predetermined value or less, it is determined that the visible light communication can be performed. If the terminal is identified as a terminal that cannot change the exposure time to the predetermined value or less, it may be determined that visible light communication is not possible.

Accordingly, as shown in FIG. 180, for example, it is possible to appropriately determine whether or not the visible light signal can be transmitted.

setting the exposure time of the image sensor to a first exposure time when capturing an image of the subject when the terminal determines that the visible light communication can be performed in the determination of the visible light communication; When the image for decoding is acquired by imaging the subject with the first exposure time, and the terminal determines that the visible light communication is not possible in the determination of the visible light communication, the When capturing an image of a subject, the exposure time of the image sensor is set to a second exposure time, and the captured image is obtained by capturing the subject with the second exposure time, and the captured image is captured at the first exposure time. may be shorter than the second exposure time.

This makes it possible to acquire a decoding image having a striped pattern by imaging with the first exposure time, and to appropriately acquire the first identification information by decoding the striped pattern. Furthermore, by imaging with the second exposure time, it is possible to acquire the normally shot image as the shot image, and appropriately acquire the second identification information from the line pattern appearing in the normally shot image. Thereby, the terminal can acquire the first identification information or the second identification information suitable for the terminal by properly using the first exposure time and the second exposure time.

Further, the subject has a rectangular shape as viewed from the image sensor, and the first identification information is transmitted by changing the brightness of a central area of the subject, and a barcode-like line pattern is formed around the periphery of the subject. is arranged, and in the determination of the visible light communication, when the terminal determines that it is possible to perform visible light communication, when imaging the subject, the plurality of exposure lines of the image sensor correspond to obtaining the decoding image including a bright line pattern composed of a plurality of bright lines, obtaining the first identification information by decoding the bright line pattern, The second identification information may be acquired from the line pattern of the captured image when the subject is captured when it is determined that communication is not possible.

As a result, the first identification information and the second identification information can be appropriately acquired from a subject whose central region changes in brightness.

Further, the first identification information obtained from the decoding image and the second identification information obtained from the line pattern may be the same information.

As a result, the same information can be obtained from the subject regardless of whether the terminal is capable of visible light communication or the terminal is not capable of visible light communication.

Further, in the determination of the visible light communication, when it is determined that the terminal is capable of performing visible light communication, the first moving image associated with the first identification information is displayed, and the first moving image is displayed. A second moving image associated with the first identification information may be displayed next to the first moving image when an operation of sliding the moving image is received.

For example, the first moving image and the second moving image are the first AR image P46 and the second AR image P46c shown in FIG. 162, respectively. Also, the first identification information is, for example, the light ID as described above. In the communication method according to the above aspect, when an operation of sliding the first moving image, that is, a swipe is accepted, the second moving image associated with the first identification information is displayed next to the first moving image. Is displayed. Therefore, it is possible to easily display an image that is beneficial to the user. Further, as shown in FIG. 194A, it is determined in advance whether or not visible light communication is possible. Processing load can be reduced.

Further, in the display of the second moving image, when an operation of sliding the first moving image in the horizontal direction is received, the second moving image is displayed and the first moving image is slid in the vertical direction. A still image associated with the first identification information may be displayed when an operation to cause the display to be performed is received.

As a result, for example, as shown in FIG. 162, the second moving image is displayed by sliding the first moving image in the horizontal direction, that is, by swiping. Furthermore, as shown in FIGS. 163 and 164, for example, by sliding the first moving image in the vertical direction, a still image associated with the first identification information is displayed. The still image is the AR image P47 shown in FIG. 164, for example. Therefore, it is possible to easily display a wide variety of images that are beneficial to the user.

Also, in each of the first moving image and the second moving image, the object in the first displayed picture may be at the same position.

Accordingly, when the second moving image is displayed instead of the first moving image, for example, as shown in FIG. and the second moving image are related to each other.

Further, when the first identification information is acquired again by imaging with the image sensor, the next moving image associated with the first identification information may be displayed next to the displayed moving image. .

As a result, as shown in FIG. 162, for example, the next moving image is displayed when the light ID, which is the first identification information, is reacquired without an operation such as slide or swipe. Therefore, moving images beneficial to the user can be displayed more easily.

Also, in each of the moving image being displayed and the next moving image, the object in the first displayed picture may be at the same position.

As a result, for example, as shown in FIG. 162, when the next moving image is displayed in place of the displayed moving image, the first displayed objects are at the same position, so that the user can It is possible to easily grasp that the moving images are related to each other.

Further, at least one of the first moving image and the second moving image is arranged such that the closer the position in the moving image is to the end of the moving image, the higher the transparency at that position. may be formed.

As a result, for example, as shown in FIG. 93 or FIG. 166, when the moving image is superimposed on the normal photographed image and displayed, an object with an ambiguous outline actually exists in the environment indicated by the normal photographed image. The captured display image can be displayed as shown in FIG.

Further, an image may be displayed outside the area in which at least one of the first moving image and the second moving image is displayed.

As a result, an image is displayed outside the area where the moving image is displayed, such as sub-image Ps46 shown in FIG. 161, so that a wide variety of useful images for the user can be easily displayed.

Also, an area composed of a pattern of a plurality of bright lines is obtained by acquiring a normal photographed image by imaging with a first exposure time by the image sensor, and by imaging with a second exposure time shorter than the first exposure time. obtaining the image for decoding including a bright line pattern area, obtaining the first identification information by decoding the image for decoding, and moving at least one of the first moving image and the second moving image. In displaying the image, a reference area located at the same position as the bright line pattern area in the decoding image is specified from the normal captured image, and the moving image is superimposed on the normal captured image based on the reference area. The moving image may be superimposed on the target area by recognizing the target area. For example, in the display of at least one of the first moving image and the second moving image, an area above, below, left or right of the reference area in the normal shot image may be the target area. can be recognized as

As a result, for example, as shown in FIGS. 50 to 52 and 172, the target region is recognized based on the reference region, and the moving image is superimposed on the target region. can be easily increased.

Further, in displaying at least one of the first moving image and the second moving image, the size of the moving image may be increased as the size of the bright line pattern area is increased.

As a result, as shown in FIG. 172, since the size of the moving image changes according to the size of the bright line pattern area, compared to the case where the size of the moving image is fixed, the object indicated by the moving image is more visible. The moving image can be displayed as if it exists in reality.

A transmitter according to an aspect of the present invention includes an illumination plate, a light source that irradiates light from the back side of the illumination plate, and a microcontroller that changes the brightness of the light source, wherein the microcontroller is the light source. By changing the luminance of the light source, the first identification information is transmitted through the illumination plate, and a barcode-shaped line pattern is arranged around the front side of the illumination plate, and the line pattern The second identification information is encoded, and the first identification information and the second identification information are the same information. For example, the illumination plate has a rectangular shape.

As a result, the same information can be transmitted to a terminal capable of performing visible light communication and to a terminal not capable of performing visible light communication.

In addition, these general or specific aspects may be realized by an apparatus, system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM. It may be implemented in any combination of circuits, computer programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that the embodiments described below are all comprehensive or specific examples. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as optional constituent elements.

(Embodiment 1)
Embodiment 1 will be described below.

(Observation of luminance of light-emitting part)
When capturing one image, instead of exposing all the imaging elements at the same timing, an imaging method is proposed in which exposure is started and ended at different times for each imaging element. FIG. 1 shows an example in which imaging elements arranged in a row are exposed simultaneously, and the exposure start time is shifted in order of proximity of the row. Here, the lines of pixels on the image corresponding to the image sensors to be exposed at the same time are referred to as exposure lines, and the lines of pixels on the image corresponding to the image sensors are referred to as bright lines.

When this imaging method is used to capture a blinking light source on the entire surface of the imaging device, bright lines (bright and dark lines of pixel values) appear on the captured image along the exposure lines, as shown in FIG. . By recognizing this bright line pattern, it is possible to estimate the light source luminance change at a speed higher than the imaging frame rate. Accordingly, by transmitting a signal as a change in light source luminance, communication can be performed at a speed equal to or higher than the imaging frame rate. When a light source expresses a signal by taking two kinds of luminance values, the lower luminance value is called low (LO), and the higher luminance value is called high (HI). The low can be without any light source, or it can be less bright than the high.

By this method, information is transmitted at a speed exceeding the imaging frame rate.

When there are 20 exposure lines whose exposure times do not overlap in one captured image, and the frame rate of imaging is 30 fps, it is possible to recognize a luminance change with a period of 1.67 milliseconds. If there are 1,000 exposure lines whose exposure times do not overlap, luminance changes with a cycle of 1/30,000 second (approximately 33 microseconds) can be recognized. Note that the exposure time is set to be shorter than 10 milliseconds, for example.

FIG. 2 shows the case where the exposure of the next exposure line is started after the exposure of one exposure line is completed.

In this case, when the number of frames per second (frame rate) is f and the number of exposure lines that make up one image is l, if information is transmitted based on whether or not each exposure line receives a certain amount of light, the maximum can transmit information at a rate of fl bits per second.

Further, when exposure is performed with a time difference for each pixel instead of for each line, communication can be performed at a higher speed.

At this time, when the number of pixels per exposure line is m pixels and information is transmitted depending on whether or not each pixel receives a certain amount of light, the maximum transmission speed is flm bits per second.

As shown in FIG. 3, if it is possible to recognize the exposure state of each exposure line by light emission of the light emitting unit at a plurality of levels, it is possible to control the light emission time of the light emitting unit in units shorter than the exposure time of each exposure line. , can transmit more information.

If the exposure state is recognizable in Elv steps, information can be transmitted at a rate of up to flElv bits per second.

In addition, by causing the light emitting unit to emit light at timing slightly shifted from the exposure timing of each exposure line, the basic period of transmission can be recognized.

FIG. 4 shows the case where the exposure of the next exposure line is started before the exposure of one exposure line is completed. In other words, the exposure times of adjacent exposure lines are partially overlapped. With such a configuration, (1) the number of samples within a predetermined time can be increased compared to the case where the exposure time of one exposure line is completed before the exposure of the next exposure line is started. By increasing the number of samples within a predetermined period of time, it becomes possible to more appropriately detect the optical signal generated by the optical transmitter, which is the subject. That is, it is possible to reduce the error rate when detecting optical signals. Furthermore, (2) the exposure time of each exposure line can be lengthened compared to the case where the exposure of the next exposure line is started after waiting for the end of the exposure time of one exposure line, so even when the subject is dark. However, it is possible to obtain a brighter image. That is, it becomes possible to improve the S/N ratio. In all exposure lines, the exposure times of adjacent exposure lines do not need to have a partial temporal overlap, and some exposure lines do not have a partial temporal overlap. It is also possible to By configuring some exposure lines so that they do not partially overlap in time, it is possible to suppress the occurrence of intermediate colors due to overlap of exposure times on the imaging screen, and it is possible to detect bright lines more appropriately. .

In this case, the exposure time is calculated from the brightness of each exposure line, and the light emission state of the light emitting unit is recognized.

When the brightness of each exposure line is determined by a binary value indicating whether or not the brightness is equal to or greater than a threshold value, the light emitting unit detects the non-emission state of each line in order to recognize the non-emission state. must continue for at least the exposure time of

FIG. 5A shows the effect of different exposure times when the exposure start times of the exposure lines are the same. 7500a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7500b is the case where the exposure time is longer than that. As in 7500b, the exposure times of adjacent exposure lines partially overlap each other, thereby making it possible to lengthen the exposure time. That is, the amount of light incident on the image sensor increases, and a bright image can be obtained. In addition, since an image with less noise can be obtained by suppressing the imaging sensitivity for imaging an image with the same brightness, communication errors can be suppressed.

FIG. 5B shows the effect of different exposure start times for each exposure line when the exposure times are the same. 7501a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7501b is the case where the exposure of the next exposure line is started earlier than the end of exposure of the previous exposure line. As in 7501b, the exposure times of adjacent exposure lines partially overlap each other, thereby increasing the number of lines that can be exposed per time. As a result, the resolution becomes higher and a large amount of information can be obtained. By increasing the sample interval (=difference in exposure start time), it is possible to estimate changes in light source brightness more accurately, reduce the error rate, and recognize changes in light source brightness over a shorter period of time. be able to. By overlapping the exposure times, it is possible to recognize blinking of the light source shorter than the exposure time by utilizing the difference in the amount of exposure of adjacent exposure lines.

Also, when the number of samples is small, that is, when the sample interval (time difference t _D shown in FIG. 5B) is long, there is a high possibility that changes in light source luminance cannot be accurately detected. In this case, the possibility can be suppressed by shortening the exposure time. That is, changes in light source luminance can be accurately detected. Also, the exposure time preferably satisfies the condition of exposure time>(sample interval−pulse width). The pulse width is the pulse width of light, which is the period during which the luminance of the light source is high. Thereby, High luminance can be appropriately detected.

As described with reference to FIGS. 5A and 5B, in a configuration in which each exposure line is sequentially exposed so that the exposure times of adjacent exposure lines partially overlap, the exposure time is set to be longer than that in the normal shooting mode. By using the bright line pattern generated by the short setting for signal transmission, it is possible to dramatically improve the communication speed. Here, by setting the exposure time at the time of visible light communication to 1/480 seconds or less, it is possible to generate an appropriate bright line pattern. Here, assuming that the frame frequency=f, the exposure time must be set as follows: exposure time<1/8×f. The blanking that occurs during shooting has a maximum size of half of one frame. That is, since the blanking time is less than half the shooting time, the shortest actual shooting time is 1/2f. Furthermore, since it is necessary to receive quaternary information within the time of 1/2f, at least the exposure time needs to be shorter than 1/(2f×4). Since the normal frame rate is 60 frames/second or less, by setting the exposure time to 1/480 second or less, it is possible to generate an appropriate bright line pattern in the image data and perform high-speed signal transmission. Become.

FIG. 5C shows the advantage of short exposure times when the exposure times of the exposure lines do not overlap. When the exposure time is long, even if the light source has a binary brightness change like 7502a, the captured image has a neutral color part like 7502e, and it tends to be difficult to recognize the brightness change of the light source. There is However, as in 7502d, after the end of exposure of one exposure line, a predetermined idle time (predetermined waiting time) _tD2 is provided until exposure of the next exposure line is started. can be easily recognized. That is, it becomes possible to detect a more appropriate bright line pattern such as 7502f. The configuration of providing a predetermined idle time during which no exposure is performed, as in 7502d, can be realized by making the exposure time _tE smaller than the time difference _tD between the exposure start times of the exposure lines. When the normal shooting mode has a structure in which the exposure times of adjacent exposure lines partially overlap, the exposure time is set shorter than in the normal shooting mode until a predetermined non-exposure period occurs. It can be realized by Further, even when the exposure end time of the previous exposure line and the exposure start time of the next exposure line are the same in the normal shooting mode, by setting the exposure time to be short until the predetermined non-exposure time occurs, can be realized. Also, by increasing the interval _tD between the exposure start times of each exposure line as in 7502g, a predetermined non-exposure time (predetermined waiting time) _tD2 can be provided. In this configuration, since the exposure time can be lengthened, a bright image can be captured, and noise is reduced, resulting in high error resistance. On the other hand, this configuration has the disadvantage that the number of exposure lines that can be exposed within a certain period of time decreases, and thus the number of samples decreases as in 7502h. For example, by using the former configuration when the imaging target is bright and using the latter configuration when the imaging target is dark, it is possible to reduce the estimation error of the light source luminance change.

In all exposure lines, the exposure times of adjacent exposure lines do not need to have a partial temporal overlap, and some exposure lines do not have a partial temporal overlap. It is also possible to Further, in all exposure lines, it is not necessary to provide a predetermined non-exposure time (predetermined waiting time) after the end of exposure of one exposure line until the start of exposure of the next exposure line. It is also possible to configure the lines to have partial temporal overlap. With such a configuration, it is possible to take advantage of the advantages of each configuration. In addition, the same readout method is used in the normal shooting mode in which shooting is performed at a normal frame rate (30 fps, 60 fps) and in the visible light communication mode in which shooting is performed with an exposure time of 1/480 seconds or less for visible light communication. Alternatively, signals may be read out in a circuit. By reading out signals with the same reading method or circuit, it is not necessary to use separate circuits for the normal imaging mode and the visible light communication mode, and the circuit scale can be reduced.

FIG. 5D shows the relationship between the minimum change time _tS of the light source luminance, the exposure time _tE , the time difference _tD between the exposure start times of the exposure lines, and the captured image. When t _E +t _D <t _S , one or more exposure lines are always imaged with the light source unchanged from the start to the end of exposure, so an image with clear brightness such as 7503d can be obtained. It is easy to recognize the brightness change of the light source. If 2t _E >t _S , a bright line pattern different from the change in luminance of the light source may be obtained, making it difficult to recognize the change in luminance of the light source from the captured image.

FIG. 5E shows the relationship between the transition time _tT of the light source luminance and the time difference _tD between the exposure start times of the exposure lines. The larger _tD is compared to _tT , the fewer exposure lines will be intermediate colors, and the easier it will be to estimate the light source luminance. When t _D >t _T , the intermediate color exposure lines are not more than two continuous lines, which is desirable. Since _tT is 1 microsecond or less when the light source is an LED and about 5 microseconds when the light source is an organic EL, _tD is set to 5 microseconds or more to facilitate estimation of the light source luminance. be able to.

FIG. 5F shows the relationship between the high-frequency noise _tHT of the light source luminance and the exposure time _tE . The larger _tE is compared to _tHT , the less the captured image is affected by high-frequency noise, and the easier it is to estimate the light source luminance. When _tE is an integral multiple of _tHT , the effect of high-frequency noise disappears, making estimation of the light source luminance the easiest. For estimating the light source luminance, it is desirable that t _E >t _HT . _The main cause of high-frequency noise is the switching power supply circuit, and _tHT is 20 microseconds or less in many switching power supplies for lamps. can be easily done.

FIG. 5G is a graph showing the relationship between the exposure time t _E and the magnitude of high frequency noise when t _HT is 20 microseconds. Considering that _tHT varies depending on the light source, from the graph, _tE is a value equal to the value when the amount of noise is maximized, 15 microseconds or more, or 35 microseconds or more, or It can be confirmed that the efficiency is good when it is defined as 54 microseconds or more or 74 microseconds or more. _From the viewpoint of high-frequency noise reduction, it is desirable that _tE is large. Therefore, when the period of change in light source luminance is 15 to 35 microseconds, _tE is 15 microseconds or more, and when the period of change in light source luminance is 35 to 54 microseconds, _tE is 35 microseconds or more. When the period of change in is 54 to 74 microseconds, _tE should be set to 54 microseconds or longer, and when the period of change in light source luminance is 74 microseconds or longer, _tE should be set to 74 microseconds or longer.

FIG. 5H shows the relationship between exposure time t _E and recognition success rate. Since the exposure time _tE has a relative meaning with respect to the time during which the brightness of the light source is constant, the value (relative exposure time) obtained by dividing the cycle _tS at which the brightness of the light source changes by the exposure time _tE is plotted on the horizontal axis. there is From the graph, it can be seen that the relative exposure time should be set to 1.2 or less if the recognition success rate is to be approximately 100%. For example, when the transmission signal is 1 kHz, the exposure time should be approximately 0.83 milliseconds or less. Similarly, if a recognition success rate of 95% or more is desired, the relative exposure time should be 1.25 or less, and if a recognition success rate of 80% or more is desired, the relative exposure time should be 1.4 or less. Recognize. In addition, the recognition success rate drops sharply around the relative exposure time of 1.5 and becomes almost 0% at 1.6, so it can be seen that the relative exposure time should not exceed 1.5. . Also, it can be seen that after the recognition rate becomes 0 at 7507c, it rises again at 7507d, 7507e, and 7507f. Therefore, if you want to capture a bright image by increasing the exposure time, use an exposure time with a relative exposure time of 1.9 to 2.2, 2.4 to 2.6, or 2.8 to 3.0. do it. For example, intermediate modes may use these exposure times.

FIG. 6A is a flowchart of an information communication method according to this embodiment.

The information communication method according to the present embodiment is an information communication method for obtaining information from a subject, and includes steps SK91 to SK93.

In other words, in this information communication method, a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor are generated in an image obtained by photographing the subject with an image sensor according to changes in luminance of the subject. a first exposure time setting step SK91 for setting a first exposure time of the image sensor; a first image acquisition step SK92 for acquiring a bright line image including a plurality of bright lines; and information acquisition for acquiring information by demodulating data specified by the pattern of the plurality of bright lines included in the acquired bright line image. In the first image acquisition step SK92, each of the plurality of exposure lines sequentially starts exposure at different times, and the exposure of adjacent exposure lines adjacent to the exposure line is completed. After a predetermined idle time has passed since then, exposure is started.

FIG. 6B is a block diagram of the information communication device in this embodiment.

An information communication device K90 in the present embodiment is an information communication device that obtains information from a subject, and includes components K91 to K93.

In other words, the information communication device K90 is arranged such that a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor appear in an image obtained by photographing the subject with the image sensor according to changes in the brightness of the subject. an exposure time setting unit K91 for setting the exposure time of the image sensor; and the image sensor for acquiring a bright line image including the plurality of bright lines by photographing the subject whose brightness changes with the set exposure time. and an information acquisition unit K93 for acquiring information by demodulating data specified by the patterns of the plurality of bright lines included in the acquired bright line image, wherein the plurality of exposure lines , start exposure at different times sequentially, and start exposure after a predetermined idle time has elapsed after the exposure of the adjacent exposure line is completed.

In the information communication method and information communication apparatus K90 shown in FIGS. 6A and 6B, as shown in FIG. 5C, for example, each of a plurality of exposure lines is exposed in an adjacent exposure line adjacent to the exposure line. Since the exposure is started after a predetermined idle time has passed since the end, it is possible to easily recognize the luminance change of the subject. As a result, information can be appropriately acquired from the subject.

In the above-described embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. For example, the program causes the computer to execute the information communication method illustrated by the flow chart of FIG. 6A.

(Embodiment 2)
In this embodiment, each application example using a receiver such as a smartphone that is the information communication device K90 in the first embodiment and a transmitter that transmits information as a blinking pattern of a light source such as an LED or an organic EL is described. explain.

In the following description, the normal shooting mode or shooting in the normal shooting mode is called normal shooting, and the visible light communication mode or shooting in the visible light communication mode is called visible light shooting (visible light communication). Also, instead of normal photography and visible light photography, intermediate mode photography may be used, and an intermediate image may be used instead of a composite image, which will be described later.

FIG. 7 is a diagram showing an example of the imaging operation of the receiver according to this embodiment.

The receiver 8000 switches the shooting mode between normal shooting, visible light communication, normal shooting, and so on. Then, the receiver 8000 synthesizes the normal photographed image and the visible light communication image to generate a synthesized image in which the bright line pattern, the subject and its surroundings are clearly displayed, and displays the synthesized image on the display. . This composite image is an image generated by superimposing the bright line pattern of the visible light communication image on the portion of the normally captured image where the signal is transmitted. In addition, the bright line pattern, the subject and its surroundings projected by this composite image are each sharp, and have a sharpness sufficiently recognized by the user. By displaying such a composite image, the user can more clearly know where or from what position the signal is being transmitted.

FIG. 8 is a diagram showing another example of the imaging operation of the receiver according to this embodiment.

Receiver 8000 includes camera Ca1 and camera Ca2. In such a receiver 8000, the camera Ca1 performs normal imaging, and the camera Ca2 performs visible light imaging. As a result, the camera Ca1 acquires the normal captured image as described above, and the camera Ca2 acquires the visible light communication image as described above. Then, the receiver 8000 combines the normal captured image and the visible light communication image to generate the above-described composite image and display it on the display.

FIG. 9 is a diagram showing another example of the imaging operation of the receiver according to this embodiment.

In the receiver 8000 having two cameras, the camera Ca1 switches the shooting mode between normal shooting, visible light communication, normal shooting, and so on. On the other hand, the camera Ca2 continues normal photography. Then, when the cameras Ca1 and Ca2 are simultaneously performing normal photography, the receiver 8000 uses stereo vision (principle of triangulation) to receive images from the normal photography images acquired by those cameras. The distance from the machine 8000 to the subject (hereinafter referred to as subject distance) is estimated. By using the subject distance estimated in this way, the receiver 8000 can superimpose the bright line pattern of the visible light communication image at an appropriate position on the normal captured image. That is, an appropriate composite image can be generated.

FIG. 10 is a diagram showing an example of the display operation of the receiver according to this embodiment.

The receiver 8000 switches the imaging mode among visible light communication, normal imaging, visible light communication, . . . as described above. Here, receiver 8000 starts an application program when performing visible light communication for the first time. Then, the receiver 8000 estimates its own position based on the signal received by visible light communication. Next, when performing normal photography, the receiver 8000 displays AR (Augmented Reality) information on the normal photography image acquired by the normal photography. This AR information is obtained based on the position estimated as described above. In addition, the receiver 8000 estimates the movement and direction change of the receiver 8000 based on the detection result of the 9-axis sensor and the motion detection of the normal captured image, and adjusts the estimated movement and direction change. to move the display position of the AR information. This allows the AR information to follow the subject image of the normal shot image.

In addition, when the imaging mode is switched from normal imaging to visible light communication, the receiver 8000 superimposes AR information on the latest normal captured image acquired during normal imaging immediately before during the visible light communication. Then, the receiver 8000 displays the normal captured image on which the AR information is superimposed. In addition, the receiver 8000 estimates the movement and direction change of the receiver 8000 based on the detection result of the 9-axis sensor, as in the normal shooting, and adjusts the estimated movement and direction change to AR Move information and normal captured images. As a result, during visible light communication, the AR information can follow the subject image of the normal photographed image in accordance with the movement of the receiver 8000, as in the case of normal photographing. Also, the normal image can be enlarged or reduced in accordance with the movement of the receiver 8000 or the like.

FIG. 11 is a diagram showing an example of the display operation of the receiver according to this embodiment.

For example, the receiver 8000 may display the composite image in which the bright line pattern is projected, as shown in FIG. 11(a). Also, as shown in FIG. 11B, the receiver 8000 normally captures a signal manifestation object, which is an image having a predetermined color for notifying that a signal is being transmitted, instead of the bright line pattern. A composite image may be generated by superimposing on the image, and the composite image may be displayed.

In addition, as shown in FIG. 11C, the receiver 8000 is normally configured such that the locations where the signals are transmitted are indicated by dotted-line frames and identifiers (for example, ID: 101, ID: 102, etc.). A photographed image may be displayed as a composite image. Also, as shown in FIG. 11(d), the receiver 8000 has a signal identification signal which is an image having a predetermined color for notifying that a specific type of signal is being transmitted instead of the bright line pattern. A composite image may be generated by superimposing the object on the normal captured image, and the composite image may be displayed. In this case, the color of the signal identification object depends on the type of signal being output from the transmitter. For example, when the signal output from the transmitter is location information, a red signal identification object is superimposed, and when the signal output from the transmitter is a coupon, a green signal identification object is superimposed. superimposed.

FIG. 12 is a diagram showing an example of the operation of the receiver according to this embodiment.

For example, when receiving a signal by visible light communication, the receiver 8000 may display a normal captured image and output a sound to notify the user that the transmitter has been found. In this case, receiver 8000 varies the type of sound output, the number of output times, or the output time depending on the number of detected transmitters, the type of received signal, or the type of information specified by the signal. You can let

FIG. 13 is a diagram showing another example of the operation of the receiver according to this embodiment.

For example, when the user touches the bright line pattern displayed in the composite image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched bright line pattern, and notifies the information. Display an image. This information notification image shows, for example, the coupon of the store, the location, and the like. Note that the bright line pattern may be a signal manifestation object, a signal identification object, or a dotted line frame shown in FIG. 11 . The same applies to bright line patterns described below.

FIG. 14 is a diagram showing another example of the operation of the receiver according to this embodiment.

For example, when the user touches the bright line pattern displayed in the composite image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched bright line pattern, and notifies the information. Display an image. This information notification image indicates, for example, the current location of the receiver 8000 using a map or the like.

FIG. 15 is a diagram showing another example of the operation of the receiver according to this embodiment.

For example, when the user swipes the receiver 8000 on which the composite image is displayed, the receiver 8000 displays a normal shot image having a dotted line frame and an identifier similar to the normal shot image shown in (c) of FIG. Along with displaying an image, a list of information is displayed following the swipe operation. This list shows information specified by a signal transmitted from a location (transmitter) indicated by each identifier. A swipe may also be an operation of moving a finger from outside to inside on the right side of the display of the receiver 8000, for example. The swipe may be an operation of moving a finger in from the upper side, the lower side, or the left side of the display.

Also, when the user taps information included in the list, the receiver 8000 may display an information notification image (for example, an image showing a coupon) showing the information in more detail.

FIG. 16 is a diagram showing another example of the operation of the receiver according to this embodiment.

For example, when the user swipes the receiver 8000 on which the composite image is displayed, the receiver 8000 displays the information notification image superimposed on the composite image so as to follow the swipe operation. This information notification image shows the subject distance with an arrow in an easy-to-understand manner for the user. Also, swiping may be, for example, an operation of moving a finger from the outside to the inside of the bottom side of the display of the receiver 8000 . A swipe may be an operation of moving a finger inward from the left side, the upper side, or the right side of the display.

FIG. 17 is a diagram showing another example of the operation of the receiver according to this embodiment.

For example, the receiver 8000 shoots a transmitter, which is a signage showing a plurality of stores, as a subject, and displays a normal shot image acquired by the shooting. Here, when the user taps the signage image of one store included in the subject displayed in the normal captured image, the receiver 8000 generates an information notification image based on the signal transmitted from the signage of the store. and the information notification image 8001 is displayed. This information notification image 8001 is an image indicating, for example, the vacant seats in the store.

An information communication method according to the present embodiment is an information communication method for acquiring information from a subject, wherein an image obtained by photographing the subject with an image sensor includes bright lines corresponding to exposure lines included in the image sensor. a first exposure time setting step of setting an exposure time of the image sensor so as to occur according to a change in brightness of the subject; a bright line image acquiring step of acquiring a bright line image, which is an image including the bright lines; An image display step of displaying a display image showing the surroundings of a subject; and an information acquisition step of acquiring transmission information by demodulating data specified by the bright line pattern included in the acquired bright line image. including.

For example, composite images or intermediate images as shown in FIGS. 7, 8 and 11 are displayed as display images. Further, in a display image in which a subject and its surroundings are projected, the spatial position of a portion where a bright line appears is identified by a bright line pattern, a signal manifestation object, a signal identification object, a dotted line frame, or the like. Therefore, by looking at such a displayed image, the user can easily find the subject that is transmitting the signal by the luminance change.

The information communication method further includes a second exposure time setting step of setting an exposure time longer than the exposure time; a normal image obtaining step of obtaining a normal photographed image; a site where the bright lines appear in the normally photographed image are specified based on the bright line image; and a synthesizing step of generating a synthetic image by superimposing it on the captured image, and the image display step may display the synthetic image as the display image.

For example, the signal object is a bright line pattern, a signal manifestation object, a signal identification object, a dotted line frame, etc. As shown in FIGS. 7, 8 and 11, a synthesized image is displayed as a display image. As a result, the user can more easily find the object transmitting the signal by the change in brightness.

Further, in the first exposure time setting step, the exposure time is set to 1/3000 second, in the bright line image acquiring step, the bright line image showing the surroundings of the subject is acquired, and in the image displaying step, , the bright line image may be displayed as the display image.

For example, a bright line image is obtained and displayed as an intermediate image. Therefore, there is no need to perform processing such as acquiring and synthesizing the normally captured image and the visible light communication image, and the processing can be simplified.

Further, the image sensor includes a first image sensor and a second image sensor, and in the normal image obtaining step, the normal image is obtained by photographing by the first image sensor, and the bright line In the image acquisition step, the bright line image may be acquired by the second image sensor taking an image simultaneously with the image taking by the first image sensor.

For example, as shown in FIG. 8, a normal captured image and a visible light communication image, which is a bright line image, are acquired by respective cameras. Therefore, compared to the case where a single camera acquires a normal shot image and a visible light communication image, these images can be acquired quickly, and the processing speed can be increased.

Further, in the information communication method, when a region where the bright line appears in the display image is designated by a user's operation, the transmission information acquired from the bright line pattern of the designated region is An information presentation step of presenting presentation information based on the information may be included. For example, the operation by the user includes tapping, swiping, an operation of keeping the fingertip on the part for a predetermined time or more, an operation of keeping the line of sight directed to the part for a predetermined time or more, and an operation of showing in association with the part. The pointer displayed on the display image is applied to the part by moving a part of the user's body in line with the arrow, touching the part with the pen tip whose brightness changes, or by touching the touch sensor. Operation.

For example, as shown in FIGS. 13 to 17, presentation information is displayed as an information notification image. Thereby, desired information can be presented to the user.

Also, in the information communication method for acquiring information from a subject, in an image obtained by photographing the subject with an image sensor, a bright line corresponding to an exposure line included in the image sensor is generated according to a change in brightness of the subject. a first exposure time setting step of setting an exposure time of the image sensor; and an information acquisition step of acquiring information by demodulating data specified by the pattern of the bright lines included in the acquired bright line image, wherein the bright line image In the obtaining step, the bright line image including a plurality of parts in which the bright lines appear is obtained by photographing a plurality of the subjects while the image sensor is being moved, and obtaining the positions of each of the plurality of subjects by demodulating the data specified by the pattern of the bright lines of the part, and the information communication method further comprises: A position estimation step of estimating the position of the image sensor based on the position and the movement state of the image sensor may be included.

This makes it possible to accurately estimate the position of a receiver including an image sensor based on changes in brightness caused by objects such as multiple lights.

Also, in the information communication method for acquiring information from a subject, in an image obtained by photographing the subject with an image sensor, a bright line corresponding to an exposure line included in the image sensor is generated according to a change in brightness of the subject. a first exposure time setting step of setting an exposure time of the image sensor; an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image; and the acquired information and an information presenting step of presenting the information, wherein the information presenting step may present an image prompting a user of the image sensor to perform a predetermined gesture as the information.

Accordingly, the user can be authenticated or the like depending on whether or not the user performs the gesture as prompted, and convenience can be improved.

Also, in the information communication method for acquiring information from a subject, in an image obtained by photographing the subject with an image sensor, a bright line corresponding to an exposure line included in the image sensor is generated according to a change in brightness of the subject. and an exposure time setting step of setting the exposure time of the image sensor, and the image sensor captures the subject whose luminance changes at the set exposure time, thereby obtaining a bright line image including the bright lines. and an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image, wherein the image acquisition step includes: acquiring the bright line image by photographing a plurality of the subjects, and in the information obtaining step, converting the bright lines into bright lines corresponding to each of the plurality of subjects according to the intensity of the bright lines included in the bright line image; The information may be obtained by separating and demodulating the data specified by the bright line pattern corresponding to each subject.

This makes it possible to acquire appropriate information from each of the subjects even when the brightness of each subject such as a plurality of lights changes.

Also, in the information communication method for acquiring information from a subject, in an image obtained by photographing the subject with an image sensor, a bright line corresponding to an exposure line included in the image sensor is generated according to a change in brightness of the subject. and an exposure time setting step of setting the exposure time of the image sensor, and the image sensor captures the subject whose luminance changes at the set exposure time, thereby obtaining a bright line image including the bright lines. and an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image, wherein the image acquisition step includes: Obtaining the bright line image by photographing the subject, the information communication method may further include a position estimation step of estimating the position of the subject based on a luminance distribution in the bright line image.

This makes it possible to estimate an appropriate subject position based on the luminance distribution.

Further, in the transmitting step, when switching the luminance change between the luminance change according to the first pattern and the luminance change according to the second pattern, a buffer time may be provided before switching.

Thereby, interference between the first signal and the second signal can be suppressed.

Also, an information communication method for transmitting a signal by means of luminance change, comprising: a determination step of determining a pattern of luminance change by modulating a signal to be transmitted; and a transmitting step of transmitting the signal to be transmitted by transmitting the signal to be transmitted, the signal being composed of a plurality of large blocks, each of the plurality of large blocks being first data and a preamble for the first data. and a check signal for the first data, wherein the first data is composed of a plurality of small blocks, and the small blocks include second data, a preamble for the second data, and the first data. and a check signal for 2 data.

As a result, both a receiver that requires a blanking period and a receiver that does not require a blanking period can appropriately acquire data.

Also, an information communication method for transmitting signals by means of luminance changes, comprising: a determination step of determining a pattern of luminance change by modulating a signal to be transmitted by each of a plurality of transmitters; a transmitting step in which a light-emitting body provided in a transmitter transmits the signal to be transmitted by changing luminance according to the determined pattern, wherein the transmitting step transmits signals having different frequencies or different protocols. You may

Thereby, interference of signals from a plurality of transmitters can be suppressed.

Also, an information communication method for transmitting signals by means of luminance changes, comprising: a determination step of determining a pattern of luminance change by modulating a signal to be transmitted by each of a plurality of transmitters; a transmitting step of transmitting the signal to be transmitted by a light-emitting body provided in a transmitter that changes in luminance according to the determined pattern, wherein the transmitting step comprises one of the plurality of transmitters; One transmitter may receive the signal transmitted from the other transmitter and transmit the other signal in a manner that does not interfere with the received signal.

Thereby, interference of signals from a plurality of transmitters can be suppressed.

(guidance at the station)
FIG. 18A shows an example of usage of the present invention on a train platform. A user holds a portable terminal up to an electronic bulletin board or a light and obtains information displayed on the electronic bulletin board, train information of the station where the electronic bulletin board is installed, premises information of the station, etc. by visible light communication. Here, the information itself displayed on the electronic bulletin board may be transmitted to the mobile terminal by visible light communication, or the ID information corresponding to the electronic bulletin board may be transmitted to the mobile terminal, and the ID information acquired by the mobile terminal may be transmitted to the mobile terminal. information displayed on the electronic bulletin board may be acquired by inquiring of the server. When the ID information is transmitted from the mobile terminal, the server transmits the contents displayed on the electronic bulletin board to the mobile terminal based on the ID information. The train ticket information stored in the memory of the mobile terminal is compared with the information displayed on the electronic bulletin board, and if the ticket information corresponding to the user's ticket is displayed on the electronic bulletin board, the The display shows an arrow indicating the destination of the train that the user plans to board. A route to a vehicle close to an exit or transfer route may be displayed when getting off.

If a seat is assigned, the route to that seat may be displayed. When displaying the arrows, the arrows can be displayed in a more comprehensible manner by displaying the arrows in the same color as the color of the train route in the map or the train guide information. In addition to displaying the arrow, it is also possible to display the user's reservation information (platform number, vehicle number, departure time, seat number). By also displaying the reservation information of the user, it is possible to prevent erroneous recognition. If the ticket information is stored in the server, the ticket information is obtained and compared by inquiring the server from the mobile terminal, or the ticket information is compared with the information displayed on the electronic bulletin board on the server side. can acquire information related to the ticket information. A target route may be estimated from the user's transfer search history, and the route may be displayed. In addition to the contents displayed on the electronic bulletin board, train information and premises information of the station where the electronic bulletin board is installed may be acquired and compared. Information related to the user may be highlighted or rewritten for display on the electronic bulletin board on the display. If the user's boarding schedule is unknown, arrows may be displayed to guide the boarding point of each route. When the premises information of the station is acquired, arrows may be displayed on the display to guide the user to shops, restrooms, and the like. The user's behavioral characteristics may be managed in advance by a server, and when the user frequently visits a store or restroom in the station, an arrow may be displayed on the display to guide the user to the store or restroom. It is possible to reduce the amount of processing by displaying an arrow that guides the user to the shop or restroom only for users who have the behavior characteristic of stopping by the shop or restroom, and not displaying it for other users. . The color of the arrow leading to the shop, restroom, etc. may be different from the color of the arrow guiding the destination to the platform. When both arrows are displayed at the same time, erroneous recognition can be prevented by using different colors. Note that FIG. 18A shows an example of a train, but it is possible to display with a similar configuration on an airplane, a bus, or the like.

Specifically, a mobile terminal such as a smartphone (that is, a receiver such as the receiver 200 described later) captures an image of an electronic bulletin board as shown in (1) of FIG. is received as an optical ID or optical data. At this time, the mobile terminal performs self-position estimation. In other words, the mobile terminal obtains the position on the map of the electronic bulletin board indicated directly or indirectly by the optical data. Then, the mobile terminal, for example, based on its own posture obtained by a 9-axis sensor, and the position, shape, and size in the image of the electronic bulletin board displayed in the image obtained by imaging, the electronic bulletin board. Calculate the relative position of the mobile terminal. The mobile terminal estimates its own position, which is the position on the map of the mobile terminal, based on the position on the map of the electronic bulletin board and its relative position. The mobile terminal searches for a route from the starting point, which is the self-position, to the destination indicated by, for example, the ticket information, and starts navigation to guide the user to the destination along the route. It should be noted that the mobile terminal may transmit information indicating its starting point and destination to the server, and obtain the above-described route retrieved by the server from the server. At this time, the mobile terminal may acquire a map including the route from the server.

In navigation, as shown in (2) to (4) of FIG. 18A , the mobile terminal repeats imaging with the camera, and sequentially displays normal captured images obtained by the imaging in real time, while arrows indicating the destination of the user are displayed. A direction indicating image such as , is superimposed on the normal photographed image. The user moves according to the displayed direction instruction image while carrying the portable terminal. Then, the mobile terminal updates its own position based on the movement of the object or the feature point displayed in each of the above-described normal captured images. For example, the mobile terminal detects the motion of the object or feature point appearing in each of the above-described normal shot images, and estimates the moving direction and moving distance of the mobile terminal based on the motion. Then, the mobile terminal updates its current self-location based on the estimated moving direction and moving distance and the estimated self-location in (1) of FIG. 18A. This update of the self-position may be performed for each frame period of the normal captured image, or may be performed for each period longer than the frame period. That is, when the mobile terminal is on an underground floor or pathway, the mobile terminal cannot acquire GPS data. Therefore, in such a case, the mobile terminal estimates or updates its own position based on the movement of the feature points of the above-mentioned normal shot images without using GPS data.

Here, as shown in (4) of FIG. 18A, the mobile terminal may guide the user to the elevator on the way to the destination. Further, as shown in (5) and (6) in FIG. 18A, the mobile terminal receives the optical data from the transmitter that transmits the optical data, or when the reflected light of the optical data is captured, and receives the optical data. The self-position is estimated in the same manner as the example shown in (1). For example, even when a user gets on an elevator, the mobile terminal receives optical data transmitted from a transmitter (that is, a transmitter such as a transmitter 100 to be described later) installed as a lighting device or the like inside the car of the elevator. receive. For example, the light data may directly or indirectly indicate the floor on which the elevator car is currently located. Therefore, the mobile terminal can identify the floor on which the mobile terminal is currently located by receiving the optical data. If the optical data does not directly indicate the current position of the car, the portable terminal transmits the information indicated by the optical data to the server, and the floor information associated with the information in the server. , from that server. Thereby, the mobile terminal identifies the floor indicated by the floor number information as the floor on which the mobile terminal is currently located. A floor identified in this way is treated as a self-location.

As a result, as shown in (7) of FIG. 18A, the terminal device replaces the self-position derived from the movement of the feature points of the normal captured image with the self-position derived using the optical data. , to reposition itself.

Then, as shown in (8) of FIG. 18A, if the user has not reached the destination after getting off the elevator, the mobile terminal performs the same processes as (2) to (4) of FIG. 18A. navigate while Also, the mobile terminal repeatedly confirms whether or not GPS data can be acquired during navigation. Therefore, the mobile terminal determines that it can acquire GPS data when it rises above ground from an underground floor or pathway. Then, the portable terminal switches the self-position estimation method from the estimation method based on the movement of the feature points to the estimation method based on the GPS data. Then, as shown in (9) of FIG. 18A, the mobile terminal continues to perform navigation while estimating its own position based on the GPS data until the user arrives at the destination. For example, when the user goes underground again, the mobile terminal cannot acquire GPS data. switch to

The example of FIG. 18A will be described in detail below.

In the example shown in FIG. 18A, a receiver implemented as a wearable device such as a smartphone or smart glasses receives the visible light signal (optical data) transmitted from the transmitter at (1) in FIG. 18A. Transmitters are implemented, for example, as lights that illuminate illuminated signs, posters, or statues. The receiver starts navigation to the destination according to the received optical data, the information preset in the receiver, and the user's instructions. The receiver transmits the light data to the server and obtains the navigation information associated with this data. The navigation information includes the following first to sixth information. The first information is information indicating the position and shape of the transmitter. The second information is information indicating the route to the destination. The third information is information of other transmitters on and near the route to the destination. Specifically, the information of another transmitter indicates the optical data that the transmitter is transmitting, the position and shape of the transmitter, and the position and shape of the reflected light. The fourth information is location specific information on and around the route. Specifically, the position specifying information is radio wave information or sound wave information for specifying an image feature amount or a position. The fifth information is information indicating the distance to the destination and the expected arrival time. The sixth information is part or all of content information for AR display. The navigation information may be pre-stored in the receiver. Note that the above shape may include a size.

The receiver determines the position of the receiver based on the relative position between the transmitter and the receiver calculated from the appearance of the transmitter in the image obtained by imaging and the sensor value of the acceleration sensor, and the position information of the transmitter. The self-position is estimated, and the self-position is used as the starting point of navigation. The receiver may start navigation by estimating its self-position based on image feature values, barcodes or two-dimensional codes, radio waves, or sound waves instead of optical data.

The receiver displays navigation to the destination as shown in (2) of FIG. 18A. This navigation display may be an AR display in which another image is superimposed on a normal shot image obtained by imaging with a camera, may be a map display, or may be an instruction by voice or vibration. A combination of these may be used. The display method may be selected by setting on the receiver, optical data, or server. Either setting may take precedence over the other. Also, if the destination (that is, the destination) is a boarding place for transportation, the receiver acquires the timetable and displays the reserved time or the departure time or boarding time near the expected arrival time. good too. Also, if the destination is a theater or the like, the receiver may display the opening time or the entry deadline.

The receiver advances navigation according to the movement of the receiver, as shown in (3) and (4) of FIG. 18A. In a situation where absolute position information cannot be obtained, the receiver calculates the moving distance of the receiver while the images were captured from the moving distance between the images of the feature points in the plurality of images. Direction may be estimated. Also, the receiver may estimate the movement distance and direction of the receiver from an acceleration sensor, radio waves, or changes in sound waves. The receiver may also estimate the moving distance and direction of the receiver by SLAM (Simultaneous Localization and Mapping) or PTAM (Parallel Tracking and Mapping).

In (5) of FIG. 18A, when the receiver receives optical data different from the optical data received in (1) of FIG. 18A, for example, outside an elevator, it sends the optical data to the server, The shape and position of the transmitter associated with the data may be obtained. Then, the receiver may estimate its own position by a method similar to (1) in FIG. 18A. As a result, the receiver corrects the current navigation position by eliminating errors in the receiver's self-position estimation that occurred in processes (3) and (4) of FIG. 18A. The receiver receives only a portion of the visible light signal, and if the complete light data is not available, the nearest transmitter from the navigation information is the transmitter that is transmitting that visible light signal. It is assumed that there is, and thereafter, the self-position estimation of the receiver is performed in the same manner as described above. This allows transmitters with poor reception conditions, e.g. small transmitters, distant transmitters, or even dark transmitters, to be used for receiver self-localization. can.

The receiver receives the optical data by the reflected light at (6) in FIG. 18A. The receiver identifies the medium of the received optical data as reflected light from the imaging direction, light intensity, or contour clarity. If it is reflected light, the receiver identifies the position of the reflected light from the navigation information (that is, the position on the map), and estimates the center of the reflected light area being imaged as the position of the reflected light. . Then, the receiver estimates its own position and corrects the current navigation position, as in (5) of FIG. 18A.

When the receiver receives a signal for identifying a position such as GPS, GLONASS, Galileo, Beidou satellite positioning system, IRNSS, etc., the receiver identifies the position of the receiver by the signal, and the current navigation position (i.e. self-position ) is corrected. If the strength of the signal is sufficient, that is, if it is stronger than a predetermined strength, the receiver estimates the self-position only by the signal, and if it is less than the predetermined strength, (3) and (3) in FIG. 18A The method used in (4) may be used together.

When the receiver receives a visible light signal, [1] a radio signal having a predetermined ID received at the same time as the visible light signal, [2] a radio signal having a predetermined ID received last, or [3] Sending the information indicating the last estimated position of the receiver to the server together with the information indicated by the visible light signal. This identifies the transmitter that transmits the visible light signal. Alternatively, the receiver receives a visible light signal by an algorithm specified by the above-described radio signal or information indicating the position of the receiver, and the server specified in the same manner as described above is indicated by the visible light signal. You may send information.

The receiver may estimate its self-location and display information about items near its self-location. Also, the receiver may navigate to the position of the product specified by the user. Also, the receiver may present the optimum route for visiting all the locations of a plurality of products specified by the user. The optimal route is the route with the shortest distance, the route with the shortest required time, or the route with the least travel effort. Also, the receiver may perform navigation so as to pass through a predetermined location in addition to the product or location specified by the user. This makes it possible to advertise a given location or advertise a product or store located at that location.

FIG. 18B is a diagram for explaining navigation by receiver 200 in an elevator in this embodiment.

For example, a receiver configured as a smartphone performs guidance using AR display, that is, AR navigation, as shown in (1) of FIG. 18B when the user is on the third basement floor (B3). AR navigation, as shown in FIG. 18A, is a navigation function that guides a user to a destination by superimposing a direction indicating image such as an arrow on a normally captured image. In addition, hereinafter, AR navigation is also simply referred to as navigation or navigation.

Then, when the user boards the elevator, the receiver receives an optical signal (i.e., visible light signal, optical data, or optical ID) from the transmitter located in the elevator car, as shown in (2) of FIG. 18B. receive. Thereby, the receiver obtains the elevator ID and the floor information based on the optical signal. The elevator ID is identification information for identifying the elevator or its car in which the transmitter is located, and the floor information is information indicating the floor (or floor number) where the car is currently located. For example, the receiver transmits the light signal (or information indicated by the light signal) to the server and obtains from the server the elevator ID and floor information associated with the light signal at the server. The transmitter may always send the same optical signal no matter what floor the elevator car is on, or may send different optical signals depending on the floor on which the car is located. Also, the transmitter is configured as, for example, a lighting device. Light from this transmitter brightly illuminates the interior of the elevator car. Thus, the receiver can receive the optical signal superimposed on such light either directly from the transmitter or indirectly via reflection from the inner wall or floor of the cage.

The receiver detects the current position of the receiver according to the elevator ID and floor information obtained based on the optical signal transmitted from the transmitter even when the car containing the receiver is ascending. Each floor is identified one by one. Then, as shown in (3) of FIG. 18B, when the floor where the receiver is currently located becomes the target floor, the receiver displays a message or an image prompting to get off the elevator on the display of the receiver. do. Also, the receiver may output a voice prompting to get off the elevator.

Then, if the target floor is a place where GPS data does not reach, such as the first basement floor, the receiver uses the above-described estimation method using the movement of the feature points of the normal photographed image ( As shown in 4), the above-described AR navigation is resumed while estimating the self-position. On the other hand, if the target floor is a location where GPS data can reach, such as the first floor above ground, the receiver determines its own position by an estimation method using the GPS data, as shown in (4) of FIG. 18B. Resume the AR navigation described above while estimating.

FIG. 18C is a diagram showing an example of a system configuration provided in an elevator according to this embodiment.

A transmitter 100, which is the transmitter described above, is installed in the car 420 of the elevator. This transmitter 100 is arranged on the ceiling of an elevator car 420 as a lighting device for the car 420 . The transmitter 100 also has a built-in camera 404 and a microphone 411 . The built-in camera 404 images the inside of the basket 420 and the microphone 411 picks up the sound inside the basket 420 .

In addition, a surveillance camera system 401 , a floor display section 414 and a sensor 403 are installed in the cage 420 . Surveillance camera system 401 is a system having at least one camera for imaging the interior of basket 420 . The floor number display portion 414 displays the floor on which the basket 420 is currently located. Sensor 403 includes, for example, at least one of an air pressure sensor and an acceleration sensor.

The elevator also includes an image recognition unit 402 , a current floor detection unit 405 , a light modulation unit 406 , a light emission circuit 407 , a radio unit 409 and a voice recognition unit 410 .

The image recognition unit 402 recognizes the characters (that is, the floor number) displayed on the floor display unit 414 from the image captured by the monitoring camera system 401 or the built-in camera 404, and converts the current floor data obtained by the recognition. Output. The current floor data indicates the floor number displayed in the floor number display section 414 .

The speech recognition unit 410 recognizes the floor on which the car 420 is currently located based on the speech data output from the microphone 411, and outputs floor data indicating the floor.

Current floor detection unit 405 detects the floor where car 420 is currently located based on data output from at least one of sensor 403 , image recognition unit 402 , and voice recognition unit 410 . Current floor detection section 405 then outputs information indicating the detected floor to light modulation section 406 .

Light modulating section 406 modulates the information indicating the floor output from current floor detecting section 405 and the signal indicating the elevator ID, and outputs the modulated signal to light emitting circuit 407 . Light emitting circuit 407 changes the brightness of transmitter 100 according to the modulated signal. As a result, transmitter 100 transmits the above-described visible light signal, light signal, light data, or light ID indicating the floor where car 420 is currently located and the elevator ID.

Similarly to optical modulation section 406, radio section 409 modulates the information indicating the floor output from current floor detection section 405 and the signal indicating the elevator ID, and transmits the modulated signal by radio. For example, the wireless unit 409 transmits signals by Wi-Fi or Bluetooth.

Accordingly, the receiver 200 can identify the floor and the elevator ID on which the receiver 200 is currently located by receiving at least one of the radio signal and the optical signal.

The elevator may also include a current floor detector 412 having the floor display 414 described above. The current floor detection unit 412 is composed of an elevator control unit 413 and a floor number display unit 414 . Elevator control unit 413 controls lifting and stopping of cage 420 . Such an elevator control unit 413 grasps the floor on which the car 420 is currently located. Then, the elevator control unit 413 may output the data indicating the grasped floor to the light modulation unit 406 and the radio unit 409 as the current floor data.

With such a configuration, receiver 200 can realize AR navigation shown in FIGS. 18A and 18B.

(Application example for directions)
FIG. 19 is a diagram showing an application example of the transmission/reception system according to the second embodiment.

The receiver 8955a receives the transmission ID of the transmitter 8955b configured as, for example, an information board, acquires the map data displayed on the information board from the server, and displays the data. At this time, the server may send advertisements suitable for the user of the receiver 8955a, and the receiver 8955a may also display this advertisement information. The receiver 8955a displays the route from the current location to the location specified by the user.

(Example of application to usage log accumulation and analysis)
FIG. 20 is a diagram showing an application example of the transmission/reception system according to the second embodiment.

The receiver 8957a receives an ID transmitted from a transmitter 8957b configured as a signboard, acquires coupon information from the server, and displays the coupon information. The receiver 8957a monitors subsequent actions of the user, such as saving the coupon, going to the store displayed on the coupon, shopping at the store, or leaving without saving the coupon. Save to 8957c. This makes it possible to analyze the subsequent behavior of the user who acquired information from the signboard 8957b, and estimate the advertising value of the signboard 8957b.

An information communication method according to the present embodiment is an information communication method for obtaining information from a subject, wherein an image obtained by photographing a first subject, which is the subject, by an image sensor includes each exposure included in the image sensor. a first exposure time setting step of setting a first exposure time of the image sensor such that a plurality of bright lines corresponding to lines are generated according to a change in brightness of the first object; a first bright line image obtaining step of obtaining a first bright line image, which is an image including the plurality of bright lines, by photographing the changing first subject at the set first exposure time; a first information acquiring step of acquiring first transmission information by demodulating data specified by the pattern of the plurality of bright lines included in the acquired first bright line image; and the first transmission information. is obtained, a door control step of causing a door opening/closing driving device to open the door by transmitting a control signal.

This allows the receiver equipped with the image sensor to be used like a door key, eliminating the need for a special electronic lock. As a result, communication can be performed between various devices including devices with low computing power.

Further, in the information communication method, the image sensor captures a second subject whose luminance changes at the set first exposure time, thereby generating a second image including a plurality of bright lines. a second bright line image acquiring step of acquiring a bright line image; and acquiring second transmission information by demodulating data specified by the plurality of bright line patterns included in the acquired second bright line image. a second information acquisition step; and an approach determination step of determining whether or not the receiving device having the image sensor is approaching the door based on the acquired first and second transmission information. In the door control step, the control signal may be transmitted when it is determined that the receiving device is approaching the door.

Thereby, the door can be opened only when the receiving device (receiver) approaches the door, that is, only at an appropriate timing.

The information communication method further includes a second exposure time setting step of setting a second exposure time longer than the first exposure time; and a normal image obtaining step of obtaining a normal image in which the third subject is projected by photographing with the second exposure time, wherein the normal image obtaining step includes an area including optical black of the image sensor. For each of the plurality of exposure lines in the above, after a predetermined time has elapsed from the time when the charge readout for the exposure line adjacent to the exposure line is performed, the charge is read out, and the first bright line image acquisition step Then, for each of a plurality of exposure lines in a region other than the optical black in the image sensor, the charge is read for the exposure line adjacent to the exposure line without using the optical black for reading the charge. After a period of time longer than the predetermined period of time has passed from the point of time when the charge is read out, the electric charge may be read out.

As a result, when the first bright line image is acquired, the charge readout (exposure) for the optical black is not performed, so the charge readout (exposure) for the effective pixel region other than the optical black in the image sensor is performed. It can take longer. As a result, the time for receiving signals in the effective pixel area can be lengthened, and many signals can be obtained.

Further, in the information communication method, the length in the direction perpendicular to each of the plurality of bright lines in the pattern of the plurality of bright lines included in the first bright line image is less than a predetermined length. a length determination step of determining whether or not there is a frame rate of the image sensor when it is determined that the length of the pattern is less than the predetermined length; a frame rate changing step of changing the frame rate to a second frame rate that is slower than the first frame rate when the image was acquired; and a third bright line image obtaining step of obtaining a third bright line image, which is an image including a plurality of bright lines, by photographing at the set first exposure time; and a third information acquisition step of acquiring the first transmission information by demodulating data specified by the plurality of bright line patterns included in the bright line image.

As a result, when the signal length indicated by the bright line pattern (bright line area) included in the first bright line image is less than, for example, one block worth of the transmitted signal, the frame rate is reduced and the bright line An image is acquired as a third bright line image. As a result, the length of the bright line pattern included in the third bright line image can be increased, and the transmitted signal for one block can be obtained.

In addition, the information communication method further includes a ratio setting step of setting a ratio of the vertical width to the horizontal width of the image obtained by the image sensor, and the first bright line image acquisition step is performed according to the set ratio, a clipping determination step of determining whether or not an end of the image in a direction perpendicular to each exposure line is clipped; and the ratio set in the ratio setting step when it is determined that the end is clipped. to a non-clipping ratio, which is a ratio at which the edge is not clipped; and an acquiring step of acquiring the image.

As a result, for example, when the ratio of the width and height of the effective pixel area of the image sensor is 4:3, and the ratio of the width and height of the image is set to 16:9, and a bright line appears along the horizontal direction, That is, when the exposure line extends in the horizontal direction, it is determined that the upper and lower edges of the image are clipped. That is, it is determined that the edge of the first bright line image is missing. In this case, the ratio of the image is changed to a non-clipped ratio, such as 4:3. As a result, it is possible to prevent the end of the first bright line image from being missing, and to obtain a large amount of information from the first bright line image.

The information communication method further includes a compression step of generating a compressed image by compressing the first bright line image in a direction parallel to each of the plurality of bright lines included in the first bright line image. and a compressed image transmission step of transmitting the compressed image.

As a result, the first bright line image can be appropriately compressed without missing information indicated by the plurality of bright lines.

Further, the information communication method further includes a gesture determining step of determining whether or not the receiving device including the image sensor has been moved in a predetermined manner, and determining that the receiving device has been moved in the predetermined manner. and sometimes an activation step of activating the image sensor.

As a result, the image sensor can be easily activated only when necessary, and power consumption efficiency can be improved.

In this embodiment, application examples using a receiver such as the above-described smartphone and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 21 is a diagram showing an application example of a transmitter and a receiver according to Embodiment 2. FIG.

Robot 8970 has, for example, a function as a self-propelled cleaner and a function as a receiver in each of the above embodiments. Lighting devices 8971a and 8971b each have a function as a transmitter in each of the above embodiments.

For example, the robot 8970 cleans while moving in the room and photographs the lighting device 8971a that illuminates the room. This lighting device 8971a transmits the ID of the lighting device 8971a by changing the luminance. As a result, the robot 8970 receives the ID from the lighting device 8971a and estimates its own position (self-position) based on the ID, as in the above embodiments. In other words, the robot 8970 moves and positions itself based on the detection result by the 9-axis sensor, the relative position of the lighting device 8971a reflected in the image obtained by shooting, and the absolute position of the lighting device 8971a specified by the ID. is estimated.

Furthermore, when the robot 8970 moves away from the lighting device 8971a, it transmits a signal (turn-off command) to turn off the lighting device 8971a. For example, the robot 8970 transmits a turn-off command when it leaves the lighting device 8971a by a predetermined distance. Alternatively, the robot 8970 transmits a turn-off command to the lighting device 8971a when the lighting device 8971a disappears from the captured image, or when another lighting device appears in the image. Upon receiving a turn-off command from the robot 8970, the lighting device 8971a turns off in accordance with the turn-off command.

Next, the robot 8970 detects that the robot 8970 has approached the lighting device 8971b while moving and cleaning based on the estimated self position. In other words, the robot 8970 holds information indicating the position of the lighting device 8971b, and when the distance between its own position and the position of the lighting device 8971b becomes equal to or less than a predetermined distance, the robot 8970 turns on the lighting device 8971b. Detects approaching Then, the robot 8970 transmits a signal (lighting command) for commanding lighting to the lighting device 8971b. Upon receiving the lighting command, the lighting device 8971b lights according to the lighting command.

As a result, the robot 8970 can light up only its surroundings while moving to facilitate cleaning.

22 is a diagram showing an application example of the transmitter and receiver in Embodiment 2. FIG.

The lighting equipment 8974 has a function as a transmitter in each of the above embodiments. This lighting device 8974 illuminates, for example, a bulletin board 8975 in a railway station while changing its brightness. A receiver 8973 directed to the bulletin board 8975 by the user photographs the bulletin board 8975 . As a result, the receiver 8973 acquires the ID of the route bulletin board 8975 and acquires detailed information about each route described on the route bulletin board 8975, which is information associated with the ID. The receiver 8973 then displays a guide image 8973a showing the detailed information. For example, the guidance image 8973a indicates the distance to the route described on the route bulletin board 8975, the direction toward that route, and the next arrival time of the train on that route.

Here, when the guide image 8973a is touched by the user, the receiver 8973 displays a supplementary guide image 8973b. This supplementary guidance image 8973b is, for example, a railway timetable, information on a route different from the route indicated by the guidance image 8973a, or detailed information on the station, which can be selected by the user. It is an image to be displayed according to.

(Embodiment 3)
Here, an application example of audio synchronous reproduction will be described below.

23 is a diagram illustrating an example of an application according to Embodiment 3. FIG.

A receiver 1800a configured as, for example, a smartphone receives a signal (visible light signal) transmitted from a transmitter 1800b configured as, for example, street digital signage. That is, the receiver 1800a receives the timing of image reproduction by the transmitter 1800b. The receiver 1800a reproduces the sound at the same timing as the image reproduction. In other words, receiver 1800a performs synchronous reproduction of audio such that the image and audio reproduced by transmitter 1800b are synchronized. Note that the receiver 1800a may reproduce the same image as the image (reproduced image) reproduced by the transmitter 1800b, or a related image related to the reproduced image, together with the sound. Also, the receiver 1800a may cause a device connected to the receiver 1800a to reproduce audio or the like. After receiving the visible light signal, the receiver 1800a may download content such as audio or related images associated with the visible light signal from the server. The receiver 1800a performs synchronous playback after the download.

As a result, even when the sound from the transmitter 1800b cannot be heard, or when the sound from the transmitter 1800b is not reproduced because the reproduction of the street sound is prohibited, the user can hear the sound matched with the display of the transmitter 1800b. can hear Also, even if there is a distance where it takes time for the sound to reach, the user can listen to the sound that matches the display.

Here, multi-language correspondence by voice synchronous playback will be described below.

24 is a diagram illustrating an example of an application according to Embodiment 3. FIG.

Each of the receivers 1800a and 1800c acquires from the server audio in the language set in the receiver and corresponds to, for example, an image such as a movie displayed on the transmitter 1800d, and reproduces the audio. do. Specifically, the transmitter 1800d transmits a visible light signal indicating an ID for identifying the displayed image to the receiver. Upon receiving the visible light signal, the receiver transmits a request signal including the ID indicated in the visible light signal and the language set to itself to the server. The receiver acquires the audio corresponding to the request signal from the server and reproduces it. Thereby, the user can enjoy the work displayed on the transmitter 1800d in the language set by the user.

Here, the audio synchronization method will be described below.

25 and 26 are diagrams showing an example of a transmission signal and an example of an audio synchronization method according to Embodiment 3. FIG.

Different data (for example, data shown in FIG. 25: 1 to 6, etc.) are associated with time at regular time intervals (N seconds). These data may be, for example, an ID for identifying time, time, or voice data (eg, 64 Kbps data). The following description assumes that the data is an ID. The different IDs may have different additional information portions attached to the IDs.

It is desirable that the packets that make up the ID are different. Therefore, it is desirable that the IDs are not continuous. Alternatively, when packetizing the ID, a packetization method is desirable in which discontinuous portions are configured as one packet. Since the error correction signal tends to have different patterns even if the IDs are continuous, the error correction signal may be distributed over a plurality of packets instead of being collected in one packet.

The transmitter 1800d transmits the ID, for example, according to the playback time of the displayed image. The receiver can recognize the reproduction time (synchronization time) of the image of the transmitter 1800d by detecting the timing when the ID is changed.

In the case of (a), since the change points of ID: 1 and ID: 2 are received, the synchronization time can be recognized accurately.

If the time period N during which the ID is transmitted is long, there are few such opportunities, and the ID may be received as in (b). Even in this case, the synchronization time can be recognized by the following method.

(b1) Assume that the midpoint of the receiving section where the ID changes is the ID change point. Also, the time after an integral multiple of the time N from the previously estimated ID change point is also estimated as an ID change point, and the middle point of a plurality of ID change points is estimated as a more accurate ID change point. With such an estimation algorithm, it is possible to gradually estimate an accurate ID change point.

(b2) In addition to the above, by estimating that the ID change point is not included in the reception interval in which the ID did not change and the time after an integer multiple of the time N, the possibility that the ID change point gradually occurs. is reduced, and an accurate ID change point can be estimated.

Accurate synchronization can be achieved by setting N to 0.5 seconds or less.

By setting N to 2 seconds or less, synchronization can be performed without causing the user to perceive a delay.

By setting N to 10 seconds or less, waste of IDs can be suppressed for synchronization.

26 is a diagram showing an example of a transmission signal in Embodiment 3. FIG.

In FIG. 26, ID waste can be avoided by synchronizing with time packets. A time packet is a packet that holds the time it was sent. When it is necessary to express a long time, the time packets are divided into a time packet 1 representing fine time and a time packet 2 representing coarse time. For example, time packet 2 indicates the hours and minutes of the time, and time packet 1 indicates only the seconds of the time. A packet indicating time may be divided into three or more time packets. Since the coarse time is less necessary, sending more fine time packets than coarse time packets allows the receiver to quickly and accurately recognize the synchronized time.

That is, in the present embodiment, the visible light signal includes second information (time packet 2) indicating the hour and minute of the time, and first information (time packet 1) indicating the second of the time. to indicate the time at which the visible light signal is transmitted from transmitter 1800d. Then, the receiver 1800a receives the second information and also receives the first information more times than it receives the second information.

Here, synchronization time adjustment will be described below.

27 is a diagram illustrating an example of a processing flow of receiver 1800a according to Embodiment 3. FIG.

Since it takes a certain amount of time for the signal to be processed by the receiver 1800a after the signal is transmitted and before the audio or video is played back, it is possible to reproduce the audio or video in anticipation of this processing time. Synchronous playback can be performed.

First, a processing delay time is designated to the receiver 1800a (step S1801). This may be held in the processing program or specified by the user. Correction by the user makes it possible to achieve more accurate synchronization that matches individual receivers. This processing delay time can be changed for each receiver model, depending on the temperature of the receiver and the percentage of CPU usage, so that synchronization can be performed more accurately.

The receiver 1800a determines whether it has received a time packet or whether it has received an associated ID for voice synchronization (step S1802). Here, when the receiver 1800a determines that the image has been received (Y in step S1802), it further determines whether or not there is an image waiting to be processed (step S1804). If it is determined that there is an image waiting to be processed (Y in step S1804), the receiver 1800a discards the image waiting to be processed, or delays the processing of the image waiting to be processed, and starts receiving processing from the latest acquired image. (step S1805). As a result, it is possible to avoid unexpected delays due to the processing waiting amount.

The receiver 1800a measures the position of the visible light signal (specifically, the bright line) in the image (step S1806). In other words, by measuring where the signal appears in the direction perpendicular to the exposure line from the first exposure line on the image sensor, the time difference (in-image delay time) from the time when the image acquisition starts until the time when the signal is received can be calculated. can be calculated.

The receiver 1800a reproduces the audio or video at the time obtained by adding the processing delay time and the intra-image delay time to the recognized synchronization time, thereby performing accurate synchronous reproduction (step S1807).

On the other hand, if the receiver 1800a determines in step S1802 that the time packet or the audio synchronization ID has not been received, it receives a signal from the captured image (step S1803).

FIG. 28 is a diagram showing an example of a user interface of receiver 1800a according to the third embodiment.

The user can adjust the above-described processing delay time by pressing any of buttons Bt1 to Bt4 displayed on the receiver 1800a, as shown in FIG. 28(a). Also, as shown in FIG. 28B, the processing delay time may be set by a swipe operation. As a result, synchronized playback can be performed more accurately based on the user's senses.

Here, earphone limited reproduction will be described below.

29 is a diagram illustrating an example of a processing flow of receiver 1800a according to Embodiment 3. FIG.

The earphone-only reproduction indicated by this processing flow enables audio reproduction without disturbing the surroundings.

The receiver 1800a confirms whether or not the setting is limited to earphones (step S1811). If the setting is limited to earphones, for example, the receiver 1800a is set to be limited to earphones. Alternatively, the received signal (visible light signal) is set to be earphone-only. Alternatively, earpiece only is recorded in the server or receiver 1800a in association with the received signal.

When receiver 1800a confirms that earphones are limited (Y in step S1811), receiver 1800a determines whether earphones are connected to receiver 1800a (step S1813).

When the receiver 1800a confirms that the earphone is not limited (N of step S1811) or determines that the earphone is connected (Y of step S1813), it reproduces the sound (step S1812). When reproducing audio, the receiver 1800a adjusts the volume so that the volume is within the set range. This setting range is set in the same manner as the setting limited to earphones.

When the receiver 1800a determines that the earphone is not connected (N in step S1813), it notifies the user to connect the earphone (step S1814). This notification is made, for example, by screen display, audio output, or vibration.

In addition, if the receiver 1800a is not set to forcibly reproduce audio, the receiver 1800a prepares an interface for forced reproduction and determines whether or not the user has performed an operation for forced reproduction ( step S1815). Here, if it is determined that the forced reproduction operation has been performed (Y in step S1815), the receiver 1800a reproduces the sound even if the earphone is not connected (step S1812).

On the other hand, if it is determined that the forced reproduction operation has not been performed (N in step S1815), the receiver 1800a retains the received audio data and the analyzed synchronization time in advance so that when the earphone is connected, Synchronous playback of audio is performed promptly.

FIG. 30 is a diagram showing another example of the processing flow of receiver 1800a according to the third embodiment.

The receiver 1800a first receives the ID from the transmitter 1800d (step S1821). That is, the receiver 1800a receives a visible light signal that indicates the ID of the transmitter 1800d or the ID of the content being displayed on the transmitter 1800d.

Next, the receiver 1800a downloads the information (content) associated with the received ID from the server (step S1822). Alternatively, receiver 1800a reads the information from a data holding unit inside receiver 1800a. This information is hereinafter referred to as related information.

Next, the receiver 1800a determines whether or not the synchronous reproduction flag included in the related information indicates ON (step S1823). Here, if it is determined that the synchronous reproduction flag does not indicate ON (N of step S1823), the receiver 1800a outputs the content indicated by the related information (step S1824). That is, if the content is an image, the receiver 1800a displays the image, and if the content is audio, the receiver 1800a outputs audio.

On the other hand, when the receiver 1800a determines that the synchronous reproduction flag indicates ON (Y in step S1823), it further determines whether the time adjustment mode included in the related information is set to the transmitter reference mode. , determines whether the absolute time mode is set (step S1825). When determining that the absolute time mode is set, the receiver 1800a determines whether or not the last time adjustment was performed within a certain period of time from the current time (step S1826). The time adjustment at this time is a process of obtaining time information by a predetermined method and using the time information to adjust the time of the clock provided in the receiver 1800a to the absolute time of the reference clock. The predetermined method is, for example, a method using GPS (Global Positioning System) radio waves or NTP (Network Time Protocol) radio waves. Note that the current time described above may be the time when the receiver 1800a, which is a terminal device, receives the visible light signal.

When receiver 1800a determines that the last time adjustment has been performed within a certain period of time (Y in step S1826), receiver 1800a outputs related information based on the time of the clock of receiver 1800a, thereby displaying it on transmitter 1800d. Synchronize the content and the related information (step S1827). If the content indicated by the related information is, for example, a moving image, the receiver 1800a displays the moving image so as to synchronize with the content displayed on the transmitter 1800d. If the content indicated by the related information is audio, for example, the receiver 1800a outputs the audio so as to synchronize with the content displayed on the transmitter 1800d. For example, if the relevant information indicates speech, the relevant information includes each frame that makes up the speech, with those frames time-stamped. The receiver 1800a outputs audio synchronized with the content of the transmitter 1800d by reproducing the frames with the time stamp corresponding to the time of its own clock.

When the receiver 1800a determines that the last time adjustment has not been performed within a predetermined period of time (N in step S1826), it attempts to obtain time information by a predetermined method, and determines whether the time information could be obtained. (step S1828). Here, if it is determined that the time information has been obtained (Y in step S1828), the receiver 1800a uses the time information to update the time of the clock of the receiver 1800a (step S1829). The receiver 1800a then executes the process of step S1827 described above.

Also, when it is determined in step S1825 that the time adjustment mode is the transmitter reference mode, or when it is determined in step S1828 that the time information could not be obtained (N in step S1828), the receiver 1800a , acquires time information from the transmitter 1800d (step S1830). That is, the receiver 1800a acquires time information, which is a synchronization signal, from the transmitter 1800d by visible light communication. For example, the synchronization signals are time packet 1 and time packet 2 shown in FIG. Alternatively, the receiver 1800a acquires the time information from the transmitter 1800d via radio waves such as Bluetooth (registered trademark) or Wi-Fi. Receiver 1800a then executes the processes of steps S1829 and S1827 described above.

In this embodiment, as in steps S1829 and S1830, a process (time adjustment) for synchronizing the clock of the terminal device, which is the receiver 1800a, with the reference clock is performed by GPS radio waves or NTP radio waves. If the received time is earlier than a predetermined time from the time when the terminal device received the visible light signal, the time indicated by the visible light signal transmitted from the transmitter 1800d causes the clock of the terminal device and the clock of the transmitter. synchronize between As a result, the terminal device can reproduce the content (video or audio) at the timing synchronized with the transmitter-side content reproduced by the transmitter 1800d.

31A is a diagram for explaining a specific method of synchronous reproduction according to Embodiment 3. FIG. Synchronous playback methods include methods a to e shown in FIG. 31A.

(Method a)
In method a, transmitter 1800d outputs a visible light signal indicating the content ID and the time during content reproduction by changing the brightness of the display, as in the above embodiments. The content playback time is the playback time of data, which is part of the content, being played back by the transmitter 1800d when the content ID was transmitted from the transmitter 1800d. If the content is a moving image, the data is pictures or sequences that make up the moving image, and if the content is audio, the data is a frame or the like that makes up the audio. The reproduction time indicates, for example, the reproduction time from the beginning of the content as time. If the content is a moving image, the playback time is included in the content as a PTS (Presentation Time Stamp). In other words, the content includes the reproduction time (display time) of each data constituting the content.

Receiver 1800a receives the visible light signal by photographing transmitter 1800d in the same manner as in the above embodiments. The receiver 1800a then transmits a request signal including the content ID indicated by the visible light signal to the server 1800f. Server 1800f receives the request signal and transmits the content associated with the content ID included in the request signal to receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a reproduces the content from the point of time (time during content reproduction+time elapsed since ID reception). The elapsed time since ID reception is the elapsed time since the content ID was received by the receiver 1800a.

(Method b)
In method b, the transmitter 1800d outputs a visible light signal indicating the content ID and the time during content reproduction by changing the brightness of the display in the same manner as in the above embodiments. Receiver 1800a receives the visible light signal by photographing transmitter 1800d in the same manner as in the above embodiments. The receiver 1800a then transmits a request signal including the content ID indicated by the visible light signal and the content reproduction time to the server 1800f. Server 1800f receives the request signal, and transmits to receiver 1800a only a portion of the content associated with the content ID included in the request signal after the content reproduction time.

When the receiver 1800a receives the part of the content, the receiver 1800a reproduces the part of the content from the point of time (elapsed time from ID reception).

(Method c)
In method c, transmitter 1800d outputs a visible light signal indicating the transmitter ID and content playback time by changing the brightness of the display, as in the above embodiments. A transmitter ID is information for identifying a transmitter.

Receiver 1800a receives the visible light signal by photographing transmitter 1800d in the same manner as in the above embodiments. Receiver 1800a then transmits a request signal including the transmitter ID indicated by the visible light signal to server 1800f.

The server 1800f holds, for each transmitter ID, a reproduction schedule, which is a timetable of content to be reproduced by the transmitter having that transmitter ID. In addition, server 1800f is equipped with a clock. When such a server 1800f receives the request signal, the content associated with the transmitter ID included in the request signal and the time (server time) of the clock of the server 1800f is displayed as the content being reproduced. , specified from the playback schedule. Server 1800f then transmits the content to receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a reproduces the content from the point of time (time during content reproduction+time elapsed since ID reception).

(Method d)
In method d, transmitter 1800d outputs a visible light signal indicating the transmitter ID and the transmitter time by changing the brightness of the display as in the above embodiments. The transmitter time is the time indicated by the clock provided in the transmitter 1800d.

Receiver 1800a receives the visible light signal by photographing transmitter 1800d in the same manner as in the above embodiments. The receiver 1800a then transmits a request signal including the transmitter ID and the transmitter time indicated by the visible light signal to the server 1800f.

The server 1800f holds the above-described playback schedule. When such a server 1800f receives the request signal, the content associated with the transmitter ID and the transmitter time included in the request signal is identified from the reproduction schedule as the content being reproduced. Furthermore, the server 1800f identifies the content playback time from the transmitter time. That is, the server 1800f finds the playback start time of the specified content from the playback schedule, and specifies the time between the transmitter time and the playback start time as the content playback time. Then, the server 1800f transmits the content and the content playback time to the receiver 1800a.

When the receiver 1800a receives the content and the time during content reproduction, the receiver 1800a reproduces the content from the point of time (the time during content reproduction+the elapsed time from reception of the ID).

Thus, in this embodiment, a visible light signal indicates the time at which the visible light signal is transmitted from transmitter 1800d. Therefore, the receiver 1800a, which is a terminal device, can receive content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter 1800d. For example, if the transmitter time is 5:43, content played at 5:43 can be received.

Also, in this embodiment, the server 1800f has a plurality of contents each associated with a time. However, there is a case where the content associated with the time indicated by the visible light signal does not exist on the server 1800f. In such a case, the receiver 1800a, which is a terminal device, is associated with the time closest to the time indicated by the visible light signal and later than the time indicated by the visible light signal among the plurality of contents. Content may be received. As a result, even if the server 1800f does not have content associated with the time indicated by the visible light signal, appropriate content can be received from among the plurality of content on the server 1800f.

Further, the reproduction method according to the present embodiment includes a signal receiving step of receiving a visible light signal from a transmitter 1800d that transmits a visible light signal by a change in luminance of a light source with a sensor of a receiver 1800a (terminal device); A transmitting step of transmitting a request signal for requesting content associated with a visible light signal from 1800a to a server 1800f, a content receiving step of receiving content from the server 1800f by the receiver 1800a, and reproducing the content. and a regeneration step. The visible light signal indicates the transmitter ID and transmitter time. The transmitter ID is ID information. The transmitter time is the time indicated by the clock of the transmitter 1800d, and is the time when the visible light signal is transmitted from the transmitter 1800d. Then, in the content receiving step, the receiver 1800a receives the content associated with the transmitter ID and the transmitter time indicated by the visible light signal. This allows the receiver 1800a to reproduce content appropriate for the transmitter ID and transmitter time.

(Method e)
In method e, transmitter 1800d outputs a visible light signal indicating the transmitter ID by changing the brightness of the display, as in the above embodiments.

Receiver 1800a receives the visible light signal by photographing transmitter 1800d in the same manner as in the above embodiments. Receiver 1800a then transmits a request signal including the transmitter ID indicated by the visible light signal to server 1800f.

The server 1800f holds the above-mentioned playback schedule and also has a clock. When such a server 1800f receives the request signal, the content associated with the transmitter ID and the server time included in the request signal is identified from the reproduction schedule as the content being reproduced. The server time is the time indicated by the clock of the server 1800f. Furthermore, the server 1800f also finds out the playback start time of the specified content from the playback schedule. Server 1800f then transmits the content and the content reproduction start time to receiver 1800a.

When the receiver 1800a receives the content and the content reproduction start time, the receiver 1800a reproduces the content from the point of (receiver time - content reproduction start time). Note that the receiver time is the time indicated by the clock provided in the receiver 1800a.

As described above, the reproduction method according to the present embodiment includes a signal receiving step of receiving a visible light signal from a transmitter 1800d that transmits a visible light signal according to a change in luminance of a light source by a sensor of a receiver 1800a (terminal device); A transmission step of transmitting a request signal for requesting content associated with the visible light signal from the receiver 1800a to the server 1800f, and the receiver 1800a includes each time and data reproduced at each time. It includes a content receiving step of receiving content from the server 1800f and a reproducing step of reproducing data corresponding to the time of the clock provided in the receiver 1800a. Therefore, the receiver 1800a can properly reproduce the data in the content at the correct time shown in the content without reproducing the data at the wrong time. Also, if content related to the content (transmitter-side content) is being reproduced in the transmitter 1800d, the receiver 1800a can appropriately synchronize the content with the transmitter-side content and reproduce it. .

It should be noted that, even in methods c to e described above, the server 1800f may transmit only a portion of the content after the content reproduction time to the receiver 1800a, as in the method b.

In the above methods a to e, the receiver 1800a transmits a request signal to the server 1800f and receives necessary data from the server 1800f. You can keep it.

FIG. 31B is a block diagram showing the configuration of a playback device that performs synchronous playback according to method e described above.

The playback device B10 is a receiver 1800a or a terminal device that performs synchronous playback by the method e described above, and includes a sensor B11, a request signal transmission unit B12, a content reception unit B13, a clock B14, and a playback unit B15. I have.

The sensor B11 is an image sensor, for example, and receives the visible light signal from the transmitter 1800d that transmits the visible light signal according to changes in luminance of the light source. The request signal transmission unit B12 transmits a request signal for requesting content associated with the visible light signal to the server 1800f. The content receiving unit B13 receives content including each time and data to be reproduced at each time from the server 1800f. The reproducing unit B15 reproduces the data corresponding to the time of the clock B14 among the contents.

FIG. 31C is a flow chart showing the processing operation of the terminal device that performs synchronous playback according to method e described above.

The playback device B10 is a receiver 1800a or a terminal device that performs synchronous playback according to the above-described method e, and executes steps SB11 to SB15.

At step SB11, the visible light signal is received from the transmitter 1800d that transmits the visible light signal according to the luminance change of the light source. At step SB12, a request signal for requesting the content associated with the visible light signal is transmitted to the server 1800f. At step SB13, contents including each time and data to be reproduced at each time are received from the server 1800f. At step SB15, the data corresponding to the time of the clock B14 is reproduced among the contents.

As described above, the reproducing apparatus B10 and the reproducing method according to the present embodiment can properly reproduce the data in the content at the correct time shown in the content without reproducing the data at the wrong time.

In addition, in the present embodiment, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. Here, the software that implements the playback device B10 and the like of the present embodiment is a program that causes a computer to execute each step included in the flowchart shown in FIG. 31C.

FIG. 32 is a diagram for explaining advance preparation for synchronous playback according to the third embodiment.

In order to perform synchronous reproduction, the receiver 1800a adjusts the time of the clock provided in the receiver 1800a with the time of the reference clock. For this time adjustment, the receiver 1800a performs the following processes (1) to (5).

(1) Receiver 1800a receives a signal. This signal may be a visible light signal transmitted by changing the brightness of the display of the transmitter 1800d, or a radio wave signal based on Wi-Fi or Bluetooth (registered trademark) from a wireless device. Alternatively, the receiver 1800a acquires position information indicating the position of the receiver 1800a by, for example, GPS, instead of receiving such a signal. Then, receiver 1800a recognizes that receiver 1800a has entered a predetermined location or building based on the position information.

(2) When receiving the signal or recognizing that the receiver 1800a has entered a predetermined location, the receiver 1800a transmits a request signal requesting data (related information) associated with the signal or location. It is transmitted to the server (visible light ID resolution server) 1800f.

(3) The server 1800f transmits to the receiver 1800a the above data and a time adjustment request for causing the receiver 1800a to adjust the time.

(4) Upon receiving the data and the time adjustment request, the receiver 1800a transmits the time adjustment request to the GPS time server, NTP server, or base station of the telecommunications carrier (carrier).

(5) Upon receiving the time adjustment request, the server or base station transmits time data (time information) indicating the current time (the time of the reference clock or the absolute time) to the receiver 1800a. Receiver 1800a adjusts the time by synchronizing the time on its own clock with the current time indicated by the time data.

Thus, in this embodiment, GPS (Global Positioning System) radio waves or NTP (Network Time Protocol) radio waves are used between the clock provided in the receiver 1800a (terminal device) and the reference clock. Synchronized. Therefore, the receiver 1800a can reproduce data corresponding to that time at an appropriate time according to the reference clock.

FIG. 33 is a diagram showing an application example of receiver 1800a according to the third embodiment.

The receiver 1800a is configured as a smart phone as described above, and is used by being held by a holder 1810 made of a member such as translucent resin or glass. This holder 1810 has a back plate portion 1810a and a locking portion 1810b erected on the back plate portion 1810a. The receiver 1800a is inserted along the back plate portion 1810a between the back plate portion 1810a and the locking portion 1810b.

34A is a front view of receiver 1800a held by holder 1810 according to the third embodiment. FIG.

Receiver 1800a is held in holder 1810 while inserted as described above. At this time, the engaging portion 1810b engages with the lower portion of the receiver 1800a and sandwiches the lower portion with the back plate portion 1810a. In addition, the back surface of the receiver 1800a faces the back plate portion 1810a, and the display 1801 of the receiver 1800a is exposed.

FIG. 34B is a rear view of receiver 1800a held by holder 1810 according to the third embodiment.

A through hole 1811 is formed in the back plate portion 1810a, and a variable filter 1812 is attached near the through hole 1811. As shown in FIG. When the receiver 1800a is held by the holder 1810, the camera 1802 of the receiver 1800a is exposed through the through hole 1811 from the back plate portion 1810a. Also, the flashlight 1803 of the receiver 1800 a faces the variable filter 1812 .

The variable filter 1812 is formed, for example, in a disc shape, and has three color filters (red filter, yellow filter, and green filter) each having a fan shape and the same size. Also, the variable filter 1812 is attached to the back plate portion 1810a so as to be rotatable around the center of the variable filter 1812. As shown in FIG. The red filter is a filter that transmits red light, the yellow filter is a filter that transmits yellow light, and the green filter is a filter that transmits green light.

Accordingly, the variable filter 1812 is rotated so that, for example, the red filter is positioned opposite the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as red light by passing through the red filter. As a result, substantially the entire holder 1810 emits red light.

Similarly, variable filter 1812 is rotated to position, for example, the yellow filter opposite flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as yellow light by passing through the yellow filter. As a result, substantially the entire holder 1810 emits yellow light.

Similarly, variable filter 1812 is rotated so that, for example, the green filter is positioned opposite flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as green light by passing through the green filter. As a result, substantially the entire holder 1810 emits green light.

That is, holder 1810 lights red, yellow, or green like a penlight.

FIG. 35 is a diagram for explaining a use case of receiver 1800a held by holder 1810 in the third embodiment.

For example, a holder-equipped receiver, which is receiver 1800a held by holder 1810, is used in amusement parks and the like. In other words, a plurality of holder-mounted receivers facing a moving float in an amusement park blink in sync with the music playing from the float. That is, the float is configured as the transmitter in each of the above embodiments, and transmits a visible light signal according to changes in brightness of the light source attached to the float. For example, a float transmits a visible light signal that indicates the identity of the float. Then, the holder-equipped receiver receives the visible light signal, that is, the ID by photographing with the camera 1802 of the receiver 1800a, as in each of the above embodiments. The receiver 1800a that has received the ID acquires the program associated with the ID from, for example, a server. This program consists of commands to turn on the flashlight 1803 of the receiver 1800a at each predetermined time. Each predetermined time is set to match (synchronize with) the music that is played from the float. Then, receiver 1800a blinks flashlight 1803a according to the program.

As a result, the holder 1810 of each receiver 1800a that has received that ID repeats lighting at the same timing in time with the music that flows from the float of that ID.

Here, each receiver 1800a blinks a flashlight 1803 according to a set color filter (hereinafter referred to as a set filter). A setting filter is a color filter facing the flashlight 1803 of the receiver 1800a. Also, each receiver 1800a recognizes the current set filter based on the user's operation. Alternatively, each receiver 1800a recognizes the currently set filter based on the color of the image captured by the camera 1802, or the like.

That is, among the plurality of receivers 1800a that have received the ID, only the holders 1810 of the plurality of receivers 1800a that recognize that the set filter is the red filter are lit at a predetermined time. At the next time, only the holders 1810 of multiple receivers 1800a that recognize that the set filter is a green filter are illuminated at the same time. Furthermore, at the next time, only the holders 1810 of the plurality of receivers 1800a recognizing that the set filter is the yellow filter light up at the same time.

In this way, the receiver 1800a held by the holder 1810 synchronizes with the music of the float and the receiver 1800a held by the other holder 1810, similar to the synchronous playback shown in FIGS. to blink the flash light 1803, ie, the holder 1810.

FIG. 36 is a flow chart showing processing operations of receiver 1800a held by holder 1810 according to the third embodiment.

Receiver 1800a receives the ID of the float indicated by the visible light signal from the float (step S1831). Next, the receiver 1800a acquires the program associated with that ID from the server (step S1832). Next, the receiver 1800a executes the program to turn on the flashlight 1803 at each predetermined time corresponding to the setting filter (step S1833).

Here, the receiver 1800a may cause the display 1801 to display an image according to the received ID or the acquired program.

FIG. 37 is a diagram showing an example of an image displayed by receiver 1800a according to the third embodiment.

For example, when the receiver 1800a receives the ID from the Santa Claus float, the receiver 1800a displays an image of Santa Claus as shown in FIG. 37(a). Furthermore, receiver 1800a may change the background color of the image of Santa Claus to the color of the setting filter at the same time that flashlight 1803 is turned on, as shown in FIG. 37(b). For example, when the color of the setting filter is red, lighting of the flashlight 1803 causes the holder 1810 to light up in red, and at the same time an image of Santa Claus with a red background color is displayed on the display 1801 . That is, the flickering of the holder 1810 and the display of the display 1801 are synchronized.

38 is a diagram showing another example of the holder according to Embodiment 3. FIG.

Holder 1820 is configured similarly to holder 1810 described above, but lacks through hole 1811 and variable filter 1812 . Such a holder 1820 holds the receiver 1800a with the display 1801 of the receiver 1800a facing the back plate portion 1820a. In this case, receiver 1800 a causes display 1801 to emit light instead of flashlight 1803 . This diffuses the light from the display 1801 over substantially the entire holder 1820 . Therefore, when receiver 1800a causes display 1801 to emit red light in accordance with the above program, holder 1820 lights red. Similarly, when receiver 1800a illuminates display 1801 with yellow light in accordance with the above program, holder 1820 lights yellow. When receiver 1800a illuminates display 1801 with green light in accordance with the above program, holder 1820 lights green. By using such a holder 1820, setting of the variable filter 1812 can be omitted.

(visible light signal)
39A to 39D are diagrams showing examples of visible light signals according to Embodiment 3. FIG.

The transmitter generates a visible light signal of 4PPM, for example, as shown in FIG. 39A, in the same manner as described above, and changes the luminance according to this visible light signal. Specifically, the transmitter allocates four slots to one signal unit and generates a visible light signal made up of a plurality of signal units. A signal unit indicates High (H) or Low (L) for each slot. The transmitter then lights brightly in the H slots and dimly lights or extinguishes in the L slots. For example, one slot is a period corresponding to 1/9600 seconds.

Also, the transmitter may generate a visible light signal in which the number of slots assigned to one signal unit is variable, as shown in FIG. 39B, for example. In this case, the signal unit consists of a signal indicating H in one or more consecutive slots and a signal indicating L in one slot following the H signal. Since the number of slots in H is variable, the total number of slots in signal units is variable. For example, as shown in FIG. 39B, the transmitter generates a visible light signal including signal units of 3 slots, 4 slots, and 6 slots in this order. The transmitter then again lights up brightly in the H slot and dimly lit or off in the L slot.

Also, the transmitter may allocate an arbitrary period (signal unit period) to one signal unit without allocating a plurality of slots to one signal unit, as shown in FIG. 39C, for example. This signal unit period consists of an H period and an L period following the H period. The period of H is adjusted according to the signal before modulation. The period of L may be fixed and may be a period corresponding to the slot. Also, the H period and the L period are each, for example, a period of 100 μs or longer. For example, as shown in FIG. 39C, the transmitter generates a visible light signal including those signal units in the order of a signal unit having a signal unit period of 210 μs, a signal unit having a signal unit period of 220 μs, and a signal unit having a signal unit period of 230 μs. to send. Then, the transmitter again emits bright light during the H period and dimly emits or extinguishes during the L period.

Also, the transmitter may generate a signal alternately showing L and H as a visible light signal, as shown in FIG. 39D, for example. In this case, the L period and the H period of the visible light signal are adjusted according to the signal before modulation. For example, as shown in FIG. 39D, the transmitter indicates H for a period of 100 μs, then L for a period of 120 μs, then H for a period of 110 μs, and then L for a period of 200 μs. transmit a visible light signal indicating Then, the transmitter again emits bright light during the H period and dimly emits or extinguishes during the L period.

40 is a diagram showing a structure of a visible light signal according to Embodiment 3. FIG.

Visible light signals include, for example, signal 1, a brightness adjustment signal corresponding to signal 1, a signal 2, and a brightness adjustment signal corresponding to signal 2. The transmitter generates signal 1 and signal 2 by modulating the pre-modulation signal, generates a brightness adjustment signal for these signals, and generates the visible light signal described above.

A brightness adjustment signal corresponding to the signal 1 is a signal that compensates for an increase or decrease in brightness due to a change in brightness according to the signal 1 . A brightness adjustment signal corresponding to the signal 2 is a signal that compensates for increase or decrease in brightness due to a change in brightness according to the signal 2 . Here, the brightness B1 is expressed by the signal 1 and the brightness change according to the brightness adjustment signal of the signal 1, and the brightness B1 is expressed by the signal 2 and the brightness adjustment signal of the signal 2. Brightness B2 is represented. The transmitter in this embodiment generates brightness adjustment signals for each of signal 1 and signal 2 as part of the visible light signal so that their brightness B1 and brightness B2 are equal. This keeps the brightness constant and suppresses flickering.

Also, when the transmitter generates signal 1 described above, the transmitter generates signal 1 including data 1, a preamble (header) following the data 1, and data 1 following the preamble. Here, the preamble is a signal corresponding to data 1 arranged before and after it. For example, this preamble is a signal that serves as an identifier for reading data 1 . In this way, since the signal 1 is composed of two data 1 and the preamble arranged between them, even if the visible light signal is read from the middle of the preceding data 1, the receiver can Data 1 (ie signal 1) can be demodulated correctly.

A reproducing method according to an aspect of the present invention includes a signal receiving step of receiving the visible light signal from a transmitter that transmits the visible light signal by a change in luminance of a light source with a sensor of a terminal device; a transmitting step of transmitting a request signal for requesting content associated with an optical signal to a server; and a reproducing step of reproducing the data corresponding to the time of the clock provided in the terminal device among the contents.

As a result, as shown in FIG. 31C, the content including each time and the data to be reproduced at each time is received by the terminal device, and the data corresponding to the clock time of the terminal device is reproduced. Therefore, the terminal device can appropriately reproduce the data in the content at the correct time shown in the content without reproducing the data at the wrong time. Specifically, as in method e in FIG. 31A, the receiver, which is a terminal device, reproduces the content from the time point of (receiver time−content reproduction start time). The data corresponding to the clock time of the terminal device described above is the data at the point of time (receiver time - content reproduction start time) in the content. Also, if content (transmitter-side content) related to the content (transmitter-side content) is reproduced in the transmitter, the terminal device can appropriately synchronize the content with the transmitter-side content and reproduce it. Note that the content is audio or image.

Further, the clock provided in the terminal device and the reference clock may be synchronized by GPS (Global Positioning System) radio waves or NTP (Network Time Protocol) radio waves.

As a result, as shown in FIGS. 30 and 32, the clock of the terminal device (receiver) and the reference clock are synchronized. data can be played back.

Also, the visible light signal may indicate the time at which the visible light signal is transmitted from the transmitter.

Thereby, as shown in method d of FIG. 31A, the terminal device (receiver) can receive the content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter. For example, if the transmitter time is 5:43, content played at 5:43 can be received.

Further, in the reproducing method, the time at which processing for synchronizing the clock of the terminal device and the reference clock is performed by the GPS radio wave or the NTP radio wave is displayed on the terminal device. If the time is earlier than a predetermined time from the time when the optical signal is received, the clock of the terminal device and the clock of the transmitter are synchronized according to the time indicated by the visible light signal transmitted from the transmitter. very good

For example, if a predetermined amount of time has elapsed since the process for synchronizing the clock of the terminal device and the reference clock was performed, the synchronization may not be properly maintained. In such a case, the terminal device may not be able to reproduce the content at the time synchronized with the transmitter-side content reproduced by the transmitter. Therefore, in the reproduction method according to one aspect of the present invention, as in steps S1829 and S1830 in FIG. Synchronized. Therefore, the terminal device can reproduce the content at the time synchronized with the transmitter-side content reproduced by the transmitter.

Further, the server has a plurality of contents each associated with a time, and in the content receiving step, if the content associated with the time indicated by the visible light signal does not exist in the server, Among the plurality of contents, the content closest to the time indicated by the visible light signal and associated with the time after the time indicated by the visible light signal may be received.

As a result, as shown in method d of FIG. 31A, even if the server does not have content associated with the time indicated by the visible light signal, appropriate content is received from among the plurality of content on the server. be able to.

Further, a signal receiving step of receiving the visible light signal from a transmitter that transmits the visible light signal according to a change in brightness of a light source by a sensor of the terminal device, and receiving content associated with the visible light signal from the terminal device. a transmitting step of transmitting a request signal for making a request to a server; a content receiving step of receiving content from the server by the terminal device; and a reproducing step of reproducing the content, wherein the visible light signal is ID information and a time at which the visible light signal is transmitted from the transmitter, and in the content receiving step, the content associated with the ID information and the time indicated by the visible light signal is received. good.

As a result, as in method d in FIG. 31A , the visible light signal is associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter from among the plurality of contents associated with the ID information (transmitter ID). received and played back. Therefore, appropriate content can be reproduced for that transmitter ID and transmitter time.

Further, the visible light signal includes second information indicating the hour and minute of the time and first information indicating the second of the time, so that the visible light signal is transmitted from the transmitter. In the signal receiving step, the second information may be received and the first information may be received a number of times greater than the number of times the second information is received.

As a result, for example, when notifying the terminal device of the time at which each packet included in the visible light signal is transmitted in units of seconds, the packet indicating the current time expressed using all of the hour, minute, and second to the terminal device each time one second elapses can be reduced. That is, as shown in FIG. 26, a packet indicating only seconds (time packet 1 ) need only be transmitted. Thus, by sending less second information, packets indicating hours and minutes (time packet 2), than first information, packets indicating seconds (time packet 1), transmitted by the transmitter, Transmission of packets containing redundant content can be suppressed.

(Embodiment 4)
In this embodiment, a display method and the like for realizing AR (Augmented Reality) using a light ID will be described.

FIG. 41 is a diagram showing an example in which the receiver according to this embodiment displays an AR image.

Receiver 200 in the present embodiment is a receiver including an image sensor and display 201 in any one of Embodiments 1 to 3, and is configured as a smart phone, for example. Such a receiver 200 obtains the captured display image Pa, which is the normal captured image, and the decoding image, which is the visible light communication image or the bright line image, by imaging the subject with the image sensor.

Specifically, the image sensor of the receiver 200 captures an image of the transmitter 100 configured as a station sign. Transmitter 100 is a transmitter according to any one of Embodiments 1 to 3 above, and includes one or more light emitting elements (eg, LEDs). The transmitter 100 changes its luminance by blinking one or more light emitting elements, and transmits a light ID (light identification information) according to the luminance change. This light ID is the visible light signal described above.

The receiver 200 captures an image of the transmitter 100 with a normal exposure time to acquire a captured display image Pa in which the transmitter 100 is displayed, and also captures the transmitter 100 with a communication exposure time shorter than the normal exposure time. to obtain a decoding image. The normal exposure time is the exposure time in the above-described normal shooting mode, and the communication exposure time is the exposure time in the above-described visible light communication mode.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P1 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. For example, the receiver 200 recognizes an area where the station name sign, which is the transmitter 100, is displayed as the target area. Then, the receiver 200 superimposes the AR image P1 on the target area, and displays the captured display image Pa on which the AR image P1 is superimposed on the display 201 . For example, if the station name board, which is the transmitter 100, describes "Kyoto Station" in Japanese as the station name, the receiver 200 displays the AR image P1 in which the station name is described in English, that is, "Kyoto Station." Acquire the AR image P1. In this case, since the AR image P1 is superimposed on the target area of the captured display image Pa, the captured display image Pa can be displayed as if a station name sign with the station name written in English actually exists. As a result, even if a user who can understand English cannot read Japanese, he/she can easily understand the station name written on the station name sign which is the transmitter 100 by looking at the captured display image Pa.

For example, the recognition information may be an image to be recognized (for example, the image of the above-mentioned station name sign), or may be feature points and feature amounts of the image. Feature points and feature quantities are obtained by image processing such as SIFT (Scale-invariant feature transform), SURF (Speed-Upped Robust Feature), ORB (Oriented-BRIEF), and AKAZE (Accelerated KAZE). Alternatively, the recognition information may be an image of a white rectangle similar to the image to be recognized, and may indicate the aspect ratio of the rectangle. Alternatively, the identification information may be random dots appearing in the image to be recognized. Additionally, the recognition information may indicate an orientation relative to a predetermined direction, such as the white squares or random dots mentioned above. The predetermined direction is, for example, the direction of gravity.

The receiver 200 recognizes an area corresponding to such recognition information from the captured display image Pa as a target area. Specifically, if the recognition information is an image, the receiver 200 recognizes an area similar to the image, which is the recognition information, as the target area. Also, if the recognition information is a feature point and a feature amount obtained by image processing, the receiver 200 detects the feature point and extracts the feature amount by performing the image processing on the captured display image Pa. . Then, receiver 200 recognizes an area having feature points and feature amounts similar to the feature points and feature amounts as the recognition information in captured display image Pa as a target area. Also, if the recognition information indicates a white square and its orientation, the receiver 200 first detects the direction of gravity using its own acceleration sensor. Then, the receiver 200 recognizes an area similar to a white square oriented in the direction indicated by the recognition information from the captured display image Pa arranged with reference to the direction of gravity as a target area.

Here, the recognition information may include reference information for specifying a reference area in the captured display image Pa, and target information indicating the relative position of the target area with respect to the reference area. The reference information is an image to be recognized, feature points and feature amounts, a white square image, random dots, or the like, as described above. In this case, when recognizing the target area, the receiver 200 first identifies the reference area from the captured display image Pa based on the reference information. Then, the receiver 200 recognizes an area in the captured display image Pa at a relative position indicated by the target information with respect to the position of the reference area as a target area. Note that the target information may indicate that the target area is located at the same position as the reference area. In this way, by including the reference information and the target information in the recognition information, it is possible to recognize the target area in a wide range. Also, the server can freely set the place where the AR image is superimposed and inform the receiver 200 of it.

Further, the reference information may indicate that the reference area in the captured display image Pa is the area where the display is shown in the captured display image. In this case, if the transmitter 100 is configured as a display such as a television, the target area can be recognized based on the area where the display is projected.

In other words, receiver 200 in the present embodiment identifies a reference image and an image recognition method based on the light ID. The image recognition method is a method of recognizing the captured display image Pa, and includes, for example, geometric feature amount extraction, spectral feature amount extraction, or texture feature amount extraction. A reference image is data indicating a reference feature amount. The feature quantity is, for example, the feature quantity of the white outer frame of the image, and more specifically, may be data representing the feature of the image by a vector. The receiver 200 extracts a feature amount from the captured display image Pa according to an image recognition method, and compares the feature amount with the feature amount of the reference image to extract the reference region or the target region from the captured display image Pa. find out

Image recognition methods may also include, for example, location-based methods, marker-based methods, and markerless methods. The location utilization method is a method utilizing GPS position information (that is, the position of the receiver 200), and the target area is recognized from the captured display image Pa based on the position information. The marker utilization method is a method of using a marker composed of black and white figures such as a two-dimensional bar code as a mark for target identification. That is, in this marker utilization method, the target area is recognized based on the marker displayed in the captured display image Pa. In the markerless method, a feature point or feature amount is extracted from the captured display image Pa by image analysis of the captured display image Pa, and the position and area of the target are specified based on the extracted feature point or feature amount. The method. That is, when the image recognition method is the markerless method, the image recognition method is the above-mentioned geometric feature amount extraction, spectral feature amount extraction, texture feature amount extraction, or the like.

Such a receiver 200 receives a light ID from the transmitter 100 and acquires a reference image and an image recognition method associated with the light ID (hereinafter referred to as a received light ID) from a server, thereby performing the reference image. Images and image recognition methods may be specified. That is, the server stores a plurality of sets including reference images and image recognition methods, and each of the plurality of sets is associated with a different light ID. As a result, one set associated with the received light ID can be identified from among the multiple sets stored in the server. Therefore, it is possible to improve the speed of image processing for superimposing an AR image. Further, the receiver 200 may acquire a reference image or the like associated with the received light ID by inquiring the server, and selects the received light ID from among a plurality of reference images held in advance by itself. A reference image associated with may be obtained.

Also, the server may hold, for each light ID, relative position information associated with the light ID together with the reference image, the image recognition method, and the AR image. The relative position information is, for example, information indicating the relative positional relationship between the reference area and the target area. Accordingly, when the receiver 200 transmits the received light ID to the server and inquires about it, the receiver 200 acquires the reference image, the image recognition method, the AR image, and the relative position information associated with the received light ID. In this case, the receiver 200 identifies the reference area from the captured display image Pa based on the reference image and the image recognition method. Then, receiver 200 recognizes an area in the direction and distance indicated by the relative position information from the position of the reference area as the target area, and superimposes the AR image on the target area. Also, if there is no relative position information, the receiver 200 may recognize the above reference area as the target area and superimpose the AR image on the reference area. That is, instead of obtaining relative position information, the receiver 200 may store in advance a program for displaying an AR image based on a reference image, and may display the AR image within a white frame that is the reference area, for example. . In this case, no relative position information is required.

There are four variations (1) to (4) below for retaining or acquiring reference images, relative position information, AR images, and image recognition methods.

(1) The server holds multiple sets of reference images, relative position information, AR images, and image recognition methods. Receiver 200 acquires one set associated with the received light ID from among these sets.

(2) The server holds multiple sets of reference images and AR images. The receiver 200 uses predetermined relative position information and image recognition method, and acquires one set associated with the received light ID from those sets. Alternatively, the receiver 200 may hold multiple sets of relative position information and image recognition methods in advance, and select one set associated with the received light ID from among the multiple sets. In this case, the receiver 200 may transmit the received light ID to the server to inquire, and acquire from the server the relative position information corresponding to the received light ID and the information for specifying the image recognition method. Then, receiver 200 selects one set based on the information obtained from the server from among a plurality of prestored sets of relative position information and image recognition methods. Alternatively, the receiver 200 selects one set associated with the received light ID from among a plurality of prestored sets of relative position information and image recognition methods, without inquiring of the server. may

(3) The receiver 200 holds multiple sets of reference images, relative position information, AR images, and image recognition methods, and selects one of these sets. The receiver 200 may select one set by querying the server as in (2) above, or may select one set associated with the receiver light ID.

(4) The receiver 200 holds multiple sets of reference images and AR images, and selects one set associated with the received light ID. Receiver 200 uses a predetermined image recognition method and relative position information.

FIG. 42 is a diagram showing an example of a display system according to this embodiment.

The display system according to the present embodiment includes, for example, transmitter 100, which is the above-described station name sign, receiver 200, and server 300. FIG.

The receiver 200 first receives the light ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Receiver 200 then transmits the light ID to server 300 .

Server 300 holds, for each light ID, an AR image and recognition information associated with that light ID. Therefore, when server 300 receives a light ID from receiver 200 , server 300 selects an AR image and recognition information associated with the received light ID, and sends the selected AR image and recognition information to receiver 200 . Send. Accordingly, the receiver 200 receives the AR image and the recognition information transmitted from the server 300, and displays the captured display image on which the AR image is superimposed.

FIG. 43 is a diagram showing another example of the display system according to this embodiment.

The display system according to the present embodiment includes, for example, transmitter 100, which is the above-described station name sign, receiver 200, first server 301, and second server 302. FIG.

The receiver 200 first receives the light ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Receiver 200 then transmits the light ID to first server 301 .

Upon receiving the light ID from the receiver 200, the first server 301 notifies the receiver 200 of the URL (Uniform Resource Locator) and the key associated with the received light ID. The receiver 200 that has received such notification accesses the second server 302 based on the URL and transfers the Key to the second server 302 .

The second server 302 holds an AR image and recognition information associated with each key. Therefore, when the second server 302 receives the Key from the receiver 200 , it selects the AR image and recognition information associated with that Key, and transmits the selected AR image and recognition information to the receiver 200 . . Accordingly, the receiver 200 receives the AR image and the recognition information transmitted from the second server 302, and displays the captured display image on which the AR image is superimposed.

FIG. 44 is a diagram showing another example of the display system according to this embodiment.

The display system according to the present embodiment includes, for example, transmitter 100, which is the above-described station name sign, receiver 200, first server 301, and second server 302. FIG.

The receiver 200 first receives the light ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Receiver 200 then transmits the light ID to first server 301 .

Upon receiving the light ID from the receiver 200, the first server 301 notifies the second server 302 of the Key associated with the received light ID.

The second server 302 holds an AR image and recognition information associated with each key. Therefore, when the second server 302 receives the Key from the first server 301, it selects the AR image and recognition information associated with the Key, and sends the selected AR image and recognition information to the first server. Send to server 301 . The first server 301 , upon receiving the AR image and recognition information from the second server 302 , transmits the AR image and recognition information to the receiver 200 . Accordingly, the receiver 200 receives the AR image and the recognition information transmitted from the first server 301, and displays the captured display image on which the AR image is superimposed.

Although the second server 302 transmits the AR image and the recognition information to the first server 301 in the above example, they may be transmitted to the receiver 200 without transmitting to the first server 301 . .

FIG. 45 is a flowchart showing an example of the processing operation of receiver 200 in this embodiment.

First, the receiver 200 starts imaging with the above-described normal exposure time and communication exposure time (step S101). Then, the receiver 200 acquires the optical ID by decoding the decoding image obtained by imaging during the communication exposure time (step S102). The receiver 200 then transmits the light ID to the server (step S103).

The receiver 200 acquires the AR image and recognition information corresponding to the transmitted light ID from the server (step S104). Next, the receiver 200 recognizes, as a target area, an area corresponding to the recognition information in the imaged display image obtained by imaging during the normal exposure time (step S105). Then, the receiver 200 superimposes the AR image on the target area, and displays the captured display image on which the AR image is superimposed (step S106).

Next, the receiver 200 determines whether or not to end the imaging and display of the captured display image (step S107). If the receiver 200 determines that the process should not end (N in step S107), it further determines whether the acceleration of the receiver 200 is equal to or greater than the threshold (step S108). This acceleration is measured by an acceleration sensor provided in receiver 200 . When the receiver 200 determines that the acceleration is less than the threshold (N in step S108), it executes the process from step S105. As a result, even if the captured display image displayed on the display 201 of the receiver 200 is shifted, the AR image can follow the target area of the captured display image. Further, when the receiver 200 determines that the acceleration is equal to or greater than the threshold (Y in step S108), it executes the processing from step S102. As a result, when the transmitter 100 is no longer displayed in the captured display image, it is possible to prevent an area in which a different subject from the transmitter 100 is displayed being erroneously recognized as the target area.

As described above, in the present embodiment, since the AR image is superimposed on the captured display image, it is possible to display an image useful to the user. Furthermore, the AR image can be superimposed on an appropriate target area while suppressing the processing load.

That is, in general augmented reality (that is, AR), by comparing a large number of pre-stored recognition target images with a captured display image, any recognition target image is included in the captured display image. It is determined whether or not Then, if it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, positioning of the AR images is performed with reference to the recognition target image. As described above, in general augmented reality, since a huge number of recognition target images and captured display images are compared, it is necessary to detect the position of the recognition target image in the captured display image for alignment. There is a problem that the amount of calculation is large and the processing load is high.

However, in the display method according to the present embodiment, the light ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the light ID transmitted from the transmitter, which is the subject, is received. Furthermore, an AR image and recognition information corresponding to this light ID are obtained from the server. Therefore, the server does not need to compare a huge number of images to be recognized and captured display images, and can select an AR image associated with the light ID in advance and transmit it to the display device. As a result, the amount of calculation can be reduced and the processing load can be greatly reduced. Furthermore, the display processing of the AR image can be speeded up.

Further, in the present embodiment, recognition information corresponding to this light ID is acquired from the server. The recognition information is information for recognizing the target area, which is the area where the AR image is superimposed in the captured display image. This recognition information may be, for example, information indicating that a white rectangle is the target area. In this case, the target area can be easily recognized, and the processing load can be further reduced. That is, the processing load can be further reduced according to the content of the recognition information. In addition, since the server can arbitrarily set the content of the recognition information in accordance with the light ID, it is possible to maintain an appropriate balance between the processing load and the recognition accuracy.

In the present embodiment, receiver 200 acquires an AR image and recognition information corresponding to the light ID from the server after receiver 200 transmits the light ID to the server. At least one of may be acquired in advance. That is, the receiver 200 collectively acquires from the server and stores a plurality of AR images and a plurality of pieces of recognition information corresponding to a plurality of light IDs that may be received. After that, when receiving the light ID, the receiver 200 selects the AR image and the recognition information corresponding to the light ID from the plurality of AR images and the plurality of recognition information stored in itself. This makes it possible to further speed up the display processing of the AR image.

FIG. 46 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

The transmitter 100 is configured as an illumination device, for example, as shown in FIG. 46, and transmits the light ID by changing the luminance while illuminating the information board 101 of the facility. Since the guide plate 101 is illuminated by the light from the transmitter 100, the luminance changes similarly to the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Pb and the decoding image in the same manner as described above by capturing the image of the guide board 101 illuminated by the transmitter 100 . The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the light ID from the information board 101 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P2 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pb as a target area. For example, the receiver 200 recognizes the area where the frame 102 is projected on the guide plate 101 as the target area. This frame 102 is a frame for indicating the waiting time of the facility. The receiver 200 then superimposes the AR image P2 on the target area, and displays the captured display image Pb on which the AR image P2 is superimposed on the display 201 . For example, the AR image P2 is an image containing the character string "30 minutes". In this case, the AR image P2 is superimposed on the target area of the captured display image Pb. Pb can be displayed. As a result, the user of the receiver 200 can be notified of the waiting time in a simple and easy-to-understand manner without providing the information board 101 with a special display device.

FIG. 47 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

Transmitter 100 consists of two illumination devices, for example, as shown in FIG. The transmitter 100 transmits the light ID by changing the brightness while illuminating the information board 104 of the facility. Since the guide plate 104 is illuminated by the light from the transmitter 100, the luminance changes similarly to the transmitter 100, and transmits the light ID. Also, the information board 104 indicates the names of a plurality of facilities such as "ABC Land" and "Adventure Land".

The receiver 200 acquires the captured display image Pc and the decoding image in the same manner as described above by capturing the image of the guide board 104 illuminated by the transmitter 100 . The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the light ID from the information board 104 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P3 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pc as a target area. For example, the receiver 200 recognizes the area where the guide plate 104 is projected as the target area. The receiver 200 then superimposes the AR image P3 on the target area, and displays the captured display image Pc on which the AR image P3 is superimposed on the display 201 . For example, the AR image P3 is an image showing the names of multiple facilities. In the AR image P3, the longer the waiting time of a facility is, the smaller the name of the facility is displayed. Conversely, the shorter the waiting time of the facility is, the larger the name of the facility is displayed. In this case, since the AR image P3 is superimposed on the target area of the imaged display image Pc, the receiver 200 displays the guide board 104 having the size corresponding to the waiting time and having the name of each facility listed there as if it actually exists. , the captured display image Pc can be displayed. As a result, the user of the receiver 200 can be notified of the waiting time of each facility simply and in an easy-to-understand manner without providing the information board 104 with a special display device.

FIG. 48 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

Transmitter 100 consists of two illumination devices, for example, as shown in FIG. The transmitter 100 transmits the light ID by changing the brightness while illuminating the castle wall 105 . Since the castle wall 105 is illuminated by the light from the transmitter 100, it changes in luminance similarly to the transmitter 100 and transmits the light ID. Also, on the castle wall 105, for example, a small mark in the shape of a character's face is engraved as a hidden character 106. - 特許庁

The receiver 200 captures the image of the castle wall 105 illuminated by the transmitter 100 to acquire the captured display image Pd and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the castle wall 105 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P4 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pd as a target area. For example, the receiver 200 recognizes an area of the castle wall 105 in which a range including the hidden character 106 is projected as the target area. Then, the receiver 200 superimposes the AR image P4 on the target area, and displays the captured display image Pd on which the AR image P4 is superimposed on the display 201 . For example, the AR image P4 is an image of a character's face. This AR image P4 is an image sufficiently larger than the hidden character 106 displayed in the captured display image Pd. In this case, since the AR image P4 is superimposed on the target area of the captured display image Pd, the receiver 200 captures the castle wall 105 in which a large mark representing the character's face is engraved as if it actually exists. A display image Pd can be displayed. Thereby, the user of the receiver 200 can be notified of the position of the hidden character 106 in an easy-to-understand manner.

FIG. 49 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

Transmitter 100 consists of two illumination devices, for example, as shown in FIG. The transmitter 100 transmits the light ID by changing the luminance while illuminating the information board 107 of the facility. Since the guide plate 107 is illuminated by the light from the transmitter 100, it changes in luminance similarly to the transmitter 100 and transmits the light ID. Further, infrared shielding paint 108 is applied to a plurality of corners of guide plate 107 .

The receiver 200 captures the image of the guide board 107 illuminated by the transmitter 100 to acquire the captured display image Pe and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the light ID from the guide plate 107 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P5 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pe as a target area. For example, the receiver 200 recognizes the area where the guide plate 107 is projected as the target area.

Specifically, the recognition information indicates that the target area is a rectangle circumscribing the infrared shielding paint 108 at a plurality of locations. Also, the infrared blocking paint 108 blocks infrared rays contained in the light emitted from the transmitter 100 . Therefore, the image sensor of the receiver 200 recognizes the infrared shielding paint 108 as a darker image than its surroundings. The receiver 200 recognizes rectangles circumscribing the infrared shielding paint 108 at a plurality of locations, each of which appears as a dark image, as a target area.

The receiver 200 then superimposes the AR image P5 on the target area, and displays the captured display image Pe on which the AR image P5 is superimposed on the display 201 . For example, the AR image P5 shows the schedule of events held at the facility on the information board 107 . In this case, since the AR image P5 is superimposed on the target area of the captured display image Pe, the receiver 200 displays the captured display image Pe so that the information board 107 on which the event schedule is written actually exists. can do. This makes it possible to inform the user of the receiver 200 of the event schedule of the facility in an easy-to-understand manner without providing the information board 107 with a special display device.

In place of the infrared shielding paint 108, the guide plate 107 may be coated with an infrared reflecting paint. The infrared reflecting paint reflects infrared rays contained in the light emitted from transmitter 100 . Therefore, the image sensor of the receiver 200 recognizes the infrared reflective paint as a brighter image than its surroundings. That is, in this case, the receiver 200 recognizes, as the target area, rectangles circumscribing the infrared reflective paint at a plurality of locations appearing as bright images.

FIG. 50 is a diagram showing another example in which receiver 200 displays an AR image according to this embodiment.

The transmitter 100 is configured as a station name sign and placed near the station exit information board 110 . The station exit guide board 110 has a light source and emits light, but unlike the transmitter 100, it does not transmit a light ID.

The receiver 200 acquires the captured display image Ppre and the decoding image Pdec by imaging the transmitter 100 and the station exit information board 110 . Since the transmitter 100 changes its brightness and the station exit guide board 110 emits light, the decoding image Pdec includes a bright line pattern area Pdec1 corresponding to the transmitter 100 and a bright area corresponding to the station exit guide board 110. Pdec2 appears. The bright line pattern region Pdec1 is a region composed of a plurality of bright line patterns that appear by exposing the plurality of exposure lines of the image sensor of the receiver 200 during the communication exposure time.

Here, as described above, the identification information includes the reference information for specifying the reference area Pbas in the captured display image Ppre and the target information indicating the relative position of the target area Ptar with respect to the reference area Pbas. there is For example, the reference information indicates that the position of the reference region Pbas in the captured display image Ppre is the same as the position of the bright line pattern region Pdec1 in the decoding image Pdec. Furthermore, the target information indicates that the position of the target area is the position of the reference area.

Therefore, the receiver 200 identifies the reference area Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 identifies an area in the captured display image Ppre at the same position as the bright line pattern area Pdec1 in the decoding image Pdec as the reference area Pbas. Further, the receiver 200 recognizes an area, in the captured display image Ppre, at a relative position indicated by the target information with respect to the position of the reference area Pbas as a target area Ptar. In the above example, the target information indicates that the position of the target area Ptar is the position of the reference area Pbas, so the receiver 200 recognizes the reference area Pbas in the captured display image Ppre as the target area Ptar.

Then, the receiver 200 superimposes the AR image P1 on the target area Ptar in the captured display image Ppre.

Thus, in the above example, the bright line pattern area Pdec1 is used to recognize the target area Ptar. On the other hand, if an attempt is made to recognize the area where the transmitter 100 is projected as the target area Ptar only from the captured display image Ppre without using the bright line pattern area Pdec1, misrecognition may occur. . That is, there is a possibility that an area in which the station exit guide plate 110 is displayed, not an area in which the transmitter 100 is displayed, of the captured display image Ppre is erroneously recognized as the target area Ptar. This is because the image of the transmitter 100 and the image of the station exit guide plate 110 are similar in the captured display image Ppre. However, when using the bright line pattern region Pdec1 as in the above example, the occurrence of erroneous recognition can be suppressed and the target region Ptar can be accurately recognized.

FIG. 51 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

In the example shown in FIG. 50, the transmitter 100 transmits the light ID by changing the brightness of the entire station name sign, and the target information indicates that the target area is the position of the reference area. However, in the present embodiment, the transmitter 100 transmits the optical ID by changing the luminance of the light-emitting elements arranged in part of the outer frame of the station name sign without changing the luminance of the entire station name sign. good too. Further, the target information only needs to indicate the relative position of the target region Ptar with respect to the reference region Pbas. may indicate.

In the example shown in FIG. 51, the transmitter 100 transmits the light ID by changing the luminance of a plurality of light emitting elements arranged along the horizontal direction under the outer frame of the station name sign. Also, the target information indicates that the position of the target area Ptar is above the reference area Pbas.

In such a case, the receiver 200 identifies the reference area Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 identifies an area in the captured display image Ppre at the same position as the bright line pattern area Pdec1 in the decoding image Pdec as the reference area Pbas. Specifically, the receiver 200 identifies a rectangular reference area Pbas that is long in the horizontal direction and short in the vertical direction. Further, the receiver 200 recognizes an area, in the captured display image Ppre, at a relative position indicated by the target information with respect to the position of the reference area Pbas as a target area Ptar. That is, the receiver 200 recognizes an area above the reference area Pbas in the captured display image Ppre as the target area Ptar. At this time, the receiver 200 identifies the orientation above the reference area Pbas based on the direction of gravity measured by the acceleration sensor provided therein.

Note that the target information may indicate not only the relative position of the target region Ptar, but also the size, shape and aspect ratio of the target region Ptar. In this case, the receiver 200 recognizes the target area Ptar of size, shape and aspect ratio indicated by the target information. Also, the receiver 200 may determine the size of the target area Ptar based on the size of the reference area Pbas.

FIG. 52 is a flowchart showing another example of the processing operation of receiver 200 in this embodiment.

The receiver 200 executes the processes of steps S101 to S104 in the same manner as in the example shown in FIG.

Next, the receiver 200 identifies a bright line pattern region Pdec1 from the decoding image Pdec (step S111). Next, the receiver 200 identifies a reference area Pbas corresponding to the bright line pattern area Pdec1 from the captured display image Ppre (step S112). Then, the receiver 200 recognizes the target area Ptar from the captured display image Ppre based on the recognition information (specifically, target information) and its reference area Pbas (step S113).

Next, similarly to the example shown in FIG. 45, the receiver 200 superimposes an AR image on the target area Ptar of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S106). . Then, the receiver 200 determines whether or not to end the imaging and display of the captured display image Ppre (step S107). If the receiver 200 determines that the process should not end (N in step S107), it further determines whether the acceleration of the receiver 200 is equal to or greater than the threshold (step S114). This acceleration is measured by an acceleration sensor provided in receiver 200 . When the receiver 200 determines that the acceleration is less than the threshold (N in step S114), the process from step S113 is executed. Accordingly, even if the captured display image Ppre displayed on the display 201 of the receiver 200 is shifted, the AR image can follow the target area Ptar of the captured display image Ppre. Further, when the receiver 200 determines that the acceleration is equal to or greater than the threshold value (Y in step S114), the receiver 200 executes the processing from step S111 or step S102. As a result, it is possible to suppress erroneous recognition of an area where a subject different from that of the transmitter 100 (for example, the station exit guide board 110) is displayed as the target area Ptar.

FIG. 53 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

When the AR image P1 in the captured display image Ppre being displayed is tapped, the receiver 200 enlarges and displays the AR image P1. Alternatively, when tapped, the receiver 200 may display a new AR image showing more detailed content than the content shown in the AR image P1 instead of the AR image P1. Further, when the AR image P1 shows information for one page of an information magazine consisting of multiple pages, the receiver 200 displays a new AR image showing information on the next page of the page of the AR image P1. It may be displayed instead of the AR image P1. Alternatively, when tapped, the receiver 200 may display a moving image related to the AR image P1 as a new AR image instead of the AR image P1. At this time, the receiver 200 may display, as an AR image, a moving image in which an object (autumn leaves in the example of FIG. 53) emerges from the target area Ptar.

FIG. 54 is a diagram showing captured display image Ppre and decoding image Pdec obtained by imaging by receiver 200 in the present embodiment.

During imaging, the receiver 200 acquires captured images such as a captured display image Ppre and a decoding image Pdec at a frame rate of 30 fps, as shown in (a1) of FIG. 54, for example. Specifically, the receiver 200 acquires the captured display image Ppre "A" at time t1, acquires the decoding image Pdec at time t2, and acquires the captured display image Ppre "B" at time t3. , the captured display image Ppre and the decoding image Pdec are acquired alternately.

Further, when displaying the captured image, the receiver 200 displays only the captured display image Ppre of the captured image, and does not display the decoding image Pdec. That is, as shown in (a2) of FIG. 54, when acquiring the decoding image Pdec, the receiver 200 displays the previously acquired captured display image Ppre instead of the decoding image Pdec. Specifically, receiver 200 displays captured display image Ppre "A" acquired at time t1, and displays captured display image Ppre "A" acquired at time t1 again at time t2. do. Thereby, the receiver 200 displays the captured display image Ppre at a frame rate of 15 fps.

Here, in the example shown in (a1) of FIG. 54, the receiver 200 alternately acquires the captured display image Ppre and the decoding image Pdec. It is not limited to such a form. That is, the receiver 200 continuously acquires N (N is an integer of 1 or more) images for decoding Pdec, and then continuously acquires M (M is an integer of 1 or more) captured display images Ppre. Obtaining may be repeated.

In addition, the receiver 200 needs to switch the captured image to be acquired between the captured display image Ppre and the decoding image Pdec, and this switching may take time. Therefore, as shown in (b1) of FIG. 54, the receiver 200 may provide a switching period when switching between acquisition of the captured display image Ppre and acquisition of the decoding image Pdec. Specifically, when the receiver 200 acquires the decoding image Pdec at time t3, the receiver 200 executes processing for switching the captured image during the switching period from time t3 to t5, and performs the process for switching the captured image at time t5. Get A. After that, the receiver 200 performs processing for switching captured images during the switching period from time t5 to t7, and acquires the decoding image Pdec at time t7.

When the switching period is provided in this way, the receiver 200 displays the captured display image Ppre acquired immediately before during the switching period, as shown in (b2) of FIG. Therefore, in this case, the frame rate for displaying the captured display image Ppre in the receiver 200 is low, for example, 3 fps. When the frame rate is low like this, even if the user moves the receiver 200 , the displayed captured display image Ppre may not move according to the movement of the receiver 200 . That is, the captured display image Ppre is not displayed as a live view. Therefore, the receiver 200 may move the captured display image Ppre according to the movement of the receiver 200 .

FIG. 55 is a diagram showing an example of captured display image Ppre displayed on receiver 200 in the present embodiment.

The receiver 200 displays an imaged display image Ppre obtained by imaging on the display 201, as shown in FIG. 55(a), for example. Here, the user moves receiver 200 to the left. At this time, if a new captured display image Ppre is not acquired by the imaging by the receiver 200, the receiver 200 moves the displayed captured display image Ppre to the right as shown in FIG. 55(b). That is, the receiver 200 includes an acceleration sensor, and moves the displayed captured display image Ppre so as to match the movement of the receiver 200 according to the acceleration measured by the acceleration sensor. Thereby, the receiver 200 can display the captured display image Ppre as a pseudo live view.

FIG. 56 is a flow chart showing another example of the processing operation of receiver 200 in this embodiment.

First, as described above, the receiver 200 superimposes the AR image on the target area Ptar of the captured display image Ppre to follow the target area Ptar (step S122). That is, an AR image that moves together with the target area Ptar in the captured display image Ppre is displayed. The receiver 200 then determines whether or not to maintain the display of the AR image (step S122). Here, when it is determined not to maintain the display of the AR image (N in step S122), if the receiver 200 acquires a new light ID by imaging, the receiver 200 captures and displays a new AR image corresponding to the light ID. It is superimposed on Ppre and displayed (step S123).

On the other hand, when determining to maintain the display of the AR image (Y in step S122), the receiver 200 repeats the processing from step S121. At this time, receiver 200 does not display other AR images even if other AR images are acquired. Alternatively, even if the receiver 200 acquires a new decoding image Pdec, the receiver 200 does not acquire the light ID by decoding the decoding image Pdec. At this time, the power consumption required for decoding can be suppressed.

By maintaining the display of the AR image in this way, it is possible to prevent the displayed AR image from being erased or from being obscured by the display of another AR image. That is, it is possible to make the displayed AR image easier for the user to see.

For example, in step S122, the receiver 200 determines to maintain the display of the AR image until a predetermined period (fixed period of time) elapses after the AR image is displayed. That is, when displaying the captured display image Ppre, the receiver 200 suppresses the display of the second AR image different from the first AR image that is the AR image superimposed in step S121, while suppressing the display of the second AR image. The first AR image is displayed only during the display period. The receiver 200 may prohibit decoding of the newly acquired decoding image Pdec during this display period.

As a result, when the user is viewing the first AR image that has been displayed once, it is possible to prevent the first AR image from being immediately replaced with a different second AR image. Furthermore, decoding the newly obtained decoding image Pdec is a wasteful process when the display of the second AR image is suppressed, so power consumption can be suppressed by prohibiting the decoding. .

Alternatively, in step S122, the receiver 200 may be provided with a face camera, and may determine to maintain the display of the AR image when detecting that the user's face is approaching based on the imaging result of the face camera. . That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 by imaging with the face camera provided in the receiver 200 . When the receiver 200 determines that the face is approaching, the receiver 200 suppresses the display of the second AR image, which is the AR image superimposed in step S121, and is different from the first AR image. Display the AR image.

Alternatively, in step S122, the receiver 200 may be provided with an acceleration sensor, and may determine to maintain the display of the AR image when detecting that the user's face is approaching based on the measurement result of the acceleration sensor. . That is, when displaying the captured display image Ppre, the receiver 200 further determines whether the user's face is approaching the receiver 200 based on the acceleration of the receiver 200 measured by the acceleration sensor. For example, when the acceleration of the receiver 200 measured by the acceleration sensor indicates a positive value in the direction perpendicular to the display 201 of the receiver 200, the receiver 200 determines that the user's face is approaching. judge. Then, when the receiver 200 determines that the face is approaching, the receiver 200 suppresses the display of the second AR image, which is the AR image superimposed in step S121, and is different from the first augmented reality image. display the AR image of

As a result, when the user brings his/her face closer to the receiver 200 to see the first AR image, it is possible to prevent the first AR image from being replaced with a different second AR image. .

Alternatively, in step S122, the receiver 200 may determine to maintain the display of the AR image when the lock button provided in the receiver 200 is pressed.

Also, in step S122, the receiver 200 determines not to maintain the display of the AR image after the above-mentioned fixed period (that is, the display period) has passed. Further, the receiver 200 determines not to maintain the display of the AR image when the acceleration sensor measures an acceleration equal to or greater than the threshold value even if the above-described certain period of time has not elapsed. That is, when the captured display image Ppre is displayed, the receiver 200 further measures the acceleration of the receiver 200 with the acceleration sensor during the display period described above, and determines whether the measured acceleration is equal to or greater than the threshold. Then, when the receiver 200 determines that it is equal to or greater than the threshold value, the receiver 200 releases the suppression of the display of the second AR image, thereby displaying the second AR image instead of the first AR image in step S123.

Accordingly, when the acceleration of the display device equal to or greater than the threshold is measured, the suppression of the display of the second AR image is released. Therefore, for example, when the user moves the receiver 200 by a large amount to aim the image sensor at another subject, the second AR image can be immediately displayed.

FIG. 57 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

The transmitter 100 is configured as an illumination device, for example, as shown in FIG. 57, and transmits the light ID by changing the luminance while illuminating a stage 111 for a small doll. Since the stage 111 is illuminated by the light from the transmitter 100, it changes in luminance similarly to the transmitter 100 and transmits the light ID.

The two receivers 200 image the stage 111 illuminated by the transmitter 100 from left and right.

The left receiver 200 of the two receivers 200 captures the stage 111 illuminated by the transmitter 100 from the left side, thereby obtaining the captured display image Pf and the decoding image in the same manner as described above. The receiver 200 on the left acquires the optical ID by decoding the decoding image. That is, the left receiver 200 receives the optical ID from the stage 111 . The receiver 200 on the left transmits its light ID to the server. Then, the receiver 200 on the left side acquires the three-dimensional AR image and the recognition information corresponding to the light ID from the server. This three-dimensional AR image is, for example, an image for stereoscopically displaying a doll. The receiver 200 on the left side recognizes an area corresponding to the recognition information in the imaged display image Pf as a target area. For example, the left receiver 200 recognizes the area above the center of the stage 111 as the target area.

Next, based on the orientation of the stage 111 displayed in the captured display image Pf, the left receiver 200 generates a two-dimensional AR image P6a corresponding to the orientation from the three-dimensional AR image. Then, the receiver 200 on the left side superimposes the two-dimensional AR image P6a on the target area, and displays the captured display image Pf on which the AR image P6a is superimposed on the display 201 . In this case, since the two-dimensional AR image P6a is superimposed on the target area of the captured display image Pf, the left receiver 200 displays the captured display image Pf as if the doll actually exists on the stage 111. can do.

Similarly, the right receiver 200 of the two receivers 200 captures the imaged display image Pg and the decoding image in the same manner as described above by capturing the stage 111 illuminated by the transmitter 100 from the right side. get. The receiver 200 on the right side acquires the optical ID by decoding the decoding image. That is, the receiver 200 on the right receives the optical ID from the stage 111 . The receiver 200 on the right transmits its light ID to the server. Then, the receiver 200 on the right side acquires the three-dimensional AR image and recognition information corresponding to the light ID from the server. The receiver 200 on the right side recognizes the area corresponding to the recognition information in the imaged display image Pg as the target area. For example, the receiver 200 on the right side recognizes the area above the center of the stage 111 as the target area.

Next, the receiver 200 on the right generates a two-dimensional AR image P6b corresponding to the orientation of the stage 111 displayed in the captured display image Pg from the three-dimensional AR image. Then, the receiver 200 on the right side superimposes the two-dimensional AR image P6b on the target area, and displays on the display 201 the captured display image Pg superimposed with the AR image P6b. In this case, since the two-dimensional AR image P6b is superimposed on the target area of the captured display image Pg, the right receiver 200 displays the captured display image Pg as if the doll actually exists on the stage 111. can do.

Thus, two receivers 200 display AR images P6a and P6b at the same position on stage 111. FIG. Also, these AR images P6a and P6b are generated according to the orientation of receiver 200 so that the virtual doll actually faces a predetermined direction. Therefore, the captured display image can be displayed as if the doll actually exists on the stage 111 regardless of which direction the stage 111 is captured.

In the above example, receiver 200 generates a two-dimensional AR image according to the positional relationship between receiver 200 and stage 111 from a three-dimensional AR image. can be obtained from the server. That is, the receiver 200 transmits information indicating the positional relationship to the server together with the light ID, and acquires the two-dimensional AR image from the server instead of the three-dimensional AR image. Thereby, the burden on the receiver 200 can be reduced.

FIG. 58 is a diagram showing another example in which receiver 200 displays an AR image in this embodiment.

The transmitter 100 is configured as an illumination device, for example, as shown in FIG. 58, and transmits the light ID by changing the brightness while illuminating a cylindrical structure 112 . Since the structure 112 is illuminated by the light from the transmitter 100, it changes in luminance in the same way as the transmitter 100 and transmits the light ID.

The receiver 200 captures the image of the structure 112 illuminated by the transmitter 100 to acquire the captured display image Ph and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the optical ID from structure 112 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P7 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Ph as a target area. For example, the receiver 200 recognizes an area in which the central portion of the structure 112 is projected as the target area. The receiver 200 then superimposes the AR image P7 on the target area, and displays the captured display image Ph on which the AR image P7 is superimposed on the display 201 . For example, the AR image P7 is an image containing the character string “ABCD”, and the character string is distorted according to the curved surface of the central portion of the structure 112 . In this case, the AR image P2 including the distorted character string is superimposed on the target area of the captured display image Ph. , the captured display image Ph can be displayed.

FIG. 59 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

The transmitter 100 transmits the light ID by changing the luminance while illuminating the menu 113 of the restaurant, as shown in FIG. 59, for example. Since the menu 113 is illuminated by the light from the transmitter 100, the brightness changes in the same way as the transmitter 100, and the light ID is transmitted. Menu 113 also shows the names of multiple dishes such as, for example, "ABC soup," "XYZ salad," and "KLM lunch."

The receiver 200 captures the image of the menu 113 illuminated by the transmitter 100 to acquire the captured display image Pi and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from menu 113 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P8 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pi as a target area. For example, the receiver 200 recognizes the area where the menu 113 is displayed as the target area. The receiver 200 then superimposes the AR image P8 on the target area, and displays the captured display image Pi on which the AR image P8 is superimposed on the display 201 . For example, the AR image P8 is an image showing the ingredients used in each of a plurality of dishes with marks. For example, the AR image P8 shows a mark in the shape of an egg for the dish "XYZ salad" that uses eggs, and a pork dish for the dish "KLM lunch" that uses pork. Show a mock mark. In this case, since the AR image P8 is superimposed on the target area of the captured display image Pi, the receiver 200 displays the captured display image Pi so that the menu 113 marked with the ingredients actually exists. be able to. As a result, the user of the receiver 200 can be notified of the ingredients for each dish simply and in an easy-to-understand manner without providing a special display device for the menu 113 .

In addition, the receiver 200 acquires a plurality of AR images, selects an AR image suitable for the user from the plurality of AR images based on user information set by the user, and superimposes the AR image. may For example, if the user information indicates that the user has an allergic reaction to eggs, the receiver 200 selects an AR image marked with an egg for dishes using eggs. Also, if the user information indicates that the consumption of pork is prohibited, the receiver 200 selects an AR image with a pig mark attached to dishes using pork. Alternatively, the receiver 200 may transmit the user information together with the light ID to the server, and acquire the AR image corresponding to the light ID and the user information from the server. This makes it possible to display, for each user, a menu for prompting the user.

FIG. 60 is a diagram showing another example in which receiver 200 displays an AR image according to the present embodiment.

Transmitter 100 is configured as a television, for example, as shown in FIG. 60, and transmits light ID by changing luminance while displaying an image on a display. A normal television 114 is arranged near the transmitter 100 . The television 114 displays an image on the display, but does not transmit the light ID.

The receiver 200 acquires the captured display image Pj and the decoding image in the same manner as described above, for example, by imaging the television 114 together with the transmitter 100 . The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P9 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pj as a target area.

For example, the receiver 200 uses the bright line pattern area of the decoding image to select the lower part of the area where the transmitter 100 transmitting the light ID is displayed in the captured display image Pj as the first target area. recognized as At this time, the reference information included in the recognition information indicates that the position of the reference area in the captured display image Pj is the same as the position of the bright line pattern area in the decoding image. Furthermore, the target information included in the recognition information indicates that there is a target area below the reference area. Receiver 200 recognizes the above-described first target region using such recognition information.

Furthermore, the receiver 200 recognizes a region whose position is fixed in advance in the lower portion of the captured display image Pj as a second target region. The second region of interest is larger than the first region of interest. Note that the target information included in the recognition information further indicates not only the position of the first target region, but also the position and size of the second target region as described above. Receiver 200 recognizes the above-described second target region using such recognition information.

Then, the receiver 200 superimposes the AR image P9 on the first target area and the second target area, and displays on the display 201 the captured display image Pj on which the AR image P8 is superimposed. In superimposing the AR image P9, the receiver 200 adjusts the size of the AR image P9 to the size of the first target area, and superimposes the size-adjusted AR image P9 on the first target area. Further, the receiver 200 adjusts the size of the AR image P9 to the size of the second region of interest and superimposes the resized AR image P9 on the second region of interest.

For example, AR image P9 indicates a subtitle for the video of transmitter 100 . Also, the language of the subtitles of the AR image P9 is a language corresponding to the user information set and registered in the receiver 200. FIG. That is, when the receiver 200 transmits the light ID to the server, the receiver 200 also transmits the user information (for example, information indicating the user's nationality or language used) to the server. Then, the receiver 200 acquires an AR image P9 showing captions in a language corresponding to the user information. Alternatively, the receiver 200 acquires a plurality of AR images P9 showing subtitles in different languages, and selects an AR image to be used for superimposition from the plurality of AR images P9 in accordance with registered user information. P9 may be selected.

In other words, in the example shown in FIG. 60, the receiver 200 obtains the captured display image Pj and the image for decoding by capturing images of a plurality of displays each displaying an image as subjects. Then, when the receiver 200 recognizes the target area, the receiver 200 recognizes the area where the transmission display (i.e., the transmitter 100), which is the display transmitting the light ID among the plurality of displays, appears in the captured display image Pj. is recognized as the region of interest. Next, the receiver 200 superimposes the first caption corresponding to the image displayed on the transmitting display as an AR image on the target area. Furthermore, the receiver 200 superimposes the second caption, which is an enlarged caption of the first caption, on an area of the captured display image Pj that is larger than the target area.

Thereby, the receiver 200 can display the captured display image Pj so that the caption actually exists in the video of the transmitter 100 . Furthermore, since the receiver 200 also superimposes a large subtitle on the lower part of the captured display image Pj, even if the subtitle attached to the image of the transmitter 100 is small, the subtitle can be easily viewed. Note that when there is no subtitle attached to the video of the transmitter 100 and only a large subtitle is superimposed at the bottom of the captured display image Pj, the superimposed subtitle may be the subtitle for the video of the transmitter 100 or the television. Therefore, it is difficult to determine whether the caption is for the video of 114. However, in the present embodiment, subtitles are added to the video from transmitter 100 that transmits the light ID, so the user can easily determine which video the superimposed subtitles correspond to. can be done.

Further, in displaying the captured display image Pj, the receiver 200 may further determine whether or not the information acquired from the server includes audio information. When the receiver 200 determines that the audio information is included, the receiver 200 preferentially outputs the audio indicated by the audio information over the first and second subtitles. As a result, audio is preferentially output, so that the user's burden of reading subtitles can be reduced.

Also, in the above example, the subtitle language is changed according to the user information (that is, the user's attribute), but the video (that is, the content) itself that is displayed on the transmitter 100 may be changed. For example, when the image displayed on the transmitter 100 is a news image, if the user information indicates that the user is Japanese, the receiver 200 displays the news image broadcast in Japan. is acquired as an AR image. The receiver 200 then superimposes the news video on the area where the display of the transmitter 100 is projected (that is, the target area). On the other hand, if the user information indicates that the user is an American, receiver 200 acquires the news video being broadcast in the United States as an AR image. The receiver 200 then superimposes the news video on the area where the display of the transmitter 100 is projected (that is, the target area). Accordingly, it is possible to display an image suitable for the user. Note that the user information indicates, for example, the nationality or the language used as user attributes, and the receiver 200 acquires the AR image described above based on the attributes.

FIG. 61 is a diagram showing an example of recognition information in this embodiment.

Even if the recognition information is, for example, the feature points and feature amounts as described above, there is a possibility of misrecognition. For example, each of the transmitters 100a and 100b is configured as a station name sign, similar to the transmitter 100. FIG. These transmitters 100a and 100b may be erroneously recognized even if the station name signs are different from each other because they are similar if they are located close to each other.

Therefore, the recognition information of each of the transmitters 100a and 100b does not indicate each feature point and each feature amount of the entire image of the transmitter 100a or 100b, but each feature point and each feature amount of only a characteristic portion of the image. Each feature amount may be indicated.

For example, the portion a1 of the transmitter 100a and the portion b1 of the transmitter 100b are significantly different, and the portion a2 of the transmitter 100a and the portion b2 of the transmitter 100b are significantly different. Therefore, if the transmitters 100a and 100b are installed within a predetermined range (that is, a short distance), the server uses the characteristics of the images of the portions a1 and a2 as the recognition information corresponding to the transmitter 100a. Holds points and features. Similarly, the server holds feature points and feature amounts of images of portions b1 and b2 as identification information corresponding to transmitter 100b.

With this, even when mutually similar transmitters 100a and 100b are close to each other (within the above-described predetermined range), receiver 200 uses their identification information to perform appropriate transmission. can recognize the target area.

FIG. 62 is a flowchart showing another example of the processing operation of receiver 200 in this embodiment.

The receiver 200 first determines whether or not the user has a visual impairment based on the user information set and registered in the receiver 200 (step S131). Here, when the receiver 200 determines that there is a visual impairment (Y in step S131), the characters of the superimposed and displayed AR image are output by voice (step S132). On the other hand, if the receiver 200 determines that there is no visual impairment (N in step S131), it further determines whether the user has hearing impairment based on the user information (step S133). Here, when the receiver 200 determines that there is hearing impairment (Y in step S133), the receiver 200 stops audio output (step S134). At this time, the receiver 200 stops outputting audio from all functions.

Note that the receiver 200 may perform the process of step S133 when determining that there is a visual impairment in step S131 (Y in step S131). In other words, when the receiver 200 determines that the user has a visual impairment and does not have a hearing impairment, the receiver 200 may output the superimposed and displayed characters of the AR image by voice.

FIG. 63 is a diagram showing an example of how receiver 200 identifies bright line pattern regions according to the present embodiment.

The receiver 200 first obtains a decoding image by capturing images of two transmitters that respectively transmit optical IDs, and decodes the decoding image to obtain an optical ID as shown in (e) of FIG. to get At this time, since the decoding image includes two bright line pattern areas X and Y, the receiver 200 determines the light ID of the transmitter corresponding to the bright line pattern area X and the transmission light ID corresponding to the bright line pattern area Y. acquire the light ID of the machine. The light ID of the transmitter corresponding to the bright line pattern area X consists of, for example, numerical values (that is, data) corresponding to addresses 0 to 9, and is represented as "5, 2, 8, 4, 3, 6, 1, 9, 4, 3”. Similarly, the light ID of the transmitter corresponding to the bright line pattern area X is also composed of numerical values corresponding to addresses 0 to 9, for example, "5, 2, 7, 7, 1, 5, 3, 2, 7, 4”.

Even if the receiver 200 obtains these light IDs once, that is, even if these light IDs are already known, the receiver 200 can determine from which bright line pattern area each light ID was obtained during imaging. Sometimes you don't know. In such a case, the receiver 200 performs the processing shown in (a) to (d) of FIG. , can be determined quickly.

Specifically, as shown in (a) of FIG. 63, the receiver 200 first acquires the decoding image Pdec11, and decodes the bright line pattern regions X and Y by decoding the decoding image Pdec11. Acquire the numerical value of the address 0 of the light ID. For example, the numerical value of the address 0 of the light ID of the bright line pattern area X is "5", and the numerical value of the address 0 of the light ID of the bright line pattern area Y is also "5". Since the numerical value of the address 0 of each light ID is "5", at this time it is impossible to determine from which bright line pattern area the known light ID was obtained.

Therefore, as shown in FIG. 63(b), the receiver 200 obtains the decoding image Pdec12, and decodes the decoding image Pdec12 to obtain the address 1 of the light ID of each of the bright line pattern regions X and Y. to get the numerical value of For example, the numerical value of the address 1 of the light ID of the bright line pattern area X is "2", and the numerical value of the address 1 of the light ID of the bright line pattern area Y is also "2". Since the numerical value of the address 1 of each light ID is "2", it cannot be determined from which bright line pattern area the known light ID was obtained.

Therefore, as shown in (c) of FIG. 63, the receiver 200 further acquires the image for decoding Pdec13, and decodes the image for decoding Pdec13 to determine the light IDs of the bright line pattern regions X and Y. Get the value of address 2. For example, the numerical value of the address 2 of the light ID of the bright line pattern area X is "8", and the numerical value of the address 2 of the light ID of the bright line pattern area Y is "7". At this time, it can be determined that the known light IDs "5, 2, 8, 4, 3, 6, 1, 9, 4, 3" are obtained from the bright line pattern region X, and the known light IDs "5, 8, 4, 3, 6, 1, 9, 4, 3" 2, 7, 7, 1, 5, 3, 2, 7, 4” are obtained from the bright line pattern region Y.

However, in order to increase reliability, the receiver 200 may further acquire the numeric value of address 3 of each light ID, as shown in (d) of FIG. That is, the receiver 200 obtains the decoding image Pdec14, and obtains the numerical value of the address 3 of the light ID of each of the bright line pattern regions X and Y by decoding the decoding image Pdec14. For example, the numerical value of the address 3 of the light ID of the bright line pattern area X is "4", and the numerical value of the address 3 of the light ID of the bright line pattern area Y is "7". At this time, it can be determined that the known light IDs "5, 2, 8, 4, 3, 6, 1, 9, 4, 3" are obtained from the bright line pattern region X, and the known light IDs "5, 8, 4, 3, 6, 1, 9, 4, 3" 2, 7, 7, 1, 5, 3, 2, 7, 4” are obtained from the bright line pattern region Y. In other words, the optical IDs of the bright line pattern regions X and Y can be identified not only by the address 2 but also by the address 3, so reliability can be improved.

Thus, in this embodiment, the numerical value of at least one address is reacquired without reacquiring the numerical values (that is, data) of all the addresses of the light ID. This makes it possible to easily and quickly determine from which bright line pattern area the known light ID was obtained.

In the examples shown in (c) and (d) of FIG. 63, the numerical value obtained for the predetermined address matches the numerical value of the known light ID. good. For example, in the example shown in (d) of FIG. 63 , the receiver 200 acquires “6” as the numerical value of the address 3 of the light ID of the bright line pattern region Y. In the example shown in FIG. The numerical value "6" of this address 3 is different from the numerical value "7" of the address 3 of the known light ID "5,2,7,7,1,5,3,2,7,4". However, since the numerical value “6” is a numerical value close to the numerical value “7”, the receiver 200 detects that the known light ID “5,2,7,7,1,5,3,2,7,4” is a bright line. It may be determined that it is obtained from the pattern area Y. Note that the receiver determines that the numerical value "6" is close to the numerical value "7" depending on whether or not the numerical value "6" is within the range of the numerical value "7"±n (n is a number equal to or greater than 1, for example). It may be determined whether

FIG. 64 is a diagram showing another example of receiver 200 in this embodiment.

The receiver 200 is configured as a smartphone in the above example, but may be configured as a head-mounted display (also referred to as glasses) having an image sensor.

Such a receiver 200 consumes a lot of power if the processing circuit for displaying an AR image (hereinafter referred to as an AR processing circuit) as described above is always activated. The AR processing circuit may be activated when the

For example, receiver 200 includes touch sensor 202 . The touch sensor 202 outputs a touch signal when touched by a user's finger or the like. Receiver 200 activates the AR processing circuit when it detects the touch signal.

Alternatively, the receiver 200 may activate the AR processing circuit when a radio signal such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) is detected.

Alternatively, the receiver 200 may include an acceleration sensor and activate the AR processing circuit when the acceleration sensor measures an acceleration equal to or greater than a threshold in the direction opposite to the direction of gravity. That is, the receiver 200 activates the AR processing circuit when detecting the signal indicating the acceleration. For example, when the user pushes up the nose pad portion of the receiver 200 configured as a glass from below with a fingertip, the receiver 200 detects the signal indicating the acceleration and activates the AR processing circuit.

Alternatively, the receiver 200 may activate the AR processing circuitry when it detects that the image sensor is pointed at the transmitter 100, such as by GPS and 9-axis sensors. That is, the receiver 200 activates the AR processing circuitry when it detects a signal indicating that the receiver 200 is oriented in a predetermined direction. In this case, if the transmitter 100 is the above-mentioned Japanese station name sign, the receiver 200 displays an AR image showing the English station name superimposed on the station name sign.

FIG. 65 is a flowchart showing another example of the processing operation of receiver 200 in this embodiment.

When the receiver 200 acquires the light ID from the transmitter 100 (step S141), the receiver 200 switches the noise cancellation mode by receiving the mode designation information according to the light ID (step S142). Then, the receiver 200 determines whether or not to end the mode switching process (step S143), and if it determines not to end it (N in step S143), it repeats the process from step S141. The switching of the noise cancellation mode is, for example, between a mode (ON) for canceling noise from an engine in an airplane and a mode (OFF) for not canceling the noise. Specifically, a user carrying receiver 200 puts earphones connected to receiver 200 to his or her ear and listens to sounds such as music output from receiver 200 . When such a user boards an airplane, receiver 200 acquires a light ID. As a result, the receiver 200 switches the noise cancellation mode from OFF to ON. As a result, the user can listen to voice that does not contain noise such as engine noise even in the cabin. The receiver 200 also acquires the light ID when the user leaves the plane. The receiver 200 that has acquired this light ID switches the noise cancellation mode from ON to OFF. It should be noted that the noise to be noise-cancelled may be not only engine noise but also any sound such as a human voice.

FIG. 66 is a diagram showing an example of a transmission system including multiple transmitters according to this embodiment.

This transmission system comprises a plurality of transmitters 120 arranged in a predetermined order. Similar to transmitter 100, these transmitters 120 are transmitters in any one of the first to third embodiments, and comprise one or more light-emitting elements (eg, LEDs). The top transmitter 120 transmits the light ID by changing the brightness of one or more light emitting elements according to a predetermined frequency (carrier frequency). Furthermore, the leading transmitter 120 outputs a signal indicating the change in brightness to the trailing transmitter 120 as a synchronization signal. A subsequent transmitter 120, upon receiving the synchronization signal, transmits the light ID by changing the brightness of one or more light emitting elements according to the synchronization signal. Furthermore, the succeeding transmitter 120 outputs a signal indicating the change in brightness to the following succeeding transmitter 120 as a synchronization signal. Thereby, all the transmitters 120 included in the transmission system synchronously transmit the light ID.

Here, the synchronization signal is passed from the leading transmitter 120 to the trailing transmitter 120, from the trailing transmitter 120 to the next trailing transmitter 120, and then to the last transmitter 120. reach. It takes, for example, about 1 μs to pass the sync signal. Therefore, if the transmission system includes N (N is an integer equal to or greater than 2) transmitters 120, it takes 1×N μ seconds for the synchronization signal to reach the last transmitter 120 from the top transmitter 120. become. As a result, the timing of transmission of the optical ID is shifted by a maximum of N μ seconds. For example, even if N transmitters 120 transmit light IDs according to a frequency of 9.6 kHz and the receiver 200 attempts to receive the light IDs at a frequency of 9.6 kHz, the receiver 200 receives light that is shifted by N μs. In order to receive the ID, the optical ID may not be received correctly.

Therefore, in the present embodiment, the leading transmitter 120 transmits the light ID earlier according to the number of transmitters 120 included in the transmission system. For example, the top transmitter 120 transmits light IDs according to a frequency of 9.605 kHz. On the other hand, receiver 200 receives the optical ID at a frequency of 9.6 kHz. At this time, even if the receiver 200 receives an optical ID that is shifted by N μs, the frequency of the leading transmitter 120 is higher than the frequency of the receiver 200 by 0.005 kHz. occurrence can be suppressed.

Also, the top transmitter 120 may control the amount of frequency adjustment by having the synchronization signal fed back from the last transmitter 120 . For example, the top transmitter 120 measures the time from outputting the synchronization signal by itself to receiving the synchronization signal fed back from the last transmitter 120 . Then, the longer the time, the leading transmitter 120 transmits the optical ID according to a frequency higher than the reference frequency (9.6 kHz, for example).

FIG. 67 is a diagram showing an example of a transmission system including a plurality of transmitters and receivers according to this embodiment.

This transmission system comprises for example two transmitters 120 and a receiver 200 . One transmitter 120 of the two transmitters 120 transmits light IDs according to a frequency of 9.599 kHz. The other transmitter 120 transmits light IDs according to a frequency of 9.601 kHz. In such a case, each of the two transmitters 120 notifies the receiver 200 of the frequency of its own light ID using radio signals.

Upon being notified of those frequencies, receiver 200 attempts decoding according to each of the notified frequencies. That is, the receiver 200 tries to decode the image for decoding according to the frequency of 9.599 kHz, and if the optical ID cannot be received by this, attempts to decode the image for decoding according to the frequency of 9.601 kHz. In this way, the receiver 200 attempts to decode the decoding image according to each of all the notified frequencies. In other words, the receiver 200 brute-forces each frequency that has been notified. Alternatively, the receiver 200 may attempt decoding according to the average frequency of all notified frequencies. That is, receiver 200 attempts decoding according to 9.6 kHz, which is the average frequency of 9.599 kHz and 9.601 kHz.

As a result, it is possible to reduce the incidence of reception errors due to the difference in frequency between receiver 200 and transmitter 120 .

FIG. 68A is a flowchart showing an example of the processing operation of receiver 200 in this embodiment.

The receiver 200 first starts imaging (step S151) and initializes the parameter N to 1 (step S152). Next, the receiver 200 decodes the decoding image obtained by the imaging according to the frequency corresponding to the parameter N, and calculates the evaluation value for the decoding result (step S153). For example, parameters N=1, 2, 3, 4, and 5 are associated in advance with frequencies such as 9.6 kHz, 9.601 kHz, 9.599 kHz, and 9.602 kHz. The evaluation value indicates a higher numerical value as the decoding result is more similar to the correct optical ID.

Next, receiver 200 determines whether the numerical value of parameter N is equal to Nmax, which is a predetermined integer of 1 or more (step S154). Here, if the receiver 200 determines that it is not equal to Nmax (N in step S154), it increments the parameter N (step S155), and repeats the processing from step S153. On the other hand, when receiver 200 determines that it is equal to Nmax (Y in step S154), receiver 200 registers the frequency for which the maximum evaluation value is calculated as the optimum frequency in the server in association with location information indicating the location of receiver 200. do. The optimum frequency and location information registered in this way are used for reception of the optical ID by the receiver 200 that has moved to the location indicated by the location information after registration. The location information may be, for example, information indicating a position measured by GPS, or may be identification information (for example, SSID: Service Set Identifier) of an access point in a wireless LAN (Local Area Network).

Also, the receiver 200 that has registered with the server displays an AR image as described above, for example, according to the optical ID obtained by decoding with the optimum frequency.

FIG. 68B is a flowchart showing an example of the processing operation of receiver 200 in this embodiment.

After being registered with the server shown in FIG. 68A, the receiver 200 transmits location information indicating its own location to the server (step S161). Next, receiver 200 acquires from the server the optimal frequency registered in association with the location information (step S162).

Next, the receiver 200 starts imaging (step S163), and decodes the decoding image obtained by the imaging according to the optimum frequency acquired in step S162 (step S164). The receiver 200 displays an AR image as described above, for example, according to the light ID obtained by this decoding.

In this way, after being registered with the server, the receiver 200 can acquire the optimum frequency and receive the light ID without executing the processing shown in FIG. 68A. Note that receiver 200 may obtain the optimum frequency by executing the process shown in FIG. 68A when the optimum frequency could not be obtained in step S162.

[Summary of Embodiment 4]
FIG. 69A is a flowchart showing a display method according to this embodiment.

The display method according to the present embodiment is a display method in which the display device, which is the receiver 200 described above, displays an image, and includes steps SL11 to SL16.

At step SL11, the image sensor picks up an image of the subject to acquire a captured display image and a decoding image. At step SL12, the light ID is obtained by decoding the decoding image. At step SL13, the light ID is transmitted to the server. At step SL14, the AR image and recognition information corresponding to the light ID are obtained from the server. At step SL15, the area corresponding to the recognition information is recognized as the target area in the captured display image. At step SL16, the captured display image in which the AR image is superimposed on the target area is displayed.

As a result, the AR image is superimposed on the captured display image and displayed, so that an image beneficial to the user can be displayed. Furthermore, the AR image can be superimposed on an appropriate target area while suppressing the processing load.

That is, in general augmented reality (that is, AR), by comparing a large number of pre-stored recognition target images with a captured display image, any recognition target image is included in the captured display image. It is determined whether or not Then, if it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, positioning of the AR images is performed with reference to the recognition target image. As described above, in general augmented reality, since a huge number of recognition target images and captured display images are compared, it is necessary to detect the position of the recognition target image in the captured display image for alignment. There is a problem that the amount of calculation is large and the processing load is high.

However, in the display method according to the present embodiment, as shown in FIGS. 41 to 68B, the light ID is obtained by decoding the decoding image obtained by imaging the subject. That is, the light ID transmitted from the transmitter, which is the subject, is received. Furthermore, an AR image and recognition information corresponding to this light ID are obtained from the server. Therefore, the server does not need to compare a huge number of images to be recognized and captured display images, and can select an AR image associated with the light ID in advance and transmit it to the display device. As a result, the amount of calculation can be reduced and the processing load can be greatly suppressed.

Further, in the display method according to the present embodiment, recognition information corresponding to this light ID is obtained from the server. The recognition information is information for recognizing the target area, which is the area where the AR image is superimposed in the captured display image. This recognition information may be, for example, information indicating that a white rectangle is the target area. In this case, the target area can be easily recognized, and the processing load can be further reduced. That is, the processing load can be further reduced according to the content of the recognition information. In addition, since the server can arbitrarily set the content of the recognition information in accordance with the light ID, it is possible to maintain an appropriate balance between the processing load and the recognition accuracy.

Here, the recognition information is reference information for specifying a reference area in the captured and displayed image, and in recognizing the target area, the reference area is specified from the captured and displayed image based on the reference information, and the captured and displayed image is identified. Among them, the target area may be recognized based on the position of the reference area.

Alternatively, the recognition information may include reference information for specifying a reference area in the captured display image and target information indicating the relative position of the target area with respect to the reference area. In this case, in recognizing the target area, the reference area is specified from the captured display image based on the reference information, and the area in the captured display image at the relative position indicated by the target information with respect to the position of the reference area is Recognize as target area.

Thereby, as shown in FIGS. 50 and 51, it is possible to expand the degree of freedom of the position of the target area recognized in the captured display image.

Further, the reference information indicates that the position of the reference region in the captured display image is the same as the position of the bright line pattern region formed by the patterns of a plurality of bright lines appearing in the image for decoding by exposure of the plurality of exposure lines of the image sensor. may be indicated.

As a result, as shown in FIGS. 50 and 51, the target area can be recognized with reference to the area corresponding to the bright line pattern area in the captured display image.

Further, the reference information may indicate that the reference area in the captured display image is the area where the display is shown in the captured display image.

As a result, as shown in FIG. 41, for example, if a station name sign is used as a display, the target area can be recognized based on the area where the display is projected.

In displaying the captured display image, the first AR image is displayed only for a predetermined display period while suppressing the display of the second AR image different from the first AR image, which is the AR image described above. may

As a result, as shown in FIG. 56, when the user views the first AR image once displayed, the first AR image is immediately replaced with a different second AR image. can be suppressed.

Further, in the display of the captured display image, the decoding of the newly obtained image for decoding may be prohibited during the display period.

As a result, as shown in FIG. 56, the decoding of the newly acquired decoding image is a wasteful process when the display of the second AR image is suppressed. Power consumption can be reduced.

Further, in the display of the captured display image, the acceleration of the display device may be measured by an acceleration sensor during the display period, and it may be determined whether or not the measured acceleration is equal to or greater than a threshold. Then, when it is determined to be equal to or greater than the threshold, the second AR image may be displayed instead of the first AR image by canceling the suppression of the display of the second AR image.

As a result, as shown in FIG. 56, suppression of display of the second AR image is released when the acceleration of the display device equal to or greater than the threshold is measured. Therefore, for example, when the user moves the display device significantly to direct the image sensor to another subject, the second AR image can be immediately displayed.

Further, in the display of the captured display image, whether or not the user's face is approaching the display device may be determined by capturing an image using a face camera provided in the display device. Then, when it is determined that the face is approaching, the first AR image may be displayed while suppressing the display of the second AR image different from the first AR image. Alternatively, in the display of the captured display image, whether or not the user's face is approaching the display device may be determined based on the acceleration of the display device measured by the acceleration sensor. Then, when it is determined that the face is approaching, the first AR image may be displayed while suppressing the display of the second AR image different from the first AR image.

As a result, as shown in FIG. 56, when the user brings the face closer to the display device to see the first AR image, the first AR image is replaced with the second AR image. can be suppressed.

Further, as shown in FIG. 60, in acquiring the captured display image and the decoding image, a plurality of displays each displaying an image are captured as subjects, and the captured display image and the decoding image are acquired. good too. At this time, in recognizing the target area, the area in which the transmitting display, which is the display transmitting the optical ID, appears among the plurality of displays is recognized as the target area. Also, in the display of the captured display image, the first caption corresponding to the image displayed on the transmission display is superimposed as an AR image on the target area, and further, in the captured display image, a larger area than the target area A second subtitle, which is an enlarged subtitle of the first subtitle, is superimposed.

As a result, since the first caption is superimposed on the image of the transmission display, it is possible for the user to easily understand which one of the plurality of displays the first caption is for. can. In addition, since the second subtitles, which are subtitles obtained by enlarging the first subtitles, are also displayed, even if the first subtitles are small and difficult to read, the display of the second subtitles makes the subtitles easier to read. can do.

Further, in the display of the captured display image, it is further determined whether or not the information obtained from the server contains audio information. Alternatively, the audio indicated by the audio information may be output preferentially.

As a result, audio is preferentially output, so that the user's burden of reading subtitles can be reduced.

FIG. 69B is a block diagram showing the configuration of the display device according to this embodiment.

The display device 10 according to the present embodiment is a display device that displays an image, and includes an image sensor 11, a decoding unit 12, a transmission unit 13, an acquisition unit 14, a recognition unit 15, and a display unit 16. Prepare. The display device 10 corresponds to the receiver 200 described above.

The image sensor 11 acquires a captured display image and a decoding image by capturing an image of a subject. The decoding unit 12 acquires the optical ID by decoding the decoding image. The transmitter 13 transmits the light ID to the server. The acquisition unit 14 acquires the AR image and the recognition information corresponding to the light ID from the server. The recognition unit 15 recognizes a region corresponding to the recognition information in the captured display image as a target region. The display unit 16 displays the captured display image in which the AR image is superimposed on the target area.

As a result, the AR image is superimposed on the captured display image and displayed, so that an image beneficial to the user can be displayed. Furthermore, the AR image can be superimposed on an appropriate target area while suppressing the processing load.

In addition, in the present embodiment, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. Here, the software that implements the receiver 200 or the display device 10 of the present embodiment is each included in the flowcharts shown in FIGS. It is a program that causes a computer to execute steps.

[Modification 1 of Embodiment 4]
Modification 1 of Embodiment 4, that is, modification 1 of a display method for realizing AR using optical IDs will be described below.

70] FIG. 70 is a diagram illustrating an example in which a receiver displays an AR image according to Modification 1 of Embodiment 4. [ FIG.

The receiver 200 obtains the captured display image Pk, which is the above-described normal captured image, and the decoding image, which is the above-described visible light communication image or bright line image, by imaging the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100c configured as a robot and the person 21 next to the transmitter 100c. The transmitter 100c is the transmitter according to any one of the first to third embodiments, and includes one or more light emitting elements (eg, LEDs) 131. FIG. The transmitter 100c changes its luminance by blinking one or more light emitting elements 131, and transmits a light ID (light identification information) according to the luminance change. This light ID is the visible light signal described above.

The receiver 200 acquires a captured display image Pk in which the transmitter 100c and the person 21 are imaged with the normal exposure time. Furthermore, the receiver 200 obtains a decoding image by capturing images of the transmitter 100c and the person 21 with a communication exposure time that is shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the light ID from the transmitter 100c. Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P10 and the recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, the receiver 200 recognizes the area on the right side of the area where the robot, which is the transmitter 100c, is displayed as the target area. Specifically, receiver 200 identifies the distance between two markers 132a and 132b of transmitter 100c displayed in captured display image Pk. Then, receiver 200 recognizes an area having width and height corresponding to the distance as a target area. That is, the recognition information indicates the shapes of the markers 132a and 132b, and the position and size of the target area with reference to those markers 132a and 132b.

Then, the receiver 200 superimposes the AR image P10 on the target area, and displays the captured display image Pk on which the AR image P10 is superimposed on the display 201 . For example, the receiver 200 acquires an AR image P10 showing another robot different from the transmitter 100c. In this case, since the AR image P10 is superimposed on the target area of the captured display image Pk, the captured display image Pk can be displayed as if another robot actually exists next to the transmitter 100c. As a result, person 21 can be photographed with transmitter 100c and other robots, even if the other robots do not actually exist.

FIG. 71 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 1 of Embodiment 4. In FIG.

The transmitter 100 is configured as an image display device having a display panel, for example, as shown in FIG. 71, and transmits the light ID by changing the luminance while displaying a still image PS on the display panel. The display panel is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

The receiver 200 acquires the captured display image Pm and the decoding image by capturing the image of the transmitter 100 in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P11 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pm as a target area. For example, the receiver 200 recognizes the area where the display panel of the transmitter 100 is displayed as the target area. Then, the receiver 200 superimposes the AR image P11 on the target area, and displays the captured display image Pm on which the AR image P11 is superimposed on the display 201 . For example, the AR image P11 is a moving image having the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 as the first picture in display order. That is, the AR image P11 is a moving image starting from the still image PS.

In this case, since the AR image P11 is superimposed on the target area of the captured display image Pm, the receiver 200 displays the captured display image Pm as if an image display device displaying a moving image actually exists. can be done.

FIG. 72 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 1 of Embodiment 4. In FIG.

The transmitter 100 is configured as a station name sign as shown in FIG. 72, for example, and transmits the light ID by changing the luminance.

Receiver 200 images transmitter 100 from a position distant from transmitter 100, as shown in FIG. 72(a). As a result, the receiver 200 acquires the captured display image Pn and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR images P12 to P14 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes two areas corresponding to the recognition information in the captured display image Pn as the first and second target areas. For example, the receiver 200 recognizes the area around the transmitter 100 as the first area of interest. The receiver 200 then superimposes the AR image P12 on the first target region, and displays the captured display image Pn on which the AR image P12 is superimposed on the display 201 . For example, AR image P12 is an arrow prompting the user of receiver 200 to approach transmitter 100 .

In this case, the AR image P12 is superimposed on the first target area of the captured display image Pn and displayed, so the user approaches the transmitter 100 while pointing the receiver 200 toward the transmitter 100 . As the receiver 200 approaches the transmitter 100 in this manner, the area of the transmitter 100 displayed in the captured display image Pn (corresponding to the reference area described above) becomes larger. When the size of the area becomes equal to or larger than the first threshold, the receiver 200 moves to the second target area where the transmitter 100 is projected, as shown in FIG. 72(b), for example. The AR image P13 is superimposed. That is, the receiver 200 displays on the display 201 the captured display image Pn on which the AR images P12 and P13 are superimposed. For example, the AR image P13 is a message that informs the user of an overview of the station and its surroundings indicated by the station name sign. Also, the AR image P13 is equal in size to the area of the transmitter 100 displayed in the captured display image Pn.

Also in this case, the AR image P12, which is an arrow, is displayed superimposed on the first target area of the captured display image Pn. Close to 100. As the receiver 200 approaches the transmitter 100 in this manner, the area of the transmitter 100 displayed in the captured display image Pn (corresponding to the reference area described above) becomes even larger. When the size of the area becomes equal to or larger than the second threshold, the receiver 200 changes the AR image P13 superimposed on the second target area to the AR image P14, as shown in (c) of FIG. 72, for example. do. Furthermore, the receiver 200 deletes the AR image P12 superimposed on the first target area.

That is, the receiver 200 displays on the display 201 the captured display image Pn on which the AR image P14 is superimposed. For example, the AR image P14 is a message that informs the user of the details of the area around the station indicated by the station name sign. Also, the AR image P14 is equal in size to the area of the transmitter 100 displayed in the captured display image Pn. The area of this transmitter 100 is larger as the receiver 200 is closer to the transmitter 100 . Therefore, the AR image P14 is larger than the AR image P13.

In this way, the closer the receiver 200 is to the transmitter 100, the larger the AR image and the more information is displayed. Also, since an arrow like the AR image P12 prompting the user to approach is displayed, the user can easily understand that a lot of information will be displayed when the transmitter 100 is approached.

FIG. 73 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 1 of Embodiment 4. In FIG.

In the example shown in FIG. 72, the receiver 200 displays a lot of information when approaching the transmitter 100, but regardless of the distance from the transmitter 100, it displays a lot of information, for example, in the form of balloons. good too.

Specifically, as shown in FIG. 73, the receiver 200 captures the image of the transmitter 100 to acquire the captured display image Po and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P15 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Po as a target area. For example, the receiver 200 recognizes the area around the transmitter 100 as the target area. Then, the receiver 200 superimposes the AR image P15 on the target area, and displays the captured display image Po on which the AR image P15 is superimposed on the display 201 . For example, the AR image P15 is a message that informs the user of the details of the area around the station indicated by the station name sign in the form of a balloon.

In this case, since the AR image P15 is superimposed on the target area of the captured display image Po, the user of the receiver 200 can display a lot of information on the receiver 200 without getting close to the transmitter 100 .

FIG. 74 is a diagram showing another example of receiver 200 in Modification 1 of Embodiment 4. In FIG.

Receiver 200 is configured as a smartphone in the above example, but may be configured as a head-mounted display (also referred to as glasses) having an image sensor, as in the example shown in FIG.

Such a receiver 200 obtains an optical ID by decoding only a partial decoding target area of the decoding image. For example, the receiver 200 includes a line-of-sight detection camera 203, as shown in FIG. 74(a). The line-of-sight detection camera 203 captures the eyes of the user wearing the head-mounted display, which is the receiver 200 . The receiver 200 detects the line of sight of the user based on the image of the eye obtained by the sight line detection camera 203 .

As shown in FIG. 74(b), the receiver 200 displays the line-of-sight frame 204 such that the line-of-sight frame 204 appears in the area to which the detected line of sight is directed, for example, in the user's field of view. . Therefore, the line-of-sight frame 204 moves according to the movement of the user's line of sight. The receiver 200 treats an area corresponding to the line-of-sight frame 204 in the decoding image as a decoding target area. That is, even if there is a bright line pattern area outside the decoding target area in the decoding image, the receiver 200 does not decode the bright line pattern area, and decodes only the bright line pattern area within the decoding target area. . As a result, even if the image for decoding has a plurality of bright line pattern areas, all of the bright line pattern areas are not decoded, so that the processing load can be reduced and the display of unnecessary AR images can be suppressed. can be done.

Further, when the decoding image includes a plurality of bright line pattern areas for each outputting sound, the receiver 200 decodes only the bright line pattern area in the decoding target area, and decodes the bright line pattern area. You may output only the sound corresponding to . Alternatively, the receiver 200 decodes each of the plurality of bright line pattern regions included in the decoding image, outputs loudly the sound corresponding to the bright line pattern region within the decoding target region, and outputs the sound corresponding to the bright line pattern region outside the decoding target region. A corresponding sound may be output at a low volume. Also, when there are a plurality of bright line pattern areas outside the decoding target area, the receiver 200 may output the sound corresponding to the bright line pattern area closer to the decoding target area, the louder the sound corresponding to that bright line pattern area.

FIG. 75 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 1 of Embodiment 4. In FIG.

The transmitter 100 is configured as an image display device having a display panel, for example, as shown in FIG. 75, and transmits an optical ID by changing luminance while displaying an image on the display panel.

The receiver 200 acquires the captured display image Pp and the decoding image by imaging the transmitter 100 in the same manner as described above.

At this time, the receiver 200 identifies, from the captured display image Pp, an area that is located at the same position as the bright line pattern area in the decoding image and has the same size as the bright line pattern area. The receiver 200 may then display a scanning line P100 that repeatedly moves from one end of the area to the other.

While the scanning line P100 is being displayed, the receiver 200 obtains a light ID by decoding the decoding image, and transmits the light ID to the server. Then, the receiver 200 acquires the AR image and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pp as a target area.

When such a target area is recognized, the receiver 200 ends the display of the scanning line P100, superimposes an AR image on the target area, and displays the captured display image Pp on which the AR image is superimposed on the display 201. .

As a result, the moving scanning line P100 is displayed after the transmitter 100 captures the image and before the AR image is displayed. can be notified to the user.

FIG. 76 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 1 of Embodiment 4. In FIG.

Each of the two transmitters 100 is configured as an image display device having a display panel, for example, as shown in FIG. 76, and transmits the light ID by changing the luminance while displaying the same still image PS on the display panel. are doing. Here, the two transmitters 100 transmit different light IDs (for example, light IDs "01" and "02") by changing their brightness in different manners.

The receiver 200 acquires the captured display image Pq and the decoding image by capturing images of the two transmitters 100, as in the example shown in FIG. Receiver 200 obtains optical IDs “01” and “02” by decoding the decoding image. That is, the receiver 200 receives the light ID “01” from one of the two transmitters 100 and receives the light ID “02” from the other. Receivers 200 transmit their light IDs to the server. Then, the receiver 200 acquires the AR image P16 and the recognition information corresponding to the light ID "01" from the server. Further, the receiver 200 acquires the AR image P17 and the recognition information corresponding to the light ID "02" from the server.

The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pq as a target area. For example, the receiver 200 recognizes the area where the display panels of the two transmitters 100 are displayed as the target area. Then, the receiver 200 superimposes the AR image P16 on the target area corresponding to the light ID "01", and superimposes the AR image P17 on the target area corresponding to the light ID "02". Then, receiver 200 displays captured display image Pq on which AR images P16 and P17 are superimposed on display 201 . For example, the AR image P16 is a moving image having, as the first picture in display order, the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 corresponding to the light ID "01". be. The AR image P17 is a moving image having, as the first picture in display order, the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 corresponding to the light ID "02". be. That is, the leading pictures of the AR image P16 and the AR image P17, which are moving images, are the same. However, the AR image P16 and the AR image P17 are moving images different from each other, and pictures other than the top of each are different.

Therefore, since the AR image P16 and the AR image P17 are superimposed on the captured display image Pq, the receiver 200 can be used as if there were actually an image display device that displays different moving images reproduced from the same picture. , the captured display image Pq can be displayed.

77 is a flowchart showing an example of the processing operation of receiver 200 according to Modification 1 of Embodiment 4. FIG. The processing operation shown by the flow chart of FIG. 77 is specifically an example of the processing operation of the receiver 200 for individually imaging the transmitters 100 when there are two transmitters 100 shown in FIG. be.

First, the receiver 200 acquires a first light ID by capturing an image of the first transmitter 100 as a first subject (step S201). Next, the receiver 200 recognizes the first subject from the captured display image (step S202). That is, the receiver 200 acquires the first AR image and the first recognition information corresponding to the first light ID from the server, and recognizes the first subject based on the first recognition information. Then, the receiver 200 starts playing the first moving image, which is the first AR image, from the beginning (step S203). That is, the receiver 200 starts reproduction from the first picture of the first moving image.

Here, the receiver 200 determines whether or not the first subject is out of the captured display image (step S204). That is, the receiver 200 determines whether or not the first subject cannot be recognized from the captured display image. Here, when it is determined that the first subject is out of the captured and displayed image (Y in step S204), the receiver 200 suspends reproduction of the first moving image, which is the first AR image (step S205).

Next, the receiver 200 captures an image of the second transmitter 100 different from the first transmitter 100 as a second object, thereby obtaining a second optical ID different from the first optical ID acquired in step S201. is acquired (step S206). Here, when the receiver 200 determines that it has acquired the second light ID (Y in step S206), it performs the same processing as in steps S202 and S203 after the acquisition of the first light ID. That is, the receiver 200 recognizes the second subject from the captured display image (step S207). Then, the receiver 200 starts playing the second moving image, which is the second AR image corresponding to the second optical ID, from the beginning (step S208). That is, the receiver 200 starts reproduction from the top picture of the second moving image.

On the other hand, when the receiver 200 determines in step S206 that the second light ID has not been acquired (N in step S206), it determines whether or not the first subject has entered the captured display image again (step S209). That is, the receiver 200 determines whether or not the first subject has been recognized again from the captured display image. Here, when the receiver 200 determines that the first subject has entered the captured display image (Y in step S209), it further determines whether or not a predetermined time (that is, a predetermined time) has passed ( step S210). That is, the receiver 200 determines whether or not a predetermined period of time has elapsed from when the first subject leaves the captured display image to when it reenters the captured display image. Here, if it is determined that the predetermined time has not passed (Y in step S210), the receiver 200 starts playing the interrupted first moving image from the middle (step S211). It should be noted that the playback restart leading picture, which is the picture of the first moving image that is displayed first when playback is started from the middle, is the next picture that was displayed last when the playback of the first moving image was interrupted. may be the pictures in the display order of . Alternatively, the playback restart leading picture may be a picture preceding the last displayed picture by n (n is an integer equal to or greater than 1) in display order.

On the other hand, if it is determined that the predetermined time has passed (N of step S210), the receiver 200 starts playing the interrupted first moving image from the beginning (step S212).

Also, in the above example, the receiver 200 superimposes the AR image on the target area of the captured display image, but at this time, the brightness of the AR image may be adjusted. That is, the receiver 200 determines whether or not the brightness of the AR image acquired from the server matches the brightness of the target area of the captured display image. Then, if the receiver 200 determines that they do not match, the receiver 200 adjusts the brightness of the AR image so that the brightness of the AR image matches the brightness of the target area. The receiver 200 then superimposes the AR image whose brightness has been adjusted on the target area of the captured display image. As a result, the AR image to be superimposed can be brought closer to the image of the object that actually exists, and the user's sense of incongruity with respect to the AR image can be suppressed. The brightness of the AR image is the spatial average brightness of the AR image, and the brightness of the target area is also the spatial average brightness of the target area.

Further, as shown in FIG. 53, when the AR image is tapped, the receiver 200 may enlarge the AR image and display it on the entire display 201. FIG. Also, in the example shown in FIG. 53, the receiver 200 switches the AR image whose AR image is tapped to another AR image, but the AR image may be automatically switched regardless of the tap. For example, the receiver 200 switches the AR image to another AR image and displays it when the time during which the AR image is displayed elapses for a predetermined period of time. Further, when the current time reaches a predetermined time, the receiver 200 switches the AR image that has been displayed until then to another AR image and displays it. As a result, the user can easily view the new AR image without performing any operation.

[Modification 2 of Embodiment 4]
Modification 2 of Embodiment 4, that is, modification 2 of the display method for realizing AR using optical ID will be described below.

FIG. 78 is a diagram showing an example of a problem when displaying an AR image assumed in receiver 200 according to Embodiment 4 or Modification 1 thereof.

For example, receiver 200 in Embodiment 4 or Modification 1 thereof captures an image of a subject at time t1. Note that the above-described subject is a transmitter such as a television that transmits light IDs according to changes in brightness, or a poster, information board, signboard, or the like illuminated by light from the transmitter. As a result, the receiver 200 displays the entire image obtained by the effective pixel area of the image sensor (hereinafter referred to as the entire captured image) on the display 201 as a captured display image. At this time, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID in the captured display image as a target area on which the AR image is superimposed. A region of interest is, for example, a region showing an image of a transmitter such as a television or an image such as a poster. The receiver 200 then superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201 . Note that the AR image may be a still image or moving image, or may be a character string consisting of one or more characters or symbols.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, at time t2, the area corresponding to the target area (hereinafter referred to as the recognition area) on the image sensor protrudes from the effective pixel area. Note that the recognition area is an area in the effective pixel area of the image sensor onto which the image of the target area in the captured display image is projected. That is, the effective pixel area and the recognition area in the image sensor respectively correspond to the imaged display image and the target area in the display 201 .

When the recognition area protrudes from the effective pixel area, the receiver 200 cannot recognize the target area from the captured display image, and cannot display the AR image.

Therefore, the receiver 200 in this modified example acquires an image with a wider angle of view than the captured display image displayed on the entire display 201 as the entire captured image.

79 is a diagram showing an example in which receiver 200 displays an AR image according to Modification 2 of Embodiment 4. FIG.

The angle of view of the entire captured image of the receiver 200 according to this modification, that is, the angle of view of the effective pixel area of the image sensor is wider than the angle of view of the captured display image displayed on the entire display 201 . In the image sensor, the area corresponding to the image range displayed on the display 201 is hereinafter referred to as the display area.

For example, receiver 200 captures an image of a subject at time t1. As a result, the receiver 200 displays on the display 201 as a captured display image only an image obtained by a display area narrower than the effective pixel area among all captured images obtained by the effective pixel area of the image sensor. At this time, the receiver 200 recognizes the area corresponding to the recognition information acquired based on the light ID as the target area on which the AR image is superimposed, in the same manner as described above. The receiver 200 then superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201 .

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, the recognition area of the image sensor expands. At time t2, the recognition area protrudes from the display area of the image sensor. That is, the image of the target area (for example, the image of the poster) protrudes from the captured display image displayed on the display 201 . However, the recognition area in the image sensor does not protrude from the effective pixel area. In other words, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from the entire captured image, and superimposes a part of the AR image corresponding to only a part of the target area within the captured display image. displayed on the display 201.

Accordingly, even if the user approaches the subject to display the AR image in a large size and the target area protrudes from the captured display image, the AR image can be displayed continuously.

FIG. 80 is a flowchart showing an example of the processing operation of receiver 200 in Modification 2 of Embodiment 4. FIG.

The receiver 200 acquires all captured images and decoding images by capturing images of the subject with the image sensor (step S301). Next, the receiver 200 acquires an optical ID by decoding the decoding image (step S302). The receiver 200 then transmits the light ID to the server (step S303). Next, the receiver 200 acquires the AR image and recognition information corresponding to the light ID from the server (step S304). Next, the receiver 200 recognizes the area corresponding to the recognition information among all the captured images as the target area (step S305).

Here, the receiver 200 determines whether or not the recognition area corresponding to the image of the target area in the effective pixel area of the image sensor protrudes from the display area (step S306). Here, if it is determined that the area protrudes (Yes in step S306), the receiver 200 extracts a part of the AR image corresponding to only a part of the target area within the captured display image. display (step S307). On the other hand, when the receiver 200 determines that it does not protrude (No in step S306), the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed. (step S308).

Then, the receiver 200 determines whether or not to end the AR image display processing (step S309), and if it determines not to end it (No in step S309), it repeats the processing from step S305.

FIG. 81 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 2 of Embodiment 4. In FIG.

The receiver 200 may switch the screen display of the AR image according to the above ratio of the size of the recognition area to the display area.

If the horizontal width of the display area of the image sensor is w1, the vertical width is h1, and the horizontal width of the recognition area is w2, and the vertical width is h2, then the receiver uses the ratio (h2/h1 ) and (w2/w1) are compared to a threshold.

For example, when the receiver 200 displays a captured display image in which an AR image is superimposed on the target area as shown in (screen display 1) in FIG. For example, compare with 0.9). Then, when the larger ratio becomes 0.9 or more, the receiver 200 enlarges and displays the AR image on the entire display 201 as shown in (screen display 2) in FIG. When the recognition area becomes larger than the display area and even when it becomes larger than the effective pixel area, the receiver 200 continues to enlarge and display the AR image on the entire display 201 .

Further, for example, when the receiver 200 displays an enlarged AR image on the entire display 201 as shown in (screen display 2) in FIG. For example, compare with 0.7). The second threshold is less than the first threshold. Then, when the ratio of the larger one becomes 0.7 or less, the receiver 200 displays the captured display image in which the AR image is superimposed on the target area, as shown in (screen display 1) in FIG.

FIG. 82 is a flowchart showing another example of the processing operation of receiver 200 in Modification 2 of Embodiment 4. FIG.

The receiver 200 first performs optical ID processing (step S301a). This optical ID process is a process including steps S301 to S304 shown in FIG. Next, the receiver 200 recognizes the area corresponding to the recognition information in the captured display image as the target area (step S311). Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed (step S312).

Next, the receiver 200 determines that the ratio of the recognition regions, that is, the larger ratio of the ratios (h2/h1) and (w2/w1) is greater than or equal to the first threshold K (for example, K=0.9). It is determined whether or not (step S313). Here, if it is determined that it is not equal to or greater than the first threshold value K (No in step S313), the receiver 200 repeats the processing from step S311. On the other hand, if it is determined to be equal to or greater than the first threshold K (Yes in step S313), the receiver 200 enlarges and displays the AR image on the entire display 201 (step S314). At this time, the receiver 200 periodically switches the power of the image sensor on and off. Power saving of the receiver 200 can be achieved by periodically turning off the power of the image sensor.

Next, the receiver 200 determines whether the recognition area ratio is less than or equal to a second threshold L (for example, L=0.7) when the image sensor is periodically powered on. do. Here, if it is determined that it is not equal to or less than the second threshold value L (No in step S315), the receiver 200 repeats the processing from step S314. On the other hand, if it is determined to be equal to or less than the second threshold L (Yes in step S315), the receiver 200 superimposes an AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed. (step S316).

Then, the receiver 200 determines whether or not the AR image display processing should be terminated (step S317), and if it is determined not to be terminated (No in step S317), the processing from step S313 is repeatedly executed.

Thus, by setting the second threshold L to a value smaller than the first threshold K, the screen display of the receiver 200 can be frequently switched between (screen display 1) and (screen display 2). can be prevented and the state of the screen display can be stabilized.

In the examples shown in FIGS. 81 and 82, the display area and the effective pixel area may be the same or different. Also, in the example, the ratio of the size of the recognition area to the display area is used, but if the display area and the effective pixel area are different, instead of the display area, A ratio may be used.

FIG. 83 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 2 of Embodiment 4. In FIG.

In the example shown in FIG. 83, similar to the example shown in FIG. 79, the image sensor of the receiver 200 has an effective pixel area wider than the display area.

For example, receiver 200 captures an image of a subject at time t1. As a result, the receiver 200 displays on the display 201 as a captured display image only an image obtained by a display area narrower than the effective pixel area among all captured images obtained by the effective pixel area of the image sensor. At this time, the receiver 200 recognizes the area corresponding to the recognition information acquired based on the light ID as the target area on which the AR image is superimposed, in the same manner as described above. The receiver 200 then superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201 .

Here, when the user changes the direction of the receiver 200 (specifically, the image sensor), the recognition area of the image sensor moves, for example, toward the upper left in FIG. 83, and protrudes from the display area at time t2. That is, the image of the target area (for example, the image of the poster) protrudes from the captured display image displayed on the display 201 . However, the recognition area in the image sensor does not protrude from the effective pixel area. In other words, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from the entire captured image, and superimposes a part of the AR image corresponding to only a part of the target area within the captured display image. and displayed on the display 201. Furthermore, the receiver 200 changes the size and position of a portion of the displayed AR image according to the movement of the recognition area on the image sensor, that is, the movement of the target area in the entire captured image.

Further, when the recognition area protrudes from the display area as described above, the receiver 200 detects the number of pixels corresponding to the distance between the edge of the effective pixel area and the edge of the display area (hereinafter referred to as the inter-area distance). is compared with a threshold.

For example, the distance between the upper side of the effective pixel area and the upper side of the display area and the distance between the lower side of the effective pixel area and the lower side of the display area, whichever is shorter (hereinafter referred to as the first Let dh be the number of pixels corresponding to distance). Also, the distance between the left side of the effective pixel area and the left side of the display area and the distance between the right side of the effective pixel area and the right side of the display area, whichever is shorter (hereinafter referred to as the second Let dw be the number of pixels corresponding to distance). At this time, the inter-region distance described above is the shorter one of the first and second distances.

That is, the receiver 200 compares the number of pixels dw and dh, whichever is smaller, with the threshold value N. FIG. Then, for example, at time t2, if the smaller number of pixels becomes equal to or less than the threshold value N, the receiver 200 changes the size and position of part of the AR image according to the position of the recognition area on the image sensor. fixed without That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 changes the size and position of the part of the AR image to be displayed to the size and position of the part of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold N. length and position.

Therefore, at time t3, even if the recognition area moves further and protrudes from the effective pixel area, receiver 200 continues to display part of the AR image as at time t2. That is, as long as the number of pixels dw and dh, whichever is smaller, is equal to or smaller than the threshold value N, the receiver 200 captures a portion of the AR image whose size and position are fixed, as at time t2. It is superimposed on the display image and continues to be displayed.

In the example shown in FIG. 83, receiver 200 changes the size and position of part of the displayed AR image according to the movement of the recognition area on the image sensor, but changes the display magnification and position of the entire AR image. You can change it.

FIG. 84 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 2 of Embodiment 4. In FIG. Specifically, FIG. 84 shows an example in which the display magnification of the AR image is changed.

For example, as in the example shown in FIG. 83, when the user changes the orientation of the receiver 200 (specifically, the image sensor) from the state of time t1, the recognition area on the image sensor moves, for example, toward the upper left in FIG. It moves out of the display area at time t2. That is, the image of the target area (for example, the image of the poster) protrudes from the captured display image displayed on the display 201 . However, the recognition area in the image sensor does not protrude from the effective pixel area. In other words, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all captured images.

Therefore, in the example shown in FIG. 84, the receiver 200 adjusts the display magnification of the AR image so that the size of the entire AR image matches the size of a part of the target area within the captured display image. to change That is, the receiver 200 reduces the AR image. Then, the receiver 200 superimposes the AR image whose display magnification has been changed (that is, reduced) on the area and displays it on the display 201 . Furthermore, the receiver 200 changes the display magnification and position of the displayed AR image according to the movement of the recognition area on the image sensor, that is, the movement of the target area in the entire captured image.

Further, when the recognition area protrudes from the display area as described above, the receiver 200 compares the smaller number of pixels dw and dh with the threshold value N. FIG. Then, for example, at time t2, if the smaller number of pixels becomes equal to or less than the threshold value N, the receiver 200 does not change the display magnification and position of the AR image according to the position of the recognition area on the image sensor. do. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 fixes the display magnification and position of the AR image to be displayed to the display magnification and position of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. .

Therefore, at time t3, even if the recognition area moves further and protrudes from the effective pixel area, the receiver 200 continues to display the AR image as at time t2. That is, as long as the number of pixels dw and dh, whichever is smaller, is equal to or less than the threshold value N, the receiver 200 converts the AR image with the fixed display magnification and position into the captured and displayed image, as at time t2. It continues to be superimposed and displayed.

In the above example, the smaller of the numbers of pixels dw and dh is compared with the threshold, but the ratio of the smaller number of pixels may be compared with the threshold. The ratio of the number of pixels dw is, for example, the ratio (dw/w0) of the number of pixels dw to the number of pixels w0 in the horizontal direction of the effective pixel area. Similarly, the ratio of the number of pixels dh is, for example, the ratio (dh/h0) of the number of pixels dh to the number of pixels h0 in the vertical direction of the effective pixel area. Alternatively, instead of the number of pixels in the horizontal or vertical direction of the effective pixel area, the number of pixels in the horizontal or vertical direction of the display area may be used to express the respective ratios of the number of pixels dw and dh. A threshold to be compared with the ratio of the numbers of pixels dw and dh is, for example, 0.05.

Alternatively, the smaller angle of view of the number of pixels dw and dh may be compared with the threshold. When the number of pixels on the diagonal line of the effective pixel area is m and the angle of view corresponding to the diagonal is θ (for example, 55°), the angle of view corresponding to the number of pixels dw is θ×dw/m, The angle of view corresponding to the number of pixels dh is θ×dh/m.

Also, in the examples shown in FIGS. 83 and 84, the receiver 200 switches the screen display of the AR image based on the inter-area distance between the effective pixel area and the recognition area. You may switch the screen display of an AR image based on the relationship of .

FIG. 85 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 2 of Embodiment 4. In FIG. Specifically, FIG. 85 shows an example of switching the screen display of the AR image based on the relationship between the display area and the recognition area. Also, in the example shown in FIG. 85, the image sensor of the receiver 200 has an effective pixel area wider than the display area, as in the example shown in FIG.

For example, receiver 200 captures an image of a subject at time t1. As a result, the receiver 200 displays on the display 201 as a captured display image only an image obtained by a display area narrower than the effective pixel area among all captured images obtained by the effective pixel area of the image sensor. At this time, the receiver 200 recognizes the area corresponding to the recognition information acquired based on the light ID as the target area on which the AR image is superimposed, in the same manner as described above. The receiver 200 then superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201 .

Here, when the user changes the orientation of the receiver 200, the receiver 200 changes the position of the displayed AR image according to the movement of the recognition area on the image sensor. Then, for example, the recognition area in the image sensor moves, for example, in the upper left direction in FIG. 85, and at time t2, part of the edge of the recognition area coincides with part of the edge of the display area. That is, the image of the target area (for example, an image of a poster or the like) is arranged at the corner of the captured display image displayed on the display 201 . As a result, the receiver 200 superimposes the AR image on the target area at the corner of the captured display image and displays it on the display 201 .

Then, when the recognition area moves further and protrudes from the display area, receiver 200 fixes the size and position of the AR image displayed at time t2 without changing. That is, the receiver 200 switches the screen display of the AR image.

Therefore, at time t3, even if the recognition area moves further and protrudes from the effective pixel area, the receiver 200 continues to display the AR image as at time t2. That is, as long as the recognition area protrudes from the display area, the receiver 200 superimposes an AR image of the same size as at time t2 on the same position as at time t2 in the captured display image. continue to display.

Thus, in the example shown in FIG. 85, the receiver 200 switches the screen display of the AR image depending on whether the recognition area protrudes from the display area. Also, the receiver 200 may use, instead of the display area, a determination area that includes the display area and is larger than the display area and smaller than the effective pixel area. In this case, the receiver 200 switches the screen display of the AR image depending on whether the recognition area protrudes from the determination area.

The screen display of the AR image has been described above using FIGS. The AR image of the size of the area may be superimposed on the captured display image and displayed.

FIG. 86 is a diagram showing another example in which receiver 200 displays an AR image according to Modification 2 of Embodiment 4. In FIG.

In the example shown in FIG. 49, the receiver 200 acquires the captured display image Pe and the decoding image in the same manner as described above by capturing an image of the guide board 107 illuminated by the transmitter 100 . The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the light ID from the guide plate 107 . However, if the entire surface of the guide plate 107 is a color that absorbs light (for example, a dark color), the surface is dark even when illuminated by the transmitter 100, so the receiver 200 correctly receives the light ID. may not be possible. Alternatively, even if the entire surface of guide plate 107 is striped like a decoding image (that is, a bright line image), receiver 200 may not be able to receive the light ID correctly.

Therefore, as shown in FIG. 86, a reflector 109 may be arranged near the guide plate 107 . Accordingly, the receiver 200 can receive light reflected by the reflector 109 from the transmitter 100 , that is, visible light (specifically, light ID) transmitted from the transmitter 100 . As a result, the receiver 200 can appropriately receive the light ID and display the AR image P5.

[Summary of Modifications 1 and 2 of Embodiment 4]
FIG. 87A is a flowchart illustrating a display method according to one aspect of the present invention.

A display method according to an aspect of the present invention includes steps S41 to S43.

In step S41, a captured image is obtained by capturing an image of an object illuminated by a transmitter that transmits a signal according to a change in luminance of light with an imaging sensor as a subject. In step S42, signals are decoded from the captured image. In step S43, the moving image corresponding to the decoded signal is read out from the memory, the moving image is superimposed on the target area corresponding to the subject in the captured image, and displayed on the display. Here, in step S43, one of a plurality of images included in the moving image is an image including the object and a predetermined display time preceding or following the image including the object. A moving image is displayed from any one of the plurality of images. For example, the predetermined number is ten frames. Alternatively, the object is a still image, and in step S43, the moving image is displayed from the same image as the still image. It should be noted that the image at which the moving image starts to be displayed is not limited to the same image as the still image, i.e., the image that is the same as the still image, i.e., the image that includes the object is located before or after the predetermined number of frames in display order. It may be an image. Also, the object is not limited to a still image, and may be a doll or the like.

Note that the imaging sensor and the captured image are, for example, the image sensor and the entire captured image in the fourth embodiment. The still image to be illuminated may be a still image displayed on the display panel of the image display device, or may be a poster, information board, signboard, or the like illuminated by the light from the transmitter.

Moreover, such a display method may further include a transmitting step of transmitting the signal to the server and a receiving step of receiving the moving image corresponding to the signal from the server.

As a result, as shown in FIG. 71, for example, a moving image can be displayed in virtual reality as if a still image starts to move, and an image beneficial to the user can be displayed.

Further, the still image has an outer frame of a predetermined color, and the display method according to one aspect of the present invention may further include a recognition step of recognizing the target area from the captured image by the predetermined color. In this case, in step S43, the moving image may be resized so as to have the same size as the recognized target area, and the resized moving image may be superimposed on the target area in the captured image and displayed on the display. good. For example, the predetermined colored outer frame is a white or black rectangular frame surrounding the still image, and is indicated by the recognition information in the fourth embodiment. Then, the AR image in Embodiment 4 is resized as a moving image and superimposed.

As a result, the moving image can be displayed more realistically so that the moving image actually exists as a subject.

Further, only the image projected onto the display area, which is smaller than the imaging area of the imaging sensor, is displayed on the display. In this case, in step S43, if the projection area in which the subject is projected in the imaging area is larger than the display area, the image obtained by the part of the projection area exceeding the display area is displayed on the display. You don't have to. Here, for example, as shown in FIG. 79, the imaging area and projection area are the effective pixel area and recognition area of the image sensor.

As a result, for example, as shown in FIG. 79, when the imaging sensor approaches a still image that is a subject, even if part of the image obtained by the projection area (recognition area in FIG. 79) is not displayed on the display, the subject In some cases, the entire still image is projected onto the imaging area. Therefore, in this case, it is possible to appropriately recognize the still image as the subject, and appropriately superimpose the moving image on the target area corresponding to the subject in the captured image.

Also, for example, the horizontal and vertical widths of the display area are w1 and h1, respectively, and the horizontal and vertical widths of the projection area are w2 and h2, respectively. In this case, in step S43, if the larger value of h2/h1 or w2/w1 is equal to or greater than a predetermined value, the moving image is displayed on the full screen of the display, and either h2/h1 or w2/w1 is displayed. When the larger value is smaller than the predetermined value, the moving image may be superimposed on the target area in the captured image and displayed on the display.

Accordingly, as shown in FIG. 81, for example, when the imaging sensor approaches the still image, which is the subject, the moving image is displayed on the full screen. No need to display. Therefore, it is possible to prevent the signal from being unable to be decoded due to the projection area (recognition area in FIG. 81) protruding from the imaging area (effective pixel area) due to the imaging sensor being too close to the still image. .

Further, the display method according to one aspect of the present invention may further include a control step of turning off the operation of the imaging sensor when the moving image is displayed on the full screen of the display.

As a result, power consumption of the image sensor can be suppressed by turning off the operation of the image sensor, for example, as shown in step S314 of FIG.

Further, in step S43, when the target area cannot be recognized from the captured image due to the movement of the imaging sensor, the moving image is displayed in the same size as the size of the target area that was recognized immediately before it became unrecognizable. You may Note that the target area cannot be recognized from the captured image is, for example, a situation in which at least part of the target area corresponding to the still image that is the subject is not included in the captured image. In this way, when the target area cannot be recognized, a moving image having the same size as the previously recognized target area is displayed, as at time t3 in FIG. 85, for example. Therefore, it is possible to prevent at least part of the moving image from not being displayed due to the movement of the imaging sensor.

Further, in step S43, when only part of the target area is included in the area displayed on the display of the captured image due to the movement of the imaging sensor, the part of the target area is displayed. A part of the spatial area of the moving image may be superimposed on a part of the target area and displayed on the display. A part of the spatial region of the moving image is a part of each picture forming the moving image.

As a result, only part of the spatial region of the moving image (the AR image in FIG. 83) is displayed on the display, for example, at time t2 in FIG. As a result, it is possible to inform the user that the imaging sensor is not properly directed to the still image that is the subject.

Further, in step S43, when the target area cannot be recognized from the captured image due to the movement of the imaging sensor, the space of the moving image corresponding to the part of the target area displayed immediately before the target area becomes unrecognizable is displayed. A portion of the region may be displayed continuously.

As a result, even when the user points the imaging sensor in a direction different from the still image as the subject, for example, at time t3 in FIG. part is displayed continuously. As a result, it is possible to make it easier for the user to understand how the imaging sensor should be directed to display the entire moving image.

Further, in step S43, the horizontal and vertical widths of the imaging area of the imaging sensor are w0 and h0, respectively, and the horizontal distance between the projection area in which the subject is projected in the imaging area and the imaging area. If the distances in the direction and the vertical direction are dh and dw, respectively, and the smaller value of dw/w0 or dh/h0 is equal to or less than a predetermined value, it is determined that the target area cannot be recognized. good. Note that the projection area is, for example, the recognition area shown in FIG. Alternatively, in step S43, the angle of view corresponding to the shorter of the horizontal and vertical distances between the projection area in which the subject is projected in the imaging area of the imaging sensor and the imaging area is predetermined. It may be determined that the target area cannot be recognized if the value is less than or equal to the value.

This makes it possible to appropriately determine whether or not the target area can be recognized.

FIG. 87B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention.

A display device A10 according to one aspect of the present invention includes an imaging sensor A11, a decoding unit A12, and a display control unit A13.

The imaging sensor A11 acquires a captured image by capturing, as a subject, a still image that is lit up by a transmitter that transmits a signal based on changes in the luminance of light.

The decoding unit A12 is a decoding unit that decodes a signal from the captured image.

The display control unit A13 reads the moving image corresponding to the decoded signal from the memory, superimposes the moving image on the target area corresponding to the subject in the captured image, and displays it on the display. Here, the display control unit A13 sequentially displays the plurality of images from the first image, which is the same image as the still image, among the plurality of images included in the moving image.

Thereby, the same effect as the display method described above can be obtained.

Further, the imaging sensor A11 may include a plurality of micromirrors and a photosensor, and the display device A10 may further include an imaging control section that controls the imaging sensor. In this case, the imaging control unit identifies a region containing a signal in the captured image as a signal region, and controls the angle of the micromirror corresponding to the identified signal region among the plurality of micromirrors. Then, the imaging control unit causes the photosensor to receive only light reflected by the micromirrors whose angles are controlled among the plurality of micromirrors.

As a result, even if a visible light signal, which is a signal represented by a change in luminance of light, contains high frequency components, the high frequency components can be correctly decoded.

In each of the above-described embodiments and modifications, each component may be implemented by dedicated hardware, or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. For example, the program causes the computer to execute the display method illustrated by the flow charts of FIGS. 77, 80, 82 and 87A.

Although the display method according to one or more aspects has been described based on the above embodiments and modifications, the present invention is not limited to these embodiments. As long as it does not deviate from the spirit of the present invention, various modifications that a person skilled in the art can think of are applied to this embodiment, and a form constructed by combining the components of different embodiments and modifications is also within the scope of the present invention. may be included in

[Modification 3 of Embodiment 4]
Hereinafter, Modification 3 of Embodiment 4, that is, Modification 3 of the display method for realizing AR using optical ID will be described.

FIG. 88 is a diagram showing an example of enlarging and moving an AR image.

As shown in (a) of FIG. 88, the receiver 200 superimposes an AR image P21 on the target area of the captured display image Ppre, as in the fourth embodiment or its first or second modification. Then, the receiver 200 displays on the display 201 the captured display image Ppre on which the AR image P21 is superimposed. For example, the AR image P21 is a moving image.

Here, as shown in FIG. 88(b), the receiver 200 changes the size of the AR image P21 according to the instruction upon receiving the instruction to change the size. For example, when receiving an enlargement instruction, the receiver 200 enlarges the AR image P21 according to the instruction. An instruction to change the size is given by the user, for example, by pinching, double-tapping, or long-pressing the AR image P21. Specifically, when the receiver 200 receives an instruction to expand by pinching out, the receiver 200 expands the AR image P21 according to the instruction. Conversely, when receiver 200 accepts an instruction to reduce by pinch-in, it reduces AR image P21 according to the instruction.

Further, as shown in (c) of FIG. 88, the receiver 200 receives an instruction to change the position, and changes the position of the AR image P21 in accordance with the instruction. The instruction to change the position is made by the user, for example, by swiping the AR image. Specifically, when the receiver 200 receives an instruction to change the position by swiping, the receiver 200 changes the position of the AR image P21 according to the instruction. That is, the AR image P21 moves.

Accordingly, by enlarging the AR image, which is a moving image, the AR image can be made easier to see, and by reducing or moving the AR image, which is a moving image, the area of the captured display image Ppre that is hidden in the AR image can be removed. can be displayed to the user.

FIG. 89 is a diagram showing an example of enlarging an AR image.

As shown in (a) of FIG. 89, the receiver 200 superimposes an AR image P22 on the target area of the captured display image Ppre, as in the fourth embodiment or its first or second modification. Then, the receiver 200 displays on the display 201 the captured display image Ppre on which the AR image P22 is superimposed. For example, the AR image P22 is a still image in which a character string is written.

Here, as shown in (b) of FIG. 89, the receiver 200 changes the size of the AR image P22 according to the instruction upon receiving the instruction to change the size. For example, when receiving an enlargement instruction, the receiver 200 enlarges the AR image P22 according to the instruction. An instruction to change the size is given by the user, for example, by pinching, double-tapping, or long-pressing the AR image P22, as described above. Specifically, when receiver 200 receives an instruction to expand by pinching out, receiver 200 expands AR image P22 in accordance with the instruction. By enlarging the AR image P22, it is possible to make the character string described in the AR image P22 easier for the user to read.

Further, as shown in (c) of FIG. 89, the receiver 200 changes the size of the AR image P22 according to the instruction when receiving the instruction to change the size. For example, when receiver 200 receives an instruction for further enlargement, receiver 200 further enlarges AR image P22 in accordance with the instruction. By enlarging the AR image P22, it is possible to make the character string described in the AR image P22 easier for the user to read.

It should be noted that receiver 200 may acquire a high-resolution AR image if the enlargement ratio of the AR image according to the instruction is equal to or greater than a threshold when receiving an enlargement instruction. In this case, the receiver 200 may enlarge the high-resolution AR image to the above-mentioned magnification ratio and display it instead of the original AR image that has already been displayed. For example, the receiver 200 displays an AR image of 1920×1080 pixels instead of an AR image of 640×480 pixels. As a result, the AR image can be enlarged as if the AR image were actually captured as a subject, and a high-resolution image that cannot be obtained with optical zoom can be displayed.

FIG. 90 is a flow chart showing an example of processing operations for enlarging and moving an AR image by the receiver 200. FIG.

First, the receiver 200 starts imaging with the normal exposure time and the communication exposure time (step S401), as in step S101 shown in the flowchart of FIG. When this imaging is started, a captured display image Ppre in the normal exposure time and a decoding image (that is, a bright line image) Pdec in the communication exposure time are obtained periodically. The receiver 200 then acquires the optical ID by decoding the decoding image Pdec.

Next, the receiver 200 performs AR image superimposition processing including the processing of steps S102 to S106 shown in the flowchart of FIG. 45 (step S402). When this AR image superimposition processing is performed, the AR image is superimposed on the captured display image Ppre and displayed. At this time, the receiver 200 reduces the light ID acquisition rate (step S403). The light ID acquisition rate is the ratio of the number of images for decoding (that is, bright line images) Pdec to the number of captured images per unit time obtained by the imaging started in step S401. For example, by decreasing the optical ID acquisition rate, the number of decoding images Pdec obtained per unit time becomes smaller than the number of captured display images Ppre obtained per unit time.

Next, the receiver 200 determines whether or not an instruction to change the size has been received (step S404). Here, if it is determined that a size change instruction has been received (Yes in step S404), the receiver 200 further determines whether or not the size change instruction is an enlargement instruction (step S405). When determining that the size change instruction is an enlargement instruction (Yes in step S405), the receiver 200 further determines whether or not the AR image needs to be acquired again (step S406). For example, when the receiver 200 determines that the enlargement ratio of the AR image in response to the enlargement instruction is greater than or equal to the threshold value, the receiver 200 determines that the AR image needs to be acquired again. Here, when the receiver 200 determines that reacquisition is necessary (Yes in step S406), it acquires a high-resolution AR image from, for example, a server, and displays the superimposed AR image as the high-resolution Replace with AR image (step S407).

Then, the receiver 200 changes the size of the AR image in accordance with the accepted size change instruction (step S408). That is, when the high-resolution AR image is acquired in step S407, the receiver 200 enlarges the high-resolution AR image. If it is determined in step S406 that the AR image does not need to be acquired again (No in step S406), the receiver 200 enlarges the superimposed AR image. Further, when it is determined in step S405 that the size change instruction is a reduction instruction (No in step S405), the receiver 200 superimposes a Reduce the displayed AR image.

On the other hand, if the receiver 200 determines in step S404 that the size change instruction has not been received (No in step S404), it determines whether or not the position change instruction has been received (step S409). Here, if it is determined that an instruction to change the position has been received (Yes in step S409), the receiver 200 changes the position of the superimposed and displayed AR image in accordance with the instruction to change the position (step S410). ). That is, receiver 200 moves the AR image. If it is determined that a position change instruction has not been received (No in step S409), the receiver 200 repeats the processing from step S404.

When the size of the AR image is changed in step S408, or when the position of the AR image is changed in step S410, the receiver 200 stops acquiring the light ID periodically acquired from step S401. It is determined whether or not (step S411). Here, if it is determined that the light ID is no longer acquired (Yes in step S411), the receiver 200 terminates the processing operations regarding enlargement and movement of the AR image. On the other hand, when determining that the light ID is still acquired (No in step S411), the receiver 200 repeats the process from step S404.

FIG. 91 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

The receiver 200 superimposes the AR image P23 on the target area in the captured display image Ppre, as described above. Here, as shown in FIG. 91, the AR image P23 is configured such that the closer each part of the AR image P23 is to the edge of the AR image P23, the higher the transmittance at that part. Transmittance is the degree to which the superimposed image is displayed transparently. For example, when the overall transmittance of the AR image is 100%, even if the AR image is superimposed on the target area of the captured display image, the AR image is not displayed on the display 201 and only the target area is displayed. means that Conversely, when the transmittance of the entire AR image is 0%, it means that the target area of the captured display image is not displayed on the display 201, and only the AR image superimposed on the target area is displayed. .

For example, when the AR image P23 is rectangular, the transmittance of each part in the AR image P23 is higher as the part is closer to the upper end, lower end, left end, or right end of the rectangle. More specifically, the transmittance at those edges is 100%. In addition, in the central part of the AR image P23, there is a rectangular area with a transmittance of 0%, which is smaller than the AR image P23. That is, in the peripheral portion of the AR image P23, the transmittance changes stepwise from 0% to 100% like gradation.

The receiver 200 superimposes such an AR image P23 on the target area in the captured display image Ppre as shown in FIG. At this time, the receiver 200 adjusts the size of the AR image P23 to the size of the target area, and superimposes the resized AR image P23 on the target area. For example, in the target area, an image of a station name sign with the same background color as the rectangular area in the center of the AR image P23 appears. The name of the station is written as 'Kyoto' in Japanese.

Here, as described above, the transmittance of each portion of the AR image P23 is higher as the portion is closer to the edge of the AR image P23. Therefore, when the AR image P23 is superimposed on the target area, even if the central rectangular area of the AR image P23 is displayed, the edge of the AR image P23 is not displayed, and the edge of the target area, that is, the station name sign, is displayed. The edges of the image are displayed.

This makes it possible to make the deviation between the AR image P23 and the target area inconspicuous. In other words, even if the AR image P23 is superimposed on the target area, a shift may occur between the AR image P23 and the target area due to the movement of the receiver 200 or the like. In this case, if the overall transmittance of the AR image P23 is 0%, the edge of the AR image P23 and the edge of the target area are displayed, and the deviation therebetween is conspicuous. However, in the AR image P23 in this modified example, since the transmittance of the portion closer to the edge is higher, the edge of the AR image P23 can be made difficult to be displayed. It is possible to make the gap between them inconspicuous. Furthermore, since the transmittance changes like a gradation in the peripheral portion of the AR image P23, it is possible to make it inconspicuous that the AR image P23 is superimposed on the target area.

FIG. 92 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

The receiver 200 superimposes the AR image P24 on the target area in the captured display image Ppre, as described above. Here, as shown in FIG. 92, the subject to be imaged is, for example, the menu of a restaurant. This menu is surrounded by a white frame, and the white frame is further surrounded by a black frame. That is, the subject includes a menu, a white frame surrounding the menu, and a black frame surrounding the white frame.

The receiver 200 recognizes an area larger than the white-framed image and smaller than the black-framed image in the captured display image Ppre as a target area. Then, the receiver 200 adjusts the size of the AR image P24 to the size of the target area, and superimposes the resized AR image P24 on the target area.

As a result, even if the superimposed AR image P24 deviates from the target area due to movement of the receiver 200, etc., the AR image P24 can continue to be displayed while being surrounded by a black frame. Therefore, it is possible to make the deviation between the AR image P24 and the target area inconspicuous.

In the example shown in FIG. 92, the color of the frame is black or white, but it is not limited to these colors and may be any color.

FIG. 93 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

For example, receiver 200 captures, as a subject, a poster depicting a castle lit up in the night sky. For example, the poster is illuminated by the above-described transmitter 100 configured as a backlight, which transmits a visible light signal (ie light ID). The receiver 200 acquires the imaged display image Ppre including the image of the subject, which is the poster, and the AR image P25 corresponding to the light ID. Here, the AR image P25 has the same shape as the image of the poster from which the area where the castle is drawn is extracted. That is, the area corresponding to the poster image castle in the AR image P25 is masked. Furthermore, the AR image P25 is configured such that the nearer each part of the AR image P25 is to the edge of the AR image P25, the higher the transmittance at that part is, as in the case of the AR image P23 described above. In addition, in the central part of the AR image P25 where the transmittance is 0%, fireworks launched into the night sky are displayed as a moving image.

The receiver 200 matches the size of the AR image P25 to the size of the target area, which is the image of the subject, and superimposes the resized AR image P25 on the target area. As a result, the castle drawn on the poster is displayed not as an AR image but as an image of the subject, and furthermore, a moving image of fireworks is displayed as an AR image.

As a result, the captured display image Ppre can be displayed as if fireworks were actually set off in the poster. Also, the transmittance of each portion of the AR image P25 is higher as the portion is closer to the edge of the AR image P25. Therefore, when the AR image P25 is superimposed on the target area, the edge of the AR image P25 is not displayed, and the edge of the target area is displayed, even if the central portion of the AR image P25 is displayed. As a result, it is possible to make the deviation between the AR image P25 and the target area inconspicuous. Furthermore, since the transmittance changes like a gradation in the peripheral portion of the AR image P25, it is possible to make it less noticeable that the AR image P25 is superimposed on the target area.

FIG. 94 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

For example, the receiver 200 takes an image of the transmitter 100 configured as a television as a subject. Specifically, this transmitter 100 displays a castle illuminated in the night sky on its display and transmits a visible light signal (that is, light ID). The receiver 200 acquires the imaged display image Ppre in which the transmitter 100 is displayed and the AR image P26 corresponding to the light ID. Here, the receiver 200 first displays the captured display image Ppre on the display 201 . At this time, the receiver 200 also displays on the display 201 a message m prompting the user to turn off the light. Specifically, the message m is, for example, "Please turn off the room lighting and darken the room."

When the display of this message m turns off the light of the user and the room in which the transmitter 100 is installed becomes dark, the receiver 200 displays the AR image P26 superimposed on the captured display image Ppre. Here, the AR image P26 has the same size as the imaged display image Ppre, and the area corresponding to the castle of the imaged display image Ppre in the AR image P26 is hollowed out. That is, the area corresponding to the castle of the captured display image Ppre in the AR image P26 is masked. Therefore, the castle of the captured display image Ppre can be shown to the user from that area. Also, in the peripheral portion of the area in the AR image P26, the transmittance may change stepwise from 0% to 100% like gradation, as described above. In this case, it is possible to make the deviation between the captured display image Ppre and the AR image P26 inconspicuous.

In the above example, by superimposing an AR image with a high peripheral transmittance on the target area of the captured display image Ppre, the shift between the AR image and the target area is made less noticeable. However, instead of such an AR image, an AR image that has the same size as the captured display image Ppre and is entirely translucent (that is, has a transmittance of 50%) may be superimposed on the captured display image Ppre. Even in this case, it is possible to make the shift between the AR image and the target area inconspicuous. Further, when the captured display image Ppre is bright as a whole, an AR image with uniformly low transparency is superimposed on the captured display image Ppre. A high AR image may be superimposed on the captured display image Ppre.

Objects such as fireworks in the AR image P25 and the AR image P26 may be represented by CG (computer graphics). In this case, masking can be dispensed with. In addition, in the example shown in FIG. 94, the receiver 200 displays a message m prompting the user to turn off the light, but the light may be turned off automatically without such a display. For example, the receiver 200 outputs a turn-off signal to the lighting device to which the transmitter 100, which is a television, is set by Bluetooth (registered trademark), ZigBee, or a specified low-power radio station. As a result, the lighting device is automatically turned off.

FIG. 95A is a diagram showing an example of an imaged display image Ppre obtained by imaging by the receiver 200. FIG.

For example, transmitter 100 is configured as a large display installed in a stadium. The transmitter 100 then displays a message indicating that, for example, fast food and drinks can be ordered with the light ID, and transmits a visible light signal (ie, light ID). When such a message is displayed, the user points the receiver 200 toward the transmitter 100 and takes an image. In other words, receiver 200 captures an image of transmitter 100, which is configured as a large display installed in a stadium, as a subject.

The receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 acquires the light ID by decoding the decoding image Pdec, and transmits the light ID and the captured display image Ppre to the server.

For each light ID, the server identifies the installation information of the imaged large display associated with the light ID transmitted from the receiver 200 from among the installation information associated with the light ID. For example, the installation information indicates the position and orientation of the large display installed, the size of the large display, and the like. Furthermore, the server identifies the number of the seat in the stadium where the imaged display image Ppre was captured, based on the size and orientation of the large display displayed in the imaged display image Ppre and the installation information. do. Then, the server causes the receiver 200 to display a menu screen including the seat number.

95B is a diagram showing an example of a menu screen displayed on display 201 of receiver 200. As shown in FIG.

The menu screen m1 includes, for each product, for example, an input field ma1 for inputting the number of orders for that product, a seat field mb1 for entering the stadium seat number specified by the server, and an order button mc1. . By operating the receiver 200, the user inputs the order quantity of the desired product in the input field ma1 corresponding to the desired product, and selects the order button mc1. As a result, the order is confirmed, and the receiver 200 transmits the order content according to the input result to the server.

When the server receives the order details, the server instructs the stadium staff to deliver the ordered number of products according to the order details to the seats with the specified numbers as described above.

FIG. 96 is a flow chart showing an example of processing operations between the receiver 200 and the server.

The receiver 200 first captures an image of the transmitter 100 configured as a large stadium display (step S421). The receiver 200 acquires the light ID transmitted from the transmitter 100 by decoding the decoding image Pdec obtained by the imaging (step S422). The receiver 200 transmits the light ID obtained in step S422 and the captured display image Ppre obtained by the imaging in step S421 to the server (step S423).

When the server receives the light ID and the captured display image Ppre (step S424), the server identifies the installation information of the large display installed in the stadium based on the light ID (step S425). For example, the server holds, for each light ID, a table indicating installation information of the large display associated with the light ID, and extracts the installation information associated with the light ID transmitted from the receiver 200 from the table. By searching, the installation information is specified.

Next, the server acquires (i.e., captures) the captured display image Ppre in the stadium based on the specified installation information and the size and orientation of the large display shown in the captured display image Ppre. The assigned seat number is specified (step S426). The server then transmits the URL (Uniform Resource Locator) of the menu screen m1 including the identified seat number to the receiver 200 (step S427).

When the receiver 200 receives the URL of the menu screen m1 transmitted from the server (step S428), the receiver 200 accesses the URL and displays the menu screen m1 (step S429). Here, the user operates the receiver 200 to input the details of the order to the menu screen m1, and selects the order button mc1 to confirm the order. Accordingly, the receiver 200 transmits the order details to the server (step S430).

When the server receives the order content transmitted from the receiver 200, the server performs order reception processing according to the order content (step S431). At this time, the server, for example, instructs the stadium staff to deliver the number of products ordered according to the content of the order to the seat number specified in step S426.

Since the seat number is specified based on the imaged display image Ppre obtained by imaging by the receiver 200, the user of the receiver 200 does not bother to input the seat number when ordering products. you don't have to. Therefore, the user can easily place an order for the product without inputting the seat number.

Although the server identifies the seat number in the above example, the receiver 200 may identify the seat number. In this case, the receiver 200 acquires the installation information from the server and identifies the seat number based on the installation information and the size and orientation of the large display shown in the captured display image Ppre.

FIG. 97 is a diagram for explaining the volume of audio reproduced by receiver 1800a.

The receiver 1800a receives a light ID (visible light signal) transmitted from a transmitter 1800b configured as, for example, street digital signage, similarly to the example shown in FIG. Then, the receiver 1800a reproduces the sound at the same timing as the image reproduction by the transmitter 1800b. That is, the receiver 1800a reproduces the audio in synchronization with the image reproduced by the transmitter 1800b. Note that the receiver 1800a may reproduce the same image as the image (reproduced image) reproduced by the transmitter 1800b, or an AR image (AR moving image) related to the reproduced image, together with the sound.

Here, when the receiver 1800a reproduces the sound as described above, the volume of the sound is adjusted according to the distance to the transmitter 1800b. Specifically, the receiver 1800a adjusts the volume to be smaller as the distance to the transmitter 1800b is longer, and adjusts the volume to be larger as the distance to the transmitter 1800b is shorter.

Receiver 1800a may identify the distance to transmitter 1800b using GPS (Global Positioning System) or the like. Specifically, receiver 1800a acquires the location information of transmitter 1800b associated with the light ID from a server or the like, and further identifies the location of receiver 1800a by GPS. Then, the receiver 1800a specifies the distance between the position of the transmitter 1800b indicated by the position information obtained from the server and the specified position of the receiver 1800a as the distance to the transmitter 1800b. . Note that the receiver 1800a may use Bluetooth (registered trademark) or the like instead of GPS to identify the distance to the transmitter 1800b.

Further, the receiver 1800a may specify the distance to the transmitter 1800b based on the size of the bright line pattern area of the above-described decoding image Pdec obtained by imaging. 51 and 52, the bright line pattern area is an area consisting of a plurality of bright line patterns appearing by exposing a plurality of exposure lines of the image sensor of the receiver 1800a during the communication exposure time. This bright line pattern area corresponds to the display area of the transmitter 1800b shown in the captured display image Ppre. Specifically, the receiver 1800a specifies a shorter distance to the transmitter 1800b as the bright line pattern area is larger, and conversely specifies a longer distance to the transmitter 1800b as the bright line pattern area is smaller. . Further, the receiver 1800a uses distance data indicating the relationship between the size of the bright line pattern area and the distance, and in the distance data, the distance associated with the size of the bright line pattern area in the captured display image Ppre is It may be specified as the distance to the transmitter 1800b. Note that the receiver 1800a may transmit the light ID received as described above to the server, and acquire the distance data associated with the light ID from the server.

In this way, since the volume is adjusted according to the distance to transmitter 1800b, the user of receiver 1800a can hear the sound reproduced by receiver 1800a like the sound actually reproduced by transmitter 1800b. can be heard.

FIG. 98 is a diagram showing the relationship between the distance from receiver 1800a to transmitter 1800b and volume.

For example, when the distance to the transmitter 1800b is between L1 and L2 [m], the volume increases or decreases in proportion to the distance within the range of Vmin to Vmax [dB]. Specifically, receiver 1800a linearly decreases the volume from Vmax [dB] to Vmin [dB] when the distance to transmitter 1800b increases from L1 [m] to L2 [m]. Further, even if the distance to transmitter 1800b becomes shorter than L1 [m], receiver 1800a maintains the volume at Vmax [dB], and the distance to transmitter 1800b becomes longer than L2 [m]. maintains the volume at Vmin [dB].

In this way, the receiver 1800a stores the maximum volume Vmax, the longest distance L1 at which the sound with the maximum volume Vmax is output, the minimum volume Vmin, and the shortest distance L2 at which the sound with the minimum volume Vmin is output. are doing. Further, the receiver 1800a may change its maximum volume Vmax, minimum volume Vmin, longest distance L1, and shortest distance L2 according to the attributes set for itself. For example, if the attribute is the age of the user and the age indicates old age, the receiver 1800a sets the maximum volume Vmax higher than the reference maximum volume and sets the minimum volume Vmin higher than the reference minimum volume. good too. Also, the attribute may be information indicating whether audio is output from speakers or earphones.

In this way, since the receiver 1800a is set to the minimum sound volume Vmin, it is possible to prevent the receiver 1800a from being too far from the transmitter 1800b to hear the sound. Furthermore, since the receiver 1800a is set to the maximum volume Vmax, it is possible to prevent the receiver 1800a from being too close to the transmitter 1800b, which results in the output of unnecessarily loud sound.

FIG. 99 is a diagram showing an example of AR image superimposition by the receiver 200. FIG.

The receiver 200 captures an image of the illuminated signboard. Here, the signboard is lit up by the illumination device, which is the transmitter 100 described above that transmits the light ID. Therefore, the receiver 200 obtains the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 obtains a light ID by decoding the decoding image Pdec, and obtains a plurality of AR images P27a to P27c associated with the light ID and recognition information from the server. Based on the recognition information, the receiver 200 recognizes the area around the area m2 where the signboard is displayed in the captured display image Ppre as the target area.

Specifically, as shown in (a) of FIG. 99, the receiver 200 recognizes an area adjacent to the left side of the area m2 as the first target area, and superimposes the AR image P27a on the first target area. do.

Next, as shown in (b) of FIG. 99, the receiver 200 recognizes the area including the lower side of the area m2 as the second target area, and superimposes the AR image P27b on the second target area. .

Next, as shown in (c) of FIG. 99, the receiver 200 recognizes the area adjacent to the upper side of the area m2 as the third target area, and superimposes the AR image P27c on the third target area.

Here, each of the AR images P27a to P27c is, for example, an image of a Yukio character, and may be a moving image.

In addition, while the receiver 200 continuously and repeatedly acquires the light ID, the receiver 200 switches the target region to be recognized to one of the first to third target regions in a predetermined order and timing. may That is, the receiver 200 may switch the target area to be recognized in order of the first target area, the second target area, and the third target area. Alternatively, the receiver 200 may switch the target area to be recognized to one of the first to third target areas in a predetermined order each time the above-described light ID is acquired. In other words, the receiver 200 first acquires the light ID, and while continuously and repeatedly acquiring the light ID, the receiver 200 recognizes the first target area as shown in (a) of FIG. Then, the AR image P27a is superimposed on the first target area. Then, when the receiver 200 cannot acquire the light ID, the receiver 200 hides the AR image P27a. Next, when the receiver 200 acquires the light ID again, the receiver 200 recognizes the second target area as shown in (b) of FIG. 99 while repeatedly acquiring the light ID. Then, the AR image P27b is superimposed on the second target area. Then, when the receiver 200 cannot acquire the light ID again, the receiver 200 hides the AR image P27b. Next, when the receiver 200 acquires the light ID again, the receiver 200 recognizes the third target area as shown in (c) of FIG. 99 while repeatedly acquiring the light ID. Then, the AR image P27c is superimposed on the third target area.

In the case where the target area to be recognized is switched each time the light ID is acquired in this way, the receiver 200 changes the displayed AR image once every N times (N is an integer equal to or greater than 2). You can change the color. N times is the number of times the AR image is displayed, and may be 200 times, for example. In other words, the AR images P27a to P27c are all images of the same white character, but an AR image of a pink character, for example, is displayed once every 200 times. The receiver 200 may give points to the user when receiving an operation on the AR image by the user while the AR image of the pink character is being displayed.

In this way, by switching the target area on which the AR image is superimposed or changing the color of the AR image at a predetermined frequency, the user's interest can be directed to the image of the signboard illuminated by the transmitter 100. , the user can repeatedly acquire the light ID.

FIG. 100 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

The receiver 200 functions as a so-called Way Finder that presents the route to be taken by the user by imaging marks M4 drawn on the floor at positions where a plurality of passages intersect in the building, for example. have The building is, for example, a hotel, and the presented route is the route taken by the user who has checked in to his room.

The mark M4 is illuminated by a lighting device, which is the transmitter 100 described above, which transmits the light ID by changing the brightness. Therefore, the receiver 200 obtains the captured display image Ppre and the decoding image Pdec by capturing the mark M4. Then, the receiver 200 acquires the light ID by decoding the decoding image Pdec, and transmits the light ID and the terminal information of the receiver 200 to the server. The receiver 200 acquires a plurality of AR images P28 and recognition information associated with the light ID and terminal information from the server. Note that the light ID and terminal information are stored in the server in association with the plurality of AR images P28 and the recognition information when the user checks-in.

Based on the recognition information, the receiver 200 recognizes a plurality of target areas around the area m4 where the mark M4 is displayed in the captured display image Ppre. Then, as shown in FIG. 100, the receiver 200 superimposes and displays an AR image P28, such as an animal's footprint, on each of the plurality of target areas.

Specifically, the recognition information indicates a course that turns right at the position of the mark M4. Based on such recognition information, the receiver 200 identifies a route in the captured display image Ppre and recognizes a plurality of target regions arranged along the route. This route is directed from the lower side of the display 201 to the area m4 and turns right at the area m4. The receiver 200 places an AR image P28 in each of the recognized regions of interest as if the animal had walked along its path.

Here, the receiver 200 may use the geomagnetism detected by its own 9-axis sensor when specifying the route in the captured display image Ppre. In this case, the recognition information indicates the azimuth to be advanced at the position of the mark M4 with reference to the orientation of the geomagnetism. For example, the recognition information indicates west as the direction to go at the position of mark M4. Based on such recognition information, the receiver 200 identifies a route from the lower side of the display 201 toward the region m4 and heading west in the region m4 in the captured display image Ppre. The receiver 200 then recognizes multiple regions of interest arranged along the path. Note that the receiver 200 identifies the lower side of the display 201 by detecting gravitational acceleration with the 9-axis sensor.

In this way, since the route of the user is presented by the receiver 200, the user can easily reach the destination by following the route. Further, since the route is displayed as an AR image in the captured display image Ppre, the route can be presented to the user in an easy-to-understand manner.

Note that the illumination device that is the transmitter 100 can appropriately transmit the light ID while suppressing the brightness by illuminating the mark M4 with short-pulse light. In addition, although the receiver 200 captures the image of the mark M4, the camera (so-called self-taking camera) arranged on the display 201 side may be used to capture the image of the lighting device. Also, the receiver 200 may image both the mark M4 and the illumination device.

FIG. 101 is a diagram for explaining an example of how the receiver 200 determines the line scan time.

When decoding the decoding image Pdec, the receiver 200 performs decoding using the line scan time. This line scan time is the time from the start of exposure of one exposure line included in the image sensor to the start of exposure of the next exposure line. If the line scan time is known, the receiver 200 decodes the decoding image Pdec using the known line scan time. However, if the line scan time is not known, the receiver 200 obtains the line scan time from the decoding image Pdec.

For example, as shown in FIG. 101, the receiver 200 finds the line with the minimum width among the multiple bright lines and multiple dark lines forming the bright line pattern in the decoding image Pdec. It should be noted that the bright lines are lines on the decoding image Pdec caused by exposure of each of the one or more continuous exposure lines when the luminance of the transmitter 100 is high. Dark lines are lines on the decoding image Pdec caused by exposure of each of the one or more continuous exposure lines when the brightness of the transmitter 100 is low.

When the receiver 200 finds the minimum width line, it determines the number of lines, or pixels, of the exposure line corresponding to the minimum width line. When the transmitter 100 uses a carrier frequency of 9.6 kHz to change the brightness for transmitting the light ID, the time during which the brightness of the transmitter 100 is high or low is 104 μs at the shortest. Therefore, the receiver 200 calculates the line scan time by dividing 104 μs by the number of minimum width pixels specified.

FIG. 102 is a diagram for explaining an example of how the receiver 200 determines the line scan time.

The receiver 200 may perform Fourier transform on the bright line pattern of the decoding image Pdec, and obtain the line scan time based on the spatial frequency obtained by the Fourier transform.

For example, as shown in FIG. 102, the receiver 200 derives a spectrum indicating the relationship between the spatial frequency and the intensity of the spatial frequency component in the decoding image Pdec by the Fourier transform described above. Receiver 200 then selects each of the multiple peaks shown in the spectrum in turn. Each time the receiver 200 selects a peak, the line scan time is set such that the spatial frequency of the selected peak (for example, the spatial frequency f2 in FIG. 102) is obtained by the temporal frequency of 9.6 kHz. It is calculated as a scan time candidate. 9.6 kHz is the carrier frequency for the brightness variation of transmitter 100 as described above. As a result, a plurality of line scan time candidates are calculated. The receiver 200 selects the most likely candidate among these multiple line scan time candidates as the line scan time.

In order to select the most likely candidate, the receiver 200 calculates the line scan time tolerance based on the frame rate in imaging and the number of exposure lines included in the image sensor. That is, the receiver 200 calculates the maximum value of the line scan time by 1×10 ⁶ [μs]/{(frame rate)×(number of exposure lines)}. Then, the receiver 200 determines the maximum value×constant K (K<1) to the maximum value as the allowable range of the line scan time. Constant K is, for example, 0.9 or 0.8.

The receiver 200 selects a candidate within this allowable range from among the plurality of line scan time candidates as the maximum likelihood candidate, that is, the line scan time.

Note that the receiver 200 may evaluate the reliability of the calculated line scan time based on whether the line scan time calculated according to the example shown in FIG. 101 is within the allowable range described above.

FIG. 103 is a flow chart showing an example of how the receiver 200 determines the line scan time.

The receiver 200 may determine the line scan time by attempting to decode the decoding image Pdec. Specifically, first, the receiver 200 starts imaging (step S441). Next, the receiver 200 determines whether or not the line scan time is known (step S442). For example, the receiver 200 may determine whether the line scan time is known by notifying the server of its own type and model and inquiring the line scan time according to the type and model. . Here, if it is determined that it is known (Yes in step S442), the receiver 200 sets the reference number of light ID acquisition times to n (n is an integer equal to or greater than 2, for example 4) (step S443). ). Next, the receiver 200 acquires the light ID by decoding the decoding image Pdec using the known line scan time (step S444). At this time, the receiver 200 obtains a plurality of optical IDs by decoding each of the plurality of decoding images Pdec sequentially obtained by the imaging started in step S441. Here, the receiver 200 determines whether or not the same light ID has been obtained for the reference number of times (that is, n times) (step S445). If it is determined that it has been acquired n times (Yes in step S445), the receiver 200 trusts the light ID and starts processing (for example, superimposing an AR image) using the light ID (step S446). On the other hand, if it is determined that the light ID has not been obtained n times (No in step S445), the receiver 200 does not trust the light ID and terminates the process.

If it is determined in step S442 that the line scan time has not been determined (No in step S442), the receiver 200 sets the reference number of optical ID acquisitions to n+k (k is an integer equal to or greater than 1) (step S447). . In other words, the receiver 200 sets a larger reference number of times of acquisition when the line scan time is unknown than when the line scan time is known. Receiver 200 then determines a tentative line scan time (step S448). Then, the receiver 200 acquires the light ID by decoding the decoding image Pdec using the provisionally determined line scan time (step S449). At this time, as described above, the receiver 200 obtains a plurality of light IDs by decoding each of the plurality of decoding images Pdec sequentially obtained by the imaging started in step S441. Here, the receiver 200 determines whether or not the same light ID has been obtained the reference number of times (that is, (n+k) times) (step S450).

If it is determined that it has been acquired (n+k) times (Yes in step S450), the receiver 200 determines that the provisionally determined line scan time is the correct line scan time. Then, the receiver 200 notifies the server of the type and model of the receiver 200 and its line scan time (step S451). As a result, the server associates and stores the type and model of the receiver with the line scan time suitable for the receiver. Therefore, when other receivers of the same type and model start imaging, they can identify their line scan times by querying the server. That is, other receivers can determine that the line scan time is known in the determination of step S442.

Then, the receiver 200 trusts the optical ID acquired (n+k) times, and starts processing (for example, superimposing an AR image) using the optical ID (step S446).

Further, if it is determined in step S450 that the same light ID has not been obtained (n+k) times (No in step S450), the receiver 200 further determines whether or not the termination condition is satisfied (step S452). . The termination condition is, for example, that a predetermined time has elapsed since the start of imaging, or that the light ID has been obtained more than the maximum number of times. When determining that such a termination condition is satisfied (Yes in step S452), the receiver 200 terminates the process. On the other hand, if it is determined that the termination condition is not satisfied (No in step S452), the receiver 200 changes the tentative line scan time (step S453). Then, the receiver 200 repeats the process from step S449 using the changed provisionally determined line scan time.

Thus, even if the line scan time is not known, the receiver 200 can obtain the line scan time as in the examples shown in FIGS. 101-103. Accordingly, regardless of the type and model of receiver 200, receiver 200 can appropriately decode decoding image Pdec to acquire the optical ID.

FIG. 104 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

Receiver 200 images transmitter 100 configured as a television. This transmitter 100 periodically transmits the light ID and the time code by changing the brightness while displaying a television program, for example. The time code is information indicating the time of transmission each time it is transmitted, and may be a time packet shown in FIG. 26, for example.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec through the above imaging. Then, the receiver 200 acquires the aforementioned light ID and time code by decoding the decoding image Pdec while displaying the captured display image Ppre that is acquired periodically on the display 201 . Receiver 200 then transmits the light ID to server 300 . When server 300 receives the light ID, server 300 transmits audio data associated with the light ID, AR start time information, AR image P29, and recognition information to receiver 200 .

When the receiver 200 acquires the audio data, the receiver 200 synchronizes the audio data with the image of the television program displayed on the transmitter 100 and reproduces it. That is, the audio data consists of a plurality of audio unit data, and the plurality of audio unit data includes time codes. The receiver 200 starts reproducing a plurality of audio unit data from the audio data including the time code indicating the same time as the time code acquired from the transmitter 100 together with the light ID. Thereby, the reproduction of the audio data is synchronized with the video of the television program. It should be noted that such synchronization of audio and video may be performed by a method similar to the audio synchronous reproduction shown in FIG. 23 and subsequent drawings.

When receiving the AR image P29 and the recognition information, the receiver 200 recognizes an area corresponding to the recognition information in the imaged display image Ppre as a target area, and superimposes the AR image P29 on the target area. For example, the AR image P29 is an image showing a crack in the display 201 of the receiver 200, and the target area is the area across the image of the transmitter 100 in the captured display image Ppre.

Here, the receiver 200 displays the captured display image Ppre superimposed with the AR image P29 as described above at a timing according to the AR start time information. The AR start time information is information indicating the time when the AR image P29 is displayed. That is, the receiver 200 receives the time code indicating the same time as the AR start time information among the time codes transmitted from the transmitter 100 at any time. Display Ppre. For example, the time indicated by the AR start time information is the time when a scene in which a witch girl casts ice magic appears in a television program. Also, at this time, the receiver 200 may output from the speaker of the receiver 200 a sound that cracks the AR image P29 by reproducing the audio data.

This allows the user to view the scenes of the TV program with a more realistic feeling.

In addition, receiver 200 may vibrate a vibrator provided in receiver 200 at the time indicated by the AR start time information, may cause the light source to emit light like a flash, or may display 201 momentarily. You can also make it brighten or flash. Also, the AR image P29 may include not only an image showing cracks but also an image showing a state where dew condensation on the display 201 has frozen.

FIG. 105 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

A receiver 200 images a transmitter 100 configured as, for example, a toy cane. This transmitter 100 has a light source, and transmits a light ID by changing the brightness of the light source.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec through the above imaging. Then, the receiver 200 acquires the aforementioned light ID by decoding the decoding image Pdec while displaying the captured display image Ppre that is acquired periodically on the display 201 . Receiver 200 then transmits the light ID to server 300 . Upon receiving the light ID, server 300 transmits to receiver 200 the AR image P30 associated with the light ID and the recognition information.

Here, the recognition information further includes gesture information indicating gestures (that is, actions) by a person holding transmitter 100 . Gesture information indicates, for example, a gesture of a person moving transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100 displayed in each captured display image Ppre with the gesture indicated by the gesture information. Then, when the gestures match, the receiver 200 moves the AR images P30 so that, for example, many star-shaped AR images P30 are arranged along the trajectory of the transmitter 100 that moves according to the gesture. is superimposed on the captured display image Ppre.

FIG. 106 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

Receiver 200 images transmitter 100, which is configured as, for example, a toy cane, similar to that described above.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 acquires the aforementioned light ID by decoding the decoding image Pdec while displaying the captured display image Ppre that is acquired periodically on the display 201 . Receiver 200 then transmits the light ID to server 300 . Upon receiving the light ID, the server 300 transmits to the receiver 200 the AR image P31 associated with the light ID and the recognition information.

Here, the recognition information includes gesture information indicating gestures made by a person holding transmitter 100, as described above. Gesture information indicates, for example, a gesture of a person moving transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100 displayed in each captured display image Ppre with the gesture indicated by the gesture information. Then, when the gestures match, the receiver 200 displays an AR image P30 showing a dress in a target area in which a person holding the transmitter 100 is displayed, for example, in the captured display image Ppre. is superimposed.

Thus, in the display method in this modified example, the gesture information corresponding to the light ID is acquired from the server. Next, it is determined whether or not the movement of the subject indicated by the captured display images acquired periodically matches the movement indicated by the gesture information acquired from the server. Then, when it is determined that they match, the captured display image Ppre on which the AR image is superimposed is displayed.

As a result, an AR image can be displayed according to the movement of a subject such as a person. That is, the AR image can be displayed at appropriate timing.

FIG. 107 is a diagram showing an example of a decoding image Pdec acquired according to the attitude of the receiver 200. As shown in FIG.

For example, as shown in (a) of FIG. 107, the receiver 200 takes an image of the transmitter 100 that transmits the light ID according to the change in luminance in a lateral posture. Note that the sideways posture is a posture in which the longitudinal direction of display 201 of receiver 200 is along the horizontal direction. Also, each exposure line of the image sensor provided in the receiver 200 is perpendicular to the longitudinal direction of the display 201 . A decoding image Pdec including a bright line pattern region X having a small number of bright lines is acquired by the imaging as described above. The number of bright lines is small in the bright line pattern region X of this decoding image Pdec. In other words, there are few portions where the brightness changes to High or Low. Therefore, the receiver 200 may not be able to properly acquire the optical ID by decoding the decoding image Pdec.

Therefore, for example, as shown in FIG. 107(b), the user changes the attitude of the receiver 200 from horizontal to vertical. Note that the vertical posture is a posture in which the longitudinal direction of display 201 of receiver 200 is along the vertical direction. When receiving an image of the transmitter 100 that transmits the light ID, the receiver 200 in such a posture can acquire the decoding image Pdec including the bright line pattern region Y with many bright lines.

As described above, the light ID may not be acquired appropriately depending on the attitude of the receiver 200. Therefore, when the receiver 200 is caused to acquire the light ID, the attitude of the receiver 200 that is capturing an image may be adjusted appropriately. should be changed. When the attitude is changed, the receiver 200 can appropriately acquire the light ID at the timing when the attitude becomes easy to acquire the light ID.

FIG. 108 is a diagram showing another example of decoding image Pdec acquired according to the attitude of receiver 200. In FIG.

For example, the transmitter 100 is configured as a digital signage for a coffee shop, displays an image relating to an advertisement for the coffee shop during the image display period, and transmits the light ID according to the luminance change during the light ID transmission period. That is, the transmitter 100 alternately and repeatedly performs image display during the image display period and light ID transmission during the light ID transmission period.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the imaging of the transmitter 100 . At this time, by synchronizing the repetition period of the video display period and the optical ID transmission period of the transmitter 100 with the repetition period of acquisition of the captured display image Ppre and the decoding image Pdec by the receiver 200, the It may not be possible to acquire the image Pdec. Furthermore, depending on the attitude of the receiver 200, it may not be possible to acquire the decoding image Pdec including the bright line pattern area.

For example, the receiver 200 captures an image of the transmitter 100 in the attitude shown in FIG. 108(a). That is, the receiver 200 approaches the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected on the entire image sensor of the receiver 200 .

Here, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 appropriately acquires the captured display image Ppre in which the transmitter 100 is displayed. .

Even if the timing at which the receiver 200 obtains the decoding image Pdec straddles the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 can obtain the decoding image Pdec including the bright line pattern region Z1. An image Pdec can be obtained.

That is, the exposure of each exposure line included in the image sensor is sequentially started from the exposure line at the upper end in the vertical direction downward. Therefore, even if the receiver 200 starts exposing the image sensor in order to obtain the decoding image Pdec during the image display period, the bright line pattern area cannot be obtained. However, when the video display period is switched to the optical ID transmission period, it is possible to obtain a bright line pattern area corresponding to each exposure line exposed during the optical ID transmission period.

Here, the receiver 200 takes an image of the transmitter 100 in the attitude shown in FIG. 108(b). That is, the receiver 200 moves away from the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected only on the area above the image sensor of the receiver 200 . At this time, as described above, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 acquires the captured display image Ppre in which the transmitter 100 is displayed. get it right. However, if the timing at which the receiver 200 acquires the decoding image Pdec straddles the video display period of the transmitter 100 and the optical ID transmission period, the receiver 200 acquires the decoding image Pdec including the bright line pattern area. may not be obtained. In other words, even if the image display period of the transmitter 100 is switched to the light ID transmission period, the image of the transmitter 100 whose brightness changes is displayed on each exposure line below the image sensor exposed during the light ID transmission period. may not be projected. Therefore, the decoding image Pdec having the bright line pattern area cannot be obtained.

On the other hand, as shown in (c) of FIG. 108, the receiver 200 is separated from the transmitter 100 so that the image of the transmitter 100 is projected only on the area below the image sensor of the receiver 200 . , the transmitter 100 is imaged. At this time, as described above, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 acquires the captured display image Ppre in which the transmitter 100 is displayed. get it right. Furthermore, even if the timing at which the receiver 200 acquires the decoding image Pdec straddles the video display period of the transmitter 100 and the optical ID transmission period, the receiver 200 acquires the decoding image Pdec including the bright line pattern area. There are things we can do. That is, when the image display period of the transmitter 100 is switched to the optical ID transmission period, the image of the transmitter 100 whose luminance changes is displayed on each exposure line below the image sensor exposed during the optical ID transmission period. projected. Therefore, the decoding image Pdec having the bright line pattern area Z2 can be obtained.

As described above, depending on the attitude of the receiver 200, it may not be possible to acquire the light ID appropriately. may be urged to In other words, when the imaging is started, the receiver 200 displays or outputs a message such as “Please move” or “Please shake” so that the attitude of the receiver 200 changes. As a result, the receiver 200 captures an image while changing its posture, and thus can appropriately acquire the optical ID.

FIG. 109 is a flow chart showing an example of the processing operation of the receiver 200. FIG.

For example, the receiver 200 determines whether or not the receiver 200 is shaken during imaging (step S461). Specifically, receiver 200 determines whether or not it is shaken based on the output of the 9-axis sensor provided in receiver 200 . Here, if the receiver 200 determines that it is shaken during imaging (Yes in step S461), it increases the above-described light ID acquisition rate (step S462). Specifically, the receiver 200 acquires all captured images per unit time obtained during imaging as images for decoding (that is, bright line images) Pdec, and decodes each of all the acquired images for decoding. . Alternatively, the receiver 200 starts acquiring and decoding the decoding image Pdec when all the captured images have been acquired as the captured display image Ppre, that is, when the acquisition and decoding of the decoding image Pdec have been stopped.

On the other hand, if the receiver 200 determines that it is not shaken during imaging (No in step S461), it acquires the decoding image Pdec at a low light ID acquisition rate (step S463). Specifically, if the light ID acquisition rate was increased in step S462 and is still at a high light ID acquisition rate, the receiver 200 increases the light ID acquisition rate because the current light ID acquisition rate is high. Lower. This reduces the frequency with which the decoding image Pdec for decoding is performed by the receiver 200, so that power consumption can be suppressed.

Then, the receiver 200 determines whether or not the termination condition for terminating the process of adjusting the light ID acquisition rate is satisfied (step S464). The process from S461 is repeatedly executed. On the other hand, when the receiver 200 determines that the termination condition is satisfied (Yes in step S464), the receiver 200 terminates the light ID acquisition rate adjustment process.

FIG. 110 is a diagram illustrating an example of camera lens switching processing by the receiver 200 .

Receiver 200 may include wide-angle lens 211 and telephoto lens 212 as camera lenses, respectively. A captured image obtained by imaging using the wide-angle lens 211 is an image with a wide angle of view, and the subject appears small in the image. On the other hand, the captured image obtained by imaging using the telephoto lens 212 is an image with a narrow angle of view, and the subject appears large in the image.

The receiver 200 as described above may switch the camera lens used for imaging by any one of the methods A to E shown in FIG. 110 when imaging.

In method A, the receiver 200 always uses the telephoto lens 212 when capturing an image, whether it is for normal imaging or when receiving an optical ID. Here, the case of normal imaging is the case of acquiring all the captured images as the captured display image Ppre by imaging. Further, the case of receiving the optical ID is the case of periodically acquiring the captured display image Ppre and the decoding image Pdec by imaging.

In method B, receiver 200 uses wide-angle lens 211 for normal imaging. On the other hand, when receiving an optical ID, receiver 200 first uses wide-angle lens 211 . Then, the receiver 200 switches the camera lens from the wide-angle lens 211 to the telephoto lens 212 if the bright line pattern area is included in the decoding image Pdec acquired when using the wide-angle lens 211 . After this switching, the receiver 200 can acquire a decoding image Pdec with a narrow angle of view, that is, a large bright line pattern area.

In method C, receiver 200 uses wide-angle lens 211 for normal imaging. On the other hand, when receiving the light ID, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212 . That is, the receiver 200 uses the wide-angle lens 211 to acquire the captured display image Ppre, and uses the telephoto lens 212 to acquire the decoding image Pdec.

In method D, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212 according to the user's operation, both in the case of normal imaging and in the case of receiving the light ID.

In method E, the receiver 200 decodes the decoding image Pdec acquired using the wide-angle lens 211 when receiving the light ID, and if it cannot be decoded correctly, changes the camera lens from the wide-angle lens 211 to the telephoto lens 212. switch. Alternatively, the receiver 200 decodes the decoding image Pdec acquired using the telephoto lens 212 , and switches the camera lens from the telephoto lens 212 to the wide-angle lens 211 if it cannot be decoded correctly. When determining whether or not the decoding image Pdec has been correctly decoded, the receiver 200 first transmits to the server the light ID obtained by decoding the decoding image Pdec. If the light ID matches the light ID registered to itself, the server notifies the receiver 200 of matching information indicating that the light ID matches. to the receiver 200. If the information notified by the server is match information, the receiver 200 determines that the decoding image Pdec has been correctly decoded. determined that it was not possible. Alternatively, the receiver 200 determines that the decoding image Pdec has been correctly decoded when the optical ID obtained by decoding the decoding image Pdec satisfies a predetermined condition. On the other hand, if the condition is not satisfied, the receiver 200 determines that the decoding image Pdec could not be correctly decoded.

By switching the camera lens in this manner, an appropriate decoding image Pdec can be obtained.

FIG. 111 is a diagram illustrating an example of camera switching processing by the receiver 200 .

For example, the receiver 200 includes an in-camera 213 and an out-camera (not shown in FIG. 111) as cameras. In-camera 213 is also called a face camera or a selfie camera, and is a camera arranged on the same surface as display 201 in receiver 200 . The out-camera is a camera arranged on the surface of the receiver 200 opposite to the surface of the display 201 .

Such a receiver 200 takes an image of the transmitter 100 configured as a lighting device with the in-camera 213 with the in-camera 213 directed upward. By this imaging, the receiver 200 acquires the image for decoding Pdec, and acquires the light ID transmitted from the transmitter 100 by decoding the image for decoding Pdec.

Next, the receiver 200 acquires the AR image and recognition information associated with the light ID from the server by transmitting the acquired light ID to the server. Receiver 200 starts a process of recognizing a target area according to the recognition information from each captured display image Ppre obtained by each of out-camera and in-camera 213 . Here, receiver 200 prompts the user to move receiver 200 when the target area cannot be recognized from both captured display image Ppre obtained by each of out-camera and in-camera 213 . A user prompted by the receiver 200 moves the receiver 200 . Specifically, the user moves the receiver 200 so that the in-camera 213 and the out-camera face the front-back direction of the user. As a result, the receiver 200 recognizes the target area from the captured display image Ppre acquired by the out-camera. That is, the receiver 200 recognizes the area where the person is projected as the target area, superimposes the AR image on the target area of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed. do.

FIG. 112 is a flow chart showing an example of processing operations between the receiver 200 and the server.

The receiver 200 acquires the light ID transmitted from the transmitter 100 by imaging the transmitter 100, which is a lighting device, with the in-camera 213, and transmits the light ID to the server (step S471). The server receives the light ID from the receiver 200 (step S472), and estimates the position of the receiver 200 based on the light ID (step S473). For example, the server stores a table indicating, for each light ID, the room, building, or space in which the transmitter 100 transmitting that light ID is located. Then, the server estimates the room or the like associated with the light ID transmitted from receiver 200 as the position of receiver 200 in the table. Further, the server transmits the AR image and recognition information associated with the estimated position to the receiver 200 (step S474).

The receiver 200 acquires the AR image and recognition information transmitted from the server (step S475). Here, the receiver 200 starts a process of recognizing a target area according to the recognition information from each captured display image Ppre obtained by each of the out-camera and the in-camera 213 . Then, the receiver 200 recognizes the target area from the imaged display image Ppre acquired by the out-camera, for example (step S476). The receiver 200 superimposes the AR image on the target area of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S477).

In the above example, when the receiver 200 acquires the AR image and the recognition information transmitted from the server, in step S476, from each captured display image Ppre obtained by the out-camera and the in-camera 213, Processing to recognize the target area has started. However, in step S476, the receiver 200 may start the process of recognizing the target area from the imaged display image Ppre obtained only by the out-camera. That is, the camera for acquiring the light ID (the in-camera 213 in the above example) and the camera for acquiring the captured display image Ppre on which the AR image is superimposed (the out-camera in the above example) must always be different. You can let

In the above example, the receiver 200 captures the transmitter 100, which is the lighting device, with the in-camera 213, but the floor surface illuminated by the transmitter 100 may be captured with the out-camera. The receiver 200 can acquire the light ID transmitted from the transmitter 100 even with such an out camera.

FIG. 113 is a diagram showing an example of AR image superimposition by the receiver 200. As shown in FIG.

The receiver 200 captures an image of the transmitter 100 configured as a microwave oven installed in a store such as a convenience store. This transmitter 100 includes a camera for capturing an image of the interior of a microwave oven and an illumination device for illuminating the interior of the microwave oven. Then, the transmitter 100 recognizes the food (that is, the object to be warmed) stored in the refrigerator by imaging with the camera. Further, when warming the food, the transmitter 100 causes the lighting device to emit light and changes the brightness of the lighting device to transmit the light ID indicating the recognized food. Although this lighting device illuminates the interior of the microwave oven, the light from the lighting device is emitted to the outside through the transparent window of the microwave oven. Therefore, the light ID is transmitted from the lighting device to the outside of the microwave oven through the window of the microwave oven.

Here, the user purchases food at a convenience store and puts the food into transmitter 100, which is a microwave oven, in order to heat the food. At this time, the transmitter 100 recognizes the food by the camera and starts warming the food while transmitting an optical ID indicating the recognized food.

The receiver 200 acquires the light ID transmitted from the transmitter 100 by capturing an image of the transmitter 100 that has started the warming, and transmits the light ID to the server. Next, the receiver 200 acquires the AR image, audio data and recognition information associated with the light ID from the server.

The AR images described above are an AR image P32a that is a moving image showing a virtual state inside the transmitter 100, an AR image P32b that shows the food and drink stored in the refrigerator in detail, and steam coming out of the transmitter 100. and an AR image P32d showing the remaining time until the food is warmed up.

For example, if the food stored in the microwave oven is pizza, the AR image P32a is an animation in which a turntable on which the pizza is placed is rotating and a plurality of small people are dancing around the pizza. is. For example, if the food stored in the refrigerator is pizza, the AR image P32b is an image showing the product name "pizza" and the ingredients of the pizza.

Based on the recognition information, the receiver 200 recognizes the area where the window of the transmitter 100 is displayed in the imaged display image Ppre as the target area of the AR image P32a, and displays the AR image P32a in the target area. superimpose. Furthermore, based on the recognition information, the receiver 200 recognizes an area above the area where the transmitter 100 is displayed in the captured display image Ppre as the target area of the AR image P32b, and An AR image P32b is superimposed on the area. Furthermore, based on the recognition information, the receiver 200 recognizes an area between the target area of the AR image P32a and the target area of the AR image P32b in the captured display image Ppre as the target area of the AR image P32c. Then, the AR image P32c is superimposed on the target area. Furthermore, based on the recognition information, the receiver 200 recognizes an area under the area where the transmitter 100 is displayed in the imaged display image Ppre as a target area of the AR image P32d, and The AR image P32d is superimposed.

Furthermore, the receiver 200 outputs the sound produced when the food is heated by reproducing the audio data.

The AR images P32a to P32d as described above are displayed by the receiver 200, and the user's interest is attracted to the receiver 200 until the warming of the food and drink is completed. . As a result, the burden on the user waiting for the completion of warming can be reduced. In addition, the AR image P32c showing steam or the like is displayed, and the sound produced when the food is heated is output, thereby giving the user a sense of sizzle. In addition, the display of the AR image P32d allows the user to easily know the remaining time until the heating of the food and drink is completed. Therefore, the user can move away from the transmitter 100, which is a microwave oven, and read a book displayed in the store until warming is completed. Also, the receiver 200 may notify the user that the warming is completed when the remaining time reaches zero.

In the above example, the AR image P32a is a moving image of a rotating turntable on which a pizza is placed and a plurality of small people dancing around the pizza. may be an image that virtually shows the Also, the AR image P32b is an image indicating the product name and ingredients of the food and drink stored in the refrigerator, but may be an image indicating nutritional components or calories. Alternatively, the AR image P32b may be an image showing a discount coupon.

As described above, in the display method according to the present modification, the subject is a microwave oven equipped with a lighting device, and the lighting device illuminates the inside of the microwave oven, and changes the luminance so that the light ID is displayed in the microwave oven. Send externally. In acquisition of the captured display image Ppre and the decoding image Pdec, the captured display image Ppre and the decoding image Pdec are acquired by capturing an image of the microwave oven transmitting the optical ID. In recognition of the target area, the window portion of the microwave oven displayed in the captured display image Ppre is recognized as the target area. In the display of the imaged display image Ppre, the imaged display image Ppre on which the AR image showing the state change in the refrigerator is superimposed is displayed.

As a result, changes in the state of the inside of the microwave oven are displayed as an AR image, so that the user of the microwave oven can be informed of the state of the inside of the microwave oven in an easy-to-understand manner.

FIG. 114 is a sequence diagram showing processing operations of a system including receiver 200, a microwave oven, a relay server, and an electronic payment server. Note that the microwave oven includes a camera and an illumination device as described above, and transmits the light ID by changing the brightness of the illumination device. In other words, the microwave oven functions as transmitter 100 .

First, the microwave oven recognizes the food and drink stored in the oven with the camera (step S481). Next, the microwave oven transmits the light ID indicating the recognized food to the receiver 200 according to the brightness change of the lighting device.

Receiver 200 receives the light ID transmitted from the microwave oven by capturing an image of the microwave oven (step S483), and transmits the light ID and the card information to the relay server. The card information is information such as a credit card stored in advance in the receiver 200, and is information necessary for electronic payment.

The relay server holds, for each light ID, a table showing the AR image, recognition information, and product information corresponding to that light ID. This product information indicates the price of the food and drink indicated by the light ID. When such a relay server receives the light ID and the card information transmitted from the receiver 200 (step S486), it finds the product information associated with the light ID from the table described above. The relay server then transmits the product information and card information to the electronic payment server (step S486). Upon receiving the product information and card information transmitted from the relay server (step S487), the electronic payment server performs electronic payment processing based on the product information and card information (step S488). When the electronic payment processing is completed, the electronic payment server notifies the relay server of the completion (step S489).

When the relay server confirms the payment completion notification from the electronic payment server (step S490), it instructs the microwave oven to start heating food (step S491). Further, the relay server transmits to the receiver 200 the AR image and recognition information associated with the light ID received in step S485 in the table described above (step S493).

Upon receiving the instruction to start warming from the relay server, the microwave oven starts warming the food stored in the oven (step S492). In addition, when the receiver 200 receives the AR image and the recognition information transmitted from the relay server, the receiver 200 selects the object corresponding to the recognition information from the imaged display image Ppre periodically acquired by the imaging started from step S483. Recognize the area. The receiver 200 then superimposes the AR image on the target area (step S494).

As a result, the user of the receiver 200 can easily finish the payment and start warming the food by putting the food in the microwave oven and taking an image. Also, if payment cannot be made, it is possible to prohibit the user from warming food. Furthermore, when warming is started, an AR image P32a shown in FIG. 113 or the like can be displayed to notify the user of the state of the inside of the refrigerator.

FIG. 115 is a sequence diagram showing processing operations of a system including a POS terminal, server, receiver 200 and microwave oven. Note that the microwave oven includes a camera and an illumination device as described above, and transmits the light ID by changing the brightness of the illumination device. In other words, the microwave oven functions as transmitter 100 . A POS (point-of-sale) terminal is a terminal installed in a store, such as a convenience store, like the microwave oven.

First, the user of the receiver 200 selects a food item as a product at a store, and goes to a place where a POS terminal is installed to purchase the food item. The store clerk operates the POS terminal and receives payment for the food and drink from the user. By the operation of the POS terminal by the store clerk, the POS terminal acquires operation input data and sales information (step S501). The sales information indicates, for example, the name, quantity and price of the product, the place of sale, and the date and time of sale. The operation input data indicates, for example, the sex and age of the user input by the store clerk. The POS terminal transmits the operation input data and sales information to the server (step S502). The server receives the operation input data and sales information transmitted from the POS terminal (step S503).

On the other hand, the user of the receiver 200 pays the food and drink to the clerk and puts the food and drink into the microwave oven to heat the food and drink. The microwave oven recognizes the food and drink stored in the oven with the camera (step S504). Next, the microwave oven transmits the light ID indicating the recognized food to the receiver 200 according to the brightness change of the lighting device (step S505). Then, the microwave oven starts warming the food (step S507).

The receiver 200 receives the light ID transmitted from the microwave oven by capturing an image of the microwave oven (step S508), and transmits the light ID and the terminal information to the server (step S509). The terminal information is information stored in advance in the receiver 200, and indicates, for example, the type of language displayed on the display 201 of the receiver 200 (for example, English or Japanese).

When the server is accessed by the receiver 200 and receives the light ID and the terminal information transmitted from the receiver 200, it determines whether or not the access from the receiver 200 is the first access (step S510). The first access is the first access within a predetermined period of time from when the process of step S503 was performed. Here, when the server determines that the access from the receiver 200 is the first access (Yes in step S510), it associates and saves the operation input data and the terminal information (step S511).

Although the server determines whether or not the access from the receiver 200 is the first access, the server may determine whether or not the product indicated by the sales information matches the food and drink indicated by the light ID. . Further, in step S511, the server may not only associate the operation input data with the terminal information, but also associate the sales information with them and store them.

(Indoor use)
FIG. 116 is a diagram showing how the system is used indoors, such as in an underground mall.

The receiver 200 receives the light ID transmitted by the transmitter 100 configured as a lighting device and estimates its own current position. The receiver 200 also displays the current position on a map to provide directions, and displays information on nearby shops.

By transmitting disaster information and evacuation information from the transmitter 100 in an emergency, even when communication is congested, when a communication base station breaks down, or when radio waves from a communication base station do not reach. You can still get this information. This is effective for hearing-impaired people who miss the emergency broadcast or who cannot hear the emergency broadcast.

In other words, the receiver 200 obtains the light ID transmitted from the transmitter 100 by capturing the image, and further obtains the AR image P33 and the recognition information associated with the light ID from the server. Then, the receiver 200 recognizes a target area corresponding to the recognition information from the captured display image Ppre obtained by the above imaging, and superimposes an arrow-shaped AR image P33 on the target area. As a result, the receiver 200 can be used as the aforementioned wayfinder (see FIG. 100).

(Displaying Augmented Reality Objects)
FIG. 117 is a diagram showing how an augmented reality object is displayed.

A stage 2718e on which augmented reality is displayed is configured as the transmitter 100 described above, and the information on the augmented reality object and the reference position for displaying the augmented reality object are displayed using the light emission patterns and position patterns of the light emitting units 2718a, 2718b, 2718c, and 2718d. to send.

Based on the received information, the receiver 200 displays an augmented reality object 2718f, which is an AR image, superimposed on the captured image.

In addition, these general or specific aspects may be realized by an apparatus, system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM. It may be implemented in any combination of circuits, computer programs or recording media. Moreover, a computer program for executing the method according to one embodiment may be stored in a recording medium of a server, and may be implemented in a mode of being distributed from the server to the terminal in response to a request from the terminal.

[Modification 4 of Embodiment 4]
118 is a diagram illustrating a configuration of a display system according to Modification 4 of Embodiment 4. FIG.

This display system 500 performs object recognition and augmented reality/mixed reality display using visible light signals.

The receiver 200 performs imaging, receives visible light signals, and extracts feature amounts for object recognition or space recognition. Extraction of a feature amount is extraction of an image feature amount from a captured image obtained by imaging. Note that the visible light signal may be a visible light adjacent carrier signal such as infrared or ultraviolet. In addition, in this modification, the receiver 200 is configured as a recognition device that recognizes an object on which an augmented reality image (that is, an AR image) is displayed. Note that in the example shown in FIG. 118, the target is, for example, an AR target 501 or the like.

The transmitter 100 transmits information such as an ID for identifying itself or the AR target 501 as a visible light signal or radio wave signal. Note that the ID is identification information such as the above-described optical ID, for example, and the AR target 501 is the above-described target area. A visible light signal is a signal transmitted by a change in luminance of the light source of the transmitter 100 .

The receiver 200 or the server 300 associates and holds the identification information transmitted by the transmitter 100, the AR recognition information, and the AR display information. The binding may be one-to-one or one-to-many. The AR recognition information is the recognition information described above, and is information for recognizing the AR object 501 for AR display. Specifically, the AR recognition information is the image feature amount (SIFT feature amount, SURF feature amount, or ORB feature amount, etc.) of the AR object 501, color, shape, size, reflectance, transmittance, or three-dimensional model, etc. The AR recognition information may also include identification information or a recognition algorithm indicating which recognition method is used for recognition. The AR display information is information for performing AR display, and includes an image (that is, the AR image described above), video, audio, three-dimensional model, motion data, display coordinates, display size, transmittance, and the like. Also, the AR display information may be absolute values or change ratios of hue, saturation, and lightness.

The transmitter 100 may also function as the server 300 . That is, the transmitter 100 may hold AR recognition information and AR display information and transmit such information by wired or wireless communication.

Receiver 200 captures an image with a camera (specifically, an image sensor). The receiver 200 also receives visible light signals or radio wave signals such as WiFi or Bluetooth (registered trademark). Also, the receiver 200 acquires information such as position information obtained by GPS or the like, information obtained by a gyro sensor or an acceleration sensor, and information such as voice from a microphone, and integrates all or part of this information. You may recognize the AR object which exists nearby. Also, the receiver 200 may recognize the AR object using only one of the information without integrating the information.

119 is a flowchart illustrating processing operations of a display system according to Modification 4 of Embodiment 4. FIG.

The receiver 200 first determines whether or not a visible light signal has already been received (step S521). That is, the receiver 200 determines whether or not the visible light signal indicating the identification information is acquired by, for example, photographing the transmitter 100 that transmits the visible light signal according to the luminance change of the light source. At this time, the photographed image of the transmitter 100 is acquired by the photographing.

Here, if the receiver 200 determines that the visible light signal has already been received (Y in step S521), the AR target (object, reference point, spatial coordinates, or position and orientation of the receiver 200). Furthermore, the receiver 200 recognizes the relative positions of AR objects. This relative position is represented by the distance and direction from the receiver 200 to the AR object. For example, the receiver 200 identifies the AR target (that is, the target area that is the bright line pattern area) based on the size and position of the bright line pattern area shown in FIG. 50, and recognizes the relative position of the AR target. do.

Then, receiver 200 transmits the information such as the ID and the relative position included in the visible light signal to server 300, and uses the information and the relative position as keys to obtain the AR recognition information registered in server 300. and AR display information are acquired (step S522). At this time, the receiver 200 acquires not only the information of the recognized AR object but also the information of other AR objects existing near the AR object (that is, the AR recognition information and the AR display information) at the same time. Also good. As a result, when another AR object existing nearby is imaged by the receiver 200, the receiver 200 can quickly and without error recognize the other AR object existing nearby. can be done. For example, other AR objects in the vicinity are different from the first recognized AR object.

Note that the receiver 200 may acquire these pieces of information from a database within the receiver 200 instead of accessing the server 300 . The receiver 200 receives this information after a certain period of time has passed since it was acquired, or after a specific process (for example, turning off the screen, pressing a button, ending or stopping an application, displaying an AR image, or performing another AR object , etc.) may be discarded. Alternatively, the receiver 200 may lower the reliability of each of the plurality of pieces of information to be acquired every time a certain period of time has passed since the acquisition of the information, and use the information with the highest degree of reliability among the plurality of pieces of information. good.

Here, the receiver 200 may preferentially acquire the AR recognition information of the AR target that is effective in relation to the relative position based on the relative position with each AR target. For example, the receiver 200 acquires a plurality of visible light signals (that is, identification information) by photographing a plurality of transmitters 100 in step S521, and corresponds to the plurality of visible light signals in step S522. A plurality of pieces of AR recognition information (that is, image feature amounts) are acquired. At this time, in step S522, the receiver 200 selects the image feature amount of the AR target closest to the receiver 200 that captures the images of the transmitter 100 from among the multiple AR targets. That is, the selected image feature amount is used to identify one AR object (that is, the first object) that is identified using the visible light signal. Thereby, even if a plurality of image feature values are acquired, an appropriate image feature value can be used to specify the first object.

On the other hand, when the receiver 200 determines that the visible light signal has not been received (N of step S521), it further determines whether or not the AR recognition information has already been acquired (step S523). If it is determined that the AR recognition information has not been acquired (N in step S523), the receiver 200 does not use identification information such as an ID indicated by a visible light signal, but uses image processing, or obtains position information or radio wave information. and other information are used to recognize candidate AR objects (step S524). This process may be performed only in receiver 200 . Alternatively, the receiver 200 may transmit the captured image or information such as the image feature amount of the captured image to the server 300, and the server 300 may recognize the AR target candidate. As a result, the receiver 200 acquires AR recognition information and AR display information corresponding to the recognized candidate from the server 300 or its own database.

After step S522, the receiver 200 determines whether or not the AR target is detected by another method, such as image recognition, which does not use identification information such as an ID indicated by the visible light signal (step S525). ). In other words, the receiver 200 determines whether or not the AR object has been recognized using a plurality of methods. Specifically, the receiver 200 identifies the AR target (that is, the first target) from the captured image using the image feature quantity acquired based on the identification information indicated by the visible light signal. Then, without using such identification information, the receiver 200 determines whether or not the AR object (that is, the second object) is specified from the captured image by image processing.

Here, when the receiver 200 determines that the AR target object has been recognized by a plurality of methods (Y in step S525), priority is given to the recognition result based on the visible light signal. That is, the receiver 200 checks whether the AR objects recognized by each method match. Then, if they do not match, the receiver 200 selects one AR object on which the AR image is superimposed in the captured image from those AR objects as the AR object recognized by the visible light signal. Determine (step S526). In other words, when the first target is different from the second target, the receiver 200 preferentially recognizes the first target as the target on which the AR image is displayed. Note that the target on which the AR image is displayed is the target on which the AR image is superimposed.

Alternatively, receiver 200 may give priority to a method given a high priority based on the order of priority given to each of a plurality of methods. That is, the receiver 200 recognizes one AR object on which the AR image is superimposed in the captured image from among the AR objects recognized by each method, for example, by the method given the highest priority. AR target object. Alternatively, the receiver 200 may determine one AR target on which the AR image is superimposed in the captured image by majority vote or majority vote with priority. If this processing overturns the recognition results up to that point, the receiver 200 performs error handling processing.

Next, based on the acquired AR recognition information, the receiver 200 determines the state of the AR target in the captured image (specifically, the absolute position, the relative position from the receiver 200, the size, the angle, the lighting conditions, etc.). , or occlusion) is recognized (step S527). Then, the receiver 200 superimposes the AR display information (that is, the AR image) on the captured image and displays it according to the recognition result (step S528). That is, the receiver 200 superimposes the AR display information on the recognized AR target in the captured image. Alternatively, the receiver 200 displays only AR display information.

These allow difficult recognition or detection by image processing alone. The difficult recognition or detection is, for example, the identification of graphically similar AR objects (such as differing only in text content), the detection of AR objects with few patterns, the detection of AR objects with high reflectance or transmittance. detection of objects, detection of AR objects whose shape or pattern changes (such as animals), or detection of AR objects from wide angles (various directions). That is, in this modified example, recognition and AR display of these AR objects can be performed. In addition, in image processing that does not use visible light signals, as the number of AR objects to be recognized increases, it takes time to search for neighborhoods of image feature values, recognition processing takes time, and the recognition rate deteriorates. . However, in this modified example, the effect of the increase in recognition time and the deterioration of the recognition rate due to the increase in the number of recognition objects does not exist or is extremely small, and effective recognition of AR objects is possible. In addition, efficient recognition is possible by using the relative position of the AR target object. For example, by using the approximate distance to the AR target, it is possible to omit the processing to make it independent of the size of the AR target when calculating the image feature amount, or to use features that depend on the size. can be done. In addition, by using the angle of the AR object, normally it is necessary to evaluate the image feature amount for many angles, but only the image feature amount corresponding to the angle of the AR object is stored and calculated. can improve computational speed or memory efficiency.

[Summary of Modification 4 of Embodiment 4]
FIG. 120 is a flowchart illustrating a recognition method according to one aspect of the present invention.

A display method according to an aspect of the present invention is a method of recognizing an object on which an augmented reality image (AR image) is displayed, and includes steps S531 to S535.

In step S531, the receiver 200 acquires identification information by photographing the transmitter 100 that transmits a visible light signal according to changes in luminance of the light source. The identification information is, for example, a light ID. In step S<b>532 , receiver 200 transmits the identification information to server 300 and acquires the image feature amount corresponding to the identification information from server 300 . The image feature amount is indicated as AR recognition information or recognition information.

In step S533, the receiver 200 identifies the first object from the captured image of the transmitter 100 using the image feature amount. In step S534, the receiver 200 identifies the second object from the captured image of the transmitter 100 by image processing without using the identification information (that is, the light ID).

In step S535, if the first object identified in step S533 is different from the second object identified in step S534, the receiver 200 gives priority to the first object to perform augmented reality. It is recognized as an object on which a sensory image is displayed.

For example, an augmented reality image, a captured image, and a target object respectively correspond to the AR image, the captured display image, and the target area in Embodiment 4 and its modifications.

As a result, as shown in FIG. 119, the first object identified using the identification information indicated by the visible light signal and the second object identified by image processing without using the identification information. are different, the first object is preferentially recognized as the object on which the augmented reality image is displayed. Therefore, the object for which the augmented reality image is displayed can be appropriately recognized from the captured image.

In addition to the image feature amount of the first object, the image feature amount includes the image feature amount of a third object located near the first object and different from the first object. You can

As a result, as shown in step S522 in FIG. 119, not only the image feature amount of the first object but also the image feature amount of the third object are acquired. The third object can be quickly identified or recognized when it appears in the captured image.

Further, in step S531, the receiver 200 acquires a plurality of identification information by photographing a plurality of transmitters, and in step S532, acquires a plurality of image feature amounts corresponding to the plurality of identification information. be. In such a case, in step S533, the receiver 200 converts the image feature amount of the object closest to the receiver 200 that captures the images of the plurality of transmitters among the plurality of objects to the first object may be used to identify the

As a result, as shown in step S522 in FIG. 119, even if a plurality of image feature amounts are acquired, an appropriate image feature amount can be used to identify the first object.

Note that the recognition device in this modification is, for example, a device provided in the receiver 200 described above, and includes a processor and a recording medium. A program for causing a processor to execute the recognition method shown in FIG. 120 is recorded in this recording medium. Also, the program in this modified example is a program that causes a computer to execute the recognition method shown in FIG.

(Embodiment 5)
FIG. 121 is a diagram showing an example of operation modes of visible light signals according to this embodiment.

There are two modes of operation of the physical (PHY) layer of visible light signals, as shown in FIG. The first operation mode is a mode in which packet PWM (Pulse Width Modulation) is performed, and the second operation mode is a mode in which packet PPM (Pulse-Position Modulation) is performed. The transmitter according to each of the above-described embodiments or modifications thereof generates and transmits a visible light signal by modulating a signal to be transmitted according to one of these operation modes.

In the packet PWM mode of operation, no RLL (Run-Length Limited) coding is performed, the optical clock rate is 100 kHz, forward error correction (FEC) is iteratively coded, and the typical data rate is 5.5 kbps. is.

In this packet PWM, the pulse width is modulated and the pulse is represented by two brightness states. The two brightness states are a bright state (Bright or High) and a dark state (Dark or Low), typically a light on and off. Physical layer signal chunks called packets (also called PHY packets) correspond to MAC (medium access control) frames. The transmitter can repeatedly send PHY packets and send multiple sets of PHY packets in no particular order.

Note that packet PWM is used to generate a visible light signal transmitted from a normal transmitter.

In the packet PPM mode of operation, no RLL coding is performed, the optical clock rate is 100 kHz, forward error correction (FEC) is iteratively coded, and the typical data rate is 8 kbps.

In this packet PPM, the position of short duration pulses is modulated. That is, this pulse is a bright pulse out of a bright pulse (High) and a dark pulse (Low), and the position of this pulse is modulated. Also, the position of this pulse is indicated by the interval between the pulse and the next pulse.

Packet PPM provides deep dimming. The format, waveform, and features of packet PPM, which are not described in each embodiment and its modifications, are the same as those of packet PWM. It should be noted that packet PPMs are used to generate visible light signals that are transmitted from transmitters having very bright light sources.

Also, in each of packet PWM and packet PPM, dimming in the physical layer of the visible light signal is controlled by the average brightness of the optional field.

122A is a flowchart showing another visible light signal generation method according to Embodiment 5. FIG. This visible light signal generating method is a method of generating a visible light signal to be transmitted by changing the luminance of a light source provided in a transmitter, and includes steps SE1 to SE3.

In step SE1, a preamble is generated as data in which first and second luminance values that are different from each other appear alternately along the time axis.

In step SE2, in the data in which the first and second luminance values appear alternately along the time axis, the interval from the appearance of the first luminance value to the appearance of the next first luminance value is determined as the object of transmission. A first payload is generated by determining according to a scheme according to the signal of .

At step SE3, a visible light signal is generated by combining the preamble and the first payload.

122B is a block diagram showing a configuration of another signal generation device according to Embodiment 5. FIG. This signal generation device E10 is a signal generation device that generates a visible light signal to be transmitted by a change in luminance of a light source provided in a transmitter, and includes a preamble generation unit E11, a payload generation unit E12, and a combination unit E13. . Further, the signal generation device E10 executes the processing of the flowchart shown in FIG. 122A.

That is, the preamble generation unit E11 generates a preamble that is data in which the first and second luminance values that are mutually different luminance values alternately appear along the time axis.

In the data in which the first and second luminance values appear alternately along the time axis, the payload generator E12 sets the interval from the appearance of the first luminance value to the appearance of the next first luminance value as follows: A first payload is generated by determining according to a scheme dependent on the signal to be transmitted.

The combiner E13 generates a visible light signal by combining the preamble and the first payload.

For example, the first and second luminance values are Bright (High) and Dark (Low), and the first payload is the PHY payload. By transmitting the visible light signal generated in this way, it is possible to increase the number of received packets and improve reliability. As a result, communication between various devices can be enabled.

For example, the time length of the first luminance value in each of the preamble and the first payload is 10 μs or less.

As a result, the average brightness of the light source can be suppressed while performing visible light communication.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from the appearance of the first luminance value to the appearance of the next first luminance value. Here each of the three intervals is 160 μs. That is, the pattern of intervals between each pulse contained in the header (SHR) in mode 1 of packet PPM is defined. Each pulse is a pulse having a first luminance value, for example.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from the appearance of the first luminance value to the appearance of the next first luminance value. Here, the first of the three intervals is 160 μs, the second is 180 μs, and the third is 160 μs. That is, the pattern of intervals between each pulse contained in the header (SHR) in mode 2 of packet PPM is defined.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from the appearance of the first luminance value to the appearance of the next first luminance value. Here, the first interval of the three intervals is 80 μs, the second interval is 90 μs, and the third interval is 80 μs. That is, the pattern of intervals between each pulse contained in the header (SHR) in mode 3 of packet PPM is defined.

Thus, the patterns of the headers of each of mode 1, mode 2 and mode 3 of the packet PPM are defined so that the receiver can properly receive the first payload in the visible light signal.

Also, the signal to be transmitted consists of 6 bits from the first bit x ₀ to the sixth bit x ₅ , and the time length of the first payload is the time length of the first payload after the appearance of the first luminance value. contains two intervals until the appearance of the luminance value of . Here, if the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} x 2 + x _{3k + 2} x 4 (where k is 0 or 1), then in generating the first payload, Each of the two intervals is determined according to the above formula, interval P _k =180+30×y _k [μsec]. That is, in mode 1 of packet PPM, the signal to be transmitted is modulated as intervals between pulses contained in the first payload (PHY payload).

The signal to be transmitted consists of 12 bits from the first bit x ₀ to the twelfth bit x ₁₁ , and the time length of the first payload is the time length of the first payload after the appearance of the first luminance value. contains four intervals until the appearance of the luminance value of . Here, if the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} x 2 + x _{3k + 2} x 4, where k is 0, 1, 2 or 3, then in generating the first payload, the first Each of the four intervals in a payload of 1 is determined according to the above scheme, interval P _k =180+30×y _k [μsec]. That is, in mode 2 of packet PPM, the signal to be transmitted is modulated as intervals between pulses included in the first payload (PHY payload).

In addition, the signal to be transmitted consists of 3n bits from the first bit x ₀ to the 3nth bit x _3n-1 (n is an integer of 2 or more), and the time length of the first payload is It contains n intervals from the appearance of a luminance value to the appearance of the next first luminance value. Here, if the parameter y _k is represented as y _k = x _3k + x _{3k + 1} x 2 + x _{3k + 2} x 4 (where k is an integer from 0 to (n-1)), the first payload generation Now determine each of the n said intervals in the first payload according to the above scheme, interval P _k =100+20×y _k [μsec]. That is, in mode 3 of packet PPM, the signal to be transmitted is modulated as intervals between pulses included in the first payload (PHY payload).

Thus, in packet PPM modes 1, 2 and 3, the signal to be transmitted is modulated as the interval between each pulse so that the receiver can properly transmit the visible light signal based on the interval. It can be demodulated to the signal of interest.

Also, the visible light signal generating method may further include generating a footer for the first payload, and combining the footer next to the first payload in generating the visible light signal. That is, in mode 3 of packet PWM and packet PPM, the footer (SFT) is transmitted following the first payload (PHY payload). As a result, the end of the first payload can be clearly specified by the footer, so visible light communication can be performed efficiently.

Also, in the generation of the visible light signal, if the footer is not transmitted, the header of the signal next to the signal to be transmitted may be combined instead of the footer. That is, in mode 3 of packet PWM and packet PPM, instead of the footer (SFT), the first payload (PHY payload) is followed by the header (SHR) for the next first payload. As a result, the end of the first payload can be clearly specified by the header for the next first payload, and the footer is not transmitted, so visible light communication can be performed more efficiently.

In each of the above-described embodiments and modifications, each component may be implemented by dedicated hardware, or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. For example, the program causes the computer to perform the visible light signal generation method illustrated by the flowchart of FIG. 122A.

Although the visible light signal generation method according to one or more aspects has been described based on the above embodiments and modifications, the present invention is not limited to these embodiments. As long as it does not deviate from the spirit of the present invention, various modifications that a person skilled in the art can think of are applied to this embodiment, and a form constructed by combining the components of different embodiments and modifications is also within the scope of the present invention. may be included in

(Embodiment 6)
In the present embodiment, a decoding method, an encoding method, and the like of a visible light signal will be described.

FIG. 123 is a diagram showing the format of a MAC frame in MPM.

The format of a MAC (medium access control) frame in MPM (Mirror Pulse Modulation) consists of MHR (medium access control header) and MSDU (medium access control service-data unit). The MHR field contains a sequence number subfield. The MSDU contains the frame payload and is of variable length. The bit length of MPDU (medium access control protocol-data unit) containing MHR and MSDU is set as macMpmMpduLength.

MPM is a modulation scheme in Embodiment 5, and is a scheme for modulating information or signals to be transmitted as shown in FIG. 121, for example.

FIG. 124 is a flowchart showing the processing operation of an encoding device that generates MAC frames in MPM. Specifically, FIG. 124 is a diagram showing how to determine the bit length of the sequence number subfield. Note that the encoding device is provided in, for example, the above-described transmitter or transmission device that transmits a visible light signal.

The sequence number subfield contains the frame sequence number (also called sequence number). The bit length of the sequence number subfield is set as macMpmSnLength. If the bit length of the sequence number subfield is set to variable length, the leading bit in the sequence number subfield is used as the last frame flag. That is, in this case, the sequence number subfield contains the last frame flag and a bit string indicating the sequence number. The last frame flag is set to 1 in the last frame and to 0 in other frames. That is, this final frame flag indicates whether or not the frame to be processed is the final frame. Note that this final frame flag corresponds to the stop bit described above. Also, the sequence number corresponds to the address described above.

First, the encoding device determines whether SN is set to a variable length (step S101a). Note that SN is the bit length of the sequence number subfield. That is, the encoding device determines whether macMpmSnLength indicates 0xf. When macMpmSnLength indicates 0xf, SN has a variable length, and when macMpmSnLength indicates other than 0xf, SN has a fixed length. When the encoding device determines that SN is not set to a variable length, that is, that SN is set to a fixed length (N in step S101a), it determines SN to be the value indicated by macMpmSnLength (step S102a). . At this time, the encoder does not use the last frame flag (ie LFF).

On the other hand, when the encoding apparatus determines that the SN is set to a variable length (Y in step S101a), it determines whether the frame to be processed is the last frame (step S103a). Here, when the encoding apparatus determines that the frame to be processed is the final frame (Y in step S103a), it determines SN to be 5 bits (step S104a). At this time, the encoder determines the last frame flag indicating 1 as the leading bit in the sequence number subfield.

If the encoding device determines that the frame to be processed is not the final frame (N in step S103a), the encoding device determines whether the value of the sequence number of the final frame is one of 1 to 15 (step S105a). The sequence number is an integer assigned to each frame in ascending order from 0. Moreover, in the case of N in step S103a, the number of frames is two or more. Therefore, in this case, the value of the sequence number of the final frame can be anywhere from 1-15, excluding zero.

When the encoding apparatus determines in step S105a that the value of the sequence number of the last frame is 1, it determines SN to be 1 bit (step S106a). At this time, the encoding device sets the value of the last frame flag, which is the leading bit in the sequence number subfield, to zero.

For example, if the last frame sequence number value is 1, then the last frame sequence number subfield is represented as (1,1) containing the last frame flag (1) and the sequence number value (1). . At this time, the encoding device determines the bit length of the sequence number subfield of the frame to be processed to be 1 bit. That is, the encoder determines a sequence number subfield that contains only the last frame flag (0).

When the encoding apparatus determines in step S105a that the value of the sequence number of the final frame is 2, it determines SN to be 2 bits (step S107a). Also at this time, the encoding device determines the value of the last frame flag to be 0.

For example, if the last frame sequence number value is 2, then the last frame sequence number subfield is represented as (1,0,1) containing the last frame flag (1) and the sequence number value (2). be done. The sequence number is indicated by a bit string, in which the leftmost bit is the LSB (least significant bit) and the rightmost bit is the MSB (most significant bit). Therefore, the sequence number value (2) is denoted as bit string (0,1). Thus, when the value of the sequence number of the final frame is 2, the encoding device determines the bit length of the sequence number subfield of the frame to be processed to be 2 bits. That is, the encoder determines a sequence number subfield containing a last frame flag (0) and a bit (0) or (1) indicating the sequence number.

When the encoding apparatus determines in step S105a that the value of the sequence number of the final frame is 3 or 4, it determines SN to be 3 bits (step S108a). Also at this time, the encoding device determines the value of the last frame flag to be 0.

When the encoding apparatus determines in step S105a that the value of the sequence number of the final frame is an integer between 5 and 8, it determines SN to be 4 bits (step S109a). Also at this time, the encoding device determines the value of the last frame flag to be 0.

When the encoding apparatus determines in step S105a that the value of the sequence number of the final frame is any integer from 9 to 15, it determines SN to be 5 bits (step S110a). Also at this time, the encoding device determines the value of the last frame flag to be 0.

FIG. 125 is a flowchart showing the processing operation of a decoding device that decodes MAC frames in MPM. Specifically, FIG. 125 is a diagram showing how to determine the bit length of the sequence number subfield. Note that the decoding device is provided in, for example, the above-described receiver or receiving device that receives the visible light signal.

Here, the decoding device determines whether SN is set to a variable length (step S201a). That is, the decoding device determines whether or not macMpmSnLength indicates 0xf. When the decoding device determines that the SN is not set to a variable length, ie, that the SN is set to a fixed length (N in step S201a), it determines the SN to be the value indicated by macMpmSnLength (step S202a). At this time, the decoding device does not use the last frame flag (ie LFF).

On the other hand, when the decoding device determines that the SN is set to a variable length (Y in step S201a), it determines whether the value of the final frame flag of the decoding target frame is 1 or 0 (step S203a). . That is, the decoding device determines whether or not the frame to be decoded is the final frame. Here, when the decoding device determines that the value of the last frame flag is 1 (1 in step S203a), it determines SN to be 5 bits (step S204a).

Further, when the decoding device determines that the value of the final frame flag is 0 (0 in step S203a), the value indicated by the bit string of the second to fifth bits in the sequence number subfield of the final frame is 1. -15 (step S205a). A final frame is a frame that has a final frame flag indicating 1 and is generated from the same source as the frame to be decoded. Also, each source is identified by its position in the captured image. Note that the source is, for example, divided into a plurality of frames (ie packets). That is, the last frame is the last frame among the multiple frames generated by splitting one source. Also, the value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield is the value of the sequence number.

When the decoding device determines in step S205a that the value indicated by the bit string is 1, it determines SN to be 1 bit (step S206a). For example, if the last frame sequence number subfield is two bits (1, 1), the last frame flag is 1 and the last frame sequence number, ie, the value indicated by the above bit string, is 1. At this time, the decoding device determines the bit length of the sequence number subfield of the decoding target frame to be 1 bit. That is, the decoding device determines the sequence number subfield of the decoding target frame to be (0).

When the decoding device determines in step S205a that the value indicated by the bit string is 2, it determines SN to be 2 bits (step S207a). For example, if the last frame sequence number subfield is 3 bits (1, 0, 1), the last frame flag is 1, and the value indicated by the last frame sequence number, i.e., the bit string (0, 1) is 2. In the above bit string, the leftmost bit is the LSB (least significant bit) and the rightmost bit is the MSB (most significant bit). At this time, the decoding device determines the bit length of the sequence number subfield of the decoding target frame to be 2 bits. That is, the decoding device determines the sequence number subfield of the decoding target frame to be (0,0) or (0,1).

When the decoding device determines in step S205a that the value indicated by the bit string is 3 or 4, it determines SN to be 3 bits (step S208a).

When the decoding device determines in step S205a that the value indicated by the bit string is any integer from 5 to 8, it determines SN to be 4 bits (step S209a).

When the decoding device determines in step S205a that the value indicated by the bit string is any integer from 9 to 15, it determines SN to be 5 bits (step S210a).

FIG. 126 is a diagram showing PIB attributes of MAC.

MAC PIB (physical-layer personal-area-network information base) attributes include macMpmSnLength and macMpmMpduLength. macMpmSnLength is any integer value in the range 0x0-0xf indicating the bit length of the sequence number subfield. Specifically, if macMpmSnLength is any integer value in the range 0x0-0xe, it indicates that integer value as the fixed bit length of the sequence number subfield. Also, when macMpmSnLength is 0xf, it indicates that the bit length of the sequence number subfield is variable.

macMpmMpduLength is any integer value in the range of 0x00-0xff and indicates the bit length of MPDU.

FIG. 127 is a diagram for explaining the MPM dimming method.

MPM has a dimming function. MPM dimming methods include, for example, (a) analog dimming method, (b) PWM dimming method, (c) VPPM dimming method, and (d) field insertion dimming method shown in FIG.

In the analog dimming method, for example, as shown in (a2), a visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, the overall brightness of the visible light signal is lowered, as shown in (a1), for example. Conversely, to brighten the visible light signal, the overall brightness of the visible light signal is increased, as shown in (a3), for example.

In the PWM dimming method, for example, as shown in (b2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (b1), the luminance is lowered for a short period during the period in which the high luminance light shown in (b2) is output. Conversely, to brighten the visible light signal, for example, as shown in (b3), the luminance is increased for a short period during the period in which the low luminance light shown in (b2) is output. Note that the fractional period mentioned above should be less than 1/3 of the original pulse width and less than 50 microseconds.

In the VPPM dimming method, a visible light signal is transmitted by changing the luminance, as shown in (c2), for example. Here, when darkening the visible light signal, the timing of the fall of luminance is advanced, as shown in (c1), for example. Conversely, when brightening the visible light signal, the fall timing of the luminance is delayed as shown in (c3), for example. Note that the VPPM modulation scheme can be used only for the PPM mode of PHY in MPM.

In the field insertion dimming method, a visible light signal including a plurality of PPDUs (physical-layer data units) is transmitted, for example, as shown in (d2). Here, to darken the visible light signal, a dimming field with a luminance lower than that of the PPDU is inserted between the PPDUs, as shown in (d1), for example. Conversely, to brighten the visible light signal, a dimming field with a higher luminance than the PPDU is inserted between the PPDUs, as shown in (d3), for example.

FIG. 128 is a diagram showing PIB attributes of PHY.

PHY (physical layer) PIB attributes include phyMpmMode, phyMpmPlcpHeaderMode, phyMpmPlcpCenterMode, phyMpmSymbolSize, phyMpmOddSymbolBit, phyMpmEvenSymbolBit, phyMpmSymbolOffset, and phyMpmSymbolUnit.

phyMpmMode is 0 or 1 and indicates the PHY mode of the MPM. Specifically, when phyMpmMode is 0, it indicates that the PHY mode is the PWM mode, and when it is 1, it indicates that the PHY mode is the PWM mode.

phyMpmPlcpHeaderMode is any integer value in the range from 0x0 to 0xf, indicating a PLCP (Physical Layer Conversion Protocol) header subfield mode and a PLCP footer subfield mode.

phyMpmPlcpCenterMode is any integer value in the range 0x0-0xf to indicate the PLCP center subfield mode.

phyMpmSymbolSize is any integer value in the range 0x0-0xf indicating the number of symbols in the payload subfield. Specifically, phyMpmSymbolSize, when 0x0, indicates that the number of symbols is variable and is referred to as N.

phyMpmOddSymbolBit is any integer value in the range 0x0-0xf indicating the bit length contained in each odd symbol of the payload subfield, referred to as _Modd .

phyMpmEvenSymbolBit is any integer value in the range 0x0-0xf indicating the bit length contained in each even symbol of the payload subfield, referred to as M _even .

phyMpmSymbolOffset is any integer value in the range 0x00-0xff indicating the symbol offset value of the payload subfield, referenced as _W1 .

phyMpmSymbolUnit is any integer value in the range 0x00-0xff indicating the symbol unit value of the payload subfield, referenced as _W2 .

FIG. 129 is a diagram for explaining the MPM.

The MPM is composed only of PSDU (PHY service data unit) fields. Also, the PSDU field contains the MPDU that is transformed by the PLCP of the MPM.

PLCP of MPM converts MPDU into 5 subfields as shown in FIG. The five subfields are the PLCP Header Subfield, Front Payload Subfield, PLCP Center Subfield, Back Payload Subfield, and PLCP Footer Subfield. The PHY mode of the MPM is set as phyMpmMode.

As shown in FIG. 129, the MPM PLCP includes a bit rearrangement unit 301a, a replication unit 302a, a front conversion unit 303a, and a back conversion unit 304a.

where (x ₀ , x ₁ , x ₂ , ...) are the bits contained in the MPDU, L _SN is the bit length of the sequence number subfield, and N is the length of each payload subfield. is the number of symbols. Bit rearrangement section 301a rearranges (x ₀ , x ₁ , x ₂ , . . . ) to (y ₀ , y ₁ , y ₂ , .

This rearrangement causes each bit contained in the sequence number subfield at the beginning of the MPDU to be moved backward by L _SN . The duplicating unit 302a duplicates the MPDU after the bit rearrangement.

The front payload subfield and the back payload subfield each consist of N symbols. Here, M _odd is the bit length included in the odd-numbered symbols, M _even is the bit length included in the even-numbered symbols, and _W1 is the symbol value offset (the offset value described above). , _W2 are the symbol value units (unit values described above). Note that N, M _odd , M _even , W ₁ , and W ₂ are set by the PHY PIB shown in FIG.

The front conversion unit 303a and the back conversion unit 304a _convert the rearranged payload bits (y0, y1, y2, .

The front transform unit 303a uses _zi to calculate the i-th symbol (that is, symbol value) of the front payload subfield by the following (Equation 6).

The back transform unit 304a uses z _i to calculate the i-th symbol (that is, the symbol value) of the back payload subfield by the following (equation 7).

FIG. 130 is a diagram showing PLCP header subfields.

As shown in FIG. 130, the PLCP header subfield consists of 4 symbols in PWM mode and 3 symbols in PPM mode.

FIG. 131 is a diagram showing the PLCP center subfield.

As shown in FIG. 131, the PLCP center subfield consists of 4 symbols in PWM mode and 3 symbols in PPM mode.

FIG. 132 is a diagram showing the PLCP footer subfield.

The PLCP footer subfield consists of 4 symbols in PWM mode and 3 symbols in PPM mode, as shown in FIG.

FIG. 133 is a diagram showing PWM mode waveforms of PHY in MPM.

In PWM mode, symbols must be transmitted in one of two states of light intensity: bright or dark. In the PWM mode of the PHY in MPM, the symbol values correspond to continuous time in microseconds. For example, as shown in FIG. 133, the first symbol value corresponds to the first bright state duration and the second symbol value corresponds to the next dark state duration. In the example shown in FIG. 133, the initial state of each subfield is bright, but it may be dark.

FIG. 134 is a diagram showing waveforms in PPM mode of PHY in MPM.

In PPM mode, the symbol values represent the time in microseconds from the start of a bright state to the start of the next bright state, as shown in FIG. The bright state duration must be less than 90% of the symbol value.

For both modes, the transmitter can only send a portion of the multiple symbols. However, the transmitter must transmit all symbols of the PLCP center subfield and at least N symbols. Each of the at least N symbols is a symbol contained in either the front payload subfield and the back payload subfield.

(Summary of Embodiment 6)
135 is a flowchart illustrating an example of a decoding method according to Embodiment 6. FIG. Note that the flowchart shown in FIG. 135 corresponds to the flowchart shown in FIG.

This decoding method is a method of decoding a visible light signal composed of a plurality of frames, and includes steps S310b, S320b, and S330b as shown in FIG. Each of these multiple frames also includes a sequence number and a frame payload.

In step S310b, based on macSnLength which is information for determining the bit length of the subfield in which the sequence number is stored in the decoding target frame, variable length determination is performed to determine whether the bit length of the subfield is variable. process.

In step S320b, the bit length of the subfield is determined based on the result of the variable length determination process. Then, in step S330b, the decoding target frame is decoded based on the determined bit length of the subfield.

Here, the determination of the bit length of the subfield in step S320b includes steps S321b to S324b.

That is, if it is determined in the variable length determination process in step S310b that the bit length of the subfield is not variable, the bit length of the subfield is determined to be the value indicated by macSnLength (step S321b). ).

On the other hand, if it is determined in the variable length determination process in step S310b that the bit length of the subfield is variable, it is determined whether or not the frame to be decoded is the last frame of the plurality of frames. A final determination process is performed (step S322b). Here, if it is determined to be the last frame (Y in step S322b), the bit length of that subfield is determined to a predetermined value (step S323b). On the other hand, if it is determined that the frame is not the final frame (N in step S322b), the bit length of the subfield is determined based on the value of the sequence number of the final frame (step S324b).

As a result, as shown in FIG. 135, regardless of whether the bit length of the subfield storing the sequence number (specifically, the sequence number subfield) is fixed or variable, the Bit length can be determined appropriately.

Here, in the final determination process in step S322b, it may be determined whether or not the decoding target frame is the final frame based on the final frame flag indicating whether or not the decoding target frame is the final frame. Specifically, in the final determination process in step S322b, when the final frame flag indicates 1, it is determined that the decoding target frame is the final frame, and when the final frame flag indicates 0, the decoding target frame is determined to be the final frame. is not the final frame. For example, the last frame flag may be included in the 1st bit of the subfield.

Accordingly, as shown in step S203a in FIG. 125, it is possible to appropriately determine whether or not the frame to be decoded is the final frame.

More specifically, in determining the bit length of the subfield in step S320b, if it is determined in the final determination processing in step S322b that the frame to be decoded is the final frame, the bit length of the subfield is determined as described above. may be determined to be 5 bits, which is a predetermined value of . That is, as shown in step S204a of FIG. 125, the bit length SN of the subfield is determined to be 5 bits.

Further, in determining the bit length of the subfield in step S320b, if it is determined in the final determination process in step S322b that the frame to be decoded is not the final frame and the value of the sequence number of the final frame is 1, The bit length of the subfield may be determined to be 1 bit. Also, when the value of the sequence number of the final frame is 2, the bit length of that subfield may be determined to be 2 bits. Also, when the value of the sequence number of the final frame is 3 or 4, the bit length of that subfield may be determined to be 3 bits. Also, when the value of the sequence number of the final frame is any integer from 5 to 8, the bit length of the subfield may be determined to be 4 bits. Also, when the value of the sequence number of the final frame is any integer from 9 to 15, the bit length of the subfield may be determined to be 5 bits. That is, as shown in steps S206a to S210a of FIG. 125, the bit length SN of the subfield is determined to be any one of 1 to 5 bits.

136 is a flowchart showing an example of an encoding method according to Embodiment 6. FIG. Note that the flowchart shown in FIG. 136 corresponds to the flowchart shown in FIG.

This encoding method is a method of encoding information to be encoded into a visible light signal composed of a plurality of frames, and as shown in FIG. including. Each of these multiple frames also includes a sequence number and a frame payload.

In step S410a, based on macSnLength, which is information for determining the bit length of the subfield in which the sequence number is stored in the frame to be processed, variable length determination is performed to determine whether the bit length of the subfield is variable. process.

In step S420a, the bit length of the subfield is determined based on the result of the variable length determination process. Then, in step S430a, a part of the information to be encoded is encoded into the processing target frame based on the determined bit length of the subfield.

Here, determination of the bit length of the subfield in step S420a includes steps S421a to S424a.

That is, if it is determined in the variable length determination process of step S410a that the bit length of the subfield is not variable, the bit length of the subfield is determined to be the value indicated by macSnLength (step S421a). ).

On the other hand, if it is determined in the variable length determination process of step S410a that the bit length of the subfield is variable, it is determined whether or not the frame to be processed is the final frame of the plurality of frames. A final determination process is performed (step S422a). Here, if it is determined to be the last frame (Y in step S422a), the bit length of that subfield is determined to a predetermined value (step S423a). On the other hand, if it is determined that the frame is not the final frame (N in step S422a), the bit length of the subfield is determined based on the value of the sequence number of the final frame (step S424a).

As a result, as shown in FIG. 136, regardless of whether the bit length of the subfield storing the sequence number (specifically, the sequence number subfield) is fixed or variable, the Bit length can be determined appropriately.

The decoding device according to the present embodiment includes a processor and memory, and the memory stores a program that causes the processor to execute the decoding method shown in FIG. 135 . The encoding device according to the present embodiment includes a processor and memory, and the memory stores a program that causes the processor to execute the encoding method shown in FIG. 136 . A program according to the present embodiment is a program that causes a computer to execute the decoding method shown in FIG. 135 or the encoding method shown in FIG.

(Embodiment 7)
In this embodiment, a transmission method for transmitting a light ID by a visible light signal will be described. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitting device) and receiver (or receiving device) in the above embodiments.

FIG. 137 is a diagram showing an example of AR image display by a receiver according to this embodiment.

Receiver 200 in the present embodiment is a receiver that includes an image sensor and display 201, and is configured as a smart phone, for example. Such a receiver 200 obtains the captured display image Pa, which is the normal captured image, and the decoding image, which is the visible light communication image or the bright line image, by imaging the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100 . The transmitter 100 has a shape like a light bulb, for example, and includes a glass bulb 141 and a light-emitting section 142 that shimmers while shining like a flame inside the glass bulb 141 . The light emitting unit 142 is illuminated by lighting of one or a plurality of light emitting elements (for example, LEDs) provided in the transmitter 100 . The transmitter 100 changes its luminance by blinking its light emitting section 142, and transmits a light ID (light identification information) according to the luminance change. This light ID is the visible light signal described above.

The receiver 200 captures an image of the transmitter 100 with a normal exposure time to acquire a captured display image Pa in which the transmitter 100 is displayed, and also captures the transmitter 100 with a communication exposure time shorter than the normal exposure time. to obtain a decoding image. The normal exposure time is the exposure time in the above-described normal shooting mode, and the communication exposure time is the exposure time in the above-described visible light communication mode.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P42 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. The receiver 200 then superimposes the AR image P42 on the target area, and displays the captured display image Pa on which the AR image P42 is superimposed on the display 201 .

For example, as in the example shown in FIG. 51, the receiver 200 recognizes the upper left area of the area where the transmitter 100 is displayed as the target area according to the recognition information. As a result, an AR image P42 showing, for example, a fairy is displayed as if it were flying around the transmitter 100. FIG.

FIG. 138 is a diagram showing another example of the captured display image Pa on which the AR image P42 is superimposed.

Receiver 200 displays captured display image Pa on which AR image P42 is superimposed on display 201, as shown in FIG.

Here, the recognition information described above indicates that the range having luminance equal to or higher than the threshold value in the captured display image Pa is the reference area. Further, the recognition information indicates that the target area is in a predetermined direction with respect to the reference area and that the target area is separated from the center (or the center of gravity) of the reference area by a predetermined distance. show.

Therefore, when the light emitting unit 142 of the transmitter 100 imaged by the receiver 200 flickers, as shown in FIG. move synchronously with That is, when the light emitting unit 142 flickers, the image 142a of the light emitting unit 142 displayed in the captured display image Pa also flickers. This image 142a is a range having luminance equal to or higher than the above threshold, and is a reference area. That is, since the reference area moves, the receiver 200 moves the target area to the moving target area so that the distance between the reference area and the target area is maintained at a predetermined distance. The AR image P42 is superimposed. As a result, when the light emitting unit 142 flickers, the AR image P42 superimposed on the target area of the captured display image Pa also moves in synchronization with the movement of the light emitting unit 142 . Note that the center position of the reference area may also move due to deformation of the light emitting section 142 . Therefore, even when the light emitting unit 142 is deformed, the AR image 42 may move so that the distance from the center position of the moving reference area is maintained at a predetermined distance.

In the above example, the receiver 200 recognizes the target area based on the recognition information and superimposes the AR image P42 on the target area, but the AR image P42 may be vibrated around the target area. . That is, the receiver 200 oscillates the AR image P42, for example, in the vertical direction according to a function that indicates changes in amplitude with respect to time. The function is a trigonometric function, eg a sine wave.

Also, the receiver 200 may change the size of the AR image P42 according to the size of the range having luminance equal to or higher than the above threshold. That is, the receiver 200 increases the size of the AR image P42 as the area of the bright area in the captured display image Pa increases, and conversely decreases the size of the AR image P42 as the area of the bright area decreases.

Alternatively, the receiver 200 increases the size of the AR image P42 as the average brightness in the range having brightness equal to or higher than the threshold is higher, and conversely, decreases the size of the AR image P42 as the average brightness is lower. may Note that instead of changing the size of the AR image P42, the transparency of the AR image P42 may be changed according to its average brightness.

Further, in the example shown in FIG. 138, any pixel in the image 142a of the light emitting unit 142 has a luminance equal to or higher than the threshold, but any pixel may have a luminance less than the threshold. That is, the range having brightness equal to or greater than the threshold value corresponding to the image 142a may be annular. In this case also, a range having luminance equal to or higher than the threshold is specified as the reference area, and the AR image P42 is superimposed on the target area that is a predetermined distance away from the center (or center of gravity) of the reference area.

FIG. 139 is a diagram showing another example in which receiver 200 displays an AR image in this embodiment.

The transmitter 100 is configured as an illumination device, for example, as shown in FIG. 139, and transmits the light ID by changing the luminance while illuminating a figure 143 consisting of three circles drawn on the wall, for example. Since the figure 143 is illuminated by the light from the transmitter 100, the luminance changes similarly to the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Pa and the decoding image in the same manner as described above by capturing the figure 143 illuminated by the transmitter 100 . The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from graphic 143 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P43 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. For example, the receiver 200 recognizes the area where the figure 143 is projected as the target area. The receiver 200 then superimposes the AR image P43 on the target area, and displays the captured display image Pa on which the AR image P43 is superimposed on the display 201 . For example, the AR image P43 is a face image of a character.

Here, the figure 143 consists of three circles as described above, but this figure 143 has few geometric features. Therefore, it is difficult to appropriately select and acquire an AR image corresponding to the graphic 143 from many images accumulated in the server only from the captured image obtained by imaging the graphic 143 . However, in the present embodiment, the receiver 200 acquires the light ID and acquires the AR image P43 corresponding to the light ID from the server. Therefore, even if many images are stored in the server, the AR image P43 corresponding to the light ID can be appropriately selected from among the many images and obtained as the AR image corresponding to the figure 143. can.

FIG. 140 is a flowchart showing the operation of receiver 200 in this embodiment.

Receiver 200 according to the present embodiment first acquires a plurality of AR image candidates (step S541). For example, the receiver 200 acquires a plurality of AR image candidates from the server through wireless communication (BTLE, Wi-Fi, etc.) different from visible light communication. Next, the receiver 200 images the subject (step S542). The receiver 200 acquires the captured display image Pa and the decoding image as described above by this imaging. However, if the subject is a photograph of the transmitter 100, the light ID is not transmitted from the subject, so the receiver 200 cannot acquire the light ID even if the decoding image is decoded. Can not.

Therefore, the receiver 200 determines whether or not the light ID could be acquired, that is, whether or not the light ID was received from the subject (step S543).

Here, if it is determined that the light ID has not been received (No in step S543), the receiver 200 determines whether or not the AR display flag set to itself is 1 (step S544). The AR display flag is a flag indicating whether or not the AR image may be displayed only from the captured display image Pa even if the light ID is not acquired. When the AR display flag is 1, the AR display flag indicates that the AR image may be displayed only from the captured display image Pa. When the AR display flag is 0, the AR display flag is , indicates that the AR image should not be displayed only from the captured display image Pa.

When determining that the AR display flag is 1 (Yes in step S544), the receiver 200 selects a candidate corresponding to the captured display image Pa from among the plurality of AR image candidates acquired in step S541 as an AR image. (step S545). That is, the receiver 200 extracts the feature amount included in the captured display image Pa, and selects a candidate associated with the extracted feature amount as the AR image.

Then, the receiver 200 displays the selected candidate AR image superimposed on the captured display image Pa (step S546).

On the other hand, when determining that the AR display flag is 0 (No in step S544), the receiver 200 does not display the AR image.

Further, when it is determined that the light ID has been received in step S543 (Yes in step S543), the receiver 200 selects candidates associated with the light ID from among the plurality of AR image candidates acquired in step S541. It is selected as an AR image (step S547). Then, the receiver 200 displays the selected candidate AR image superimposed on the captured display image Pa (step S546).

Although the AR display flag is set in the receiver 200 in the above example, it may be set in the server. In this case, the receiver 200 inquires of the server whether the AR display flag is 1 or 0 in step S544.

Thus, when the receiver 200 performs imaging but does not receive the light ID, it is possible to control whether or not to display an AR image on the receiver 200 using the AR display flag.

FIG. 141 is a diagram for explaining the operation of transmitter 100 in this embodiment.

For example, transmitter 100 is configured as a projector. Here, the intensity of the light emitted from the projector and reflected on the screen changes depending on various factors such as aging deterioration of the light source of the projector and the distance from the light source to the screen. When the light intensity is low, it becomes difficult for the receiver 200 to receive the light ID transmitted from the transmitter 100 .

Therefore, transmitter 100 according to the present embodiment adjusts the parameters for causing the light source to emit light in order to suppress changes in light intensity according to each factor. This parameter is at least one of the value of the current input to the light source for causing the light source to emit light and the light emission time (more specifically, the light emission time per unit time). For example, the intensity of the light from the light source increases as the current value increases and the light emission time increases.

In other words, the transmitter 100 adjusts the parameters so that the light from the light source is strengthened as the light source deteriorates over time. Specifically, the transmitter 100 has a timer, and adjusts the parameters so that the longer the light source usage time measured by the timer is, the stronger the light from the light source is. That is, the longer the transmitter 100 is used, the higher the current value of the light source or the longer the light emission time. Alternatively, the transmitter 100 detects the intensity of light emitted from the light source and adjusts parameters so that the intensity of the detected light does not decrease. That is, the transmitter 100 adjusts the parameters so that the lower the intensity of the detected light, the stronger the light.

Further, the transmitter 100 adjusts the parameters so that the longer the irradiation distance from the light source to the screen, the stronger the light from the light source. Specifically, the transmitter 100 detects the intensity of the light emitted and reflected on the screen, and adjusts the parameters so that the lower the intensity of the detected light, the stronger the light from the light source. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the intensity of the detected light decreases. This adjusts the parameters so that the intensity of the reflected light is constant regardless of the irradiation distance. Alternatively, the transmitter 100 detects the irradiation distance from the light source to the screen with a range sensor, and adjusts the parameters so that the longer the detected irradiation distance is, the stronger the light from the light source is.

Also, the transmitter 100 adjusts the parameters so that the darker the screen, the stronger the light from the light source. Specifically, the transmitter 100 detects the color of the screen by capturing an image of the screen, and adjusts the parameters so that the blacker the detected color, the stronger the light from the light source. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the detected color becomes darker. This adjusts the parameters so that the intensity of the reflected light is constant regardless of the color of the screen.

Further, the transmitter 100 adjusts the parameters so that the stronger the external light, the stronger the light from the light source. Specifically, the transmitter 100 detects the difference between the brightness of the screen when the light source is turned on and light is emitted and the screen brightness when the light source is turned off and no light is emitted. Then, the transmitter 100 adjusts the parameters so that the light from the light source is strengthened as the difference in brightness is smaller. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the difference in brightness decreases. As a result, the parameters are adjusted so that the S/N ratio of the light ID is constant regardless of the outside light. Alternatively, if the transmitter 100 is configured as an LED display, for example, the intensity of sunlight may be detected, and parameters may be adjusted such that the stronger the intensity of the sunlight, the stronger the light from the light source. .

Note that the adjustment of the parameters as described above may be performed when an operation is performed by the user. For example, the transmitter 100 includes a calibration button, and performs the parameter adjustments described above when the calibration button is pressed by the user. Alternatively, the transmitter 100 may perform the above parameter adjustments on a regular basis.

FIG. 142 is a diagram for explaining another operation of transmitter 100 in the present embodiment.

For example, the transmitter 100 may be configured as a projector and illuminates the screen through the front member with light from a light source. If the projector is a liquid crystal projector, its front member is a liquid crystal panel, and if the projector is a DLP (registered trademark) projector, its front member is a DMD (Digital Mirror Device). That is, the front member is a member that adjusts the brightness of the image for each pixel. Also, the light source irradiates the front member with light, and the intensity of the light is switched between High and Low. In addition, the light source adjusts the time-average brightness by adjusting the High time per unit time.

Here, if the transmittance of the front member is, for example, 100%, the light source is darkened so that the image projected from the projector onto the screen is not too bright. That is, the light source shortens the High time per unit time.

At this time, the light source widens the pulse width of the light ID when transmitting the light ID according to the luminance change.

On the other hand, if the transmittance of the front member is, for example, 20%, the light source will be bright so that the image projected from the projector onto the screen is not too dark. That is, the light source lengthens the High time per unit time.

At this time, the light source narrows the pulse width of the light ID when transmitting the light ID according to the luminance change.

Thus, when the light source is dark, the pulse width of the light ID is widened, and conversely, when the light source is bright, the pulse width of the light ID is narrowed. It is possible to prevent the intensity of the image from being too weak or too bright.

Although the transmitter 100 is a projector in the above example, it may be configured as a large LED display. A large LED display comprises a pixel switch and a common switch. An image is expressed by turning on and off the pixel switch, and a light ID is transmitted by turning on and off the common switch. In this case, functionally, the pixel switch corresponds to the front member and the common switch corresponds to the light source. When the average luminance by the pixel switch is high, the pulse width of the optical ID by the common switch may be shortened.

FIG. 143 is a diagram for explaining another operation of transmitter 100 in the present embodiment. Specifically, FIG. 143 shows the relationship between the dimming degree of the transmitter 100 configured as a spotlight with a dimming function and the current (specifically, the peak current value) input to the light source of the transmitter 100. Show relationship.

The transmitter 100 receives a dimming level specified for the light source provided therein, and causes the light source to emit light at the specified dimming level. The dimming degree is the ratio of the average luminance of the light source to the maximum average luminance. The average luminance is the luminance over time average, not the instantaneous luminance. Further, adjustment of the degree of dimming is realized by adjusting the value of the current input to the light source or by adjusting the time during which the brightness of the light source is Low. The time during which the luminance of the light source is Low may be the time during which the light source is turned off.

Here, when transmitting a signal to be transmitted as an optical ID, transmitter 100 generates an encoded signal by encoding the signal to be transmitted in a predetermined mode. Transmitter 100 transmits the encoded signal as an optical ID (that is, a visible light signal) by changing the luminance of a light source provided therein according to the encoded signal.

For example, when the specified dimming degree is 0% or more and x3(%) or less, the transmitter 100 generates an encoded signal by encoding the signal to be transmitted in PWM mode with a duty ratio of 35%. . x3(%) is, for example, 50%. In this embodiment, the PWM mode with a duty ratio of 35% is also called the first mode, and x3 is also called the first value.

In other words, when the designated dimming degree is 0% or more and x3(%) or less, the transmitter 100 maintains the duty ratio of the visible light signal at 35%, and sets the dimming degree of the light source to the value of the peak current. Adjust by

Further, when the specified dimming degree is greater than x3(%) and equal to or less than 100%, the transmitter 100 encodes the signal to be transmitted in PWM mode with a duty ratio of 65% to generate the encoded signal. Generate. Note that in the present embodiment, the PWM mode with a duty ratio of 65% is also referred to as the second mode.

In other words, when the specified dimming degree is greater than x3(%) and equal to or less than 100%, the transmitter 100 maintains the duty ratio of the visible light signal at 65% while increasing the dimming degree of the light source to the peak current adjusted by the value of

In this way, transmitter 100 in the present embodiment accepts the dimming level specified for the light source as the specified dimming level. Then, when the designated dimming degree is equal to or less than the first value, the transmitter 100 transmits the signal encoded in the first mode by changing the luminance while causing the light source to emit light at the designated dimming degree. Further, when the value of the designated dimming degree is greater than the first value, the transmitter 100 transmits the signal encoded in the second mode by changing the luminance while causing the light source to emit light at the designated dimming degree. do. Specifically, the duty ratio of the signal encoded in the second mode is greater than the duty ratio of the signal encoded in the first mode.

Here, since the duty ratio of the second mode is greater than the duty ratio of the first mode, the change rate of the peak current with respect to the dimming level in the second mode is defined as the change in the peak current with respect to the dimming level in the first mode. can be smaller than the rate.

Also, when the specified dimming degree exceeds x3 (%), the mode is switched from the first mode to the second mode. Therefore, at this time, the peak current can be reduced instantaneously. That is, when the specified dimming level is x3 (%), the peak current is y3 (mA). can be reduced to Note that y3 (mA) is, for example, 143 mA, and y2 (mA) is, for example, 100 mA. As a result, it is possible to suppress the peak current from becoming larger than y3 (mA) in order to increase the dimming degree, and it is possible to suppress deterioration of the light source due to the flow of a large current.

Also, when the designated dimming degree exceeds x4 (%), the peak current becomes larger than y3 (mA) even if the mode is the second mode. However, if the specified dimming degree rarely exceeds x4 (%), deterioration of the light source can be suppressed. Note that, in the present embodiment, x4 described above is also referred to as a second value. Also, in the example shown in FIG. 143, x4(%) is less than 100%, but may be 100%.

In other words, in the transmitter 100 of the present embodiment, when the designated dimming degree is greater than the first value and equal to or less than the second value, the signal encoded in the second mode is transmitted according to the luminance change. is less than the peak current value of the light source for transmitting the signal encoded in the first mode with luminance change when the specified dimming level is the first value.

Thus, by switching the signal encoding mode, the value of the peak current of the light source when the designated dimming degree is greater than the first value and less than or equal to the second value is less than the value of the peak current of the light source in some cases. Therefore, as the designated dimming degree is increased, it is possible to suppress a large peak current from flowing through the light source. As a result, deterioration of the light source can be suppressed.

Furthermore, when the designated dimming degree is x1 (%) or more and smaller than x2 (%), transmitter 100 in the present embodiment operates in the first mode while causing the light source to emit light at the designated dimming degree. while transmitting a signal encoded with by luminance change, and maintaining a constant value of peak current for a specified dimming change. x2(%) is smaller than x3(%). Note that, in the present embodiment, the above x2 is also referred to as a third value.

In other words, when the designated dimming degree is less than x2 (%), the transmitter 100 increases the time during which the light source is turned off as the designated dimming degree becomes smaller. The light source is turned on and the peak current value is maintained at a constant value. Specifically, transmitter 100 lengthens the cycle of transmitting each of the encoded signals while maintaining the duty ratio of the encoded signals at 35%. As a result, the time during which the light source is turned off, that is, the extinguishing period is lengthened. As a result, it is possible to reduce the degree of dimming while maintaining a constant peak current value. In addition, even when the designated dimming level becomes small, the value of the peak current is kept constant, which makes it easier for the receiver 200 to receive a visible light signal (i.e., light ID), which is a signal transmitted according to changes in brightness. be able to.

Here, the transmitter 100 determines the time for turning off the light source so that one period, which is the sum of the time for transmitting the encoded signal with the luminance change and the time for turning off the light source, does not exceed 10 milliseconds. . For example, if the light source is turned off for too long and one period exceeds 10 milliseconds, the human eye may perceive the luminance change of the light source for transmitting the encoded signal as flickering. . Therefore, in the present embodiment, the time during which the light source is turned off is determined so that one cycle does not exceed 10 milliseconds, so that it is possible to prevent flickering from being perceived by people.

Furthermore, even when the designated dimming degree is less than x1(%), the transmitter 100 transmits the signal encoded in the first mode by luminance change while causing the light source to emit light at the designated dimming degree. At this time, the transmitter 100 reduces the value of the peak current as the designated dimming degree decreases, thereby causing the light source to emit light at the decreased designated dimming degree. x1(%) is smaller than x2(%). Note that, in the present embodiment, the above x1 is also referred to as a fourth value.

As a result, even if the designated dimming degree is smaller, the light source can be caused to emit light appropriately at the designated dimming degree.

Here, in the example shown in FIG. 143, the maximum peak current value in the first mode (ie y3 (mA)) is higher than the maximum peak current value in the second mode (ie y4 (mA)). Smaller, but may be the same. That is, the transmitter 100 encodes the signal to be transmitted in the first mode up to x3a(%), which is greater than x3(%). Then, when the specified dimming degree is x3a (%), the transmitter 100 turns on the light source with the same peak current value as the maximum peak current value (that is, y4 (mA)) in the second mode. light up. In this case, x3a is the first value. The maximum peak current value in the second mode is the peak current value when the specified dimming degree is the maximum value, that is, 100%.

That is, in the present embodiment, the value of the peak current of the light source when the designated dimming degree is the first value is the same as the value of the peak current of the light source when the designated dimming degree is the maximum value. There may be. In this case, since the range of the dimming degree that causes the light source to emit light with a peak current of y3 (mA) or more is widened, the light ID can be easily received by the receiver 200 in a wide range of dimming degree. In other words, even in the first mode, since a large peak current can be passed through the light source, it is possible to make it easier for the receiver to receive the signal transmitted by the luminance change of the light source. In this case, the period during which a large peak current flows becomes longer, so the light source is likely to deteriorate.

FIG. 144 is a diagram showing a comparative example for explaining the easiness of receiving an optical ID in this embodiment.

In this embodiment, as shown in FIG. 143, the first mode is used when the degree of dimming is small, and the second mode is used when the degree of dimming is large. The first mode is a mode that increases the peak current even if the increase in the degree of dimming is small, and the second mode is the mode that suppresses the increase in the peak current even if the increase in the degree of dimming is large. Therefore, the second mode suppresses the flow of a large peak current to the light source, thereby suppressing deterioration of the light source. Furthermore, in the first mode, a large peak current flows through the light source even if the degree of dimming is small, so that the receiver 200 can easily receive the light ID.

On the other hand, when the second mode is used even when the degree of dimming is small, as shown in FIG. difficult to receive.

Therefore, in transmitter 100 of the present embodiment, it is possible to achieve both suppression of deterioration of the light source and ease of reception of the light ID.

Further, when the value of the peak current of the light source exceeds the fifth value, the transmitter 100 may stop transmitting the signal due to the luminance change of the light source. The fifth value may be y3 (mA), for example.

This makes it possible to further suppress deterioration of the light source.

Also, the transmitter 100 may measure the usage time of the light source, as in the example shown in FIG. Then, when the usage time is equal to or longer than the predetermined time, the transmitter 100 may transmit the signal by changing the brightness using the value of the parameter for causing the light source to emit light at a dimming level greater than the specified dimming level. . In this case, the value of the parameter may be the value of the peak current or the time to turn off the light source. As a result, it is possible to prevent the light ID from being difficult to be received by the receiver 200 due to deterioration of the light source over time.

Alternatively, the transmitter 100 may measure the usage time of the light source, and if the usage time is longer than or equal to the predetermined time, the pulse width of the current of the light source may be made larger than when the usage time is less than the predetermined time. . As a result, as described above, it is possible to prevent the light ID from being difficult to receive due to deterioration of the light source.

In the above-described embodiment, the transmitter 100 is switched between the first mode and the second mode according to the specified dimming degree. good. That is, when the switch is operated by the user, the transmitter 100 switches the first mode to the second mode, or conversely switches the second mode to the first mode. Also, the transmitter 100 may notify the user when the mode is switched. For example, the transmitter 100 may notify the user of mode switching by emitting a sound, blinking a light source in a cycle visible to a person, or lighting a notification LED. Also, the transmitter 100 may notify the user that the relationship changes not only when the mode is switched, but also when the relationship between the peak current and the dimming level changes. This point is, for example, the point at which the degree of dimming changes from x1 (%) shown in FIG. 143 or the point at which the degree of dimming changes from x2 (%).

FIG. 145A is a flowchart showing the operation of transmitter 100 in this embodiment.

The transmitter 100 first receives the dimming degree designated for the light source as the designated dimming degree (step S551). Next, the transmitter 100 transmits a signal according to the brightness change of the light source (step S552). Specifically, when the specified dimming level is equal to or lower than the first value, the transmitter 100 causes the light source to emit light at the specified dimming level, while transmitting the signal encoded in the first mode according to the luminance change. Send. Further, when the designated dimming degree is greater than the first value, the transmitter 100 transmits the signal encoded in the second mode according to the luminance change while causing the light source to emit light at the designated dimming degree. Here, when the specified dimming degree is greater than the first value and less than or equal to the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by luminance change is It is less than the value of the peak current of the light source for transmitting the signal encoded in the first mode with luminance change when the specified dimming level is the first value.

FIG. 145B is a block diagram showing the configuration of transmitter 100 in this embodiment.

The transmitter 100 includes a reception section 551 and a transmission section 552 . The receiving unit 551 receives the dimming level specified for the light source as the specified dimming level (step S551). The transmission unit 552 transmits a signal according to the luminance change of the light source. Specifically, when the specified dimming level is equal to or lower than the first value, the transmitting unit 552 causes the light source to emit light at the specified dimming level, while transmitting the signal encoded in the first mode according to the luminance change. Send. Further, when the designated dimming level is greater than the first value, the transmitting section 552 transmits the signal encoded in the second mode according to the luminance change while causing the light source to emit light at the designated dimming level. Here, when the specified dimming degree is greater than the first value and less than or equal to the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by luminance change is It is less than the value of the peak current of the light source for transmitting the signal encoded in the first mode with luminance change when the specified dimming level is the first value.

As a result, as shown in FIG. 143, by switching the signal encoding mode, the value of the peak current of the light source when the designated dimming level is greater than the first value and equal to or less than the second value is changed to the designated dimming level. It is smaller than the value of the peak current of the light source when the luminous intensity is the first value. Therefore, as the designated dimming degree is increased, it is possible to suppress a large peak current from flowing through the light source. As a result, deterioration of the light source can be suppressed.

FIG. 146 is a diagram showing another example in which receiver 200 displays an AR image in this embodiment.

The receiver 200 obtains the captured display image Pk, which is the above-described normal captured image, and the decoding image, which is the above-described visible light communication image or bright line image, by imaging the subject with the image sensor.

Specifically, the image sensor of receiver 200 captures transmitter 100 configured as signage and person 21 next to transmitter 100 . Transmitter 100 is the transmitter in each of the above-described embodiments, and includes one or more light-emitting elements (for example, LEDs) and translucent plate 144 having translucency such as frosted glass. The one or more light emitting elements emit light inside the transmitter 100, and the light from the one or more light emitting elements is transmitted through the translucent plate 144 and emitted to the outside. As a result, the translucent plate 144 of the transmitter 100 shines brightly. Such a transmitter 100 changes its luminance by blinking one or a plurality of light emitting elements, and transmits light ID (light identification information) according to the luminance change. This light ID is the visible light signal described above.

Here, the transparent plate 144 has a message “Hold your smartphone over here”. Therefore, the user of the receiver 200 makes the person 21 stand next to the transmitter 100 and instructs the person 21 to put his/her arm over the transmitter 100 . Then, the user directs the camera (that is, the image sensor) of the receiver 200 toward the person 21 and the transmitter 100 and takes an image. The receiver 200 acquires a captured display image Pk in which the transmitter 100 and the person 21 are imaged with normal exposure time. Further, the receiver 200 obtains a decoding image by imaging the transmitter 100 and the person 21 with a communication exposure time that is shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P44 and recognition information corresponding to the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, receiver 200 recognizes an area where signage, which is transmitter 100, is displayed as a target area.

The receiver 200 then superimposes the AR image P44 on the captured display image Pk so that the target area is covered with the AR image P44, and displays the captured display image Pk on the display 201. FIG. For example, the receiver 200 acquires an AR image P44 showing a soccer player. In this case, since the AR image P44 is superimposed so as to cover the target area of the captured display image Pk, the captured display image Pk can be displayed as if the soccer player actually exists next to the person 21. . As a result, the person 21 can be photographed with the soccer player even if the soccer player is not next to them. More specifically, the person 21 can be photographed with the soccer player, with the arm of the person 21 on the soccer player's shoulder.

(Embodiment 8)
In this embodiment, a transmission method for transmitting a light ID by a visible light signal will be described. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitting device) and receiver (or receiving device) in the above embodiments.

FIG. 147 is a diagram for explaining the operation of transmitter 100 in this embodiment. Specifically, FIG. 147 shows the relationship between the dimming degree of the transmitter 100 configured as a spotlight with a dimming function and the current (specifically, the value of the peak current) input to the light source of the transmitter 100. Show relationship.

Transmitter 100 according to the present embodiment encodes a signal to be transmitted in a PWM mode with a duty ratio of 35% when the specified dimming degree is 0% or more and x14(%) or less. to generate That is, when the designated dimming degree changes from 0% to x14(%), the transmitter 100 increases the peak current value while maintaining the duty ratio of the visible light signal at 35%. , causes the light source to emit light at its specified dimming level. Note that the PWM mode with a duty ratio of 35% is also referred to as the first mode as in the seventh embodiment, and x14 described above is also referred to as the first value. For example, x14(%) is a value within the range of 50-60%.

Further, when the specified dimming degree is x13(%) or more and 100% or less, the transmitter 100 generates a coded signal by coding the signal to be transmitted in PWM mode with a duty ratio of 65%. . In other words, when the designated dimming level changes from 100% to x13(%), the transmitter 100 maintains the duty ratio of the visible light signal at 65% and suppresses the value of the peak current. Causes the light source to emit light at the specified dimming level. Note that the PWM mode with a duty ratio of 65% is also referred to as the second mode as in the seventh embodiment, and x13 described above is also referred to as the second value. Here, x13(%) is a value smaller than x14(%), for example, within a range of 40 to 50%.

Thus, in the present embodiment, when the specified dimming degree increases, the PWM mode changes from the PWM mode with a duty ratio of 35% to the PWM mode with a duty ratio of 65% at the dimming degree of x14 (%). can be switched. On the other hand, when the specified dimming degree decreases, the PWM mode changes from a PWM mode with a duty ratio of 65% to a PWM mode with a duty ratio of 35% at a dimming degree of x13 (%) smaller than the dimming degree of x14 (%). can be switched to That is, in the present embodiment, the dimming degree at which the PWM mode is switched differs depending on whether the designated dimming degree increases or decreases. Hereinafter, the dimming level at which the PWM mode is switched is referred to as a switching point.

Therefore, in this embodiment, frequent switching of the PWM mode can be suppressed. In the example shown in FIG. 143 of Embodiment 7, the PWM mode switching point is 50%, which is the same when the designated dimming degree increases and when the designated dimming degree decreases. . As a result, in the example shown in FIG. 143, when the specified dimming degree is repeatedly increased and decreased by around 50%, the PWM mode frequently switches between the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65%. can be switched. However, in the present embodiment, the switching point of the PWM mode differs depending on whether the designated dimming degree increases or decreases, so frequent switching of the PWM mode is avoided. can be suppressed.

Further, in the present embodiment, similarly to the example shown in FIG. 143 of Embodiment 7, when the designated dimming degree is small, the PWM mode with a small duty ratio is used. When is large, a high duty ratio PWM mode is used.

Therefore, when the specified dimming degree is large, the PWM mode with a large duty ratio is used, so that the rate of change of the peak current with respect to the dimming degree can be reduced, and the small peak current causes the light source to emit light at a large dimming degree. can be made For example, in PWM mode with a small duty ratio, such as 35% duty ratio, the light source cannot emit light at 100% dimming unless the peak current is 250 mA. However, in the present embodiment, a PWM mode with a large duty ratio, such as a duty ratio of 65%, is used for large dimming levels. % dimming. In other words, it is possible to prevent the life of the light source from being shortened due to overcurrent flowing through the light source.

Moreover, when the designated dimming degree is small, the PWM mode with a small duty ratio is used, so that the change rate of the peak current with respect to the dimming degree can be increased. As a result, the visible light signal can be transmitted with a large peak current while the light source is illuminated with a small dimming degree. The light source emits brighter light as the input current increases. Therefore, when a visible light signal is transmitted with a large peak current, the receiver 200 can easily receive the visible light signal. In other words, the range of dimming levels over which visible light signals receivable to the receiver 200 can be transmitted can be extended to smaller dimming levels. For example, as shown in FIG. 195, if the peak current is Ia (mA) or more, the receiver 200 can receive the visible light signal transmitted by the peak current. In this case, in the PWM mode with a large duty ratio such as 65%, the range of dimming degree in which a visible light signal that can be received can be transmitted is x12(%) or more. However, in the PWM mode with a small duty ratio such as a duty ratio of 35%, the range of dimming degree that can transmit a visible light signal that can be received must be x11 (%) or more, which is smaller than x12 (%). can be done.

By switching the PWM mode in this way, the life of the light source can be extended and the visible light signal can be transmitted in a wide dimming range.

FIG. 148A is a flow chart showing a transmission method according to this embodiment.

The transmission method according to the present embodiment is a transmission method for transmitting a signal according to changes in luminance of a light source, and includes reception step S561 and transmission step S562. In receiving step S561, the transmitter 100 receives the dimming degree designated for the light source as the designated dimming degree. In the transmission step S562, the transmitter 100 transmits the signal encoded in the first mode or the second mode according to luminance change while causing the light source to emit light at the specified dimming degree. Here, the duty ratio of the signal encoded in the second mode is greater than the duty ratio of said signal encoded in the first mode. Further, in transmission step S562, when the designated dimming level is changed from a small value to a large value, the transmitter 100 selects the mode used for signal encoding when the designated dimming level is the first value, Switch from the first mode to the second mode. Further, when the designated dimming level is changed from a large value to a small value, the transmitter 100 changes the mode used for signal encoding from the second mode to the second mode when the designated dimming level is the second value. Switch to the first mode. Here, the second value is less than the first value.

For example, the first mode and the second mode are the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65%, respectively, shown in FIG. Also, the first value and the second value are x14(%) and x15(%) shown in FIG. 147, respectively.

As a result, the designated dimming degree (that is, the switching point) at which switching between the first mode and the second mode is performed differs depending on whether the designated dimming degree increases or decreases. Therefore, frequent switching between these modes can be suppressed. That is, it is possible to suppress the occurrence of so-called chattering. As a result, the operation of transmitter 100 that transmits signals can be stabilized. Also, the duty ratio of the signal encoded in the second mode is greater than the duty ratio of the signal encoded in the first mode. Therefore, similarly to the transmission method shown in FIG. 143, the greater the designated dimming degree, the more it is possible to suppress the flow of a large peak current to the light source. As a result, deterioration of the light source can be suppressed. In addition, since deterioration of the light source is suppressed, long-term communication can be performed between various devices. Also, when the designated dimming degree is small, the first mode with a small duty ratio is used. Therefore, the above peak current can be increased, and a signal that is easily received by the receiver 200 can be transmitted as a visible light signal.

Further, in the transmission step S562, when switching from the first mode to the second mode, the transmitter 100 sets the peak current of the light source for transmitting the encoded signal by changing the luminance to the second mode. A current value of 1 is changed to a second current value that is smaller than the first current value. Furthermore, when switching from the second mode to the first mode, the transmitter 100 changes the peak current from the third current value to a fourth current value that is greater than the third current value. change. Here, the first current value is greater than the fourth current value, and the second current value is greater than the third current value.

For example, the first current value, second current value, third current value, and fourth current value are current value Ie, current value Ic, current value Ib, and current value Id shown in FIG. be.

Thereby, it is possible to appropriately switch between the first mode and the second mode.

FIG. 148B is a block diagram showing the configuration of transmitter 100 in this embodiment.

Transmitter 100 according to the present embodiment is a transmitter that transmits a signal according to changes in luminance of a light source, and includes receiving section 561 and transmitting section 562 . The receiving unit 561 receives the dimming level specified for the light source as the specified dimming level. The transmission unit 562 transmits the signal encoded in the first mode or the second mode according to the luminance change while causing the light source to emit light at the specified dimming degree. Here, the duty ratio of the signal encoded in the second mode is greater than the duty ratio of said signal encoded in the first mode. Further, when the designated dimming level is changed from a small value to a large value, the transmitting unit 562 changes the mode used for signal encoding from the first mode to the first mode when the designated dimming level is the first value. Switch to the second mode. Furthermore, when the designated dimming level is changed from a large value to a small value, the transmitting unit 562 changes the mode used for signal encoding from the second mode to the second mode when the designated dimming level is the second value. Switch to the first mode. Here, the second value is less than the first value.

Such a transmitter 100 implements the transmission method of the flowchart shown in FIG. 148A.

FIG. 149 is a diagram showing an example of detailed configuration of a visible light signal in this embodiment.

Such a visible light signal is a PWM mode signal.

A visible light signal packet consists of an L data portion, a preamble, and an R data portion. Note that the L data portion and the R data portion each correspond to a payload.

The preamble alternately indicates High and Low luminance values along the time axis. That is, the preamble indicates a high luminance value for a time length _C0 , indicates a low luminance value for the next time length _C1 , indicates a high luminance value for the next time length _C2 , and indicates a high luminance value for the next time length C2. Only ₃ indicates a Low luminance value. Note that the time lengths _C0 and _C3 are, for example, 100 μs. Also, the time lengths _C1 and _C2 are, for example, 90 μs shorter than the time lengths _C1 and _C3 by 10 μs.

The L data section alternately indicates high and low luminance values along the time axis and is placed immediately before the preamble. That is, the L data part indicates a high luminance value for the time length D' ₀ , indicates a low luminance value for the next time length D' ₁ , indicates a high luminance value for the next time length D' ₂ , A low luminance value is shown for the next time length D' ₃ . Note that the time lengths D' ₀ to D' ₃ are determined according to mathematical formulas according to the signal to be transmitted. This formula is D' ₀ = W ₀ + W ₁ × (3-y ₀ ), D' ₁ = W ₀ + W ₁ × (7-y ₁ ), D' ₂ = W ₀ + W ₁ × (3-y ₂ ), and D′ ₃ =W ₀ +W ₁ ×(7−y ₃ ). Here, the constant _W0 is, for example, 110 μs, and the constant _W1 is, for example, 30 μs. The variables y ₀ and y ₂ are any integers from 0 to 3 represented by 2 bits, and the variables y ₁ and y ₃ are any integers from 0 to 7 represented by 3 bits. Variables y ₀ to y ₃ are signals to be transmitted. In FIGS. 149 to 152, "*" is used as a symbol representing multiplication.

The R data portion alternately indicates High and Low luminance values along the time axis, and is placed immediately after the preamble. That is, the R data part indicates a high luminance value for the time length _D0 , indicates a low luminance value for the next time length _D1 , indicates a high luminance value for the next time length _D2 , and indicates a high luminance value for the next time length D2. A length D of ₃ indicates the Low luminance value. Note that the time lengths D ₀ to D ₃ are determined according to mathematical formulas according to the signal to be transmitted. The formulas are _D0 = _W0 + _W1 * _y0 , _D1 = _W0 + _W1 * _y1 , _D2 = _W0 + _W1 * _y2 , and _D3 = _W0 + _W1 * _y3 .

Here, the L data portion and the R data portion have a complementary relationship with respect to brightness. That is, if the brightness of the L data section is bright, the brightness of the R data section is dark, and conversely, if the brightness of the L data section is dark, the brightness of the R data section is bright. That is, the sum of the time length of the L data portion and the time length of the R data portion is constant regardless of the signal to be transmitted. In other words, regardless of the signal to be transmitted, the time-average brightness of the visible light signal transmitted from the light source can be made constant.

Also, D' ₀ = W ₀ + W ₁ × (3-y ₀ ), D' ₁ = W ₀ + W ₁ × (7-y ₁ ), D' ₂ = W ₀ + W ₁ × (3-y ₂ ), and by changing the ratio of 3 and 7 in D′ ₃ =W ₀ +W ₁ ×(7−y ₃ ), the duty ratio of the PWM mode can be changed. Note that the ratio between 3 and 7 corresponds to the ratio between the maximum values of variables _y0 and _y2 and the maximum values of variables _y1 and _y3 . For example, when the ratio is 3:7, a PWM mode with a small duty ratio is selected, and conversely, when the ratio is 7:3, a PWM mode with a large duty ratio is selected. Therefore, by adjusting the ratio, the PWM mode can be switched between the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65% shown in FIGS. A preamble may also be used to notify the receiver 200 of which PWM mode has been switched to. For example, transmitter 100 notifies receiver 200 of the switched PWM mode by including in a packet a preamble of a pattern associated with the switched PWM mode. Note that the preamble pattern is changed by the time lengths C ₀ , C ₁ , C ₂ and C ₃ .

However, since the visible light signal shown in FIG. 149 includes two data parts in the packet, it takes time to transmit the packet. For example, if transmitter 100 is a DLP projector, transmitter 100 projects red, green, and blue images in a time division manner. Here, it is preferable that the transmitter 100 transmits a visible light signal when projecting a red image. This is because the visible light signal transmitted at this time has a red wavelength and is easily received by the receiver 200 . The period during which the red image is continuously projected is, for example, 1.5 ms. This period is hereinafter referred to as a red projection period. It is difficult to transmit a packet consisting of the L data portion, preamble and R data portion described above during such a short red projection period.

A packet having only the R data portion of the two data portions is then recalled.

FIG. 150 is a diagram showing another example of detailed configuration of a visible light signal in this embodiment.

Unlike the example shown in FIG. 149, the visible light signal packet shown in FIG. 150 does not include the L data portion. Instead, the visible light signal packet shown in FIG. 150 contains invalid data and an average brightness adjustment.

The invalid data alternately shows high and low luminance values along the time axis. That is, the invalid data indicates a high luminance value for the time length _A0 , and indicates a low luminance value for the next time length _A1 . The length of time A ₀ is for example 100 μs and the length of time A ₁ is for example denoted by A ₁ =W ₀ −W ₁ . Such invalid data indicates that the packet does not contain the L data portion.

The average luminance adjusting section alternately indicates High and Low luminance values along the time axis. That is, the invalid data indicates a high luminance value for the time length _B0 , and indicates a low luminance value for the next time length _B1 . The length of time B ₀ is for example given by B ₀ =100+W ₁ ×((3−y ₀ )+(3−y ₂ )) and the length of time B ₁ is for example B ₁ =W ₁ ×((7−y ₁ )+(7−y ₃ )).

With such an average brightness adjustment unit, the average brightness in a packet can be made constant regardless of the signals y ₀ to y ₃ to be transmitted. That is, the sum of the lengths of time in which the luminance value is High in the packet (that is, the total ON time) can be A ₀ +C ₀ +C ₂ +D ₀ +D ₂ +B ₀ =790. Furthermore, the sum of the lengths of time when the luminance value is Low in the packet (that is, the total OFF time) can be A ₁ +C ₁ +C ₃ +D ₁ +D ₃ +B ₁ =910.

However, even with such a visible light signal configuration, the effective time length _E1 , which is a part of the total time length _E0 in the packet, cannot be shortened. This effective time length _E1 is the time from when the high luminance value first appears in the packet to when the last high luminance value ends, and the receiver 200 demodulates or decodes the packet of the visible light signal. is the time required to Specifically, the valid time length _E1 is _E1 = _A0 + _A1 + _C0 + _C1 + _C2 + _C3 + _D0 + _D1 + _D2 + _D3 + _B0 . Note that the total time length E ₀ is E ₀ =E ₁ +B ₁ .

That is _, since the maximum effective time length _E1 is 1700 μs even for the visible light signal having the configuration shown in FIG. It is difficult to send a single packet over

Therefore, in order to shorten the effective time length _E1 and keep the average brightness of the packet constant regardless of the signal to be transmitted, not only the time length of each High and Low brightness value but also the High brightness value It is also envisioned to adjust

FIG. 151 is a diagram showing another example of detailed configuration of a visible light signal in this embodiment.

In the visible light signal packet shown in FIG _. 151, unlike the example shown in _FIG . Regardless, it is fixed at the shortest 100 μs. Instead, in the visible light signal packet shown in FIG. 151, the High luminance value is changed according to the variables _y0 and _y2 included in the signal to be transmitted, that is, according to the time lengths _D0 and _D2 . adjusted. For example, when the time lengths _D0 and _D2 are short, the transmitter 100 adjusts the High luminance value to a large value as shown in (a) of FIG. 151 . Also, when the time lengths _D0 and _D2 are long, the transmitter 100 adjusts the High luminance value to a small value as shown in (b) of FIG. 151 . Specifically, when each of the time lengths D ₀ and D ₂ is the shortest W ₀ (for example, 110 μs), the High luminance value is 100% brightness. Also, when the time lengths D ₀ and D ₂ are respectively the maximum “W ₀ +3W ₁ ” (for example, 200 μs), the High luminance value is 77.2% brightness.

In such a visible light signal packet, the sum of the time lengths during which the luminance value is High (that is, the total ON time) can be, for example, A ₀ +C ₀ +C ₂ +D ₀ +D ₂ +B ₀ =610 to 790. . Furthermore, the sum of the lengths of time when the luminance value is Low (that is, the total OFF time) can be A ₁ +C ₁ +C ₃ +D ₁ +D ₃ +B ₁ =910.

However, in the visible light signal shown in FIG. 151, the shortest time lengths of the total time length _E0 and the valid time length _E1 in the packet can be made shorter than the example shown in FIG. length cannot be shortened.

Therefore, in order to shorten the effective time length _E1 and keep the average brightness of the packet constant regardless of the signal to be transmitted, the data part included in the packet may be the L data part or the L data part, depending on the signal to be transmitted. It is recalled that the R data section can be used to find out.

FIG. 152 is a diagram showing another example of detailed configuration of a visible light signal in this embodiment.

In the visible light signal shown in FIG. 152, unlike the examples shown in FIGS. ₁₄₉ to 151, in order to shorten the effective time length, L data _portion and a packet containing the R data part are used properly.

In other words, when the sum of variables y ₀ to y ₃ is 7 or more, transmitter 100 generates a packet containing only the L data portion of the two data portions, as shown in FIG. 152(a). . This packet is hereinafter referred to as an L packet. Also, when the sum of variables y ₀ to y ₃ is 6 or less, transmitter 100 generates a packet containing only the R data part of the two data parts, as shown in FIG. 152(b). . Hereinafter, this packet will be called an R packet.

The L packet, as shown in (a) of FIG. 152, includes an average luminance adjustment portion, an L data portion, a preamble, and invalid data.

The L-packet average luminance adjustment unit does not indicate a High luminance value, but indicates a Low luminance value for a time length _B'0 . The time length B' ₀ is given by, for example, B' ₀ =100+W ₁ ×(y ₀ +y ₁ +y ₂ +y ₃ -7).

The invalid data of the L packet alternately indicate High and Low luminance values along the time axis. In other words, the invalid data indicates a high luminance value for the time length _A'0 , and indicates a low luminance value for the next time length _A'1 . The time length A' ₀ is denoted by A' ₀ =W ₀ -W ₁ and is for example 80 µs and the time length A' ₁ is for example 150 µs. Such invalid data indicates that the packet having the invalid data does not contain the R data portion.

For such an L-packet, the total time length _E'0 is _E'0 = _5W0 + _12W1 + 4b + 230 = 1540 µs, regardless of the signal to be transmitted. Also, the effective time length _E'1 is a time length corresponding to the signal to be transmitted, and is in the range of 900 to 1290 μs. In addition, while the total time length _E′0 is a constant 1540 μs, the sum of the time lengths during which the luminance value is High (that is, the total ON time) varies within the range of 490 to 670 μs depending on the signal to be transmitted. do. Therefore, in this L packet as well, as in the example _shown in _FIG . Vary in the range of .1%.

As in the example shown in FIG. 150, the R packet includes invalid data, a preamble, an R data portion, and an average brightness adjustment portion, as shown in FIG. 152(b).

Here, in the R packet shown in FIG. 152(b), in order to shorten the valid time length _E1 , the time length _B0 of the High luminance value in the average luminance adjustment unit is the shortest regardless of the signal to be transmitted. is fixed at 100 μs. In addition, the time length B ₁ of the low luminance value in the average luminance adjustment unit is, for example, B ₁ =W ₁ ×(6−(y ₀ +y ₁ +y ₂ +y ₃ ) in order to keep the total time length E ₁ constant. Furthermore _, in the R packet _shown in (b) _of Fig. 152 , _the High is adjusted.

For such an R packet, the total time length E ₀ is E ₀ =4W ₀ +6W ₁ +4b+260=1280 μs, regardless of the signal to be transmitted. Also, the effective time length _E1 is a time length corresponding to the signal to be transmitted, and is in the range of 1100 to 1280 μs. In addition, while the total time length _E0 is a constant 1280 μs, the sum of the time lengths when the luminance value is High (that is, the total ON time) varies within the range of 610 to 790 μs depending on the signal to be transmitted. . Therefore _, in this L packet as well, as in the example shown in _FIG . Vary in the range of ~62.1%.

Thus, in the visible light signal shown in FIG. 152, the maximum valid time length in the packet can be shortened. Thus, the transmitter 100 can transmit one packet continuously for its effective time length E ₁ or E′ ₁ during the red projection period described above.

Here, in the example shown in FIG. 152, transmitter 100 generates L packets when the sum of variables y ₀ to y ₃ is 7 or more, and generates L packets when the sum of variables y ₀ to y ₃ is 6 or less. , R packets. In other words, since the sum of variables y ₀ -y ₃ is an integer, transmitter 100 generates L packets when the sum of variables y ₀ -y ₃ is greater than 6, and variable y ₀ -y ₃ is 6 or less, generate an R packet. Thus, in this example, the threshold for switching packet types is six. However, the threshold for such packet type switching is not limited to 6, and may be any value from 3 to 10. FIG.

FIG. 153 is a diagram showing the relationship between the sum of variables y ₀ to y ₃ and the total time length and valid time length. The total time length shown in FIG. 153 is the larger one of the total time length _E0 of the R packet and the total time length _E′0 of the L packet. The valid time length shown in FIG. 153 is the larger one of the maximum value of the valid time length _E1 of the R packet and the maximum value of the valid time length _E'1 of the L packet. Note that in the example shown in FIG. 153, the constants W ₀ , W ₁ and b are W ₀ =110 μs, W ₁ =15 μs and b=100 μs, respectively.

The total time length varies according to the sum of variables y ₀ to y ₃ as shown in FIG. Also, as shown in FIG. 153, the valid time length varies according to the sum of variables y ₀ to y ₃ , but is minimized when the sum is about 3.

Thus, the threshold for packet type switching may be set in the range of 3 to 10, depending on which of the total length of time and the valid length of time is to be shortened.

FIG. 154A is a flow chart showing a transmission method according to this embodiment.

The transmission method according to the present embodiment is a transmission method for transmitting a visible light signal according to changes in luminance of a light emitter, and includes determination step S571 and transmission step S572. In decision step S571, the transmitter 100 determines the pattern of luminance change by modulating the signal. In a transmitting step S572, the transmitter 100 transmits a visible light signal by varying the intensity of the red color represented by the light source contained in its luminary according to the determined pattern. Here, the visible light signal includes data, preamble and payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value The length of time one lasts is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values alternates along the time axis. In the payload, the first and second luminance values alternate along a time axis, the duration of each of the first and second luminance values being greater than a first predetermined value, and It is determined according to the above-mentioned signal and predetermined scheme.

For example, the data, preamble and payload are invalid data, preamble, and L data section or R data section shown in (a) and (b) of FIG. 152, respectively. Also, for example, the first predetermined value is 100 μs.

As a result, as shown in (a) and (b) of FIG. 152, the visible light signal has one payload (that is, the L data portion or the R data portion) of a waveform determined according to the signal to be modulated. contains and does not contain two payloads. Therefore, the visible light signal, ie the packet of the visible light signal, can be shortened. As a result, for example, even if the emission period of red light represented by the light source included in the light emitter is short, it is possible to transmit packets of visible light signals during the emission period.

Also, in the payload, the first luminance value of the first time length, the second luminance value of the second time length, the first luminance value of the third time length, the second luminance value of the fourth time length, Each luminance value may appear in luminance value order. In this case, in the transmitting step S572, the transmitter 100 sets the first time length and the third time length if the sum of the first time length and the third time length is smaller than the second predetermined value. is larger than the second predetermined value, the current value flowing through the light source is increased. Here, the second predetermined value is greater than the first predetermined value. Note that the second predetermined value is, for example, a value greater than 220 μs.

Accordingly, as shown in FIGS. 151 and 152, when the sum of the first time length and the third time length is small, the current value of the light source is increased, and the first time length and the third time length are increased. If the sum of lengths is large, the current value of the light source is reduced. Therefore, the average brightness of a packet consisting of data, preamble and payload can be kept constant regardless of the signal.

Also, in the payload, the first luminance value of the first duration _D0 , the second luminance value of the second duration _D1 , the first luminance value of the third duration _D2 , the fourth Each luminance value may appear in order of the second luminance value of duration _D3 . In this case, when the sum of the four parameters y _k (k=0, 1, 2, 3) obtained from the signal is equal to or less than the third predetermined value, the first to fourth time lengths D ₀ to D ₃ Each is determined according to D _k =W ₀ +W ₁ ×y _k (W ₀ and W ₁ are integers equal to or greater than 0). For example, the third predetermined value is 3, as shown in FIG. 152(b).

As a result, as shown in FIG. 152(b), it is possible to generate a payload with a short waveform according to the signal while setting each of the first to fourth time lengths D ₀ to D ₃ to W ₀ or more.

Also, if the sum of the four parameters y _k (k=0, 1, 2, 3) is less than or equal to the third predetermined value, in transmission step S572, the data, preamble and payload are converted to You can send them in order. In the example shown in FIG. 152(b), the payload is the R data portion.

As a result, as shown in (b) of FIG. 152, the receiver 200 that receives the packet is notified by the data that the packet of the visible light signal containing the data (that is, invalid data) does not contain the L data part. can inform

Further, when the sum of the four parameters y _k (k=0, 1, 2, 3) is greater than the third predetermined value, each of the first to fourth time lengths D ₀ to D ₃ is D ₀ =W ₀ +W ₁ ×(A−y ₀ ), D ₁ =W ₀ +W ₁ ×(B−y ₁ ), D ₂ =W ₀ +W ₁ ×(A−y ₂ ), and D ₃ =W ₀ +W It may be determined according to ₁ ×(B−y ₃ ) (A and B are each an integer equal to or greater than 0).

As a result, as shown in (a) of FIG. 152, each of the first to fourth time lengths D ₀ to D ₃ (that is, the first to fourth time lengths D' ₀ to D' ₃ ) is W ₀ or more. However, even if the above sum is large, it is possible to generate a short waveform payload according to the signal.

Also, if the sum of the four parameters y _k (k=0, 1, 2, 3) is greater than the third predetermined value, then in transmission step S572, the data, preamble and payload are converted to the payload, preamble and data. You can send them in order. In the example shown in FIG. 152(a), the payload is the L data portion.

As a result, as shown in (a) of FIG. 152, the packet of the visible light signal containing the data (that is, the invalid data) does not contain the R data portion. can let you know.

In addition, the light emitter has a plurality of light sources including a red light source, a blue light source, and a green light source. You may send.

As a result, the light emitter can display an image using a red light source, a blue light source, and a green light source, and can transmit a visible light signal having a wavelength that can be easily received by the receiver 200 .

Note that the light emitter may be, for example, a DLP projector. A DLP projector may have multiple light sources, including a red light source, a blue light source, and a green light source, as described above, but may also have only one light source. That is, the DLP projector may comprise one light source, a DMD (Digital Micromirror Device), and a color wheel arranged between the light source and the DMD. In this case, the DLP projector outputs the red light, the blue light, and the green light in a time division manner from the light source to the DMD via the color wheel, and during the period when the red light is output, the visible Send packets of optical signals.

FIG. 154B is a block diagram showing the configuration of transmitter 100 in this embodiment.

Transmitter 100 according to the present embodiment includes a transmitter that transmits a visible light signal according to a change in luminance of a light emitter, determining section 571 , and transmitting section 572 . The determination unit 571 determines the pattern of luminance change by modulating the signal. Transmitter 572 transmits a visible light signal by varying the intensity of red represented by the light source included in the light emitter according to the determined pattern. Here, the visible light signal includes data, preamble and payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value The length of time one lasts is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values alternates along the time axis. In the payload, the first and second luminance values alternate along a time axis, the duration of each of the first and second luminance values being greater than a first predetermined value, and It is determined according to the above-mentioned signal and predetermined scheme.

Such a transmitter 100 implements the transmission method of the flowchart shown in FIG. 154A.

(Embodiment 9)
In the present embodiment, a display method and a display device that realize AR (Augmented Reality) using optical IDs will be described, as in the case of the fourth embodiment. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitting device) and receiver (or receiving device) in the above embodiments. Further, the receiver in this embodiment is configured as a display device.

155 shows a configuration of a display system according to Embodiment 9. FIG.

This display system 500 performs object recognition and augmented reality/mixed reality display using visible light signals.

The transmitter 100 is configured as a lighting device, for example, as shown in FIG. 155, and transmits the light ID by changing the luminance while illuminating the AR target 501. FIG. Since the AR object 501 is illuminated by the light from the transmitter 100, the brightness changes similarly to the transmitter 100, and the light ID is transmitted.

The receiver 200 images the AR target 501 . That is, the receiver 200 images the AR target 501 in each of the normal exposure time and the communication exposure time described above. As a result, the receiver 200 acquires the captured display image and the decoding image, which is the visible light communication image or the bright line image, in the same manner as described above.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the light ID from the AR object 501. FIG. Receiver 200 transmits the light ID to server 300 . The receiver 200 then acquires the AR image P11 and recognition information associated with the light ID from the server 300 . The receiver 200 recognizes the area corresponding to the recognition information in the captured display image as the target area. For example, the receiver 200 recognizes the area where the AR target 501 is projected as the target area. Then, the receiver 200 superimposes the AR image P11 on the target area, and displays the captured display image on which the AR image P11 is superimposed on the display. For example, the AR image P11 is a moving image.

When the display or reproduction of all the moving images of the AR image P11 is completed, the receiver 200 notifies the server 300 of the completion of reproduction of the moving image. The server 300 that has received the notification of the completion of the reproduction gives rewards such as points to the receiver 200 . When the receiver 200 notifies the server 300 of the completion of playback of the moving image, the receiver 200 may notify not only the playback completion but also the personal information of the user of the receiver 200, and the wallet ID for storing the reward. may notify you. The server 300 gives points to the receiver 200 by receiving these notifications.

FIG. 156 is a sequence diagram showing processing operations of receiver 200 and server 300 .

The receiver 200 acquires a light ID as a visible light signal by imaging the AR target 501 (step S51). Receiver 200 then transmits the light ID to server 300 (step S52).

When the server 300 acquires the light ID (step S53), it transmits the recognition information and the AR image P11 associated with the light ID to the receiver 200 (step S54).

According to the recognition information, the receiver 200 recognizes an area where, for example, the AR target object 501 is displayed in the imaged display image as a target area, and displays the imaged display image in which the AR image P11 is superimposed on the target area. to display. Then, the receiver 200 starts playing the moving image, which is the AR image P11 (step S56).

Next, the receiver 200 determines whether or not the reproduction of the moving image has been completed (step S57). When it is determined that the reproduction has been completed (Yes in step S57), the receiver 200 notifies the server 300 of the completion of reproduction of the moving image (step S58).

When the server 300 receives the reproduction completion notification from the receiver 200, the server 300 gives points to the receiver 200 (step S59).

Here, server 300 may make the conditions for giving points to receiver 200 stricter, as shown in FIG.

FIG. 157 is a flow chart showing processing operations of the server 300 .

Server 300 first acquires a light ID from receiver 200 (step S60). Next, server 300 transmits the recognition information associated with the light ID and AR image P11 to receiver 200 (step S61).

Then, the server 300 determines whether or not it has received a notification from the receiver 200 that the moving image, which is the AR image P11, has been reproduced (step S62). Here, when the server 300 determines that it has received the notification of the completion of the moving image reproduction (Yes in step S62), it further determines whether or not the same AR image P11 has been reproduced in the receiver 200 in the past (step S63). ). When determining that the same AR image P11 has not been reproduced in the receiver 200 in the past (No in step S63), the server 300 gives points to the receiver 200 (step S66). On the other hand, when determining that the same AR image P11 has been reproduced in the receiver 200 in the past (Yes in step S63), the server 300 further determines whether or not a predetermined period has elapsed since the past reproduction. (Step S64). For example, the predetermined period may be one month, three months, one year, or any other period.

Here, the server 300 determines that the predetermined period has not passed (No in step S64), and does not give points to the receiver 200. FIG. On the other hand, when the server 300 determines that the predetermined period of time has passed (Yes in step S64), the current location of the receiver 200 is changed to the location where the same AR image P11 was played back in the past (hereinafter referred to as the past playback location). location) is determined (step S65). When the server 300 determines that the current location of the receiver 200 is different from the past playback location (Yes in step S65), it gives points to the receiver 200 (step S66). On the other hand, if the server 300 determines that the current location of the receiver 200 is the same as the past playback location (No in step S65), the server 300 does not give points to the receiver 200. FIG.

As a result, since points are given to the receiver 200 by reproducing all of the AR image P11, the user of the receiver 200 can be motivated to reproduce all of the AR image P11. For example, in order to obtain the AR image P11 with a large amount of data from the server 300, a high packet charge is required, and the user may interrupt the reproduction of the AR image P11 in the middle. However, by giving points, the entire AR image P11 can be reproduced. Note that the points may be discounts on packet charges. Further, points may be given to the receiver 200 according to the data amount of the AR image P11.

FIG. 158 is a diagram showing an example of communication when transmitter 100 and receiver 200 are mounted on a vehicle.

A vehicle 200n includes the receiver 200 described above, and a plurality of vehicles 100n includes the transmitter 100 described above. Also, the plurality of vehicles 100n are traveling ahead of the vehicle 200n, for example. Furthermore, the vehicle 200n wirelessly communicates with any one vehicle 100n of the plurality of vehicles 100n.

Here, the vehicle 200n transmits a visible light signal to the vehicle 100n with which the vehicle 200n is communicating in order to know which of the vehicles 100n ahead of it is in wireless communication with. request by radio communication.

When receiving the request from the vehicle 200n, the vehicle 100n of the communication partner transmits a visible light signal rearward. For example, the vehicle 100n of the communication partner transmits a visible light signal by blinking the backlight.

The vehicle 200n captures an image of the front using an image sensor. As a result, the vehicle 200n acquires the captured display image and the decoding image in the same manner as described above. A plurality of vehicles 100n traveling in front of the vehicle 200n are displayed in the captured display image.

The vehicle 200n identifies the position of the bright line pattern area in the decoding image, and superimposes a marker at the same position as the bright line pattern area in the captured display image, for example. Then, the vehicle 200n displays the captured display image on which the marker is superimposed on the in-vehicle display. For example, an imaged display image is displayed in which a marker is superimposed on the backlight of any one vehicle 100n among the plurality of vehicles 100n. As a result, an occupant such as a driver of the vehicle 200n can easily grasp which vehicle 100n is the communication partner by viewing the captured display image.

FIG. 159 is a flow chart showing the processing operation of the vehicle 200n.

The vehicle 200n starts wireless communication with the vehicle 100n around the vehicle 200n (step S71). At this time, the occupant of the vehicle 200n can determine which of the plurality of vehicles is in the image captured by the image sensor of the vehicle 200n and which of the vehicles is in wireless communication. communication partner cannot be determined. Therefore, the vehicle 200n requests the vehicle 100n, which is the communication partner, to transmit a visible light signal by wireless communication (step S72). The vehicle 100n of the communication partner that has received this request transmits a visible light signal. Vehicle 200n captures an image of its surroundings with an image sensor, and receives the visible light signal transmitted from vehicle 100n of the communication partner (step S73). That is, the vehicle 200n acquires the captured display image and the decoding image as described above. Then, the vehicle 200n specifies the position of the bright line pattern area in the decoding image, and superimposes a marker on the same position as the bright line pattern area in the captured display image. As a result, even if a plurality of vehicles are displayed in the captured display image, the vehicle 200n can identify the vehicle on which the marker is superimposed among the plurality of vehicles as the communication partner vehicle 100n (step S74).

FIG. 160 is a diagram showing an example in which receiver 200 displays an AR image in this embodiment.

The receiver 200 repeatedly acquires a captured display image Pk and a decoding image as shown in FIG. 54, for example, by capturing an image of a subject with the image sensor.

Specifically, the image sensor of receiver 200 captures transmitter 100 configured as signage and person 21 next to transmitter 100 . Transmitter 100 is the transmitter in each of the above-described embodiments, and includes one or more light-emitting elements (for example, LEDs) and translucent plate 144 having translucency such as frosted glass. The one or more light emitting elements emit light inside the transmitter 100, and the light from the one or more light emitting elements is transmitted through the translucent plate 144 and emitted to the outside. As a result, the translucent plate 144 of the transmitter 100 shines brightly. Such a transmitter 100 changes its luminance by blinking one or more of its light emitting elements, and transmits light ID (that is, light identification information) according to the luminance change. This light ID is the visible light signal described above.

Here, the transparent plate 144 has a message “Hold your smartphone over here”. Therefore, the user of the receiver 200 makes the person 21 stand next to the transmitter 100 and instructs the person 21 to extend his/her arm to the transmitter 100 . Then, the user directs the camera (that is, the image sensor) of the receiver 200 toward the person 21 and the transmitter 100 and takes an image. The receiver 200 acquires a captured display image Pk in which the transmitter 100 and the person 21 are imaged with normal exposure time. Further, the receiver 200 obtains a decoding image by imaging the transmitter 100 and the person 21 with a communication exposure time that is shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. Then, the receiver 200 acquires the AR image P45 and the recognition information associated with the light ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, receiver 200 recognizes an area where signage, which is transmitter 100, is displayed as a target area.

Then, the receiver 200 superimposes the AR image P45 on the captured display image Pk so that the target area is covered with the AR image P45, and displays the captured display image Pk on the display 201 . For example, the receiver 200 acquires an AR image P45 showing a soccer player. In this case, since the AR image P45 is superimposed so as to cover the target area of the captured display image Pk, the captured display image Pk can be displayed as if the soccer player actually exists next to the person 21. . As a result, the person 21 can be photographed with the soccer player even if the soccer player is not next to them.

Here, the AR image P45 shows a soccer player shaking hands. Therefore, the person 21 presents his or her hand to the transmitter 100 so that the AR image P45 and the picked-up display image Pk of shaking hands can be obtained. However, the person 21 cannot see the AR image P45 superimposed on the captured display image Pk, and does not know whether the person 21 is shaking hands with the soccer player in the AR image P45.

Therefore, the receiver 200 in the present embodiment transmits the captured display image Pk as a live view to the display device D5, and displays the captured display image Pk on the display of the display device D5. The display of display device D5 is directed to person 21 . Therefore, the person 21 can recognize that the person 21 is shaking hands with the soccer player in the AR image P45 by looking at the captured display image Pk displayed on the display device D5.

FIG. 161 is a diagram showing another example in which receiver 200 displays an AR image in this embodiment.

As shown in FIG. 161, the transmitter 100 is configured, for example, as a digital signage for an album of music content, and transmits the light ID by changing the luminance.

By imaging the transmitter 100, the receiver 200 repeatedly acquires the captured display image Pr and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, receiver 200 receives the light ID from transmitter 100 . Receiver 200 transmits the light ID to the server. The receiver 200 then obtains from the server the first AR image P46, the recognition information, the first music content and the sub-image Ps46 associated with the album identified by the light ID.

The receiver 200 starts playing the first music content obtained from the server. As a result, the first song, which is the first music content, is output from the speaker of the receiver 200 .

Further, the receiver 200 recognizes the area corresponding to the recognition information in the captured display image Pr as the target area. For example, the receiver 200 recognizes the area where the transmitter 100 is projected as the target area. Receiver 200 then superimposes first AR image P46 on the target area, and further superimposes sub-image Ps46 outside the target area. Receiver 200 displays on display 201 captured display image Pr on which first AR image P46 and sub-image Ps46 are superimposed. For example, the first AR image P46 is a moving image related to the first song of the first music content, and the sub-image Ps46 is a still image related to the above album. The receiver 200 reproduces the moving image of the first AR image P46 in synchronization with the first music content.

162 is a diagram showing processing operations of the receiver 200. FIG.

For example, as in the state shown in FIG. 161, the receiver 200 synchronizes and reproduces the first AR image P46 and the first music content, as shown in FIG. 162(a). Here, the user of the receiver 200 operates the receiver 200 . For example, as shown in (b) of FIG. 162, the user performs a swipe. Specifically, the user moves the fingertip laterally while placing the fingertip on the first AR image P46 displayed on the display 201 of the receiver 200 . In other words, the user slides the first AR image P46 in the horizontal direction. In this case, the receiver 200 sets the second music content associated with the above light ID next to the first music content, and the first AR image P46 next associated with the above light ID. A second AR image P46c is obtained from the server. For example, the second music content is the second song, and the second AR image P46c is a moving image related to the second song.

The receiver 200 then switches the music content to be played back from the first music content to the second music content. That is, the receiver 200 stops playing the first music content and starts playing the second song, which is the second music content.

At this time, the receiver 200 switches the image superimposed on the target area of the captured display image Pr from the first AR image P46 to the second AR image P46c. That is, the receiver 20 stops reproducing the first AR image P46 and starts reproducing the second AR image P46c.

Here, the first displayed picture of the second AR image P46c is the same as the first displayed picture of the first AR image P46.

Therefore, as shown in (a) of FIG. 162, the receiver 200 first displays the same picture as the first picture of the first AR image P46 when starting to reproduce the second song. After that, the receiver 200 sequentially displays the second and subsequent pictures included in the second AR image P46c, as shown in FIG. 162(b).

Here again, the user swipes the receiver 200 as shown in FIG. 162(b). In response to the operation, as described above, the receiver 200 outputs the third music content associated with the above-described light ID next to the second music content, and the second AR image P46c next to the above-described A third AR image P46d associated with the light ID is obtained from the server. For example, the third music content is the third song, and the third AR image P46d is a moving image related to the third song.

The receiver 200 then switches the music content to be played back from the second music content to the third music content. That is, the receiver 200 stops playing the second music content and starts playing the third song, which is the third music content.

At this time, the receiver 200 switches the image superimposed on the target area of the captured display image Pr from the second AR image P46c to the third AR image P46d. That is, the receiver 20 stops playing the second AR image P46c and starts playing the third AR image P46d.

Here, the first displayed picture of the third AR image P46d is the same as the first displayed picture of the first AR image P46.

Therefore, as shown in FIG. 162(a), receiver 200 first displays the same picture as the first picture of first AR picture P46 when starting to reproduce the third song. After that, the receiver 200 sequentially displays the second and subsequent pictures included in the third AR image P46d, as shown in (d) of FIG.

In the above example, as shown in (b) of FIG. 162, receiver 200 displays the next moving image when receiving an operation (that is, swiping) to slide an AR image that is a moving image. However, the receiver 200 may display the next moving image instead of the operation when the light ID is retaken. The time when the optical ID is reacquired means the time when the optical ID is acquired again by imaging with the image sensor. In other words, the receiver 200 repeatedly obtains the captured display image and the decoding image by performing imaging. is retaken. For example, when the image sensor of the receiver 200 directed toward the transmitter 100 is directed in another direction, the bright line pattern area disappears from the image for decoding. Then, when the image sensor is directed toward the transmitter 100 again, a bright line pattern area appears in the image for decoding. At this time, the light ID is reacquired.

As described above, in the display method according to the present embodiment, receiver 200 acquires a visible light signal as a light ID (ie, identification information) by imaging with an image sensor. Receiver 200 then displays first AR image P46, which is a moving image associated with the light ID. Next, when receiving the operation of sliding the first AR image P46, the receiver 200 slides the second AR image P46c, which is the moving image associated with the light ID, next to the first AR image P46. indicate. Therefore, it is possible to easily display an image that is beneficial to the user.

In addition, in the display method according to the present embodiment, objects in the first displayed picture may be located at the same position in each of the first AR image P46 and the second AR image P46c. For example, in the example shown in FIG. 162, the first displayed picture of each of the first AR image P46 and the second AR image P46c is the same. Therefore, the objects in those pictures are co-located. For example, as shown in (a) of FIG. 162, the singer, which is an object, is at the same position in the first displayed picture of each of the first AR image P46 and the second AR image P46c. As a result, the user can easily understand that the first AR image P46 and the second AR image P46c are related to each other. In the example shown in FIG. 162, the first displayed picture of each of the first AR image P46 and the second AR image P46c is the same. Those pictures can be different.

Further, in the display method according to the present embodiment, when the receiver 200 acquires the light ID again by imaging with the image sensor, the next moving image associated with the light ID is displayed next to the displayed moving image. indicate. This makes it possible to more easily display moving images that are beneficial to the user.

Further, in the display method according to the present embodiment, receiver 200, as shown in FIG. Outside, a sub-image Ps46 is displayed. This makes it possible to easily display a wide variety of images that are more beneficial to the user.

FIG. 163 is a diagram showing an example of an operation on receiver 200. FIG.

For example, as shown in FIGS. 161 and 162, when an AR image is displayed on display 201 of receiver 200, the user performs vertical swiping as shown in FIG. Specifically, the user moves the fingertip vertically while placing the fingertip on the AR image displayed on the display 201 of the receiver 200 . In other words, the user slides an AR image such as the first AR image P46 in the vertical direction. In this case, the receiver 200 obtains from the server another AR image associated with the light ID described above.

164 is a diagram showing an example of an AR image displayed on receiver 200. FIG.

When a swipe operation as shown in FIG. 163 is performed, the receiver 200 displays an AR image P47 acquired from the server as the other AR image described above, superimposed on the captured display image Pr.

For example, receiver 200 displays AR image P47, which is a still image showing the singer of the music content, superimposed on captured display image Pr, as in the examples shown in FIGS. Here, the AR image P47 is superimposed on the target area of the captured display image Pr, that is, the area where the transmitter 100, which is digital signage, is displayed. Therefore, similarly to the examples shown in FIGS. 146 and 160, when a person stands next to the transmitter 100, the captured display image Pr can be displayed as if the singer actually exists next to the person. . As a result, a person can be photographed with the singer without the singer being next to them.

As described above, in the display method according to the present embodiment, when receiver 200 receives an operation to slide first AR image P46 in the horizontal direction, receiver 200 displays second AR image P46c and displays first AR image P46c. When the operation of sliding P46 in the vertical direction is accepted, the AR image P47, which is a still image associated with the light ID, is displayed. Therefore, it is possible to easily display a wide variety of images that are beneficial to the user.

FIG. 165 is a diagram showing an example of an AR image superimposed on a captured display image.

As shown in FIG. 165, when superimposing the AR image P48 on the captured display image Pr1, the receiver 200 trims a part of the AR image P48 and superimposes only the part on the captured display image Pr1. good too. For example, the receiver 200 may cut out the peripheral area of the rectangular AR image P48 and superimpose only the round area in the center of the AR image P48 on the captured display image Pr1.

FIG. 166 is a diagram showing an example of an AR image superimposed on a captured display image.

The receiver 200 captures an image of the transmitter 100 configured as digital signage in a coffee shop, for example. By the imaging, the receiver 200 acquires the captured display image Pr2 and the decoding image in the same manner as described above. In the captured display image Pr2, the transmitter 100, which is the digital signage, is displayed as a signage image 100i. The receiver 200 obtains the light ID by decoding the decoding image, and obtains the AR image P49 associated with the light ID from the server. Then, the receiver 200 recognizes the area above the signage image 100i in the captured display image Pr2 as the target area, and superimposes the AR image P49 on the target area. The AR image P49 is, for example, a moving image of coffee flowing down from a pot. The AR image P49, which is a moving image, is formed such that the closer the position in the region of the coffee falling from the pot to the lower end of the AR image P49, the higher the transparency at that position. As a result, the AR image P49 can be displayed as if coffee were actually running down.

It should be noted that such an AR image P49 may be any moving image as long as it has an ambiguous outline, and may be, for example, a moving image of flames. When the AR image P49 is a flame moving image, the transparency of the peripheral portion of the AR image P49 gradually increases toward the outside. Moreover, the transparency may change with time. As a result, the AR image P49 can be displayed as a flickering flame in a realistic manner.

Also, at least one moving image of the first AR image P46, the second AR image P46c, and the third AR image P46d shown in FIG. 162 is formed to have transparency as shown in FIG. may

That is, in the display method according to the present embodiment, the closer the position in the moving image of at least one of the first AR image P46 and the second AR image P46c is to the edge of the moving image, the more You may form so that the transparency in the said position may become high. As a result, when the moving image is superimposed on the normal photographed image and displayed, the captured display image is displayed so that an object with an ambiguous outline actually exists in the environment indicated by the normal photographed image. can be done.

FIG. 167 is a diagram showing an example of transmitter 100 in this embodiment.

The transmitter 100 is configured to be able to transmit information as an image ID even to a receiver incapable of imaging in the visible light communication mode, that is, a receiver that does not support optical communication. That is, the transmitter 100 is configured as a digital signage, for example, and transmits the light ID by changing the brightness, as described above. Further, line patterns 151 to 154 are drawn on transmitter 100 . Each of these line patterns 151 to 154 is an arrangement pattern of a plurality of short straight lines extending in the horizontal direction, and the plurality of straight lines are arranged apart from each other in the vertical direction. That is, each of the line patterns 151-154 is configured like a bar code. The line pattern 151 is arranged on the left side of the letter "A" drawn on the transmitter 100, and the line pattern 152 is arranged on the right side of the letter "A". The line pattern 153 is arranged in the letter B drawn on the transmitter 100 , and the line pattern 154 is arranged in the letter C drawn on the transmitter 100 . Note that the letters A, B, and C are merely examples, and any letters or images may be drawn on transmitter 100 .

A receiver that does not support optical communication cannot set the exposure time of the image sensor to the communication exposure time described above. However, the receiver obtains a normal photographed image (that is, a photographed display image) in which the line patterns 151 to 154 are displayed by imaging the transmitter 100, and obtains an image ID from the line patterns 151 to 154. be able to. Therefore, a receiver that does not support optical communication can acquire the image ID even if it cannot acquire the light ID from the transmitter 100, and by using the image ID instead of the light ID, the above-described , the AR image can be superimposed on the captured display image and displayed.

The same image ID may be obtained from each of the line patterns 151 to 154, or different image IDs may be obtained.

FIG. 168 is a diagram showing another example of a transmitter in this embodiment.

Transmitter 100 e in the present embodiment includes transmitter main body 115 and lenticular lens 116 . 168(a) shows a top view of the transmitter 100e, and FIG. 168(b) shows a front view of the transmitter 100e.

Transmitter body 115 has the same configuration as transmitter 100 shown in FIG. In other words, letters A, B, and C and line patterns associated with these letters are drawn on the front of the transmitter body 115 .

The lenticular lens 116 is attached to the transmitter main body 115 so as to cover the front surface of the transmitter main body 115, that is, the surface on which the letters A, B, and C and the line pattern are drawn.

Therefore, line patterns 151 to 154 seen from the front left side of transmitter 100e shown in FIG. 168(c) and line patterns 151 to 154 seen from the front right side of transmitter 100e shown in FIG. 168(d) are different. can let

FIG. 169 is a diagram showing another example of transmitter 100 in this embodiment. FIG. 169(a) shows an example in which the receiver 200a captures an image of the transmitter 100 configured as a genuine product. FIG. 169(b) shows an example in which a receiver 200a captures an image of a transmitter 100f configured as a counterfeit transmitter 100. FIG.

As shown in FIG. 169(a), the genuine transmitter 100 is configured to be able to transmit an image ID to a receiver that does not support optical communication, similar to the example shown in FIG. . That is, on the front of the transmitter 100, letters A, B, and C, line patterns 154, and the like are drawn. Furthermore, a character string 161 may be drawn on the front of the transmitter 100 . This character string 161 is formed by applying an infrared reflecting paint, an infrared absorbing paint, or an infrared blocking paint. Therefore, although this character string 161 is invisible to the human eye, it appears in the normal captured image obtained by the image sensor of the receiver 200a.

The receiver 200a is a receiver that does not support optical communication. Therefore, even if the visible light signal is transmitted from the transmitter 100, the receiver 200a cannot receive the visible light signal. However, by imaging the transmitter 100, the receiver 200a can acquire the image ID from the line pattern appearing in the normal shot image obtained by the imaging. Further, receiver 200a can determine that transmitter 100 is genuine if character string 161, for example, "Please hold your smartphone over here" appears in the normal captured image. That is, the receiver 200 can determine that the acquired image ID is not illegal. In other words, the receiver 200 can authenticate the image ID depending on whether the character string 161 appears in the normal captured image. When the receiver 200 determines that the image ID is not invalid, the receiver 200 performs processing using the image ID, such as transmitting the image ID to the server.

On the other hand, the transmitter 100 as described above may be illegally duplicated. In other words, instead of the genuine transmitter 100, a transmitter 100f configured as a counterfeit of the transmitter 100 may be installed. The letters A, B and C and a line pattern 154f are drawn on the front of such a fake transmitter 100f. The letters A, B and C and line pattern 154f were drawn by a malicious person to resemble the letters A, B and C and line pattern 154 drawn on the authentic transmitter 100. is That is, line pattern 154f is similar to line pattern 154, but different.

However, if a malicious person illegally duplicates the genuine transmitter 100, the character string 161 drawn with infrared reflective paint, infrared absorbing paint, or infrared blocking paint cannot be visually recognized. Therefore, the character string 161 is not drawn on the front of the fake transmitter 100f.

Therefore, if the receiver 200a takes an image of such a counterfeit transmitter 100f, the receiver 200a acquires an incorrect image ID from the line pattern appearing in the normal shot image obtained by taking the image. However, as shown in FIG. 169(b), receiver 200a can determine that the image ID is invalid because character string 161 is not displayed in the normal shot image. As a result, the receiver 200 can prohibit processing using the illegal image ID.

FIG. 170 is a diagram showing an example of a system using a receiver 200 compatible with optical communication and a receiver 200a not compatible with optical communication.

For example, the receiver 200 a that does not support optical communication captures the transmitter 100 . In this transmitter 100, the line pattern 154 is drawn as in the example shown in FIG. 167, but the character string 161 shown in FIG. 168 is not drawn. Therefore, the receiver 200a can acquire the image ID from the line pattern appearing in the normal captured image acquired by imaging, but cannot authenticate the image ID. Therefore, even if the image ID is invalid, the receiver 200a trusts the image ID and performs processing using the image ID. For example, the receiver 200a requests the server 300 for the procedure associated with the image ID. The procedure is, for example, transferring money to an unauthorized bank account.

On the other hand, the receiver 200 compatible with optical communication obtains an optical ID, which is a visible light signal, as well as an image ID by capturing an image of the transmitter 100 in the same manner as described above. Therefore, the receiver 200 determines whether or not the image ID matches the light ID. Here, if it is determined that the image ID is different from the light ID, the receiver 200 requests the server 300 to discard the procedure request associated with the image ID.

Therefore, even if the server 300 receives a request for a procedure associated with the image ID from the receiver 200a that does not support optical communication, when the server 300 receives a request from the receiver 200 that supports optical communication, it requests the procedure. Discard.

As a result, even if a malicious person draws a line pattern 154 that gives an incorrect image ID on the transmitter 100, the request for the procedure associated with the image ID can be properly discarded. .

171 is a flow chart showing the processing operation of the receiver 200. FIG.

The receiver 200 acquires a normal captured image by imaging with the transmitter 100 (step S81). Then, the receiver 200 acquires the image ID from the line pattern displayed in the normal shot image (step S82).

Next, the receiver 200 acquires the light ID from the transmitter 100 by visible light communication (step S83). That is, the receiver 200 acquires the image for decoding by imaging the transmitter 100 in the visible light communication mode, and acquires the light ID by decoding the image for decoding.

The receiver 200 then determines whether or not the image ID obtained in step S82 matches the light ID obtained in step S83 (step S84). Here, if it is determined that they match (Yes in step S84), the receiver 200 requests the server 300 to perform the procedure associated with the light ID (step S85). On the other hand, if it is determined that they do not match (No in step S84), the receiver 200 requests the server 300 to discard the procedure request associated with the light ID (step S86).

FIG. 172 is a diagram showing an example of AR image display.

For example, the transmitter 100 is shaped like a saber, and transmits a visible light signal as an optical ID by changing the luminance of a portion other than the handle of the saber.

The receiver 200 captures an image of the transmitter 100 from near the transmitter 100, as shown in FIG. 172(a). While performing the imaging, the receiver 200 repeatedly obtains the captured display image Pr3 and the decoding image as described above. When the receiver 200 acquires the light ID by decoding the decoding image, the receiver 200 transmits the light ID to the server. As a result, the receiver 200 acquires the AR image P50 and recognition information associated with the light ID from the server. The receiver 200 recognizes the area corresponding to the recognition information in the captured display image Pr3 as the target area. For example, the receiver 200 recognizes, as the target area, an area in the imaged display image Pr3 above the area in which the portion other than the hilt of the saber is displayed.

Specifically, as shown in the examples of FIGS. 50 to 52, the identification information indicates reference information for specifying the reference area in the captured display image Pr3 and the relative position of the target area with respect to the reference area. It contains subject information. For example, the reference information indicates that the position of the reference area in the captured display image Pr3 is the same as the position of the bright line pattern area in the decoding image. Furthermore, the target information indicates that the position of the target area is above the reference area.

Therefore, the receiver 200 identifies the reference area from the captured display image Pr3 based on the reference information. That is, the receiver 200 identifies an area in the captured display image Pr3 at the same position as the bright line pattern area in the decoding image as the reference area. In other words, the receiver 200 identifies, as the reference area, an area in the imaged display image Pr3 in which a portion other than the handle of the saber is displayed.

Further, the receiver 200 recognizes, as a target area, an area in the captured display image Pr3 at a relative position indicated by the target information with respect to the position of the reference area. In the above example, the target information indicates that the position of the target area is above the reference area, so the receiver 200 recognizes the area above the reference area in the captured display image Pr3 as the target area. . In other words, the receiver 200 recognizes, as the target area, the area above the area in which the part other than the handle of the saber is displayed in the imaged display image Pr3.

Then, the receiver 200 superimposes the AR image P50 on the target area, and displays on the display 201 the captured display image Pr3 on which the AR image P50 is superimposed. For example, the AR image P50 is a moving image of a person.

Here, the receiver 200 moves away from the transmitter 100 as shown in FIG. 172(b). Therefore, the saber displayed in the captured display image Pr3 becomes smaller. That is, the size of the bright line pattern area of the decoding image is reduced. As a result, the receiver 200 reduces the size of the AR image P50 so as to match the size of the bright line pattern area. That is, the receiver 200 adjusts the size of the AR image P50 so as to keep the size ratio between the bright line pattern area and the AR image P50 constant.

Thereby, the receiver 200 can display the captured display image Pr3 as if a person actually exists on the saber.

As described above, in the display method according to the present embodiment, receiver 200 obtains a normally captured image by imaging with the image sensor during the normal exposure time (that is, the first exposure time). Further, the receiver 200 obtains an image for decoding including a bright line pattern area, which is an area composed of a plurality of bright line patterns, by imaging with a communication exposure time (that is, the second exposure time) shorter than the normal exposure time. Then, the optical ID is obtained by decoding the decoding image. Next, the receiver 200 identifies a reference area located at the same position as the bright line pattern area in the decoding image from the normal captured image, and based on the reference area, the area where the moving image is superimposed in the normal captured image. is recognized as the region of interest. Then, receiver 200 superimposes a moving image on the target area. Note that the moving image may be at least one of the first AR image P46 and the second AR image P46c shown in FIG. 162 and the like.

Also, the receiver 200 may recognize an area above, below, left, or right of the reference area in the normally captured image as the target area.

As a result, for example, as shown in FIGS. 50 to 52 and 172, the target region is recognized based on the reference region, and the moving image is superimposed on the target region. can be easily increased.

Further, in the display method according to the present embodiment, receiver 200 may change the size of the moving image according to the size of the bright line pattern area. For example, the receiver 200 increases the size of the moving image as the size of the bright line pattern area increases.

As a result, as shown in FIG. 172, since the size of the moving image changes according to the size of the bright line pattern area, compared to the case where the size of the moving image is fixed, the object indicated by the moving image is more visible. The moving image can be displayed as if it exists in reality.

(Summary of Embodiment 9)
FIG. 173A is a flowchart illustrating a display method according to one aspect of the present invention.

A display method according to an aspect of the present invention is a display method for displaying an image, and includes steps SG1 to SG3. That is, the display device, which is the receiver 200 described above, acquires a visible light signal as identification information (that is, light ID) by imaging with an image sensor (step SG1). Next, the display device displays the first moving image associated with the light ID (step SG2). Then, when receiving the operation of sliding the first moving image, the display device displays the second moving image associated with the light ID next to the first moving image (step SG3).

FIG. 173B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention.

A display device G10 according to an aspect of the present invention is a device that displays an image, and includes an acquisition unit G11 and a display unit G12. Note that the display device G10 is the receiver 200 described above. The acquisition unit G11 acquires a visible light signal as identification information (that is, light ID) by imaging with an image sensor. Next, the display unit G12 displays the first moving image associated with the light ID. Then, when receiving an operation to slide the first moving image, the display unit G12 displays the second moving image associated with the light ID next to the first moving image.

For example, the first moving image and the second moving image are the first AR image P46 and the second AR image P46c shown in FIG. 162, respectively. In the display method and display device G10 shown in FIGS. 173A and 173B, when an operation of sliding the first moving image, that is, a swipe is accepted, the second moving image associated with the identification information is displayed next to the first moving image. A moving image is displayed. Therefore, it is possible to easily display an image that is beneficial to the user.

In the above-described embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. For example, the program causes the computer to execute the display method illustrated by the flow charts of FIGS. 156, 157, 159, 171 and 173A.

(Embodiment 10)
In this embodiment, as in the fourth and ninth embodiments, a display method and a display device for realizing AR (Augmented Reality) using a light ID will be described. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitting device) and receiver (or receiving device) in the above embodiments. Further, the receiver in this embodiment is configured as a display device.

174 is a diagram showing an example of an image drawn on a transmitter in this embodiment. FIG. FIG. 175 is a diagram showing another example of an image drawn on the transmitter according to this embodiment.

As in the example shown in FIG. 167, the transmitter 100 can transmit information as an image ID to a receiver that cannot perform imaging in the visible light communication mode, that is, a receiver that does not support optical communication. is configured to That is, the transmitter 100 draws a substantially rectangular transmission image Im1 or Im2. Further, the transmitter 100 is configured as a signage, for example, and transmits the light ID by changing the brightness, as described above. Alternatively, the transmitter 100 may include a light source and directly transmit the light ID to the receiver 200 according to changes in luminance of the light source. Alternatively, the transmitter 100 includes a light source, irradiates the transmission image Im1 or Im2 with light from the light source, and reflects the light on the transmission image Im1 or Im2, thereby transmitting the light to the receiver 200 as a light ID. may

Such transmission image Im1 or Im2 of transmitter 100 is formed in a substantially rectangular shape as shown in FIGS. 174 and 175. FIG. The transmission image Im1 or Im2 has a substantially rectangular base image Bi1 or Bi2 and a line pattern 155a or 155b added to the base image.

In the example shown in FIG. 174, the line pattern 155a is an array pattern of a plurality of short straight lines arranged along each of the four sides of the base image Bi1. is orthogonal to That is, when a logotype is drawn in the base image of the transmitter 100, the signal is embedded around the logotype. A short straight line included in the line pattern is hereinafter referred to as a short line.

Also, in the example shown in FIG. 174, the plurality of short lines included in the line pattern 155a are formed such that the density decreases toward the center of the transmitter 100, that is, toward the center of the base image Bi1. Accordingly, even if the line pattern 155a is added to the base image Bi1, the line pattern 155a can be made inconspicuous.

Although the line pattern 155a is not arranged at the corner of the base image Bi1 in the example shown in FIG. 174, it may be arranged at that corner as well. Further, when the corner of the base image Bi1 is rounded, the line pattern 155a does not have to be arranged at the corner.

On the other hand, in the example shown in FIG. 175, the line pattern 155b is arranged within the frame line w on the periphery of the base image Bi2. For example, the base image Bi2 is formed by drawing a rectangular frame w so as to surround the logotype (specifically, the ABC character string). The line pattern 155b is an arrangement pattern of a plurality of short lines arranged along the frame line w of the rectangle, and these short lines are orthogonal to the frame line w. Also, these short lines are arranged within the frame line w.

Although the line pattern 155b is not arranged at the corner of the frame line w in the example shown in FIG. 175, it may be arranged at that corner. Further, when the corner of the frame line w is rounded, the line pattern 155b does not have to be arranged at the corner.

FIG. 176 is a diagram showing an example of transmitter 100 and receiver 200 in this embodiment.

For example, similar to the example shown in FIG. 168, transmitter 100 may include lenticular lens 116, as shown in FIG. Such a lenticular lens 116 is attached to the transmitter 100 so as to cover an area of the transmission image Im2 drawn on the transmitter 100 excluding the frame line w.

The receiver 200 acquires a normal captured image (that is, a captured display image) in which the line pattern 155b is displayed by the transmitter 100, and acquires an image ID from the line pattern 155b. Here, receiver 200 prompts the user of receiver 200 to move receiver 200 . For example, the receiver 200 displays the message "Please move the receiver" while the transmitter 100 is being imaged. As a result, the receiver 200 is moved by the user. At this time, the receiver 200 authenticates the acquired image ID by determining whether or not the base image Bi2 of the transmitter 100, that is, the transmission image Im2 displayed in the normal captured image changes. . For example, when the receiver 200 determines that the logotype of the base image Bi2 has changed from "ABC" to "DEF", the receiver 200 determines that the acquired image ID is the correct ID.

The transmission image Im1 or Im2 described above may be drawn on the transmitter 100 that transmits the light ID. Alternatively, the transmission image Im1 or Im2 described above may be illuminated with light containing the light ID from the transmitter 100, and the light ID may be transmitted by reflecting the light. In this case, the receiver 200 can acquire the image ID of the transmission image Im1 or Im2 and the light ID by imaging. At this time, the light ID and the image ID may be the same, and a part of each of the light ID and the image ID may be the same.

Further, the transmitter 100 may be turned on when the transmission switch is turned on, and may be turned off 10 seconds after the start of lighting. Transmitter 100 transmits the light ID during this lighting period. In such a case, the receiver 200 acquires the image ID, and when the transmission switch is turned on and the brightness of the transmission image displayed in the normal photographed image suddenly changes, the image ID is the correct ID. In addition, the receiver 200 acquires the image ID, and when the transmission switch is turned on, the transmitted image displayed in the normal photographed image becomes brighter, and if it becomes dark after a predetermined time elapses, the image ID is It may be determined that the ID is correct. This makes it possible to prevent transmission image Im1 or Im2 from being illegally copied and used.

FIG. 177 is a diagram for explaining the fundamental frequency of line patterns.

A coding device for generating a transmission image Im1 or Im2 determines the fundamental frequency of the line pattern. At this time, for example, when the base image to which the line pattern is added is a horizontally long rectangle as shown in FIG. So, transform into a square. At this time, for example, the shape of the rectangular base image is modified so that the short sides of the rectangular base image have the same length as the long sides.

Next, as shown in (c) of FIG. 177, the encoding device sets the length of the diagonal of the base image transformed into a square as the fundamental period, and sets the reciprocal of the fundamental period as the fundamental frequency. decide. A base image transformed into a square is hereinafter referred to as a square base image.

FIG. 178A is a flowchart showing processing operations of the encoding device. Also, FIG. 178B is a diagram for explaining the processing operation of the encoding device.

First, the encoding device adds an error detection code (also called an error correction code) to information to be processed (step S171). For example, the encoding device adds an 8-bit error detection code to a 13-bit bit string that is information to be processed, as shown in FIG. 178B.

Next, the encoding device divides the information to which the error detection code is added into (k+1) values xk each consisting of N bits. Note that k is an integer of 1 or more. For example, as shown in FIG. 178B, the encoding device divides the information into 7 values xk each consisting of N=3 bits when k=6. That is, the information is divided into values x0, x1, x2, . . . , x6 each indicated by a 3-bit binary number. For example, the values x0, x1 and x2 are x0=010, x1=010 and x2=100.

Next, for each of the values x0 to x6, that is, for the value xk, the encoding device calculates the frequency fk corresponding to the value xk (step S173). For example, the encoding device calculates a value (A+B×xk) times the fundamental frequency for the value xk as the frequency fk corresponding to the value xk. Note that A and B are positive integers. As a result, frequencies f0 to f6 are calculated for the values x0 to x6, respectively, as shown in FIG. 178B.

Next, the encoding device adds the positioning frequency fP to the beginning of the frequencies f0 to f6 (step S174). At this time, the encoding device sets the positioning frequency fP to a value less than A times the fundamental frequency or greater than (A+B×2 ^N−1 ) times the fundamental frequency. As a result, as shown in FIG. 178B, the positioning frequency fP different from these frequencies is arranged at the beginning of the frequencies f0 to f6.

Next, the encoding device sets (k+2) specified regions around the perimeter of the square base image. Then, for each of the specified regions, the encoding device changes the luminance value (or color) of the specified region at the frequency fk along the direction of the sides of the square base image based on the original color of the specified region. (step S175). For example, as shown in (a) or (b) of FIG. 178B, (k+2) specified regions JP, J0 to J6 are set on the periphery of the square base image. If the square base image has a frame line around its perimeter, the frame line is divided into (k+2) regions, and each of the (k+2) regions is set as the specified region. More specifically, the (k+2) specified regions are set clockwise along the four sides of the square base image in the order of specified regions JP, JP0, JP1, JP2, JP3, JP4, JP5, and JP6. . The encoding device changes the luminance value (or color) at the frequency fk in each of the specified regions set in this way. A line pattern is added to the square base image by this change in luminance value.

Next, the encoding device restores the aspect ratio of the square base image with the line pattern to the aspect ratio of the original base image (step S176). For example, a square base image with a line pattern shown in (a) of FIG. 178B is transformed into a base image with a line pattern shown in (c) of FIG. 178B. In this case, the square base image with the line pattern is shrunk vertically. Therefore, as shown in FIG. 178(c), in a base image with a line pattern, the width of the line patterns above and below the base image is smaller than the width of the line patterns on the left and right.

Therefore, when adding line patterns to the square base image in step S175, as shown in (b) of FIG. and may have different widths. For example, the inverse aspect ratio of the original base image may be used to vary the width. That is, the encoding device adds the width obtained by multiplying the width of the specified area or line pattern added to the left and right of the square base image by the inverse ratio described above to the top and bottom of the square base image. Determines the width of the specified area or line pattern to be used. As a result, even if the aspect ratio of the square base image with the line pattern is restored in step S176, as shown in (d) of FIG. The width of each of the patterns can be the same.

Furthermore, the encoding device may add a frame around the line pattern attached base image, that is, outside the (k+2) designated regions, in a color different from those of the designated regions (step S177). For example, a black frame Q1 is added as shown in FIG. 178B. This makes it easier to detect the (k+2) specified regions.

FIG. 179 is a flowchart showing processing operations of receiver 200 as a decoding device.

First, the receiver 200 captures a transmission image (step S181). Next, the receiver 200 performs edge detection from the normal shot image obtained by the imaging (step S182), and further extracts the contour (step S183).

Then, the receiver 200 selects, from among the extracted contours, a region having a rectangular contour of a predetermined size or more, or a region having a rounded rectangular contour of a predetermined size or more as follows: The processing of steps S184 to S187 is executed.

That is, the receiver 200 perspectively transforms the area into a square area (step S184). Specifically, when the area to be transformed is a quadrilateral area, the receiver 200 performs perspective transformation based on the vertices of the quadrilateral. Also, when the area to be transformed is a rectangular area with rounded corners, the receiver 200 extends each side of the area and performs perspective transformation based on the point where the two sides intersect.

Next, receiver 200 obtains the frequency of luminance change in each of a plurality of designated areas included in the square area (step S185).

Next, the receiver 200 finds a designated region of frequency fP and arranges the frequencies fk of each designated region arranged in clockwise order around the perimeter of the square region with reference to the designated region of frequency fP ( step S186).

Then, receiver 200 decodes the line pattern by performing the reverse processing of steps S171 to S174 shown in FIG. 178A on the frequency array (step S187). That is, the receiver 200 can acquire information to be processed.

In such a processing operation by the receiver 200, in step S184, by perspectively transforming the transmission image into a square area, even when the transmission image is captured not only from the front but also from a direction other than the front, the line pattern of the transmission image can be changed. can be decoded correctly. In step S186, the frequencies of the designated regions are arranged in order based on the fundamental frequency fP, so that the line pattern of the transmission image can be reproduced even when the transmission image is captured horizontally or upside down. can be decoded correctly.

FIG. 180 is a flow chart showing the processing operation of the receiver 200. FIG.

First, the receiver 200 determines whether or not the exposure time can be set to a communication exposure time shorter than the normal exposure time (step S191). That is, the receiver 200 determines whether it is a device that does not support optical communication or a device that supports optical communication. Here, when the receiver 200 determines that the communication exposure time cannot be set (N in step S191), it receives an image signal (that is, image ID) (step S193). The communication exposure time is, for example, 1/2000 seconds or less.

On the other hand, when the receiver 200 determines that the communication exposure time can be set (Y in step S191), the receiver 200 determines whether the line scan time is registered in the terminal (that is, the receiver 200) or the server. (Step S192). Note that the line scan time is the time from the start of exposure of one exposure line included in the image sensor to the start of exposure of the next exposure line, as shown in the examples of FIGS. be. If the line scan time is registered, the receiver 200 uses the registered line scan time to decode the image for decoding.

When the receiver 200 determines that the line scan time is not registered (N of step S192), the process of step S193 is performed. On the other hand, when the receiver 200 determines that the line scan time is registered (Y in step S192), it uses the line scan time to receive the light ID, which is a visible light signal (step S194).

Upon receiving the visible light signal, the receiver 200 verifies the identity between the image signal and the visible light signal if the receiver 200 itself is set to the visible light signal identity verification mode (step S195). Here, if the image signal and the visible light signal are different, the receiver 200 displays on the display a message or an image indicating that the signals are different. Alternatively, the receiver 200 notifies the server that the signals are different.

FIG. 181A is a diagram showing an example of the configuration of a system according to this embodiment.

The system according to this embodiment includes multiple transmitters 100 and receivers 200 . Transmitter 100 is configured as a self-propelled robot. For example, the robot is an automatic cleaning robot, or a robot that communicates with humans. Receiver 200 is configured as a camera, such as a surveillance camera or an environmental camera. Hereinafter, the transmitter 100 will be referred to as the robot 100 and the receiver 200 will be referred to as the camera 200 .

Robot 100 transmits a light ID, which is a visible light signal, to camera 200 . Camera 200 receives the light ID transmitted from robot 100 .

FIG. 181B is a diagram showing processing of camera 200 in the present embodiment.

Each of the plurality of robots 100 is automatically traveling. In such a case, the camera 200 first performs imaging in the normal imaging mode, and detects a moving object as the robot 100 from the normal imaging image obtained by the imaging (step S221). Next, the camera 200 transmits an ID transmission request signal prompting the robot 100 to transmit an ID by radio wave communication (step S225). When the robot 100 receives this ID transmission request signal, the robot 100 starts transmitting the ID of the robot 100 (that is, the light ID) by visible light communication.

Next, camera 200 changes the shooting mode from normal shooting mode to visible light recognition mode (step S226). This visible light recognition mode is a kind of visible light communication mode. Specifically, in the visible light recognition mode, among all the exposure lines included in the image sensor of the camera 200, only a plurality of specific exposure lines that capture the image of the robot 100 are lines in the exposure time for communication. Used for scanning. In other words, the camera 200 performs line scanning only for the specified plurality of exposure lines, and does not perform exposure for other exposure lines. Through such line scanning, camera 200 detects an ID (that is, optical ID) from robot 100 (step S227).

Next, the camera 200 recognizes the current position of the robot 100 based on the position of the visible light signal in the decoding image (i.e., bright line image), that is, the position where the bright line pattern appears, and the imaging direction of the camera 200. (Step S228). Camera 200 notifies robot 100 and the server of the ID and current position of robot 100 and the detection time of the ID.

The camera 200 then changes the imaging mode from the visible light recognition mode to the normal imaging mode (step S230).

Here, each of the plurality of robots 100 may be traveling automatically while transmitting the robot detection signal. Although this robot detection signal is a visible light signal, it is a light signal of a frequency that can be recognized even in the normal shooting mode of the camera 200 . That is, the frequency of the robot detection signal is lower than the frequency of the optical ID.

In such a case, instead of detecting a moving object as the robot 100, the camera 200 detects the robot detection signal from the normal captured image (step S223), and executes the processing of steps S225 to S230. good too.

Further, each of the plurality of robots 100 may be traveling automatically while transmitting the position recognition request signal by radio wave communication or the like and transmitting the ID by visible light communication.

In such a case, camera 200 may perform the processing of steps S226 to S230 when receiving the position recognition request signal (step S224). Note that there is a case where the robot 100 is not displayed in the normal captured image when the camera 200 receives the position recognition request signal. In such a case, the camera 200 may notify the robot 100 that the robot 100 is not captured. In other words, the camera 200 may notify the robot 100 that the position of the robot 100 cannot be recognized.

FIG. 182 is a diagram showing another example of the configuration of the system according to this embodiment.

For example, the transmitter 100 may include a plurality of light sources 171 and transmit light IDs from each of the plurality of light sources 171 by varying the brightness of the light sources 171 . Thereby, blind spots of the camera 200 can be reduced. That is, the camera 200 can easily receive the light ID. In addition, when a plurality of light sources 171 are imaged by the camera 200, the camera 200 can more appropriately recognize the position of the robot 100 by multipoint survey. That is, the position recognition accuracy of the robot 100 can be improved.

Also, the robot 100 may transmit different light IDs from each of the plurality of light sources 171 . In this case, even if the camera 200 captures an image of only a portion of the light sources 171 (for example, only one light source 171) instead of all of the plurality of light sources 171, the light ID of the portion of the light sources 171 is , the position of the robot 100 can be accurately recognized.

Further, when the robot 100 is notified of the current position of the robot 100 from the camera 200 , the robot 100 may give rewards such as points to the camera 200 .

FIG. 183 is a diagram showing another example of an image drawn on the transmitter according to this embodiment.

174 and 175, the transmitter 100 can also transmit information as an image ID to a receiver that cannot perform imaging in the visible light communication mode, i.e., a receiver that does not support optical communication. is configured so that Note that the image ID is also called a frame ID. In other words, a substantially rectangular transmission image Im3 is drawn on the transmitter 100 . Further, the transmitter 100 is configured as a signage, for example, and transmits the light ID by changing the brightness, as described above. Alternatively, the transmitter 100 may include a light source and directly transmit the light ID to the receiver 200 according to changes in luminance of the light source. Specifically, the transmission image Im3 is drawn on the front surface of a translucent plate, and the light from the light source is emitted toward the back surface of the plate. As a result, the luminance change of the light source appears as the luminance change of the transmission image Im3, and the light ID is transmitted to the receiver 200 as a visible light signal by the luminance change of the transmission image Im3. Alternatively, the transmitter 100 may be a display device having a display such as a liquid crystal display or an organic EL display. The transmitter 100 transmits the light ID by changing the brightness of the display while displaying the transmission image Im3 on the display. Alternatively, the transmitter 100 may include a light source, irradiate the transmission image Im3 with light from the light source, and reflect the light from the transmission image Im3, thereby transmitting the light to the receiver 200 as the light ID.

Such transmission image Im3 of transmitter 100 is formed in a substantially rectangular shape like transmission images Im1 and Im2 shown in FIGS. 174 and 175. FIG. The transmission image Im3 has a substantially rectangular base image Bi3 and a line pattern 155c added to the base image Bi3.

In the example shown in FIG. 183, the line pattern 155c is an array pattern of a plurality of short straight lines arranged along each of the four sides of the base image Bi3. are each orthogonal to its edge. Also, the line pattern 155c is made up of 32 blocks (the above-mentioned designated area). These blocks are also called PHY symbols, as described below. The frequency index for each of the 32 blocks is -1, 0, 1, 2, or 3. An index of -1 indicates 200 times the fundamental frequency, an index of 0 indicates 210 times the fundamental frequency, an index of 1 indicates 220 times the fundamental frequency, and an index of 2 indicates 230 times the fundamental frequency. and an index of 3 indicates 240 times the fundamental frequency. Here, the fundamental frequency is the reciprocal of the diagonal length (that is, fundamental period) of the base image Bi3, as described above. That is, in the block corresponding to the index of -1, short lines are arranged at the frequency of the fundamental frequency×200. In other words, the interval between adjacent short lines in that block is 1/200 of the diagonal of the base image Bi3. Therefore, each PHY symbol (that is, block) in the present embodiment indicates any one of -1, 0, 1, 2 and 3 depending on the arrangement pattern.

Such a transmission image Im3 is captured by the image sensor of the receiver 200 as a subject. In other words, the subject has a rectangular shape as seen from the image sensor, and when the brightness of the light in the central area of the subject changes, a visible light signal is transmitted, and a barcode-like line pattern is arranged around the periphery of the subject. ing.

FIG. 184 is a diagram showing an example format of a MAC frame forming a frame ID.

A MAC (medium access control) frame consists of a MAC header and a MAC payload. The MAC header consists of 4 bits. The MAC payload consists of variable length padding, variable length ID1 and fixed length ID2. ID2 consists of 5 bits when the MAC frame consists of 44 bits, and consists of 3 bits when the MAC frame consists of 70 bits. The padding consists of, for example, "0000000000001", "0001", "01", or "1", and is the portion from the leftmost bit to the first appearance of 1.

ID1 is the frame ID described above, which is the same information as the light ID, which is identification information indicated by the visible light signal. That is, the visible light signal and the signal acquired from the line pattern are the same identification information. Accordingly, even if the receiver 200 cannot receive the visible light signal, if the transmission image Im3 is captured, the receiver 200 can acquire the same identification information as the visible light signal from the line pattern 155c of the transmission image Im3. can be done.

FIG. 185 is a diagram showing an example of the configuration of a MAC header.

For example, the bit at address '0' in the MAC header indicates the header version. Specifically, if the bit at address '0' is '0', the header version is 1;

Two bits of address "1-2" in the MAC header indicate the protocol. Specifically, when the two bits of the address "1-2" are "00", the protocol of the MAC frame conforms to the IEC (International Electrotechnical Commission), and the two bits of the address "1-2" are In the case of "01", the protocol of the MAC frame conforms to LinkRay (registered trademark) Data. Also, when two bits of the address "1-2" are "10", the protocol of the MAC frame conforms to IEEE (The Institute of Electrical and Electronics Engineers, Inc.).

The bit at address '3' in the MAC header indicates another protocol. Specifically, when the MAC frame conforms to IEC and the bit of address "3" is "0", the number of bits per packet is 4 bits. Also, if the MAC frame conforms to IEC and the bit of address "3" is "1", the number of bits per packet is 8 bits. On the other hand, if the MAC frame complies with LinkRay Data and the bit at address "3" is "0", the number of bits per packet is 32 bits. Note that the number of bits per packet described above is the length of the DATAPART (that is, the datapart length).

FIG. 186 is a diagram showing an example of a table for deriving the number of packet divisions.

The receiver 200 decodes the frame ID, which is ID1 included in the MAC frame, from the line pattern 155c, and derives the number of divisions corresponding to the frame ID. This is because, in visible light communication involving luminance changes, the information to be sent and received is defined by the light ID and the number of packet divisions, and in order to maintain compatibility with visible light communication even in communication using transmitted images, , because the number of divisions is required.

Receiver 200 according to the present embodiment refers to the table shown in FIG. 186 and derives the number of divisions for that frame ID using a set of the number of bits of ID1 (hereinafter referred to as ID length) and datapart length. . The receiver 200 identifies, for example, how many bits the datapart length is based on the bit of address "3" in the MAC header, and further identifies the ID length, which is the length of ID1 of the MAC frame. Receiver 200 then finds the number of divisions associated with the identified set of datapart length and ID length in the table shown in FIG. 186 to derive the number of divisions. Specifically, if the datapart length is 4 bits and the ID length is 10 bits, the division number "5" is derived.

If receiver 200 cannot derive the number of divisions from the table shown in FIG. The number of divisions may be determined to be 0.

Also, in the table shown in FIG. 186, division numbers "6" and "7" are associated with a set of datapart length "4 bits" and ID length "14 bits". Therefore, for example, when the frame ID is encoded, if the ID length is 15 bits, the number of divisions may be set to "7". Then, when the receiver 200 decodes the frame ID, if the datapart length is 4 bits and the ID length is 15 bits, the receiver 200 derives the division number "7". Furthermore, the receiver 200 may ignore the leading 1 bit in the 15-bit ID1 and derive the 14-bit ID1 as the final frame ID.

Note that if the frame ID conforms to the IEEE protocol, the receiver 200 may provisionally derive the number of divisions "0", for example. Note that the number of divisions "0" indicates that division is not performed.

As a result, the light ID and the number of divisions used in visible light communication involving changes in brightness can be appropriately applied as the frame ID and the number of divisions to communication using a transmitted image. In other words, it is possible to maintain compatibility between visible light communication involving luminance changes and communication using a transmitted image.

FIG. 187 is a diagram showing PHY encoding.

First, the encoding device that encodes the frame ID adds an ECC (Error Check Code) to the MAC frame. The encoder then divides the ECC-added MAC frame into multiple blocks. The number of bits of these blocks is N (N is 2 or 3, for example). For each of the plurality of blocks, the encoding device converts the value indicated by the N bits contained in that block into a Gray code. Note that the Gray code is a code in which only one bit is different between adjacent numerical values. In other words, the Gray code has the property that the Hamming distance between adjacent codes is always 1. Errors are most likely to occur between symbols that are adjacent to each other, but if the Gray code is used, a plurality of bits do not differ between symbols, so the efficiency of error detection can be improved.

Then, the encoding device converts the values converted to Gray code into PHY symbols corresponding to the values for each of the plurality of blocks. This generates, for example, 30 PHY symbols each assigned a symbol number (0-29). This PHY symbol corresponds to the block of the line pattern 155c shown in FIG. 183, and is a pattern in which short lines are arranged at regular intervals (that is, a striped pattern). For example, if the value converted to Gray code indicates 1, a block (ie, PHY symbol) having a frequency 220 times the fundamental frequency is generated as shown in FIG.

FIG. 188 is a diagram showing an example of a transmission image Im3 having PHY symbols.

Around the base image Bi3, as shown in FIG. 188, the above-mentioned 30 PHY symbols and two header symbols are arranged. A header symbol is a symbol having a function as a header among PHY symbols. The two header symbols consist of a header symbol for rotational alignment and a header symbol for specifying the PHY version. The frequency index of these header symbols is -1. That is, as shown in Figure 183, the frequency of these header symbols is 200 times the fundamental frequency. The header symbol for rotational alignment is a symbol for making receiver 200 recognize the arrangement of 30 PHY symbols. The receiver 200 recognizes the arrangement of each PHY symbol based on the position of the header symbol for rotational alignment. For example, such a header symbol for rotational alignment is placed at the upper left corner of the base image Bi3.

A header symbol for specifying the PHY version is a symbol for specifying the PHY version. For example, the PHY version is specified by the relative position of the header symbol for specifying the PHY version from the header symbol for rotational alignment. The above 30 PHY symbols other than the header symbol are arranged clockwise around the base image Bi3 from the right side of the header symbol for rotational alignment in ascending order of symbol numbers.

FIG. 189 is a diagram for explaining two PHY versions.

PHY versions include PHY version 1 and PHY version 2. In PHY version 1, a header symbol for specifying the PHY version is arranged to the right of the header symbol for rotational alignment. In PHY version 2, no header symbol for specifying the PHY version is arranged to the right of the header symbol for rotational alignment. That is, in PHY version 2, the header symbol for specifying the PHY version is arranged so that the PHY symbol with the symbol number "0" is sandwiched between the header symbol for rotational alignment and the header symbol for specifying the PHY version. be done. Thus, the header symbol for specifying the PHY version indicates the PHY version by its arrangement.

In PHY version 1, the number of bits N per PHY symbol is 2, the ECC is 16 bits, and the MAC frame is 44 bits. The PHY body consists of a MAC frame and ECC and is 60 bits. Also, the maximum ID length (length of ID1) is 34 bits, and ID2 is 5 bits.

In PHY version 2, the number of bits N per PHY symbol is 3, the ECC is 20 bits, and the MAC frame is 70 bits. The PHY body consists of a MAC frame and ECC and is 90 bits. Also, the maximum ID length (length of ID1) is 62 bits, and ID2 is 3 bits.

FIG. 190 is a diagram for explaining the Gray code.

In PHY version 1, the number of bits N is two. In this case, in the Gray code conversion shown in FIG. 187, the binary numbers "00, 01, 10, 11" corresponding to "0, 1, 2, 3" expressed in decimal are converted to Gray code "00, 01". , 11, 10”.

In PHY version 2, the number of bits N is three. In this case, in the Gray code conversion shown in FIG. 187, binary numbers "000, 001, 010, 011, 100, 101, 110, 111" is converted to Gray code "000, 001, 011, 010, 110, 111, 101, 100".

191 is a diagram showing an example of decoding processing by the receiver 200. FIG.

The receiver 200 captures the transmission image Im3 of the transmitter 100, and recognizes the PHY version based on the position of the header symbol (PHY header symbol) included in the line pattern 155c of the captured transmission image Im3 (step S601). Note that the receiver 200 may determine whether or not visible light communication is possible, and capture the transmission image Im3 when it is determined that visible light communication is not possible. In this case, the receiver 200 acquires a captured image by capturing an image of a subject with an image sensor, and extracts at least one contour by performing edge detection on the captured image. Further, the receiver 200 selects, from among the at least one contour, a region having a rectangular contour of a predetermined size or more, or a region having a rounded rectangular contour of a predetermined size or more, as a selection region. select. There is a high possibility that the transmission image Im3, which is the subject, is displayed in this selected area. Therefore, in step S601, receiver 200 recognizes the PHY version based on the position of the header symbol included in line pattern 155c of the selected area.

Further, when the receiver 200 determines that the visible light communication is possible in the determination of the above-described visible light communication, when capturing an image of the subject, the exposure time of the image sensor is set to the first value as in the above embodiments. By setting the exposure time and capturing an image of the subject with the first exposure time, a decoding image including the identification information is obtained. Specifically, when the receiver 200 determines that the visible light communication is possible in the determination of the above-described visible light communication, when capturing an image of the subject, as in each of the above-described embodiments, the receiver 200 uses a plurality of A decoding image including a bright line pattern composed of a plurality of bright lines corresponding to an exposure line is obtained, and a visible light signal is obtained by decoding the bright line pattern. On the other hand, receiver 200 sets the exposure time of the image sensor to the second exposure time when capturing an image of the object when it is determined that visible light communication is not possible in the determination of the above-described visible light communication. A normal image is obtained as a captured image by capturing an image of the subject with the second exposure time. Here, the first exposure time described above is shorter than the second exposure time.

Next, receiver 200 restores the MAC frame to which the ECC is added from a plurality of PHY symbols forming line pattern 155c, and confirms the ECC (step S602). The receiver 200 thereby receives the MAC frame from the transmitter 100 . When the receiver 200 confirms that the same MAC frame has been received a specified number of times within a specified time (step S603), it calculates the number of divisions (that is, the number of packet divisions) (step S604). That is, receiver 200 refers to the table shown in FIG. 186 and derives the number of divisions for the MAC frame using a set of ID length and datapart length in the MAC frame. As a result, the number of divisions and the frame ID, which is ID1 of the MAC frame, are decoded. That is, the receiver 200 acquires the identification information from the line pattern of the selection area described above. Specifically, when the receiver 200 determines that the visible light communication is not possible in the determination of the above-described visible light communication, the receiver 200 acquires a signal from the line pattern of the normal image when imaging the subject. Here, the visible light signal and the signal are the same identification information.

By the way, the transmitter 100 having the transmitted image Im3 may be illegally duplicated. For example, a device such as a smart phone with a camera and display may masquerade as the transmitter 100 with the transmitted image Im3. Specifically, the smartphone captures the transmission image Im3 from the transmitter 100 with a camera, and displays the captured transmission image Im3 on the display. Thereby, the smartphone, like the transmitter 100, can transmit the frame ID to the receiver 200 by displaying the transmission image Im3.

Therefore, the receiver 200 determines whether or not the transmission image Im3 displayed on a device such as a smartphone is unauthorized. Decoding of the ID or use of that frame ID may be prohibited.

FIG. 192 is a diagram for explaining a method of fraud detection of transmission image Im3 by receiver 200. FIG.

The transmission image Im3 is, for example, a rectangle. If the transmitted image Im3 is invalid, it is highly likely that the rectangular frame of the transmitted image Im3 is tilted within the same plane with respect to the frame of the display that displays the transmitted image Im3. On the other hand, if the transmission image Im3 is valid, the rectangular frame of the transmission image Im3 is not tilted within the same plane with respect to the above-described frame.

Also, if the transmission image Im3 is an illegal transmission image Im3, there is a high possibility that the rectangular frame of the transmission image Im3 is tilted in the depth direction with respect to the frame of the display that displays the transmission image Im3. On the other hand, if the transmission image Im3 is valid, the rectangular frame of the transmission image Im3 is not tilted in the depth direction with respect to the above-described frame.

The receiver 200 detects the fraudulent transmission image Im3 based on the difference between the valid transmission image Im3 and the fraudulent transmission image Im3 as described above.

Specifically, as shown in (a) of FIG. 192, the receiver 200 captures the frame of the transmission image Im3 (the broken-line rectangle shown in (a) of FIG. 192) and the frame of the transmission image Im3. is displayed, for example, the frame of the display of the smartphone (the solid-line square shown in (a) of FIG. 192) is recognized. Next, the receiver 200 is included in each combination including any one of the two diagonal lines of the frame of the transmission image Im3 and any one of the two diagonal lines of the frame of the display. Calculate the angle formed by two diagonal lines. Then, the receiver 200 determines whether or not the angle with the smallest absolute value among the angles calculated for each combination is equal to or greater than a first threshold value (for example, 5 degrees). Determine whether Im3 is invalid. In other words, the receiver 200 determines whether or not the transmission image Im3 is invalid depending on whether or not the rectangular frame of the transmission image Im3 is tilted within the same plane with respect to the frame of the display. The receiver 200 determines that the transmitted image Im3 is invalid if the angle formed by the minimum absolute value is greater than or equal to the first threshold, and determines that the transmitted image Im3 is invalid if the angle formed is less than the first threshold. is correct.

In addition, as shown in (b) of FIG. 192, the receiver 200 sets the ratio (a/b) of the two sides facing each other in the vertical direction of the frame of the transmission image Im3 and the vertical direction of the frame of the display of the smartphone. and the ratio (A/B) of the two sides facing each other. Receiver 200 then compares those ratios. Specifically, the receiver 200 divides the smaller ratio of the ratio (a/b) and the ratio (A/B) by the larger ratio. Then, the receiver 200 determines whether or not the transmission image Im3 is illegal by determining whether or not the value obtained by the division is equal to or greater than the second threshold value (for example, 0.9). In other words, the receiver 200 determines whether or not the transmission image Im is invalid depending on whether or not the rectangular frame of the transmission image Im3 is tilted in the depth direction with respect to the frame of the display. If the value obtained by the division is less than the second threshold, the receiver 200 determines that the transmission image Im3 is invalid. Determine that there is.

The receiver 200 decodes the frame ID from the transmission image Im3 only when the transmission image Im3 is valid, and prohibits the decoding of the frame ID from the transmission image Im3 when the transmission image Im3 is invalid.

FIG. 193 is a flowchart showing an example of decoding processing including fraud detection of transmission image Im3 by receiver 200. FIG.

First, the receiver 200 captures the transmission image Im3 and detects the frame of the transmission image Im3 (step S611). Next, the receiver 200 performs processing for detecting a rectangular frame that includes the frame of the transmission image Im3 (step S612). A rectangular frame is a frame formed by the outer periphery of a rectangular display that a device such as the above-described smartphone has. Here, the receiver 200 determines whether or not a rectangular frame is desired to be detected by the detection processing in step S612 (step S613). If the receiver 200 determines that no rectangular frame has been detected (No in step S613), it prohibits decoding of the frame ID (step S619).

On the other hand, if it is determined that a rectangular frame has been detected (Yes in step S613), the receiver 200 calculates the angle between the diagonals of the frame of transmission image Im3 and the detected rectangular frame (step S614). The receiver 200 then determines whether or not the formed angle is less than the first threshold (step S615). Here, if it is determined that the formed angle is equal to or greater than the first threshold (No in step S615), the receiver 200 prohibits decoding of the frame ID (step S619).

On the other hand, if it is determined that the formed angle is less than the second threshold (Yes in step S615), the receiver 200 sets the ratio (a/b) of the two sides of the frame of the transmission image Im3 and the ratio of the two sides of the rectangular frame. Division using the ratio (A/B) is performed (step S616). Receiver 200 then determines whether the value obtained by the division is less than the second threshold (step S617). Here, if it is determined that the obtained value is equal to or greater than the second threshold (No in step S617), the receiver 200 decodes the frame ID (step S618). On the other hand, when the receiver 200 determines that the obtained value is less than the second threshold (Yes in step S617), it prohibits decoding of the frame ID (step S619).

Note that, in the above example, the receiver 200 prohibits decoding of the frame ID according to the determination result of steps S613, S615, or S617. However, the receiver 200 may perform frame ID decoding first and then perform the above steps. In this case, receiver 200 prohibits the use of the decoded frame ID or discards the frame ID, depending on the determination result of steps S613, S615, or S617.

Also, a prism sticker may be attached to the transmission image Im3. In this case, as in the example shown in FIG. 176, the receiver 200 determines whether the movement of the receiver 200 changes the pattern or color of the prism seal of the transmission image Im3. When the receiver 200 determines that the transmission image Im3 is changed, the receiver 200 determines that the transmission image Im3 is valid, and decodes the frame ID from the transmission image Im3. On the other hand, if it is determined that there is no change, the receiver 200 determines that the transmission image Im3 is illegal, and prohibits decoding of the frame ID from the transmission image Im3. As described above, the receiver 200 may first decode the frame ID and then determine the pattern or color change. In this case, if the receiver 200 determines that the pattern or color does not change, it prohibits the use of the decoded frame ID or discards the frame ID.

Also, the receiver 200 may determine whether or not the transmission image Im3 is valid by having the user bring the receiver 200 closer to the transmission image Im3. For example, the transmitter 100 transmits the visible light signal by turning on the transmission image Im3 and changing the luminance of the transmission image Im3. Therefore, when the receiver 200 captures the transmission image Im3, the receiver 200 displays on the display a message prompting the user to move the receiver 200 closer to the transmission image Im3. In response to the message, the user brings the camera (that is, image sensor) of receiver 200 closer to transmission image Im3. At this time, the camera of the receiver 200 sets, for example, the minimum exposure time of the image sensor because the amount of light received from the transmission image Im3 increases. As a result, a striped pattern appears in the image displayed on the display when the transmission image Im3 is picked up by the receiver 200 . Note that if the receiver 200 is compatible with optical communication, the striped pattern clearly appears as a bright line pattern. On the other hand, even if the receiver 200 does not support optical communication, the striped pattern does not appear clearly as a bright line pattern, but appears vaguely. It can be determined whether or not That is, the receiver 200 determines that the transmission image Im3 is valid if the striped pattern appears, and determines that the transmitted image Im3 is invalid if the striped pattern does not appear.

As described above, the receiver 200 may first decode the frame ID and then determine the striped pattern. In this case, if the receiver 200 determines that the striped pattern does not appear, the receiver 200 prohibits the use of the decoded frame ID or discards the frame ID.

(Modification)
Receiver 200 in the present embodiment may be a display device having the functions of receiver 200 in the ninth embodiment. That is, the display device determines whether or not visible light communication is possible, and if it is possible, similar to the receiver 200 in each of the above-described embodiments including the ninth embodiment, the display device performs visible light or light ID processing. conduct. On the other hand, when the visible light communication is impossible, the display device performs the processing related to the transmission image or the frame ID described above. In addition, the visible light communication is obtained by transmitting a signal according to the luminance change of the subject, and decoding the bright line pattern corresponding to each exposure line of the image sensor, which is obtained by imaging the subject with the image sensor. It is a communication method for receiving the signal.

FIG. 194A is a flowchart showing a display method according to this modification.

A display method according to an aspect of the present invention is a display method for displaying an image, and includes steps SG1 to SG4. That is, the display device, which is the receiver 200 described above, first determines whether visible light communication is possible (step SG4). Here, when it is determined that visible light communication is possible (Yes in step SG4), the display device acquires a visible light signal as identification information (that is, light ID) by capturing an image of the subject with the image sensor (step SG1). Next, the display device displays the first moving image associated with the light ID (step SG2). Then, when receiving the operation of sliding the first moving image, the display device displays the second moving image associated with the light ID next to the first moving image (step SG3).

FIG. 194B is a block diagram showing the configuration of the display device according to this modification.

A display device G10 according to an aspect of the present invention is a device that displays an image, and includes a determination unit G13, an acquisition unit G11, and a display unit G12. Note that the display device G10 is the receiver 200 described above. The determination unit G13 determines whether or not visible light communication is possible. The acquiring unit G11 acquires a visible light signal as identification information (that is, a light ID) by imaging a subject with an image sensor when the determining unit G13 determines that visible light communication is possible. Next, the display unit G12 displays the first moving image associated with the light ID. Then, when receiving an operation to slide the first moving image, the display unit G12 displays the second moving image associated with the light ID next to the first moving image.

For example, the first moving image and the second moving image are the first AR image P46 and the second AR image P46c shown in FIG. 162, respectively. In the display method and display device G10 shown in FIGS. 194A and 194B, when an operation of sliding the first moving image, that is, a swipe is accepted, the second moving image associated with the identification information is displayed next to the first moving image. A moving image is displayed. Therefore, it is possible to easily display an image that is beneficial to the user. In addition, since it is determined in advance whether or not visible light communication is possible, it is possible to omit useless processing to acquire visible light signals even when it is not possible, thereby reducing the processing load. can.

Here, the display device G10 may acquire identification information (that is, frame ID) from the transmission image Im3 when it determines that the visible light communication is not possible in the determination of the visible light communication. In this case, the display device G10 acquires a captured image by capturing an image of a subject with an image sensor, and extracts at least one contour by performing edge detection on the captured image. Next, the display device G10 selects, from among the at least one contour, a region having a rectangular contour of a predetermined size or more, or a region having a rounded rectangular contour of a predetermined size or more. Select as Then, the display device G10 acquires identification information from the line pattern of the selected area. In addition, the above-mentioned rounded quadrilateral is a quadrilateral in which the outer periphery of each of the four corners is arcuate.

As a result, for example, the transmission images shown in FIGS. 183 and 188 are picked up as the subject, the area of the transmission image is selected as the selection area, and the identification information is acquired from the line pattern of the transmission image. Therefore, even when visible light communication is impossible, the identification information can be properly obtained.

Further, when the display device G10 determines that the visible light communication is possible in determining the visible light communication, the display device G10 sets the exposure time of the image sensor to the first exposure time when capturing an image of the subject, and sets the exposure time to the first exposure time. A decoding image including identification information is obtained by capturing an image of the subject with the exposure time. When the display device G10 determines that the visible light communication is not possible, the display device G10 sets the exposure time of the image sensor to the second exposure time when capturing an image of the subject. By imaging the subject with an exposure time of , a normal image is obtained as the captured image. Here, the first exposure time described above is shorter than the second exposure time.

Accordingly, by switching the exposure time, acquisition of identification information by visible light communication and acquisition of identification information by capturing a transmission image can be appropriately switched.

In addition, the above-described subject has a rectangular shape as viewed from the image sensor, and when the central area of the subject changes in brightness, a visible light signal is transmitted, and a barcode-like line pattern is arranged around the periphery of the subject. ing. When the display device G10 determines that the visible light communication is possible in the judgment of the visible light communication, when the subject is imaged, a bright line pattern composed of a plurality of bright lines corresponding to a plurality of exposure lines of the image sensor. is obtained, and a visible light signal is obtained by decoding the bright line pattern. A visible light signal is, for example, a light ID. When the display device G10 determines that the visible light communication is not possible, the display device G10 obtains a signal from the line pattern of the normal image when imaging the subject. Here, the visible light signal and the signal are the same identification information.

Accordingly, since the identification information indicated by the visible light signal and the identification information indicated by the line pattern signal are the same, the identification information indicated by the visible light signal can be appropriately used even if visible light communication is impossible. can be obtained.

FIG. 194C is a flowchart showing a communication method according to this modification.

A communication method according to an aspect of the present invention is a communication method using a terminal equipped with an image sensor, and includes steps SG11 to SG13. That is, the terminal, which is the receiver 200 described above, first determines whether or not the terminal is capable of performing visible light communication (step SG11). Here, when the terminal determines that it is possible to perform visible light communication (Yes in step SG11), the terminal executes the process of step SG12. That is, the terminal acquires the image for decoding by capturing an image of the subject whose luminance changes with the image sensor, and acquires the first identification information transmitted by the subject from the striped pattern appearing in the image for decoding (step SG12). On the other hand, when the terminal determines that it is not capable of performing visible light communication in the determination of visible light communication in step SG11 (No in step SG11), the terminal executes the process of step SG13. That is, the terminal acquires a captured image by capturing an image of a subject with an image sensor, performs edge detection on the captured image, extracts at least one contour, and selects a predetermined edge from the at least one contour. is specified, and the second identification information transmitted by the subject is obtained from the line pattern of the specified area (step SG13). The first identification information is, for example, the light ID, and the second identification information is, for example, the image ID or the frame ID.

FIG. 194D is a block diagram showing the configuration of a communication device according to this modification.

A communication device G20 according to one aspect of the present invention is a communication device using a terminal having an image sensor, and includes a determination unit G21, a first acquisition unit G22, and a second acquisition unit G23.

The determination unit G21 determines whether or not the terminal can perform visible light communication.

When the determination unit G21 determines that the terminal is capable of performing visible light communication, the first acquisition unit G22 acquires an image for decoding by capturing an image of a subject whose luminance changes with the image sensor. , the first identification information transmitted by the subject is obtained from the striped pattern appearing in the decoding image.

A second acquisition unit G23 acquires a captured image by capturing an image of a subject with an image sensor when the determination unit G21 determines that the terminal is not capable of performing visible light communication. At least one contour is extracted by performing edge detection, a predetermined specific area is specified from the at least one contour, and second identification information transmitted by the subject is acquired from the line pattern of the specific area. do.

Note that the terminal may be included in the communication device G20 or may be outside the communication device G20. Alternatively, the terminal may include the communication device G20. That is, each step of the flowchart shown in FIG. 194C may be executed by the terminal or by the communication device G20.

Accordingly, a terminal such as the receiver 200 can acquire the first identification information or the second identification information from the subject such as the transmitter regardless of whether visible light communication is possible. That is, when the terminal can perform visible light communication, the terminal acquires, for example, the light ID from the subject as the first identification information. On the other hand, even if the terminal cannot perform visible light communication, the terminal can acquire, for example, the image ID or the frame ID from the subject as the second identification information. Specifically, for example, the transmission images shown in FIGS. 183 and 188 are picked up as a subject, the area of the transmission image is selected as a specific area (that is, the selection area), and the second identification information is obtained from the line pattern of the transmission image. is obtained. Therefore, even when visible light communication is impossible, the second identification information can be obtained appropriately.

Further, in identifying the above-mentioned specific area, the terminal identifies, as the specific area, an area having a rectangular outline of a predetermined size or more, or an area having a rounded rectangular outline of a predetermined size or more. may

As a result, for example, as shown in FIG. 179, a square or rounded square area can be appropriately specified as the specified area.

Also, in the determination of visible light communication described above, the terminal determines that visible light communication is possible when the terminal is specified as a terminal that can change the exposure time to a predetermined value or less. However, if the terminal is identified as a terminal that cannot change the exposure time to the predetermined value or less, it may be determined that visible light communication is not possible.

Accordingly, as shown in FIG. 180, for example, it is possible to appropriately determine whether or not the visible light signal can be transmitted.

Further, when the terminal determines that it is possible to perform visible light communication in determining visible light communication, the terminal sets the exposure time of the image sensor to the first exposure time when capturing an image of the subject, and sets the exposure time of the image sensor to the first exposure time. An image for decoding may be acquired by imaging the subject with the first exposure time. Further, the terminal sets the exposure time of the image sensor to the second exposure time when capturing an image of the subject when the terminal determines that the visible light communication is not possible in determining the visible light communication, A captured image may be obtained by capturing an image of the subject with the second exposure time. Here, the first exposure time is shorter than the second exposure time.

This makes it possible to obtain a decoding image having a bright line pattern area by imaging with the first exposure time, and appropriately obtain the first identification information by decoding the bright line pattern area. Furthermore, by imaging with the second exposure time, it is possible to acquire the normally shot image as the shot image, and appropriately acquire the second identification information from the line pattern appearing in the normally shot image. Thereby, the terminal can acquire the first identification information or the second identification information suitable for the terminal by properly using the first exposure time and the second exposure time.

In addition, the subject has a rectangular shape as seen from the image sensor, and the brightness of the center area of the subject changes to transmit the first identification information, and a barcode-like line pattern is arranged around the periphery of the subject. ing. Then, when the terminal determines that it is possible to perform visible light communication in determining visible light communication, when capturing an image of a subject, the terminal is composed of a plurality of bright lines corresponding to a plurality of exposure lines of the image sensor. The first identification information is obtained by obtaining a decoding image including the bright line pattern to be obtained, and decoding the bright line pattern. Further, when the terminal determines that visible light communication is not possible in determining visible light communication, the terminal acquires the second identification information from the line pattern of the captured image when capturing an image of the subject. good too.

As a result, the first identification information and the second identification information can be appropriately acquired from a subject whose central region changes in brightness.

Also, the first identification information obtained from the decoding image and the second identification information obtained from the line pattern may be the same information.

As a result, the same information can be obtained from the subject regardless of whether the terminal is capable of visible light communication or the terminal is not capable of visible light communication.

194E is a block diagram showing a configuration of a transmitter according to Embodiment 10 and its modification. FIG.

Transmitter G30 corresponds to transmitter 100 described above. This transmitter G30 comprises a light source G31, a microcontroller G32 and an illumination plate G33. The light source G31 irradiates light from the back side of the illumination plate 33 . Microcontroller G32 varies the brightness of light source G31. The illumination plate G33 is a plate that transmits light from the light source G31, that is, a plate that has translucency. Moreover, the shape of the illumination plate G33 is, for example, a rectangular shape.

The microcontroller G32 transmits the first identification information from the light source G31 through the illumination plate G33 by changing the brightness of the light source G31. Further, a barcode-like line pattern G34 is arranged around the front side of the illumination plate G33, and the second identification information is encoded in the line pattern G34. Furthermore, the first identification information and the second identification information are the same information.

As a result, the same information can be transmitted to a terminal capable of performing visible light communication and to a terminal not capable of performing visible light communication.

In the above-described embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. For example, the program causes the computer to execute the display method illustrated by the flowcharts of FIGS. 191, 193, 194A, and 194C.

(Embodiment 11)
The server management method according to the present embodiment is a method capable of providing appropriate services to mobile terminal users.

FIG. 195 is a diagram showing an example of the configuration of a communication system including servers according to this embodiment.

This communication system comprises a transmitter 100 , a receiver 200 , a first server 301 , a second server 302 and a store system 310 . Transmitter 100 and receiver 200 in the present embodiment may have the same functions as transmitter 100 and receiver 200 in the above embodiments, respectively. Further, the transmitter 100 is configured as a store signage, for example, and transmits the light ID as a visible light signal by changing the luminance. Store system 310 has at least one computer for managing a store having transmitter 100 . Also, the receiver 200 is, for example, a mobile terminal configured as a smartphone equipped with a camera and a display.

For example, the user of the receiver 200 performs reservation processing in advance for the store system 310 by operating the receiver 200 . This reservation processing is processing for registering user information, which is information about the user such as the user's name, and the menu ordered by the user in the store system 310 before the user visits the store. Note that the user does not have to perform such reservation processing.

Then, when the user visits a store, the receiver 200 takes an image of the transmitter 100, which is the signage of the store. Accordingly, the receiver 200 receives the light ID from the transmitter 100 by visible light communication. Receiver 200 then transmits the light ID to second server 302 via wireless communication. Upon receiving the light ID from the receiver 200, the second server 302 transmits store information associated with the light ID to the receiver 200 via wireless communication. The store information is information about the store displaying the signage.

When the store information is received from the second server 302, the receiver 200 transmits the user information and the store information to the first server 301 via wireless communication. When the first server 301 receives the user information and the shop information, it inquires of the shop system 310 indicated by the shop information whether the reservation processing by the user indicated by the user information has been completed. judge.

Here, when the first server 301 determines that the reservation process has been completed, it notifies the store system 310 of the user's arrival at the store via wireless communication. On the other hand, when determining that the reservation process has not been completed, the first server 301 transmits the menu list of the store to the receiver 200 via wireless communication. Upon receiving the menu list, receiver 200 displays the menu list on the display and accepts menu selection from the user. Then, receiver 200 notifies first server 301 of the menu selected by the user as a selection menu via wireless communication.

When the first server 301 receives notification of the selection menu from the receiver 200, the first server 301 notifies the store system 310 of the selection menu via wireless communication.

196 is a flow chart showing a management method by the first server 301. FIG.

First, the first server 301 receives store information from the mobile terminal, which is the receiver 200 (step S621). Next, the first server 301 determines whether or not reservation processing for the store indicated by the store information has been completed (step S622). Here, when the first server 301 determines that the reservation process has been completed (Yes in step S622), it notifies the store system 310 of the arrival of the user of the portable terminal at the store (step S623). On the other hand, when the first server 301 determines that the reservation process has not been completed (No in step S622), it notifies the mobile terminal of the menu list of the store (step S624). Further, when a selection menu, which is a menu selected from the menu list, is notified from the mobile terminal, the first server 301 notifies the store system 310 of the selection menu (step S625).

As described above, in the management method of the server (that is, the first server 301) according to the present embodiment, the server receives store information from the mobile terminal and, based on the store information, stores the information about the store by the user of the mobile terminal. determines whether or not the reservation processing for the menu of (1) is completed, and if the reservation processing is completed, notifies the store system that the user of the portable terminal has arrived at the store. I do. Further, in the management method, when the reservation processing is not completed, the server notifies the mobile terminal of the menu list of the store, and when the menu selection is accepted from the mobile terminal, the Notifies the selected menu to the store system. Further, in the management method, the mobile terminal acquires a visible light signal as identification information by photographing a subject installed in the store, transmits the identification information to another server, and receives the information from the other server. Store information corresponding to the identification information is received, and the received store information is transmitted to the server.

As a result, if the user of the portable terminal makes a reservation in advance, upon arrival at the store, the user can immediately start cooking the ordered menu at the store, and can eat and drink freshly prepared food. can. In addition, the user can select a menu from the menu list and place an order with the store as soon as they arrive at the store, even if they have not made a reservation.

Note that the receiver 200 transmits identification information (that is, light ID) to the first server 301 instead of store information, and the first server 301 determines whether or not the reservation process has been completed based on the identification information. You can check. In this case, the identification information is transmitted from the mobile terminal to the first server 301 without transmitting the identification information to the second server 302 .

(Embodiment 12)
In the present embodiment, as in the above embodiments, a communication method and a communication device using an optical ID will be described. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitting device) and receiver (or receiving device) in the above embodiments.

FIG. 197 is a diagram showing a lighting system according to this embodiment.

For example, as shown in FIG. 197(a), this lighting system includes a plurality of first lighting devices 100p and a plurality of second lighting devices 100q. Such lighting systems are, for example, mounted on the ceilings of large stores. Also, the plurality of first lighting devices 100p and the plurality of second lighting devices 100q are each formed in an elongated shape and arranged in a line along the longitudinal direction thereof. Also, the first lighting devices 100p and the second lighting devices 100q are alternately arranged.

The first illumination device 100p is configured as the transmitter 100 in each of the above-described embodiments, emits light for illumination, and transmits a visible light signal as a light ID. In addition, the second lighting device 100q emits illumination light and also transmits a dummy signal. That is, the second lighting device 100q emits light for lighting and transmits a dummy signal by periodically changing the luminance. When the receiver captures an image of the lighting system in the visible light communication mode, the decoding image, which is the above-described visible light communication image or bright line image obtained by the imaging, includes the following in the region corresponding to the first lighting device 100p: A bright line pattern area appears. However, no bright line pattern area appears in the area corresponding to the second illumination device 100q in the decoding image.

Thus, in the lighting system shown in FIG. 197(a), the second lighting device 100q is arranged between two first lighting devices 100p adjacent to each other. As a result, the receiver that receives the visible light signal can appropriately identify the edge of the bright line pattern region in the decoding image, and distinguishes and receives the visible light signal transmitted from each of the first lighting devices 100p. can do.

Also, the average luminance when the second lighting device 100q emits light (that is, when transmitting a dummy signal) and the average luminance when the first lighting device 100p emits light (that is, when transmitting a visible light signal). is equal to the average luminance when Therefore, it is possible to suppress variation in brightness of each lighting device in the lighting system. The brightness of each lighting device is the brightness perceived by a person. This makes it difficult for people in the store to perceive variations in brightness in the lighting system. Further, when the brightness of the second lighting device 100q is changed by switching between lighting (on) and lighting (off), even if the second lighting device 100q does not have a dimming function, The average brightness of the second lighting device 100q can be adjusted by the duty ratio between on and off.

Further, the lighting system may include a plurality of first lighting devices 100p and may not include the second lighting device 100q, as shown in FIG. 197(b), for example. In this case, the plurality of first lighting devices 100p are arranged in a row along their longitudinal direction and spaced apart from each other.

Therefore, even in the illumination system shown in FIG. 197(b), the receiver that receives the visible light signal detects the edge of the bright line pattern region in the decoding image as in the illumination system shown in FIG. 197(a). can be properly identified. As a result, the receiver can distinguish and receive visible light signals transmitted from the respective first lighting devices 100p.

Alternatively, the plurality of first lighting devices 100p may be arranged adjacent to each other, and the border between two adjacent first lighting devices 100p may be covered with a cover. This cover blocks the light emitted from the boundary portion. Alternatively, each of the plurality of first lighting devices 100p may have a structure in which light is not emitted from both ends in the longitudinal direction of the first lighting device 100p.

In such a lighting system shown in (a) and (b) of FIG. 197, the receiver uses the longitudinal length of the first lighting device 100p included in the lighting system to detect the first lighting device. The distance from 100p can be calculated. Therefore, the receiver can accurately estimate its position.

FIG. 198 is a diagram illustrating an example of an arrangement of lighting devices and a decoding image.

For example, as shown in (a) of FIG. 198, the first lighting device 100p and the second lighting device 100q are arranged adjacent to each other. Here, the second lighting device 100q transmits a dummy signal by switching on and off at a period of 100 μs or less.

The receiver captures the decoding image shown in (b) of FIG. 198 by capturing the images of the first lighting device 100p and the second lighting device 100q. Here, the cycle of switching on and off in the second illumination device 100q is too short compared to the exposure time of the receiver. Therefore, the brightness of the area corresponding to the second illumination device 100q (hereinafter referred to as the dummy area) in the decoding image is uniform. In addition, the brightness of this dummy area is higher than the area corresponding to the background other than the first lighting device 100p and the second lighting device 100q. Also, the luminance of this dummy area is lower than the high luminance of the area corresponding to the first illumination device 100p, that is, the bright line pattern area.

Therefore, the receiver can distinguish the illumination device corresponding to the dummy area from the illumination device corresponding to the bright line pattern area.

FIG. 199 is a diagram showing another example of the arrangement of lighting devices and a decoding image.

For example, as shown in (a) of FIG. 199, the first lighting device 100p and the second lighting device 100q are arranged adjacent to each other. Here, the second lighting device 100q transmits a dummy signal by switching on and off with a period exceeding 100 μs.

The receiver picks up an image for decoding shown in (b) of FIG. 199 by picking up images of the first lighting device 100p and the second lighting device 100q. Here, the cycle of switching on and off in the second illumination device 100q is long compared to the exposure time of the receiver. Therefore, the luminance of the dummy area in the decoding image is not uniform, and bright areas and dark areas alternately appear in the dummy area. For example, if a dark area wider than a predetermined maximum width appears in the image for decoding, the receiver can recognize that there is a dummy area in the range including the dark area.

Therefore, the receiver can distinguish the illumination device corresponding to the dummy area from the illumination device corresponding to the bright line pattern area.

FIG. 200 is a diagram for explaining position estimation using the first lighting device 100p.

The receiver 200 can estimate the position of the receiver 200 by imaging the first lighting device 100p as described above.

However, the receiver 200 may notify the user of an error if the height above the floor at its estimated position is higher than the allowable range. For example, the receiver 200 determines the position of the first lighting device 100p based on the longitudinal length of the first lighting device 100p displayed in the decoding image or the normal shot image, the output of the acceleration sensor, and the like. and orientation. Further, receiver 200 identifies the height from the floor at the position of receiver 200 using the height from the floor to the ceiling where first lighting device 100p is installed. Then, the receiver 200 notifies an error when the height at the position of the receiver 200 is higher than the allowable range. Note that the position and orientation of the first lighting device 100 p described above are relative to the receiver 200 . Therefore, it can be said that the position and orientation of the receiver 200 are identified by identifying the position and orientation of the first lighting device 100p.

FIG. 201 is a flow chart showing processing operations of the receiver 200 .

First, the receiver 200 estimates the position of the receiver 200 as shown in (a) of FIG. 201 (step S231). Next, receiver 200 derives the height from the floor to the ceiling (step S232). For example, the receiver 200 derives its floor-to-ceiling height by reading the height stored in memory. Alternatively, receiver 200 derives the height from the floor to the ceiling by receiving information transmitted by radio waves from nearby transmitters.

Next, the receiver determines that the height from the floor to the receiver 200 is within the allowable range based on the position of the receiver 200 estimated in step S231 and the height from the floor to the ceiling derived in step S232. It is determined whether or not it is within (step S233). Here, when the receiver determines that the height is within the allowable range (Yes in step S233), it displays the position and orientation of the receiver 200 (step S234). On the other hand, if the receiver determines that the height is not within the allowable range (No in step S233), it displays only the orientation of the receiver 200 (step S235).

Alternatively, receiver 200 may execute step S236 instead of step S235, as shown in FIG. 201(b). That is, when the receiver determines that the height is not within the allowable range (No in step S233), it notifies the user that an error has occurred in position estimation (step S236).

FIG. 202 is a diagram illustrating an example of a communication system according to this embodiment.

The communication system comprises receiver 200 and server 300 . The receiver 200 receives location information or transmitter ID transmitted by GPS, radio waves or visible light signals. Note that the position information is, for example, information indicating the position of the transmitter or receiver, and the transmitter ID is identification information for identifying the transmitter. Receiver 200 then transmits the received location information or transmitter ID to server 300 . Server 300 transmits the map or content associated with the location information or transmitter ID to receiver 200 .

FIG. 203 is a diagram for explaining self-position estimation processing by receiver 200 in this embodiment.

Receiver 200 performs self-position estimation at predetermined intervals. This self-position estimation consists of a plurality of processes. The period described above is, for example, the frame period of imaging by the receiver 200 .

For example, the receiver 200 acquires the result of self-position estimation performed in the previous frame period as the previous self-position. Then, receiver 200 estimates the moving distance and moving direction from the previous self-position based on outputs from the acceleration sensor, the gyro sensor, and the like. Furthermore, the receiver 200 estimates its own position in the current frame period by changing its previous self-position according to the estimated moving distance and moving direction. As a result, the first self-position estimation result is obtained. On the other hand, receiver 200 estimates its own position in the current frame period based on at least one of radio waves, visible light signals, and outputs from the acceleration sensor and direction sensor. As a result, a second self-position estimation result is obtained. Receiver 200 then adjusts the second self-position estimation result based on the first self-position estimation result using, for example, a Kalman filter. This gives the final self-localization result for the current frame period.

FIG. 204 is a flowchart showing self-position estimation by receiver 200 in this embodiment.

First, the receiver 200 estimates the position of the receiver 200 based on the strength of radio waves (step S241). As a result, the estimated position A of the receiver 200 is obtained.

Next, receiver 200 measures the moving direction and the moving direction of receiver 200 based on outputs from the acceleration sensor, gyro sensor, direction sensor, and the like (step S242).

Next, receiver 200 receives the visible light signal, and measures the position of receiver 200 based on the received visible light signal and outputs from the acceleration sensor, direction sensor, and the like (step S243).

Then, receiver 200 calculates estimated position A obtained in step S241 using the moving distance and moving direction of receiver 200 measured in step S242 and the position of receiver 200 measured in step S243. Update (step S243). An algorithm such as the Kalman filter is used to update the estimated position A. Then, the processes after step S242 are repeatedly executed.

FIG. 205 is a flow chart showing an outline of self-position estimation processing of receiver 200 in this embodiment.

First, the receiver 200 estimates a rough position of the receiver 200, for example, based on the strength of radio waves such as Bluetooth (registered trademark) (step S251). Next, the receiver 200 estimates the detailed position of the receiver 200 using visible light signals or the like (step S252). As a result, the self-position can be estimated within an error range of ±10 cm, for example.

Note that the number of light IDs that can be assigned to each transmitter is small, and it is not possible to give each transmitter in the world a unique light ID. However, in the present embodiment, as in the processing of step S251 described above, the area in which the transmitter transmitting the radio wave exists is narrowed down based on the strength of the radio wave. Then, if a plurality of transmitters each having the same light ID does not exist in that area, receiver 200 identifies one transmitter from that area by the process of step S252, that is, based on the light ID. be able to.

For each transmitter, the server associates and stores the optical ID possessed by the transmitter, the location information indicating the position of the transmitter, and the ID of the radio wave possessed by the transmitter.

FIG. 206 is a diagram showing the relationship between radio wave IDs and optical IDs in this embodiment.

For example, the radio ID includes the same information as the optical ID. The radio wave ID is identification information used for Bluetooth (registered trademark), Wi-Fi (registered trademark), or the like. That is, the transmitter transmits the ID of the radio wave by radio waves, and also transmits information matching at least part of the ID of the radio waves as the optical ID. For example, several low-order bits included in the radio wave ID match the light ID. This allows the server to centrally manage radio wave IDs and optical IDs.

Also, the receiver 200 can confirm via radio waves whether or not there are a plurality of transmitters transmitting the same light ID in the vicinity of the receiver 200 . Then, when the receiver 200 confirms that a plurality of transmitters exist, the receiver 200 may change the light ID of any one of the transmitters via radio waves.

FIG. 207 is a diagram for explaining an example of imaging by receiver 200 in this embodiment.

For example, as shown in (a) of FIG. 207, the receiver 200 captures an image of the first lighting device 100p at position A in the visible light communication mode. Further, the receiver 200 captures an image of the first lighting device 100p at the position B in the visible light communication mode. Here, the position A and the position B have a point-symmetrical relationship with respect to the first lighting device 100p. In this case, the receiver 200 generates the same image for decoding at both position A and position B, as shown in FIG. 207(b). Therefore, receiver 200 cannot distinguish whether receiver 200 is at position A or at position B only from the decoding image shown in FIG. 207(b). Therefore, receiver 200 may present position A and position B as candidates for self-position estimation. Further, receiver 200 may narrow down one candidate from a plurality of candidates based on the past position of receiver 200 and the direction of movement from that position. Further, when two or more illumination devices are displayed in the image for decoding, the image for decoding obtained at the position A and the image for decoding obtained at the position B are different. Therefore, in this case, the candidate positions for the receiver 200 can be narrowed down to one.

Note that the receiver 200 can narrow down to one position from the positions A and B based on the output from the azimuth sensor. However, even in such a case, when the reliability of the orientation sensor is low, receiver 200 may present position A and position B as candidates for the position of receiver 200 .

FIG. 208 is a diagram for explaining another example of imaging by receiver 200 in this embodiment.

For example, a mirror 901 may be arranged around the first lighting device 100p. Thereby, the image for decoding obtained by imaging at the position A and the image for decoding obtained by imaging at the position B can be made different. In other words, it is possible to suppress the occurrence of a situation where the position of the receiver 200 cannot be narrowed down to one by self-position estimation based on the decoding image.

FIG. 209 is a diagram for explaining a camera used by receiver 200 in this embodiment.

For example, the receiver 200 has a plurality of cameras and selects a camera to be used for visible light communication from the plurality of cameras. Specifically, receiver 200 identifies the orientation of receiver 200 based on the output data from the acceleration sensor, and selects an upward facing camera from a plurality of cameras. Alternatively, receiver 200 may select one or a plurality of cameras that are capable of capturing an upward image relative to the horizontal direction based on the orientation of receiver 200 and the angles of view of the multiple cameras. In addition, when a plurality of cameras are selected, receiver 200 may further select one camera having the widest upward area in the angle of view from among the plurality of cameras. In addition, the receiver 200 does not need to perform processing for self-position estimation or light ID reception on a part of the image captured by the camera. The partial area may be an area that is downward from the horizontal direction, or an area that is downward from the horizontal direction by a predetermined angle.

Thereby, the calculation load of the receiver 200 can be reduced.

FIG. 210 is a flow chart showing an example of processing for changing the visible light signal of the transmitter by receiver 200 according to the present embodiment.

The receiver 200 first receives a visible light signal A as a visible light signal (step S261).

Next, the receiver 200 transmits, by radio wave, a command to "change to visible light signal B if visible light signal A is being transmitted" (step S262).

The transmitter 100 then receives the command transmitted in step S262 from the receiver. When the visible light signal transmitted by itself is set to visible light signal A based on the command, the transmitter, which is the first lighting device 100p, makes the set visible light signal A visible. It is changed to optical signal B (step S263).

FIG. 211 is a flowchart showing another example of processing for changing the visible light signal of the transmitter by receiver 200 according to the present embodiment.

Receiver 200 first receives visible light signal A as a visible light signal (step S271).

Next, receiver 200 searches for transmitters capable of radio communication by receiving ambient radio waves, and creates a list of the transmitters (step S272).

Next, receiver 200 rearranges the order of the plurality of transmitters shown in the created list in a predetermined order (step S273). This predetermined order is, for example, in order of increasing radio wave intensity, in random order, or in decreasing order of transmitter ID.

Next, the receiver 200 radio-waves the first transmitter shown in the list to transmit the visible light signal B for a predetermined period of time (step S274). The receiver 200 then determines whether or not the visible light signal A received in step S271 has been changed to the visible light signal B (step S275). Here, when receiver 200 determines that the change has been made (Y in step S275), receiver 200 instructs the first transmitter shown in the list to continue transmission of visible light signal B (step S276). .

On the other hand, if it is determined that visible light signal A is not changed to visible light signal B (N in step S275), receiver 200 returns the visible light signal to the signal before change for the first transmitter on the list. (step S277). Receiver 200 then deletes the first transmitter shown in the list from the list, and moves up the order of the second and subsequent transmitters by one (step S278). Receiver 200 then repeats the process from step S274.

Such processing allows the receiver 200 to properly identify the transmitter transmitting the visible light signal currently being received by the receiver 200 and cause the transmitter to modify the visible light signal. can.

(Embodiment 13)
Receiver 200 performs self-position estimation and navigation using the estimation results, as in the example shown in FIGS. 18A to 18C of the second embodiment. The receiver 200 uses the size and position of the bright line pattern area included in the decoding image when estimating the self-position. That is, the receiver 200 performs reception for the transmitter 100 based on the attitude of the receiver 200, the size and shape of the transmitter 100, and the size, shape and position of the bright line pattern region included in the decoding image. Identify the relative position of the aircraft 200 . Then, receiver 200 estimates its own position using the position of transmitter 100 on the map specified by the visible light signal from transmitter 100 and the above specified relative position. The attitude of receiver 200 is, for example, the orientation of the camera of receiver 200 specified by output data from sensors such as an acceleration sensor and a direction sensor provided in receiver 200 .

FIG. 212 is a diagram for explaining navigation by the receiver 200. FIG.

The transmitter 100 is, for example, configured as a digital signage for guiding a bus stop and set in an underground mall, as shown in FIG. 212(a). Transmitter 100 transmits a visible light signal as in the example shown in FIGS. 18A to 18C of the second embodiment. Here, an image prompting AR navigation is displayed on the transmitter 100 . When the user of the receiver 200 sees the transmitter 100 and wants to be guided to the bus stop by AR navigation, the user activates the AR navigation application installed in the receiver 200 configured as a smartphone. . By this activation, the receiver 200 causes the built-in camera to alternately switch between imaging in the visible light communication mode and imaging in the normal imaging mode. Then, the receiver 200 displays a normal shot image obtained by the shooting on the display of the receiver 200 each time an image is shot in the normal shooting mode. The user then points the camera of the receiver 200 toward the transmitter 100 . As a result, the receiver 200 acquires the image for decoding at the timing when imaging is performed in the visible light communication mode, and decodes the bright line pattern area included in the image for decoding. Receiving visible light signals. Receiver 200 then transmits the information indicated by the visible light signal (i.e., the light ID) to a server, and receives data indicating the map location of transmitter 100 associated with that information from the server. . Further, the receiver 200 estimates its own position using the position of the transmitter 100 on the map, and transmits the estimated self-position to the server. The server searches for the position of the receiver 200 and the route to the bus stop, which is the destination, and transmits data indicating the map and the route to the receiver 200 . The position of receiver 200 obtained by self-position estimation at this time is the starting point for guiding the user to the destination.

Next, the receiver 200 starts navigation according to the retrieved route, as shown in FIG. 212(b). At this time, the receiver 200 superimposes the direction instruction image 431 on the normal photographed image and displays it on the display. This direction indication image 431 is generated based on the searched route, the current position of the receiver 200, and the orientation of the camera, and is configured as an arrow directed toward the destination.

Then, when moving through the underground mall, the receiver 200 estimates the current self-position based on the movements of the feature points shown in the normal captured images, as shown in (c) and (d) of FIG. do.

Furthermore, as shown in (e) of FIG. 212, the receiver 200 receives a visible light signal from a transmitter 100 different from the transmitter 100 shown in (a) of FIG. correct the self-position. That is, the receiver 200 updates its own position by re-estimating its own position using the visible light signal.

Then, the receiver 200 guides the user to the destination, the bus stop, as shown in (f) of FIG.

In this way, the receiver 200 may first perform self-position estimation based on the visible light signal at the starting point, and periodically update the self-position obtained by the estimation. For example, as shown in (c) and (d) of FIG. The self-position may be updated based on the amount of movement of the feature points displayed in the captured image. Then, the receiver 200 periodically performs imaging in the visible light communication mode while imaging is being performed in the normal imaging mode. As shown in (e) of FIG. 212, the receiver 200, if the bright line pattern area is displayed in the image for decoding obtained by the imaging in the visible light communication mode, the receiver 200 updates its self recently updated at that time. The position may be corrected based on its projected bright line pattern area.

Here, the receiver 200 can estimate its own position by decoding the bright line pattern area even if it cannot receive visible light signals. In other words, even if the receiver 200 cannot completely decode the bright line pattern area displayed in the image for decoding, the receiver 200 can decode the bright line pattern area based on the bright line pattern area or the striped area such as the bright line pattern area. You may perform self-position estimation based on.

FIG. 213 is a flowchart showing an example of self-position estimation by the receiver 200. FIG.

The receiver 200 acquires the map and the transmitter data of each of the plurality of transmitters 100 from the server or the recording medium of the receiver 200 (step S341). The transmitter data indicates the location of the transmitter 100 on the map and the shape and size of the transmitter 100 .

Next, the receiver 200 takes an image in the visible light communication mode (that is, short-time exposure), and detects a striped area (that is, the area A) from the image for decoding obtained by the imaging (step S342).

The receiver 200 then determines whether or not there is a possibility that the striped area is a visible light signal (step S343). That is, the receiver 200 determines whether or not the striped area is a bright line pattern area that appears due to the visible light signal. Here, when the receiver 200 determines that there is no possibility that the signal is a visible light signal (N in step S343), the process ends. On the other hand, when the receiver 200 determines that it may be a visible light signal (Y in step S343), it further determines whether or not the visible light signal could be received (step S344). That is, the receiver 200 decodes the bright line pattern area of the image for decoding, and determines whether or not the light ID has been acquired as a visible light signal by the decoding.

Here, when the receiver 200 determines that the visible light signal could be received (Y of step S344), it acquires the shape, size and position of the area A in the decoding image (step S347). That is, the receiver 200 obtains the shape, size and position of the transmitter 100 displayed as a striped image in the decoding image by imaging in the visible light communication mode.

Then, the receiver 200 calculates the relative position between the transmitter 100 and the receiver 200 based on the transmitter data of the transmitter 100 and the acquired shape, size and position of the area A, 200 current position (that is, current self-position) is updated (step S348). For example, the receiver 200 selects the transmitter data of the transmitter 100 corresponding to the received visible light signal from the transmitter data of each transmitter 100 acquired in step S341. That is, the receiver 200 selects the transmitter 100 corresponding to the visible light signal from among the plurality of transmitters 100 shown on the map as the transmitter 100 to be imaged displayed as the image of the area A. . Then, based on the shape, size, and position of transmitter 100 acquired in step S347 and the shape and size indicated in the transmitter data of transmitter 100 to be imaged, receiver 200 Calculate the position relative to transmitter 100 . After that, the receiver 200 updates its own position based on the relative position, the map acquired in step S341, and the position on the map indicated by the transmitter data of the transmitter 100 to be imaged.

On the other hand, when the receiver 200 determines in step S344 that the visible light signal cannot be received (N in step S344), the receiver 200 estimates which position or range on the map is captured by the camera of the receiver 200. (step S345). That is, the receiver 200 estimates the position or range captured on the map based on the current self position estimated at that time and the orientation or direction of the camera that is the imaging unit of the receiver 200. do. Then, the receiver 200 determines that the transmitter 100 with the highest possibility of being imaged among the plurality of transmitters 100 shown on the map is the transmitter 100 displayed as the image of the area A. regarded (step S346). That is, the receiver 200 selects the transmitter 100 with the highest probability of being imaged from among the plurality of transmitters 100 shown on the map as the transmitter 100 to be imaged. Note that the transmitter 100 most likely to be imaged is, for example, the transmitter 100 closest to the imaging position or range estimated in step S345.

FIG. 214 is a diagram for explaining visible light signals received by the receiver 200. FIG.

A bright line pattern area included in the image for decoding appears in two cases. In the first case, the receiver 200 directly images the transmitter 100, for example a lighting device placed on the ceiling, thereby revealing the bright line pattern area. In other words, in the first case, the light that causes bright line pattern areas to appear is direct light. In the second case, receiver 200 indirectly images transmitter 100 to reveal the bright line pattern area. That is, the receiver 200 captures an image of a reflective area such as a wall or floor that reflects the light from the transmitter 100 without capturing an image of the transmitter 100 such as a lighting device. As a result, a bright line pattern area appears in the decoding image. In other words, in the second case, the light that causes the bright line pattern area to appear is the reflected light.

Therefore, if the decoding image has a bright line pattern area, receiver 200 in the present embodiment determines whether the bright line pattern area appears in the first case or the second case. That is, the receiver 200 determines whether the bright line pattern area appears due to direct light from the transmitter 100 or reflected light from the transmitter 100 .

Then, when the receiver 200 determines that it is the first case, the receiver 200 uses the bright line pattern area in the decoding image as the transmitter 100 displayed in the decoding image, and Identify relative positions. That is, the receiver 200 performs triangulation or geometrical A survey method identifies the relative position of the receiver 200 .

On the other hand, when the receiver 200 determines that it is the second case, the receiver 200 uses the bright line pattern area in the decoding image as the reflection area displayed in the decoding image to Locate. That is, the receiver 200 captures the orientation and angle of view of the camera used for imaging, the shape, size and position of the bright line pattern area, the position and orientation of the floor or wall indicated by the map, and the shape of the transmitter 100. and magnitude are used to identify the relative position of the receiver 200 by triangulation or geometrical survey methods. At this time, the receiver 200 may use the center of the bright line pattern area as the position of the bright line pattern area.

FIG. 215 is a flowchart showing another example of self-position estimation by receiver 200. FIG.

The receiver 200 first receives a visible light signal by imaging in the visible light communication mode (step S351). Then, receiver 200 acquires the map and transmitter data of each of transmitters 100 from a server or a recording medium (that is, a database) of receiver 200 (step S352).

Next, the receiver 200 determines whether or not the visible light signal received in step S351 is received as reflected light (step S353).

Then, when the receiver 200 determines in step S353 that the reflected light has been received (Y in step S353), the receiver 200 places the central portion of the striped region in the decoding image obtained by the imaging in step S351 on the floor or wall. This is regarded as the position of the transmitter 100 being displayed (step S354).

Next, receiver 200 calculates the relative position between transmitter 100 and receiver 200, and updates the current position of receiver 200, as in step S348 of FIG. 213 (step S355). On the other hand, if the receiver 200 determines in step S353 that the reflected light is not received (N in step S353), it updates the current position of the receiver 200 without considering the reflection of the floor or wall.

FIG. 216 is a flowchart illustrating an example of determination of reflected light by the receiver 200. FIG.

The receiver 200 detects a striped area or bright line pattern area as an area A from the decoding image (step S641). Next, the receiver 200 identifies the orientation of the camera when the decoding image was captured by the acceleration sensor (step S642). Next, receiver 200 determines from the map data whether or not transmitter 100 exists in the direction of the camera specified in step S642 from the position of receiver 200 already estimated at that time on the map. Identify (step S643). That is, the receiver 200 transmits based on the estimated position of the receiver 200 on the map at that time, the orientation or direction of the imaging of the receiver 200, and the position of each transmitter 100 on the map. It is determined whether or not the aircraft 100 is directly imaged.

When the receiver 200 determines that the transmitter 100 exists (Yes in step S644), the receiver 200 determines that the light in the region A, that is, the light used for receiving the visible light signal is direct light (step S645). . On the other hand, when the receiver 200 determines that the transmitter 100 does not exist (No in step S644), the receiver 200 determines that the light in the area A, that is, the light used for receiving the visible light signal is the reflected light (step S646). ).

In this way, the receiver 200 uses the acceleration sensor to determine whether the light causing the appearance of the bright line pattern area is direct light or reflected light. Further, the receiver 200 may determine that the light is direct light when the camera is facing upward, and determine that the light is reflected light when the camera is facing downward.

Further, the receiver 200 determines whether the light is direct light or reflected light based on the intensity, position, or size of the light in the bright line pattern area included in the decoding image instead of the output of the acceleration sensor. good too. For example, if the light intensity is less than a predetermined intensity, the receiver 200 determines that the light that causes the bright line pattern area to appear is the reflected light. Alternatively, the receiver 200 determines that the light is reflected light if the position of the bright line pattern area is below the decoding image. Alternatively, if the size of the bright line pattern area is larger than a predetermined size, the receiver 200 determines that the light is reflected light.

217 is a flowchart showing an example of navigation by the receiver 200. FIG.

The receiver 200 is, for example, a smartphone equipped with an out-camera, an in-camera, and a display, and performs navigation by displaying an image on the display for guiding the user to a destination. That is, the receiver 200 executes AR navigation as in the example shown in FIGS. 18A to 18C of the second embodiment. At this time, the receiver 200 captures an image with the out-camera, and detects danger in the surroundings based on the image obtained by the capture. The receiver 200 then determines whether the user is in a dangerous situation (step S361).

Here, when receiver 200 determines that the user is in a dangerous situation (Y in step S361), receiver 200 displays a warning message on the display of receiver 200 or stops navigation (step S364). .

On the other hand, when the receiver 200 determines that the user is not in a dangerous situation (N of step S361), it determines whether walking smartphones are prohibited in the area where the receiver 200 is located (step S362). For example, the receiver 200 refers to the map data and determines whether or not the current position of the receiver 200 is included in the area where walking smartphones are prohibited indicated by the map data. Here, if the receiver 200 determines that walking smartphones are not prohibited (N in step S362), it continues navigation (step S366). On the other hand, when the receiver 200 determines that walking smartphone is prohibited (Y in step S362), it determines whether the user is looking at the receiver 200 by recognizing the line of sight of the user with the in-camera. (Step S363). Here, when the receiver 200 determines that the receiver 200 is not seen (N of step S363), it continues navigation (step S366). On the other hand, when receiver 200 determines that the user is looking at receiver 200 (Y in step S363), receiver 200 displays a warning message on the display of receiver 200 or stops navigation (step S364).

Then, receiver 200 determines whether the user has escaped from a dangerous situation, or whether the user has escaped from the state of gazing at receiver 200 (step S365). Here, when the receiver 200 determines that it has escaped from the situation or state (Y in step S365), it continues navigation (step S366). On the other hand, when the receiver 200 determines that the situation or state has not escaped (N of step S365), it repeats the process of step S364.

Also, the receiver 200 may detect the moving speed based on outputs from an acceleration sensor, a gyro sensor, or the like. In this case, the receiver 200 may determine whether or not the moving speed is equal to or above the threshold, and may stop navigation if it is above the threshold. At this time, receiver 200 may display a message to notify that walking at that speed is dangerous. As a result, it is possible to avoid the danger of using a smartphone while walking.

Here, transmitter 100 may be configured as a projector.

FIG. 218 is a diagram showing an example of transmitter 100 configured as a projector.

For example, transmitter 100 projects image 441 onto a floor or wall. Further, the transmitter 100 transmits the visible light signal by changing the brightness of the light used for the projection while projecting the image 441 . It should be noted that the projected image 441 may include text or the like prompting AR navigation. Receiver 200 receives the visible light signal by capturing an image 441 projected onto its floor or wall. The receiver 200 may then perform self-position estimation using the projected image 441 . For example, the receiver 200 acquires the position on the map of the image 441 associated with the visible light signal from the server, and uses the position of the image 441 to estimate its own position. Alternatively, the receiver 200 obtains from the server the position on the map of the transmitter 100 associated with the visible light signal, and converts the image 441 projected on the floor or wall to the second case described above. Self-position estimation may be performed by treating the light as reflected light.

FIG. 219 is a flowchart showing another example of self-position estimation by receiver 200. FIG.

The receiver 200 first captures the transmitter 100, a predetermined image, or a predetermined code (two-dimensional code, etc.) (step S371). Note that in imaging the transmitter 100 , the receiver 200 receives visible light signals from the transmitter 100 .

Next, receiver 200 acquires the position of the subject imaged in step S371 (that is, the position on the map). Then, receiver 200 estimates the position of receiver 200, i.e., its own position, based on the position, shape, and size of the subject and the position, shape, and size of the subject in the image obtained by imaging in step S371. (step S372).

Next, receiver 200 starts navigation to guide the user to a predetermined position indicated by the image obtained by imaging in step S371 (step S373). If the subject is the transmitter 100, the predetermined position is the position specified by the visible light signal. If the subject is a given image, the given position is the position obtained by analyzing the given image. If the object is a code, its predetermined position is the position obtained by decoding the code. During navigation, the receiver 200 repeats image pickup by the camera and sequentially displays the normal photographed images obtained by the photographing in real time, while displaying a direction instruction image such as an arrow indicating the user's destination on the normal photographed images. superimposed on The user, while carrying receiver 200, starts moving according to the displayed direction instruction image.

Next, the receiver 200 determines whether or not position information such as GPS (that is, GPS data) can be received (step S374). Here, when receiver 200 determines that reception is possible (Y in step S374), receiver 200 estimates the current self-position of receiver 200 using the position information such as GPS (step S375). On the other hand, when the receiver 200 determines that position information such as GPS cannot be received (N in step S374), the reception The self-position of the aircraft 200 is estimated (step S376). For example, receiver 200 detects the motion of an object or feature point appearing in each of the above-described normal shot images, and estimates the moving direction and moving distance of receiver 200 based on the motion. Receiver 200 then estimates the current self-position of receiver 200 based on the estimated moving direction and moving distance and the position estimated in step S372.

Next, receiver 200 determines whether or not the most recently estimated self-position is within a predetermined distance from a predetermined position, which is the destination (step S377). Here, when receiver 200 determines that the self-position is within that distance (Y in step S377), it determines that the user has arrived at the destination, and terminates the navigation process. On the other hand, if the receiver 200 determines that the self-position is not within the distance (N of step S377), it determines that the user has not arrived at the destination, and repeats the process from step S374.

Further, when the receiver 200 loses sight of the current self-position during navigation, that is, when the self-position cannot be estimated, the receiver 200 stops superimposing the direction instruction image on the normal captured image and The estimated self-position may be displayed on the map. Alternatively, receiver 200 may display a surrounding map including the last estimated self-position.

FIG. 220 is a flowchart showing an example of processing by the transmitter 100. FIG. In the example shown in FIG. 220, transmitter 100 is a lighting device installed in an elevator.

The transmitter 100 determines whether elevator operation information indicating the operation status of the elevator can be obtained from the elevator (step S381). The elevator operation information may indicate whether the elevator is ascending, descending, or stopped, the floor where the elevator is currently located, the floor where the elevator is scheduled to stop, and the like.

Here, when the transmitter 100 determines that the elevator operation information is obtained (Y in step S381), it transmits all or part of the elevator operation information by a visible light signal (step S386). Alternatively, the transmitter 100 may associate the visible light signal (that is, light ID) transmitted from the transmitter 100 with the elevator operation information and cause the server to hold the information.

On the other hand, when the transmitter 100 determines that the elevator operation information cannot be obtained (N in step S381), the acceleration sensor recognizes whether the elevator is stopped, ascending, or descending ( step S382). Furthermore, the transmitter 100 determines whether or not the floor on which the elevator is currently located can be identified from the floor display of the elevator (step S383). The floor display section corresponds to the floor number display section shown in FIG. 18C. Here, if it is determined that the floor has been specified (Y in step S383), the transmitter 100 executes the process of step S386 described above. On the other hand, if it is determined that the floor cannot be specified (N in step S383), the transmitter 100 further captures the image of the floor display section with the camera, and from the image obtained by the image capturing, the current position of the elevator is determined. It is determined whether or not the floor can be recognized (step S384).

If the transmitter 100 determines that the floor has been recognized (Y in step S384), the transmitter 100 executes the process of step S386 described above. On the other hand, when the transmitter 100 determines that the floor cannot be recognized (N of step S384), it transmits a predetermined visible light signal (step S385).

221 is a flowchart showing another example of navigation by the receiver 200. FIG. In the example shown in FIG. 221, transmitter 100 is a lighting device installed in an elevator.

Receiver 200 first determines whether or not the current position of receiver 200 is on an escalator (step S391). The escalator may be a slope escalator or a horizontal escalator.

Here, if it is determined that it is on the escalator (Y of step S391), the receiver 200 estimates the movement of the receiver 200 (step S392). This motion is the motion of the receiver 200 with reference to a fixed floor or wall other than the escalator. That is, the receiver 200 first acquires the direction and speed of movement of the escalator from the server. Then, the receiver 200 adds the movement of the receiver 200 on the escalator to the movement of the receiver 200 on the escalator recognized by inter-frame image processing such as SLAM (Simultaneous Localization and Mapping). Estimate motion.

On the other hand, if the receiver 200 determines that it is not on the escalator (N of step S391), it determines whether the current position of the receiver 200 is inside the elevator (step S393). Here, if the receiver 200 determines that it is not in the elevator (N of step S393), the process ends. On the other hand, when the receiver 200 determines that it is inside the elevator (Y in step S393), the receiver 200 detects the floor on which the elevator (specifically, the elevator car) is currently located by a visible light signal, radio signal, or other means. It is determined whether or not it can be specified (step S394).

Here, if it is determined that the floor cannot be specified (N of step S394), the receiver 200 displays the floor where the user plans to get off the elevator (step S395). Furthermore, the receiver 200 recognizes whether or not the receiver 200 has exited the elevator when the user gets out of the elevator, and determines the floor where the receiver 200 is currently located by a visible light signal, radio signal, or other means. recognize. Then, if the recognized floor is different from the planned floor, the receiver 200 notifies the user that the user got off at the wrong floor (step S396).

Further, when the receiver 200 determines in step S394 that the floor on which the elevator is currently located has been identified (Y in step S394), the receiver 200 moves to the floor where the user plans to get off the elevator, that is, the target floor. (step S397). Here, when it is determined that the receiver 200 is on the target floor (Y in step S397), the receiver 200 displays a message or the like prompting the user to get off (step S399). Alternatively, the receiver 200 displays an advertisement for the target floor. Also, the receiver 200 may display a warning message when the user does not want to get off.

On the other hand, when the receiver 200 determines that the receiver 200 is not on the target floor (N in step S397), it displays a message or the like warning the user not to get off (step S398). Alternatively, the receiver 200 may display advertisements. Also, the receiver 200 may display a warning message when the user is about to get off.

222 is a flowchart illustrating an example of processing by the receiver 200. FIG.

In the flowchart shown in FIG. 222, the receiver 200 uses both the visible light signal and the normal exposure image information (that is, the normal shot image).

For example, the receiver 200 configured as a wearable device such as a smart phone or smart glasses acquires the image A (that is, the image for decoding described above) by imaging with an exposure time shorter than the normal exposure time (step S631). The receiver 200 then receives the visible light signal by decoding the image A (step S632). Receiver 200 identifies the current position of receiver 200 as an example based on the received visible light signal, and starts navigation to a predetermined position.

Receiver 200 acquires image B (that is, the above-described normal shot image) by imaging with an exposure time longer than the above-described short exposure time (for example, exposure time based on automatic exposure setting) (step S633). Image A is not suitable for object detection or feature quantity extraction. Therefore, the receiver 200 alternately repeats acquisition of the image A by imaging with the above-described short exposure time and acquisition of the image B by imaging with the above-described long exposure time a predetermined number of times. Accordingly, the receiver 200 performs image processing such as the above-described object detection or feature amount extraction using the obtained plurality of images B (step S634). For example, receiver 200 detects a specific object from image B to correct the position of receiver 200 . Also, for example, the receiver 200 extracts feature points from each of two or more images B, and identifies how the same feature point has moved between the images. As a result, the receiver 200 can recognize the distance and direction of movement of the receiver 200 between the time points of each of the two or more images B and correct the current position of the receiver 200. .

FIG. 223 is a diagram showing an example of a screen displayed on the display of receiver 200. As shown in FIG.

When the navigation application is activated, the receiver 200 displays the logo of the transmitter 100 as shown in FIG. 223, for example. The logo mark is, for example, a logo mark described as "AR Navi". Then, receiver 200 may guide the user to take an image of the logo mark. The transmitter 100 is configured as a digital signage, for example, and displays the logo while changing the brightness in order to transmit the visible light signal. Alternatively, the transmitter 100 is configured as a projector, for example, and projects the above-described logo mark onto the floor or wall while changing the brightness in order to transmit visible light. The receiver 200 receives the visible light signal from the transmitter 100 by imaging the logo mark in the visible light communication mode. Note that the receiver 200 may display a picture of a nearby lighting device or landmark configured as the transmitter 100 instead of the logo mark.

Also, the receiver 200 may display the telephone number of a call center for when the user is in trouble. At this time, the receiver 200 may notify the call center server of the language used by the user and the estimated self-location. The language used by the user may be registered in the receiver 200 in advance, or may be set by the user's operation. As a result, the call center can respond quickly to the user of the receiver 200 when the user calls. For example, a call center can direct a user to a destination over the phone.

Further, the receiver 200 may correct the self-position based on the form of a pre-registered landmark, the size of the landmark, and the position of the landmark on the map. That is, when the normal photographed image is acquired, the receiver 200 detects an image area in which the landmark is displayed from the normal photographed image. Receiver 200 then estimates its own position based on the shape, size, and position of the image area in the normal captured image, and the size and position of the landmark on the map.

The receiver 200 may also recognize or detect landmarks on the ceiling or behind using the in-camera. Also, the receiver 200 may use only a region of an image captured by a camera that is at a predetermined angle or more (or a predetermined angle or less) with respect to the horizontal direction, for example. For example, if the transmitter 100 or landmarks are often placed on the ceiling side, the receiver 200 uses only the area in which the subject above the horizontal direction is captured in the image of the camera. The receiver 200 detects a bright line pattern area or a landmark image area only from that area. Thereby, the processing amount of the receiver 200 can be reduced.

Further, as shown in the example of FIG. 212, the receiver 200 superimposes the direction instruction image 431 on the normal shot image when performing AR navigation, but may also superimpose a character.

FIG. 224 is a diagram showing a character display example by the receiver 200. FIG.

When the receiver 200 receives a visible light signal from the transmitter 100 configured as digital signage, for example, it acquires the character 432 corresponding to the visible light signal as the above-described AR image from a server, for example. Then, as shown in FIG. 224, the receiver 200 displays not only the direction instruction image 431 but also its character 432 superimposed on the normal photographed image. For example, the character 432 is an advertisement character for a drinking water manufacturing and sales company, and is displayed as an image of a can containing the drinking water. Also, the character 432 is a character for advertisement of drinking water sold on the route to the user's destination. Such a character 432 may be displayed to guide the user in the direction or position indicated by the directional image 431 . Advertising with such characters may be realized by affiliates.

A character may also be in the form of an animal or a person. In this case, the receiver 200 may superimpose a character that walks on the direction indicating image on the normal captured image. Also, a plurality of characters may be superimposed on the normal shot image. Furthermore, the receiver 200 may superimpose a moving image for advertisement as a commercial on the normal shot image instead of the character or together with the character.

Also, the receiver 200 may change the size and display time of the advertisement character according to the advertisement fee paid for the company's advertisement. Also, when a plurality of characters for advertisement are displayed, the receiver 200 may determine the display order of the characters in the depth direction according to the advertisement fee paid for each character. In addition, the receiver 200 may perform electronic payment when entering a store where merchandise of the displayed character is sold.

Further, when the receiver 200 receives another visible light signal from another digital signage while displaying the character 432, the receiver 200 displays the displayed character 432 according to the other visible light signal. You may change to a different character.

The receiver 200 may superimpose a moving image serving as a commercial for the company on the normally captured image. The advertiser may be billed according to the display time and number of times the moving image of the commercial or the advertisement is displayed. The receiver 200 may display the language of the user as the language of the commercial, and the text or voice link for the user to convey the purchase intention of the commercial product to the store staff is displayed in the language of the store staff. may be displayed. Also, the receiver 200 may display the price of the product in the user's currency.

225 is a diagram showing another example of a screen displayed on the display of receiver 200. FIG.

Receiver 200 displays a message in English, which is the language used by the user, for notifying that there is a bookstore with the store name "XYZ" in front, as shown in (a) of FIG. 225, for example. Such messages may be displayed as animated commercials as described above. When the receiver 200 enters the bookstore, for example, as shown in (b) of FIG. You may display with English which is a language.

Also, the receiver 200 may prompt the user to take a detour during navigation. In this case, the receiver 200 may suggest a detour according to the available time. In the example of FIG. 212, the margin time is the difference between the time when the bus departs from the bus stop and the time when the user arrives at the bus stop.

The receiver 200 may also display promotions for nearby stores. In this case, the receiver 200 may display an advertisement for the store next door or along the road. Further, the receiver 200 may calculate the timing of starting the reproduction of the moving image so that the receiver 200 is located next to the store corresponding to the commercial when the reproduction of the moving image of the commercial ends. The receiver 200 may also stop displaying advertisements for shops that you pass by.

Furthermore, when the user makes a detour to a store or the like, the store is equipped with a lighting device or the like serving as the transmitter 100 for obtaining the starting point so that the receiver 200 can return to guidance to the original destination. good too. Alternatively, the receiver 200 may display a button labeled "Restart in front of XYZ store". Also, the receiver 200 may apply a discount price only to those who see the commercial and visit the store, or may display a coupon. In addition, the receiver 200 may display a bar code using an application and perform electronic payment in order to pay for product purchases.

Also, the server may perform flow line analysis based on the results of navigation by the receiver 200 of each user.

Further, when the camera is not used for navigation, the receiver 200 may switch the self-position estimation method to PDR (Pedestrian Dead Reckoning) using an acceleration sensor or the like. For example, when the navigation application is turned off, or when the receiver 200 is in the user's pocket or the like and the image obtained by the camera is completely dark, the method of self-localization is switched to PDR. The receiver 200 may also use radio waves (Bluetooth or Wi-Fi) or sound waves for self-position estimation.

Also, the receiver 200 may notify the user by vibration or sound when the user tries to go in the wrong direction. For example, the receiver 200 may make different types of vibrations or sounds depending on whether the user tries to go in the right direction or the wrong direction at an intersection. It should be noted that the receiver 200 may provide the above-described vibration or sound notification when it is turned in the wrong direction or when it is turned in the right direction without moving. As a result, it is possible to improve usability even for the visually impaired. The correct direction is the direction toward the destination along the searched route, and the wrong direction is any direction other than the correct direction.

Also, the receiver 200 is configured as a smart phone in the above example, but may be configured as a smart watch or smart glasses. If the receiver 200 is configured as smart glasses, it is possible to prevent the camera-based navigation of the receiver 200 from being interrupted by applications unrelated to the navigation.

Also, the receiver 200 may terminate the navigation after a certain period of time has passed since the navigation was started. The certain period of time may be changed according to the distance to the destination. Alternatively, receiver 200 may terminate navigation upon entering a location where GPS data is available. Alternatively, the receiver 200 may end navigation when the GPS location is a certain distance away. The receiver 200 may also display the estimated time of arrival or the remaining distance to arrival. Furthermore, in the example of FIG. 212, the receiver 200 may display the time at which the bus departs from the destination bus stop.

Also, the receiver 200 may alert the user at a position such as a stairway or an intersection, and may guide the user to the elevator instead of the stairs according to the user's wishes or health condition. For example, the receiver 200 may guide the user to the elevator while avoiding stairs if the user is elderly (e.g., in his 80s). Also, when the receiver 200 determines that the user has a large piece of luggage, the receiver 200 may guide the user to the elevator while avoiding the stairs. For example, the receiver 200 may determine whether the walking speed of the user is slower than the normal walking speed based on the output from the acceleration sensor, and if it is slow, determine that the user is carrying a large luggage. . Alternatively, the receiver 200 may determine whether or not the user's stride is shorter than the usual stride based on the output from the acceleration sensor, and if it is shorter, determine that the user is carrying a large load. Furthermore, the receiver 200 may guide the user to a safe course if the user is female. A safe course is indicated in the map data.

Receiver 200 may also recognize obstacles such as people or vehicles around receiver 200 based on images captured by the camera. Then, if the user is likely to collide with the obstacle, receiver 200 may prompt the user to avoid the obstacle. For example, receiver 200 may prompt the user to stop or avoid obstacles by playing a sound.

Also, when performing navigation, the receiver 200 may correct the estimated arrival time based on the past travel times of other users. At this time, the receiver 200 may be modified based on the user's age and gender. For example, the receiver 200 adjusts the expected arrival time forward if the user is in their 20s, and adjusts the expected arrival time backward if the user is in their 80s.

Moreover, even when receiver 200 captures an image of digital signage that is the same transmitter 100 at the start of navigation, the destination may differ depending on the user. For example, the receiver 200 may change the location of the toilet, which is the destination, according to the gender of the user, and may change the immigration or return counter, which is the destination, according to the nationality of the user. Alternatively, the receiver 200 may change the platform of the destination train or plane according to the user's ticket. Also, the receiver 200 may change the seat of the destination show according to the user's ticket. Also, the receiver 200 may change the destination prayer space according to the sect of the user.

Also, when starting navigation, the receiver 200 may display a dialog such as "Would you like to start guidance to XYZ? Yes/No" without starting navigation right away. Also, the receiver 200 may inquire of the user where the guidance destination (appearance gate, lounge, store, etc.) is.

Also, the receiver 200 may suppress notifications or incoming calls from other applications during navigation. This makes it possible to prevent navigation from being interrupted.

Also, the receiver 200 may guide the user to a meeting place as a destination.

FIG. 226 is a diagram showing a system configuration for navigating to a meeting place.

For example, a user having a receiver 200a and a user having a receiver 200b gather at a meeting place. Each of receiver 200a and receiver 200b has the function of receiver 200 described above.

When a meeting is performed as described above, the receiver 200a, as shown in FIG. ) to the server 300. The number may be a telephone number or any identification number as long as it can identify the receiver. Information other than numbers may also be used.

Upon receiving the above information from receiver 200a, server 300 transmits the position and number of receiver 200a to receiver 200b, as shown in FIG. 226(b). Then, the server 300 inquires of the receiver 200b whether or not to accept the meeting invitation from the receiver 200a. Here, the user of receiver 200b accepts the invitation by operating receiver 200b. That is, the receiver 200b notifies the server 300 of acceptance of the meeting, as shown in FIG. 226(c). Receiver 200b then notifies server 300 of the position of receiver 200b obtained by self-position estimation, as shown in (d) of FIG.

As a result, server 300 identifies the respective positions of receiver 200a and receiver 200b. Then, server 300 sets an intermediate point between each position as a meeting place (that is, a destination), and notifies receiver 200a and receiver 200b of the route to the meeting place. As a result, the receiver 200a and the receiver 200b each perform AR navigation to the meeting place. In the above example, the midpoint between the positions of receiver 200a and receiver 200b is set as the destination, but another location may be set as the destination. For example, among a plurality of places where landmarks are installed, the place with the shortest travel time may be set as the destination. Note that the travel time is the estimated time until the receiver 200a and the receiver 200b reach the location.

As a result, the meeting can be smoothly performed.

Here, when the receiver 200a arrives near the destination, the receiver 200a may superimpose an image for identifying the user who has the receiver 200b on the normal captured image.

FIG. 227 is a diagram showing an example of a screen displayed on the display of receiver 200a.

For example, the server 300 periodically transmits the position of the receiver 200b to the receiver 200a. The position of this receiver 200b is a position obtained by self-position estimation by the receiver 200b. Therefore, the receiver 200a knows the position of the receiver 200b on the map. Therefore, when the position of receiver 200b is displayed in the normal captured image, receiver 200a may superimpose an arrow 433 pointing to the position on the normal captured image, as shown in FIG. Note that the receiver 200b may also superimpose an arrow pointing to the position of the receiver 200a on the normal captured image, similarly to the receiver 200a.

As a result, even when there are many people at the meeting place, it is possible to find the meeting partner easily.

Note that in the above example, the indicators such as the arrow 433 are used for meeting, but may be used for purposes other than meeting. If the user of the receiver 200b is in trouble with something such as where to go regardless of the meeting, the user may notify the server 300 by operating the receiver 200b. In this case, the server 300 may display the image shown in the example of FIG. 227 on the display of the receiver 200a owned by the call center staff. At this time, the server 300 may display a question mark instead of the arrow 433 . This allows the call center staff to easily confirm that the user of the receiver 200b is in trouble.

Also, the receiver 200 may guide the inside of a concert hall.

FIG. 228 is a diagram showing the inside of a concert hall.

The receiver 200 may acquire, for example, a map of the inside of the concert hall shown in FIG. 228 and a route 434 from the entrance of the concert hall to the seats from the server. For example, the receiver 200 estimates its own position by receiving visible light signals from the transmitter 100 located at its entrance and guides the user along its path 434 to his seat. Here, if there are stairs inside the concert hall, the receiver 200 identifies how many stairs the user has climbed or descended based on the output of an acceleration sensor or the like. The self-position may be updated based on the number of stages.

Further, in the above example, the receiver 200 performed self-position estimation based on the movement of feature points when no visible light signal was received, but it is possible to detect feature points from a normal captured image. If it becomes impossible, the output from the acceleration sensor may be used. Specifically, the receiver 200 estimates the movement distance as described above when the feature points can be detected from the normal captured image, and combines the movement distance with the output data from the acceleration sensor during movement. to learn the relationship between Machine learning such as DNN (Deep Neural Network) may be used for this learning. Then, when the detection of the feature point becomes impossible, the receiver 200 derives the movement distance using the learning result and the output data from the acceleration sensor during movement. Alternatively, when the feature point detection becomes impossible, the receiver 200 assumes that the receiver 200 is moving at the same speed as the previous moving speed, and derives the moving distance based on the assumption. good too.

[First aspect]
In the communication method, it is determined whether or not the tilt of the terminal is greater than a predetermined angle with respect to a plane parallel to the ground, it is determined that the angle is less than the predetermined angle, and an object whose brightness changes is captured by the out-camera. In this case, the exposure time of the image sensor of the out-camera is set to a first exposure time, and the image for decoding is acquired by capturing the subject with the first exposure time using the image sensor, and the When the first signal transmitted by the subject can be decoded from the image for decoding, the first signal is decoded from the image for decoding, the position specified by the first signal is acquired, and the decoding is performed. If the signal transmitted by the subject cannot be decoded from the image for use, map information stored in the terminal, including the positions of a plurality of transmitters, and the position of the terminal are used to determine the position of the terminal. Identify locations relative to transmitters within range of .

FIG. 229 is a flow chart showing an example of the communication method according to the first aspect of the present invention.

First, the terminal, which is receiver 200, determines whether or not the tilt of the terminal is greater than a predetermined angle with respect to a plane parallel to the ground (step SG21). Note that the plane parallel to the ground may be, for example, a horizontal plane. Specifically, the terminal determines whether or not the tilt is large by detecting the tilt of the terminal using the output data from the acceleration sensor. This tilt is the front or back tilt of the terminal.

Here, when the terminal determines that the tilt is greater than the predetermined angle and images the subject whose luminance changes with the out-camera (Yes in step SG21), the exposure time of the image sensor of the out-camera is set to the first (step SG22). Then, the terminal obtains a decoding image by imaging the subject with the first exposure time using the image sensor (step SG23).

The image for decoding obtained here is an image obtained when the inclination of the terminal is greater than a predetermined angle with respect to a plane parallel to the ground, so an image obtained by capturing an object on the ground side. isn't it. Therefore, in capturing the image for decoding, the transmitter 100 capable of transmitting visible light signals, such as a lighting device arranged on the ceiling side or a digital signage installed on the wall, is captured as a subject. likely to be. In other words, there is a low possibility that the reflected light from the transmitter 100 is captured as the subject when capturing the decoding image. Therefore, it is possible to appropriately acquire a decoding image in which there is a high possibility that the transmitter 100 is captured as an object. That is, as shown in step S353 in FIGS. 214 and 215, it is possible to appropriately determine whether the reflected light from the floor or the wall is being imaged, or the direct light from the transmitter 100 is being imaged. can.

Next, the terminal determines whether or not the first signal transmitted by the subject can be decoded from the decoding image (step SG24). Here, if the first signal can be decoded (Yes in step SG24), the terminal decodes the first signal from the decoding image (step SG25) and acquires the position specified by the first signal. (step SG26). On the other hand, if the signal transmitted by the subject cannot be decoded from the decoding image (No in step SG24), the terminal stores map information including the positions of a plurality of transmitters and the position of the terminal. is used to specify the position associated with the transmitter within a predetermined range from the position of the terminal (step SG27).

As a result, as shown in steps S344 to S348 in FIG. 213, even if the first signal, which is, for example, a visible light signal, can or cannot be received, the position of the transmitter, which is the subject, is determined. can be specified. As a result, the current self-position of the terminal can be properly estimated.

[Second aspect]
In the communication method of the second aspect, in the first communication method, the first exposure time is set such that bright lines corresponding to a plurality of exposure lines included in the image sensor appear in the decoding image. be.

Thereby, the first signal can be appropriately decoded from the decoding image.

[Third aspect]
In a communication method according to a third aspect, in the communication method according to the first aspect, the subject is reflected light that is reflected on a floor surface from a first transmitter that transmits a signal according to changes in brightness.

Thereby, even if the image for decoding is obtained by imaging the reflected light and the first signal cannot be decoded from the image for decoding, the position of the first transmitter can be specified.

[Fourth aspect]
A communication method according to a fourth aspect is the communication method according to the first aspect, wherein the exposure time of the image sensor of the out-camera is set to a second exposure time longer than the first exposure time; A plurality of normal images are acquired by imaging with an exposure time of 2, a plurality of spatial feature amounts are calculated from the plurality of normal images, and the position of the terminal is calculated using the plurality of spatial feature amounts. .

Note that the normal image is the above-described normal captured image.

As a result, as shown in (c) and (d) of FIG. 212, even if the terminal is in an underground mall where GPS data does not reach, the current self-position of the terminal can be appropriately determined from a plurality of normal images. can be estimated. Note that the spatial feature amount may be a feature point.

[Fifth aspect]
In the communication method according to the fifth aspect, in the communication method according to the fourth aspect, an image for decoding is acquired by imaging the second transmitter with the first exposure time, and the image for decoding is obtained from the image for decoding. Decoding a second signal transmitted by a second transmitter, obtaining a position specified by the second signal, setting the position specified by the second signal as a movement start position in the map information, The position of the terminal is specified by calculating the amount of movement of the terminal using the plurality of spatial feature amounts.

As a result, as shown in (a) of FIG. 212, the position of the terminal is specified based on the amount of movement from the starting point, which is the movement start position, so that more accurate self-position estimation can be performed.

INDUSTRIAL APPLICABILITY The communication method of the present invention has the effect of enabling communication between various devices, and can be used, for example, in display devices such as smartphones, glasses, and tablets.

G20 communication device G21 determination unit G22 first acquisition unit G23 second acquisition unit

Claims (18)

A communication method using a terminal equipped with an image sensor,
determining whether the terminal is capable of performing visible light communication;
When the terminal determines that it is possible to perform visible light communication, the image sensor acquires an image for decoding by capturing an image of a subject whose luminance changes, and from the striped pattern appearing in the image for decoding, the Acquiring the first identification information transmitted by the subject,
In the determination of the visible light communication, when the terminal determines that the visible light communication is not possible, the image sensor acquires a captured image by capturing an image of the subject, and edge detection of the captured image. extracts at least one contour, identifies a predetermined specific region from the at least one contour, and obtains second identification information transmitted by the subject from the line pattern of the specific region ,
Communication method. In identifying the specific region,
Identifying a region having a rectangular contour of a predetermined size or more, or a region having a rounded rectangular contour of a predetermined size or more as the specific region;
The communication method according to claim 1. In the judgment of visible light communication,
determining that visible light communication can be performed when the terminal is identified as a terminal capable of changing the exposure time to a predetermined value or less;
determining that visible light communication is not possible when the terminal is identified as a terminal that cannot change the exposure time to the predetermined value or less;
The communication method according to claim 1. In the judgment of the visible light communication, when the terminal judges that the visible light communication is possible, when the subject is imaged, the exposure time of the image sensor is set to the first exposure time, and the exposure time of the image sensor is set to the first exposure time. Acquiring the image for decoding by imaging the subject with an exposure time of 1;
In the determination of the visible light communication, if the terminal determines that it is not possible to perform visible light communication, when imaging the subject, the exposure time of the image sensor is set to a second exposure time, Acquiring the captured image by capturing the subject with a second exposure time;
wherein the first exposure time is shorter than the second exposure time;
The communication method according to any one of claims 1 to 3. The subject has a rectangular shape as seen from the image sensor, and the first identification information is transmitted by changing the brightness of the central area of the subject, and a barcode-like line pattern is arranged around the periphery of the subject. has been
In the judgment of the visible light communication, when the terminal judges that it is possible to perform visible light communication, when the subject is imaged, a plurality of bright lines corresponding to the plurality of exposure lines of the image sensor are used. obtaining the decoding image including a bright line pattern, and obtaining the first identification information by decoding the bright line pattern;
In the judgment of the visible light communication, when the terminal judges that the visible light communication is not possible, when the subject is imaged, the second identification information is obtained from the line pattern of the imaged image. ,
The communication method according to claim 4. The first identification information obtained from the decoding image and the second identification information obtained from the line pattern are the same information.
The communication method according to claim 5. displaying a first moving image associated with the first identification information when determining that the terminal is capable of performing visible light communication in the determination of the visible light communication;
displaying a second moving image associated with the first identification information next to the first moving image when an operation of sliding the first moving image is received;
The communication method according to claim 1. In the display of the second moving image,
receiving an operation to slide the first moving image in the horizontal direction, displaying the second moving image;
8. The communication method according to claim 7, wherein a still image associated with the first identification information is displayed when an operation of sliding the first moving image in the vertical direction is received. In each of the first moving image and the second moving image, an object in the first displayed picture is at the same position.
The communication method according to claim 8. When the first identification information is acquired again by imaging with the image sensor, the next moving image associated with the first identification information is displayed next to the displayed moving image.
The communication method according to claim 7 or 8. In each of the moving image being displayed and the next moving image, the object in the first displayed picture is at the same position.
The communication method according to claim 10. At least one of the first moving image and the second moving image is formed such that the closer a position within the moving image is to an end of the moving image, the higher is the transparency at that position. ing,
The communication method according to claim 11. displaying an image outside an area in which at least one of the first moving image and the second moving image is displayed;
The communication method according to claim 12. Acquiring a normal shot image by imaging with the first exposure time by the image sensor,
The decoding image including a bright line pattern region, which is a region composed of a plurality of bright line patterns, is acquired by imaging with a second exposure time shorter than the first exposure time, and the decoding of the decoding image is performed to obtain the decoding image. Acquire the identification information of 1,
In the display of at least one of the first moving image and the second moving image, a reference area located at the same position as the bright line pattern area in the decoding image is specified from the normal captured image. death,
recognizing an area where the moving image is superimposed in the normal captured image as a target area based on the reference area, and superimposing the moving image on the target area;
The communication method according to claim 7. In displaying at least one of the first moving image and the second moving image, an area above, below, left, or right of the reference area in the normal image is recognized as the target area. do,
15. A communication method according to claim 14. In displaying at least one of the first moving image and the second moving image, the size of the moving image is increased as the size of the bright line pattern area increases.
15. A communication method according to claim 14. A communication device using a terminal equipped with an image sensor,
a determination unit that determines whether the terminal is capable of performing visible light communication;
When the determination unit determines that the terminal is capable of performing visible light communication, the image sensor acquires an image for decoding by imaging an object whose luminance changes, and converts the image for decoding into the image for decoding. a first acquisition unit that acquires first identification information transmitted by the subject from the appearing striped pattern;
When the determination unit determines that the terminal is not capable of performing visible light communication, the image sensor acquires a captured image by capturing an image of the subject, and edge detection is performed on the captured image. extracting at least one contour, identifying a predetermined specific region from the at least one contour, and acquiring second identification information transmitted by the subject from the line pattern of the specific region; and an obtaining unit for
Communication device. A program for executing a communication method using a terminal equipped with an image sensor,
determining whether the terminal is capable of performing visible light communication;
When the terminal determines that it is possible to perform visible light communication, an image for decoding is obtained by capturing an image of an object whose brightness changes with the image sensor, and the striped pattern appearing in the image for decoding is detected from the object. obtain the first identification information sent by
In the determination of the visible light communication, when the terminal determines that the visible light communication is not possible, the image sensor acquires a captured image by capturing an image of the subject, and edge detection of the captured image. extracts at least one contour, identifies a predetermined specific area from the at least one contour, and obtains second identification information transmitted by the subject from the line pattern of the specific area ,
A program that makes a computer do something.

Images