JP2016045814A - Virtual reality service providing system and virtual reality service providing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual reality service providing system capable of presenting the world of virtual reality fused with a real world.SOLUTION: A virtual reality service providing system includes: object storage means for storing a plurality of virtual reality objects; available service information storage means for storing information indicating a virtual reality service available for a user; object reading means for reading a virtual reality object to be used for the virtual reality service available for the user from the object storage means; object arrangement position determination means for determining the arrangement position of the virtual reality object in a virtual space on the basis of surrounding situation condition information indicating the surrounding situations of the user; and display video generation means for generating the coded data of a video to be displayed on the display panel of an HMD(Human Mounted Display) device mounted on the head of the user on the basis of user condition information indicating the condition of the user and the arrangement position of the virtual reality object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a virtual reality service providing system, a virtual reality service providing method, a virtual reality service providing server, an information processing system, an information processing method, an information processing server, an information processing program, a virtual reality presenting device, a head mounted display device, a virtual reality The present invention relates to a presentation system and a virtual reality presentation method.

  In recent years, various electronic devices have been developed as devices for displaying images in addition to conventional stationary displays such as televisions, and are referred to as mobile phone terminals, tablet terminals, and head mounted displays (HMDs). Electronic devices have already been developed.

  Prior art document 1 discloses a virtual reality presentation device for displaying CG (Computer Graphics) on a real landscape in an HMD.

JP 2006-338163 A

  The real world where we live is subject to physical and social constraints. For example, it is difficult to realize interplanetary travel due to the physical restriction that a substance with mass cannot exceed the speed of light, and the physical body said that the law of conservation of energy governs the sky in the living body freely. I can't fly around.

  Even if it is a physically possible event, human behavior is socially restricted by ethical norms and various laws set to give peace and order to real human society. . Thus, while human beings are in a position to enjoy freedom, in the real world they only enjoy conditional freedom subject to various restrictions.

  The real world is subject to various restrictions, but the world imagined by human beings is not bound by these restrictions and has an infinite degree of freedom. In the imaginary world, it is possible to exceed the speed of light or fly around the sky. Among creators scattered all over the world, some create fictional worlds in the form of movies, and some build fictional worlds in the form of novels, comics, animations, and games. By watching movies and reading novels, general consumers reconstruct the fictional world created by creators in their brains and experience the world in a simulated manner.

  These content created by creators, such as movies and novels, can be viewed as a supplementary material that allows general consumers to build a world full of imagination in their own brains. Differences in content such as novels, movies, anime, manga, and games are nothing but differences in the form of expression that users use to build the fictional world represented by each content in their brains. For example, a movie is a form that directly expresses the world imagined by the creator scenario writer, whereas a novel is a degree of freedom in which consumers can complement the world imagined by the creator novelist with their own imagination. It can be said that it is the expression form that gave.

  Each consumer links to this auxiliary material through his own visual and auditory senses and reconstructs the fictional world created by the creator in his brain. Thus, for example, even after reading a comic, a consumer can imagine in the brain the world represented by that comic, and only the creator can see the same scale of the same world as the fictional world built from nothing. , Consumers can also build in the brain.

  Here, the consumer recognizes that the real world and the imaginary world that he creates in his brain through the content are two different worlds that are clearly separated. This will not change no matter how realistic the world created by existing content is.

  On the other hand, we often experience the experience of identifying the real world and the imaginary world in daily life. That is a dream. While we dream, we are not aware that it is an imaginary world that is different from the real world. The dream world is sometimes wonderful and we humans often experience that when we wake up from a dream, we feel sorry that it was a dream, not a reality.

  As expected, in order to have a dreamy time, the user needs to be placed in a state where the boundary between the real world and the imaginary world cannot be clearly recognized. The imaginary world that is fused with the real world is more perceived by humans and is recognized as more wonderful than the imaginary world that is separated from the real world. We think that the final form of expression of existing content will be aggregated into a form that expresses the fictional world created by the creator in a state where it is fused with the real world of the consumer.

  Regardless of the form of expression, such as painting, sculpture, movie, animation, manga, game, novel, etc., by linking with the world created by creators full of creativity, consumers with poor creativity can share the same world in their brains. If the main purpose of consumption for art is to make it possible to construct a product, the world created by the creator can be appropriately presented to consumers when it takes a form that is integrated with the real world. It is necessary to build a simple system. This is close to saying that if it is a movie, the sound environment of the movie theater will be prepared, and if it is a game, the platform will be prepared. In the prior art, sufficient examination for such a viewpoint has not been made, and only a limited virtual reality has been presented.

  In view of the above problems, the present invention provides a virtual reality service providing system and a virtual reality service providing method capable of presenting an imaginary world created by creative creators with a real world where a user lives. It is a thing.

  A virtual reality service providing system according to an aspect of the present invention includes a virtual reality object storage unit that stores a plurality of virtual reality objects, and an available virtual reality service information storage that stores information indicating virtual reality services that can be used by a user. A virtual reality object reading means for reading out a virtual reality object used for a virtual reality service available to the user from the virtual reality object storage means, and a surrounding situation information indicating a surrounding situation of the user. Virtual reality object arrangement position determining means for determining an arrangement position of the virtual reality object, a head-mounted display that the user wears on the head based on user state information indicating a user state and the arrangement position of the virtual reality object Displayed on the display panel of the device. It comprises a display image generating means for generating encoded data of the video, the.

  A virtual reality service providing method according to an aspect of the present invention includes a virtual reality object storage database that stores a plurality of virtual reality objects, and an available virtual reality service information storage that stores information indicating virtual reality services that can be used by a user. A virtual reality service providing method used in a virtual reality service providing system comprising a database. The virtual reality service providing method includes: a virtual reality object used for a virtual reality service used by the user based on information indicating the virtual reality service available to the user stored in the available virtual reality service information storage database. A virtual reality object reading position for reading out the virtual reality object from the virtual reality object storage database, and a virtual reality object placement position for determining a placement position of the read virtual reality object in the virtual space based on ambient situation information indicating a situation around the user Based on the determination step, the user state information indicating the user state, and the arrangement position of the virtual reality object, the encoded data of the video displayed on the display panel of the head mounted display device worn by the user on the head is generated. Display image It has formed a step, a.

  ADVANTAGE OF THE INVENTION According to this invention, the imaginary world which a creator creates can be appropriately shown in the state united with the real world where a user lives.

It is a block diagram which shows the function structure of the virtual reality service provision system which concerns on Embodiment 1. FIG. It is a figure which respectively shows the scenery of the real world around the user wearing the virtual reality presentation device, the video displayed on the display unit, and the virtual reality world sensed by the user. It is a figure which shows an example of usable virtual reality service information. It is an example of the management file which manages the virtual reality object memorize | stored in virtual reality object storage DB. It is a figure explaining an example of the model video data of a virtual reality object. It is a block diagram which shows the function structure of the virtual reality service provision system which concerns on Embodiment 2. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality service provision system which concerns on Embodiment 2. FIG. It is a block diagram which shows the function structure of the virtual reality service provision system which concerns on Embodiment 3. FIG. It is an external appearance perspective view of the HMD apparatus which concerns on Embodiment 3 from diagonally forward and diagonally backward. It is sectional drawing of the HMD apparatus which concerns on Embodiment 3. FIG. It is an example of contractor information. It is a figure explaining the structure of virtual reality content. It is a figure explaining the classification | category of a virtual reality control program. It is an example of a save data management file. FIG. 10 is a flowchart showing a flow of virtual reality service start processing in the virtual reality service providing system according to the third embodiment. FIG. 10 is a flowchart illustrating a flow of virtual space control processing in the virtual reality service providing system according to the third embodiment. FIG. 10 is a flowchart illustrating a flow of virtual reality object video generation processing in the virtual reality service providing system according to the third embodiment. FIG. 10 is a flowchart illustrating a flow of a virtual reality object sound generation process in the virtual reality service providing system according to the third embodiment. FIG. 10 is a flowchart illustrating a flow of virtual reality service interruption / resumption processing in the virtual reality service providing system according to the third embodiment. FIG. 10 is a flowchart showing a flow of a virtual reality service end process in the virtual reality service providing system according to the third embodiment. It is a figure which shows an example of the data structure of the various requests | requirements (packets) issued to the virtual reality service provision server from the HMD apparatus which concerns on Embodiment 3. It is a figure which shows an example of the data structure of the various notification information (packet) output to the HMD apparatus from the virtual reality service provision server which concerns on Embodiment 3. FIG. It is sectional drawing of the modification of the HMD apparatus which concerns on Embodiment 3. FIG. It is an external appearance perspective view from the diagonally forward and diagonally backward of the modification of the HMD apparatus which concerns on Embodiment 3. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality service provision system which concerns on Embodiment 3. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality service provision system which concerns on Embodiment 3. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality service provision system which concerns on Embodiment 3. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality service provision system which concerns on Embodiment 3. FIG. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 4. FIG. 10 is a flowchart illustrating an operation flow of the virtual reality presentation system according to the fourth embodiment. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 5. FIG. FIG. 10 is a flowchart illustrating an operation flow of the virtual reality presentation system according to the fifth embodiment. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 6. FIG. FIG. 10 is a flowchart illustrating an operation flow of the virtual reality presentation system according to the sixth embodiment. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 7. FIG. FIG. 10 is a flowchart illustrating an operation flow of the virtual reality presentation system according to the seventh embodiment. It is a figure explaining the presentation method of the virtual reality world in the virtual reality presentation apparatus of this invention. It is a figure explaining how the virtual reality presentation apparatus of the present invention relates to the virtual reality world presented to the user and the real world where the user exists. It is a figure explaining how a virtual reality world presented to a user by an ideal virtual reality presentation device relates to a real world where a user exists. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 8. FIG. It is a figure which shows the scenery of the real world around the user wearing the virtual reality presentation device, the video presented to the user, and the virtual reality world sensed by the user. It is a figure explaining an example of the model video data of a virtual reality object. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 9. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 9. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 9. FIG. It is a figure explaining a mode in case the image | video for interruption is displayed. It is a figure explaining the time passage of the image | video displayed when a user's state changes at the time of a virtual reality presentation, and the image | video which a user senses. It is a figure explaining the time passage of the image | video displayed when a user's state changes at the time of a virtual reality presentation interruption, and the image | video which a user senses. It is an example of the management file which manages a virtual reality object and audio data for virtual reality presentation associated with the virtual reality object. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 9. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 9. FIG. It is a figure which shows an example of the video data management file for interruption, and the audio data management file for interruption. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a figure which shows an example of the interruption condition management file. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a figure which shows an example of the application interruption condition management file which manages the interruption condition to apply. It is a figure explaining a user's illusion degree. It is a figure which shows the time passage of a user's illusion degree. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 11. FIG. It is the figure which represented the image | video displayed when alerting is performed, and the scenery which a user senses in time series form. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 11. FIG. It is a block diagram which shows the function structure of the virtual reality presentation part periphery in the virtual reality presentation system which concerns on Embodiment 11. FIG. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is a figure which shows an example of an alert condition management file. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is a block diagram which shows the function structure of the modification of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is an example of a data management file for alerting that manages alerting video data and alerting audio data. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 13. FIG. It is a figure explaining the hierarchical structure of virtual reality content. It is a figure explaining the hierarchical structure of a virtual reality control program. It is a flowchart figure which shows the flow of the attention determination process in the virtual reality presentation system which concerns on Embodiment 13. FIG. FIG. 20 is a flowchart showing a flow of virtual reality object arrangement / update processing in a virtual reality presentation system according to Embodiment 13; It is a flowchart figure which shows the flow of the alerting process in the virtual reality presentation system which concerns on Embodiment 13. FIG. It is a figure explaining the virtual reality of the virtual reality world which a virtual reality presentation system provides. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 14. FIG. It is a block diagram which shows the function structure which concerns on the modification of the virtual reality presentation system which concerns on Embodiment 14. FIG. It is a block diagram which shows the function structure which concerns on the modification of the virtual reality presentation system which concerns on Embodiment 14. FIG. It is a figure explaining the image | video displayed when a video correction process is performed, and the scenery which a user senses. It is a figure explaining the relationship between the virtual reality which can be shown according to area | region, the virtual reality in the image | video containing a virtual reality object, and the virtual reality in the image | video after a virtual reality correction process. It is a block diagram which shows the function structure of the virtual reality presentation system which concerns on Embodiment 15. FIG. It is a figure explaining the image | video of the virtual reality object produced | generated with the virtual reality presentation apparatus which concerns on a modification, and the synthetic | combination image | video displayed on the display panel according to every alert level. It is a block diagram which shows the function structure of the virtual reality presentation system of Embodiment 16. It is a figure explaining the detection method of the user state by a virtual reality experience dome.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, parts denoted by the same reference numerals have substantially the same functions. For the sake of clarification of the invention, explanations of overlapping parts are omitted as appropriate.

<Embodiment 1>
FIG. 1 is a block diagram showing a configuration of a virtual reality service providing system 1000 according to Embodiment 1 of the present invention. The virtual reality service providing system 1000 is a system that provides a virtual reality service to a user wearing the virtual reality presentation device 100, and includes an available virtual reality service information storage database 1100, a virtual reality object storage database 1200, A virtual reality service providing server 1300. The virtual reality service providing server 1300 is connected to the virtual reality presentation device 100 worn by the user via a communication network 10 such as the Internet.

  Here, virtual reality (VR) refers to a world with reality created artificially by combining computer graphics (CG) and sound effects, and is a virtual reality service. Is a service that allows the user to experience such a virtual reality world.

  Humans sense events that occur around them using the five senses, and process the sensed information in the brain to grasp the situation. Therefore, a virtual reality world can be created by displaying images and outputting sounds so as to deceive human senses. Since the virtual reality world is an artificially created world, there are various forms of virtual reality services.

  FIG. 2 shows the correspondence between the real world, the displayed image, and the virtual reality world. The virtual reality presentation device 100 displays a video of a dog that is a virtual reality object for a user in a room where the sofa is placed, so that the world that the user sees (the world that the user perceives) is on the sofa. It becomes a virtual reality world different from the real world where the dog is sleeping.

  In the first embodiment, the virtual reality presentation device 100 will be described as a head mounted display (HMD) device that a user wears on the head. The virtual reality presentation device 100 is a device that presents a virtual reality world to a user, and includes a user state detection unit 110, an ambient state detection unit 120, a communication unit 130, a display control unit 140, and a display unit 150. .

  The user status detection unit 110 detects the status of the user who is using the virtual reality presentation device 100 and generates user status information indicating the status of the user. For example, the user state detection unit 110 includes an inclination sensor that detects the inclination of the user's head, an acceleration sensor that detects the acceleration of the user's head, and the like. For example, the user state detection unit 110 includes the user's head tilt, user head acceleration, user blood pressure, user heart rate, user heart rate waveform, user sweating amount, user posture, A user's body temperature, a user's brain wave, etc. are detected and user state information is generated based on the detection result.

  The surrounding situation detection unit 120 detects the surrounding situation of the user who is using the virtual reality presentation device 100, and generates the surrounding situation information indicating the surrounding situation of the user. For example, the ambient condition detection unit 120 includes a line-of-sight camera that captures a scene corresponding to the user's line of sight, a surrounding camera that captures a range of scenes that cannot be captured by the line-of-sight camera, such as on both sides or behind the user. For example, the surrounding state detection unit 120 includes surrounding image data, a positional relationship between the user and an object existing around the user, a moving speed and a moving direction of an object existing around the user, ambient brightness (illuminance), and surroundings. A situation such as whether or not another person exists is detected, and ambient situation information is generated based on the detection result.

  The communication unit 130 communicates with the virtual reality service providing server 1300 via the communication network 10 such as the Internet. The communication unit 130 includes a wireless transmission unit 131 that wirelessly transmits various types of information generated by the virtual reality presentation device 100, a wireless reception unit 132 that wirelessly receives various types of information transmitted from the virtual reality service providing server 1300, Is provided.

  The wireless transmission unit 131 converts various information generated by the virtual reality presentation device 100 into a wireless signal and transmits the wireless signal to the virtual reality service providing server 1300. The wireless transmission unit 131 transmits at least user state information generated by the user state detection unit 110 and ambient state information generated by the ambient state detection unit 120 to the virtual reality service providing server 1300.

  The wireless reception unit 132 wirelessly receives various pieces of information transmitted from the virtual reality service providing server 1300 and outputs the various information to each unit of the virtual reality presentation device 100 corresponding to the received information. The wireless reception unit 132 wirelessly receives at least encoded video data generated by the virtual reality service providing server 1300 and outputs the encoded data to the display control unit 140.

  The display control unit 140 performs control to display the video of the virtual reality object on the display unit 150 by decoding the encoded data received by the wireless reception unit 132.

  The display unit 150 displays the video of the virtual reality object based on the display control of the display control unit 140.

  Next, each device constituting the virtual reality service providing system 1000 will be described in detail. The available virtual reality service information storage database 1100 stores available virtual reality service information (usable service identification information) that is information indicating virtual reality services available to the user.

  The virtual reality service providing system 1000 is configured to be able to provide various types of virtual reality services, and each user preliminarily selects some virtual reality services that the user wants to use from among those virtual reality services. Selected.

  FIG. 3 shows an example of available virtual reality service information. Each user is stored in a state of being associated with a virtual reality service that can be used by the user. For example, user B indicates that the virtual reality service identified in S024 is available. Here, it is assumed that the virtual reality service identified in S024 is a service that presents a virtual reality world where a pet is present to a user by, for example, arranging a virtual reality object of a dog in a virtual space.

  The virtual reality object storage database 1200 stores a plurality of virtual reality objects. A virtual reality object refers to a virtual person or building represented in the virtual reality world by being arranged in a virtual space. By setting the virtual space set by the virtual reality service providing server 1300 to correspond to the real space around the user, the virtual reality world is presented to the user.

  Each type of virtual reality object includes 3D polygon data that defines an appearance shape by 3D computer graphics, texture data pasted on the surface of the shape formed by the polygon data, and the like. Moreover, it is more preferable that the configuration includes joint point data, skeleton data, and the like for operating the virtual reality object. The joint point data and the vertex of the polygon data are associated, and changing the position and angle of the joint point changes the position of the vertex of the associated polygon data, and the appearance of the virtual reality object The shape changes. Thus, various virtual reality objects have attributes set according to their types. By changing the position coordinates and angle of the joint point, which is one of the attributes, by the method, the position coordinates of various vertices of the polygon data are changed, and the appearance shape of the virtual reality object is changed.

  Virtual reality objects can be classified into human objects, animal objects, non-animal objects, and other objects according to the types of objects arranged in the virtual reality world. Specifically, it is classified into a plurality of classes such as a person class, an animal class, and a non-animal class, and various virtual reality objects such as a person object, an animal object, and a non-animal object are generated as instances of each class. Each virtual reality object is assigned an object ID for uniquely identifying the virtual reality object.

  A person object is an object such as a real person, an imaginary / imaginary person, a historical person, a creative person, etc., and is at least operable, and preferably speaks (outputs sound). It has the feature that it is possible. In other words, by executing a predetermined method on the person object, the appearance shape of the person object is changed, or the sound data set in the person object is reproduced to operate the person object. It is possible to make them flaunt or fool.

  The animal object is an object such as a real animal, an imaginary / fantasy animal, or a creative animal, and has a feature that it can operate. In addition, a part of the animal object can be configured to speak (output a sound). Since it is a virtual animal placed in the virtual reality world, it is possible to adopt a configuration that speaks like a human.

  A non-animal object is a static object such as a plant, furniture, or building and is basically an object that does not move. However, the plant may be configured to partially move in conjunction with the wind blowing in the real world.

  Other objects are, for example, background objects. Other objects may be used alone to create a virtual reality world, or may be used in combination with a person object, an animal object, or the like.

  FIG. 4 shows a management file for managing virtual reality objects stored in the virtual reality object storage database 1200. Each virtual reality object is classified according to a plurality of types. For example, type 1 indicates a rough division of an object, type 2 indicates whether it is real (person) or imaginary (person), and type 3 is a person included in the object. Showing gender. For example, data of a virtual reality object of a sleeping female lion is stored in the storage address xx010110101 of the management number No. 000012. Here, the virtual reality object with the management number No. 000012 is an instance of the lion class that inherits the Real class and the female class, which are subclasses of the Human class, as superclasses.

  FIG. 5 shows a virtual reality object of object ID number: No. 000009, which is a kind of virtual reality object, and 3D polygon data constituting the virtual reality object. Here, for the virtual reality object with the object ID number: No. 000009, the texture data assigned to the surface of each region connecting the polygon data, which is a set of the position coordinates of the vertices of the appearance shape, and the position coordinates of the polygon data is attribute data. Is defined as

  Next, the virtual reality service providing server 1300 will be described. The virtual reality service providing server 1300 includes a communication unit 1310, a virtual reality service provision managing unit 1320, a virtual reality object reading unit 1330, a virtual reality object arrangement position determining unit 1340, and a display video generation processing unit 1350. .

  The communication unit 1310 communicates with the virtual reality presentation device 100 worn by each user via the communication network 10 such as the Internet. The communication unit 1310 includes a receiving unit 1311 that receives various information transmitted from the virtual reality presentation device 100, and a transmission unit 1312 that transmits various information to the virtual reality presentation device 100.

  The receiving unit 1311 receives at least user status information indicating the user status and ambient status information indicating the status of the user. The reception unit 1311 outputs the received ambient state information to the virtual reality object arrangement position determination unit 1340 and the user state information to the display video generation unit 1350, respectively.

  The virtual reality service provision management unit 1320 refers to the available virtual reality service information stored in the available virtual reality service information storage database 1100, and provides virtual reality provided to the user from the virtual reality services available to the user. Process to determine the service. The virtual reality service provision management unit 1320 may determine a virtual reality service to be provided to the user by periodically performing scheduling related to provision of the virtual reality service, or may determine based on a request from the user. .

  The virtual reality object reading unit 1330 reads from the virtual reality object storage database 1300 a virtual reality object used for the virtual reality service provided to the user determined by the virtual reality service provision management unit 1320.

  The virtual reality object placement position determination unit 1340 determines the placement position of the virtual reality object in the virtual space based on the surrounding situation information indicating the surrounding situation of the user. The virtual reality object arrangement position determination unit 1340 arranges the virtual reality object read by the virtual reality object reading unit 1330 in the virtual space based on the surrounding situation information indicating the surrounding situation of the user.

  The display video generation processing unit 1350 generates a video to be displayed on the display panel of the virtual reality presentation device 100 worn by the user on the head based on the user status information indicating the user status and the arrangement position of the virtual reality object. To do. Specifically, the display video generation processing unit 1350 projects the virtual reality object onto an image plane placed between the viewpoint in the virtual space determined based on the user state information and the virtual reality object. To generate video.

  The viewpoint in the virtual space corresponds to the position of the user's eyes in the real space, and the movement of the user's head is sequentially input to the display video generation processing unit 1350 as user state information, and the viewpoint in the virtual space is controlled. . The display video generation processing unit 1350 performs coordinate conversion processing for projecting each virtual reality object arranged in the virtual space onto the image plane based on the viewpoint corresponding to the position of the user's eye indicated by the user state information. By performing clipping processing, hidden surface processing, shading processing, texture mapping processing, anti-aliasing processing, etc., a video to be displayed on the display unit 150 of the virtual reality presentation device 100 is generated. . By performing encoding using a predetermined encoding method such as H.264 / AVC, encoded data of the video is generated. The encoded data of the video generated by the display video generation processing unit 1350 is output to the transmission unit 1312.

  The transmission unit 1312 transmits the encoded data of the video generated by the display video generation processing unit 1350 to the virtual reality presentation device 100. The display control unit 140 of the virtual reality presentation device 100 that has received the encoded data of the video performs decoding processing, and the video of the virtual reality object is displayed on the display unit 150.

  As described above, according to the virtual reality service providing system according to the first embodiment, the imaginary world created by the creator can be expressed in the virtual reality world fused with the real world. That is, each creator can create the world that he / she imagines in a representation form called a virtual reality object, and store the created virtual reality object in the virtual reality object storage DB 1200. For a user who wants to enjoy the virtual reality world created by the creator, the virtual reality service providing server 1300 as a virtual reality service appropriately arranges the virtual reality object so as to merge with the real world of the user. The virtual reality world can be presented to the user.

(Embodiment 2)
FIG. 6 is a block diagram illustrating a configuration of the virtual reality service providing system 2000 according to the second embodiment. The virtual reality service providing system 2000 includes an available virtual reality service information storage DB 1100, a virtual reality object storage DB 1200, a virtual reality service providing server 2300, and a virtual reality service use condition storage DB 2400. The virtual reality service providing server 2300 is connected in a communicable state with the virtual reality presentation device 200 via a communication network such as the Internet.

  The virtual reality presentation device 200 includes a user state detection unit 110, an ambient condition detection unit 120, a communication unit 130, a display control unit 140, a display unit 150, and a virtual reality presentation control unit 260.

  The user status detection unit 110 detects the user status and generates user status information. In the virtual reality presentation device 200, the user state detection unit 110 detects an inclination of the user's head and generates inclination information, and an acceleration that detects acceleration of the user's head and generates acceleration information. A sensor 112 and a current position detection unit 113 that detects the current position of the user and generates position information are provided. The inclination information, acceleration information, and position information that are periodically generated by each unit are output to the communication unit 130 as user status information indicating the user status. It is preferable that the user status information includes time information (time stamp) indicating the time when each piece of information included in the user status information is generated.

  The surrounding situation detection unit 120 detects the surrounding situation of the user and generates the surrounding situation information. In the virtual reality presentation device 200, the surrounding state detection unit 120 includes a line-of-sight camera (line-of-sight video photographing unit) 121 and a surrounding camera (ambient video photographing unit) 122. The line-of-sight camera 121 captures a scene in front of the user corresponding to the line of sight of the user, and acquires a front scene image. The ambient camera 122 captures a scene in a range that cannot be captured by the line-of-sight camera 121, such as on both sides or behind the user, and acquires an ambient scene image. The forward scenery image photographed by the line-of-sight camera 121 and the ambient scenery image periodically photographed by the peripheral camera 122 are each appropriately encoded such as compression, and then output to the communication unit 130 as ambient state information. The

  The ambient situation information includes a time information (time stamp) indicating the time when each image included in the ambient situation information is captured, and camera identification information for identifying the camera that captured the image. It is preferable. The ambient condition detection unit 120 may be referred to as an ambient video capturing unit in this specification.

  The communication unit 130 inputs user state information indicating the current state of the user, such as tilt information, acceleration information, and position information output from the user state detection unit 110, and transmits the user state information to the virtual reality service providing server 2300. In addition, the communication unit 130 inputs ambient situation information indicating a situation around the user, such as a forward scenery image and an ambient scenery image output from the ambient situation detection unit 120, and transmits the information to the virtual reality service providing server 2300. In addition, the communication unit 130 receives encoded video data transmitted from the virtual reality service providing server 2300 and outputs the encoded data to the display control unit 140.

  The display control unit 140 performs a decoding process on the encoded data of the video received by the communication unit 130, and performs control to display the decoded video on the display unit 150. The display unit 150 displays an image according to the display control from the display control unit 140.

  The virtual reality presentation control unit 260 performs overall control for presenting virtual reality to the user. The virtual reality presentation control unit 260 starts control for presenting the virtual reality to the user when the virtual reality service start notification information for notifying the start of provision of the virtual reality service is received by the communication unit 130.

  Specifically, the virtual reality presentation control unit 260 instructs the user state detection unit 110 and the surrounding state detection unit 120 to start generating the user state information and the surrounding state information, respectively, based on the virtual reality service start notification information. . The virtual reality presentation control unit 260 instructs the communication unit 130 to periodically transmit the user state information and the surrounding state information to the virtual reality service providing server 2300. In addition, the virtual reality presentation control unit 260 instructs the display control unit 140 to start control for displaying an image on the display unit 150.

  Also, the virtual reality presentation control unit 260 performs control for ending the virtual reality presentation when the communication unit 130 receives virtual reality service end notification information for notifying the end of provision of the virtual reality service. Specifically, the virtual reality presentation control unit 260 instructs the user state detection unit 110 and the surrounding state detection unit 120 to end the detection process. In addition, the display control unit 140 is instructed to end the display control process.

  The virtual reality service providing server 2300 includes a communication unit 1310, a virtual reality service provision management unit 1320, a virtual reality object 1330, a virtual reality object arrangement position determination unit 1340, a display video generation processing unit 1350, and an image analysis processing unit. 2360.

  The image analysis processing unit 2360 analyzes the front scenery image and the surrounding scenery image received from the virtual reality presentation device 200 by the communication unit 1310 as the surrounding situation information, and performs processing for extracting a predetermined component from the image.

  The image analysis processing unit 2360 includes an object position specifying unit 2361 that extracts information related to the positional relationship between objects, and an optical extraction unit 2362 that extracts optical components such as brightness.

  The object position specifying unit 2361 inputs a front scene image acquired by a plurality of line-of-sight cameras arranged at locations corresponding to the positions of the left eye and the right eye, and a surrounding scene image captured by a plurality of surrounding cameras. , Boundary determination processing is performed by calculating a difference between a color difference signal and a luminance signal between adjacent pixels and blocks. Based on the boundary determination process, the object position specifying unit 2361 specifies each object, wall, floor, sky, and the like shown in the image, and labels each specified object. The object position specifying unit 2361 determines the distance to the object by comparing the pixel positions of the objects with the same label in the images acquired by the cameras, and exists around the user. The positional relationship of an object etc. is specified.

  The object position specifying unit 2361 outputs information indicating the positional relationship of objects existing around the specified user to the virtual reality object arrangement position determining unit 1340 and the drawing processing unit 1353 of the display video generation processing unit 1350, respectively. Moreover, it is good also as a structure which each outputs the information which shows the positional relationship of the object which exists around the specified user to the virtual reality service provision start process part 1321 and the virtual reality service provision end process part 1322 as needed.

  Preferably, the object position specifying unit 2361 generates model data indicating the physical environment around the user by modeling each object specifying the size and the distance to the user from the input image. That is, the object position specifying unit 2361 models the physical environment around the user in the real world with 3D polygon data. The object position specifying unit 2361 outputs the generated model data to the virtual reality object arrangement position determining unit 1340 and the drawing processing unit 1353 of the display video generation processing unit 1350, respectively.

  The optical component extraction unit 2362 extracts an optical component from the input front scene image. For example, the optical component extraction unit 2362 obtains information indicating the overall brightness (illuminance) around the user by calculating the average value of the luminance signal in each pixel of the front scene image. The optical component extraction unit 2362 obtains information indicating the overall color around the user by obtaining an average value of the color difference signals of the respective pixels. These pieces of information correspond to the brightness (illuminance) and color of the infinity light source controlled by the light source control unit 1352 described later.

  In addition, the optical component extraction unit 2362 compares the brightness of each pixel or block of the front scene image, and identifies a shadowed portion, so that the position of the light source around the user and the brightness of the light source Get information about.

  The optical component extraction unit 2362 outputs information related to the optical component extracted from the front scenery image input as the surrounding state information to the light source control unit 1352. The optical component extraction unit 2362 performs the same processing on the surrounding scene image input as the surrounding state information, extracts information on the optical component related to the position and intensity of the light source in the virtual space, and controls the light source. It may be configured to output to the unit 1352.

  In the virtual reality service provision server 2300, the virtual reality service provision management unit 1320 includes a virtual reality service provision start processing unit 1321 and a virtual reality service provision end processing unit 1322.

  The virtual reality service provision start processing unit 1321 refers to the available service information stored in the available virtual reality service information storage DB 1100, and provides virtual reality provided to the user from among virtual reality services available to the user. Select a service.

  For example, when the virtual reality service start request for starting the virtual reality service is received by the communication unit 1310, the virtual reality service provision start processing unit 1321 is designated by the virtual reality service start request. As a virtual reality service to be provided. In addition, the virtual reality service provision start processing unit 1321 periodically performs scheduling related to provision of the virtual reality service, and refers to available service information, and provides virtual services provided from virtual reality services available to the user. Real service may be actively selected.

  The virtual reality service provision start processing unit 1321 determines whether or not the selected virtual reality service can be provided with reference to the virtual reality service use condition stored in the virtual reality service use condition storage DB 2400.

  The virtual reality service use condition storage DB 2400 stores virtual reality service use conditions that are conditions for using the virtual reality service. The virtual reality service usage condition may be set differently for each virtual reality service, or a common condition may be set for each virtual reality service.

  For example, the virtual reality service identified by service No. S001 can be used only in the time zone between 09: 00 and 17:00, and the time zone in which the virtual reality service can be used is virtual. It may be set as a real service use condition. For example, in the case of a virtual reality service linked to a public service, a time zone in which the virtual reality service can be used is set so as to be linked to a public service time zone. In this case, the virtual reality service provision start processing unit 1321 determines whether or not the current time is an available time zone that is a virtual reality service usage condition. Similarly, a time zone in which the virtual reality service cannot be used may be set as the virtual reality service usage condition.

  Also, for example, the virtual reality service identified in service No. S002 can be used only within a predetermined location range, and the location range where the virtual reality service can be used is the virtual reality service usage condition. It may be set. For example, the condition regarding the position range is set when the virtual reality service identified by No: S002 can be used only at a specific event venue. In this case, the virtual reality service provision start processing unit 1321 determines whether the current location of the user indicated by the location information received by the communication unit 1310 is within a location range that is a virtual reality service usage condition. Similarly, a location range where the virtual reality service cannot be used may be set as the virtual reality service usage condition.

  Also, the virtual reality service usage conditions may be set differently for each user who provides the virtual reality service. The user previously sets a condition for using the virtual reality service in advance as a virtual reality service use condition. For example, when the user wants to use the virtual reality service only at home, the location range of the home is set as a virtual reality service usage condition. The set virtual reality service use condition is stored in the virtual reality service use condition storage DB 2400, and is used for the determination process of whether to provide the virtual reality service. The virtual reality service provision start processing unit 1321 refers to the virtual reality service use conditions set for the user stored in the virtual reality service use condition storage DB 2400, and indicates the position information received by the communication unit 1310. It is determined whether the current position of the user is within the position range of the home set as the virtual reality service usage conditions.

  Note that the virtual reality service provision start processing unit 1321 inputs information regarding the positional relationship of objects around the user from the object position specifying unit 2361 of the image analysis processing unit 2360, and the user can use the virtual reality service. It may be determined. For example, when it is set as a virtual reality service usage condition in the selected virtual reality service that there is no interference object in a place within a predetermined distance from the user, the user specified by the object position specifying unit 2361 It is determined whether or not the condition is satisfied based on the positional relationship between the surrounding objects. In addition, it may be set as a virtual reality service use condition for starting the use of the virtual reality service that there is no step higher than a predetermined height around the user. The virtual reality service provision start processing unit 1321 performs determination processing on the above-described plurality of types of virtual reality service usage conditions, and determines whether to start provision of the virtual reality service.

  When the virtual reality service provision start processing unit 1321 determines that the selected virtual reality service can be provided as a result of the determination process, that is, when the user determines that the selected virtual reality service is available. The virtual reality object reading unit 1330 is instructed to read a virtual reality object used for the virtual reality service.

  Further, the virtual reality service provision start processing unit 1321 generates virtual reality service start notification information for notifying that the provision of the virtual reality service is started, and outputs the virtual reality service start notification information to the communication unit 1310. The virtual reality service start notification information is transmitted from the communication unit 1310 to the virtual reality presentation device 200, and various controls in the virtual reality presentation control unit 260 of the virtual reality presentation device 200 are started.

  The virtual reality service provision end processing unit 1322 performs processing to end the virtual reality service being provided to the user. The virtual reality service provision end processing unit 1322 performs processing to end the virtual reality service when the communication unit 1310 receives a virtual reality service end request for requesting termination of the currently provided virtual reality service.

  Also, the virtual reality service provision end processing unit 1322 refers to the virtual reality service usage conditions stored in the virtual reality service usage condition storage DB 2400, and when the virtual reality service usage conditions are not satisfied, the virtual reality service termination is performed Execute the process. For example, if the location range is set as a virtual reality service usage condition, a process of terminating the currently provided virtual reality service when the current location of the user deviates from the location range specified as the condition. Do. Also, for example, when a time zone is set as a virtual reality service usage condition, a process of terminating the currently provided virtual reality service when the current time has passed the time zone specified as the condition I do.

  Further, for example, when the virtual reality service use condition is set to 2 hours as a virtual reality service use condition, the virtual reality service provision end processing unit 1322 performs virtual processing in the virtual reality service provision start processing unit 1321. When the provision of the real service is started, the timer for the time specified as the virtual reality service use condition is set and the time measurement is started. The virtual reality service provision end processing unit 1322 performs a termination process of the virtual reality service being provided when the time measurement is completed.

  The virtual reality service provision end processing unit 1322 terminates the provision of the virtual reality service by instructing the drawing processing unit 1353 of the display video generation processing unit 1350 to end the drawing processing. In addition, the virtual reality service provision end processing unit 1322 generates virtual reality service end notification information for notifying that provision of the virtual reality service is to be ended, and outputs the virtual reality service end notification information to the communication unit 1310. The virtual reality service end notification information is transmitted from the communication unit 1310 to the virtual reality presentation device 200, and the virtual reality presentation control unit 260 of the virtual reality presentation device 200 performs control to end the presentation of the virtual reality.

  The virtual reality object reading unit 1330 reads from the virtual reality object storage DB 1200 a virtual reality object used for the virtual reality service determined to start providing to the user by the virtual reality service provision start processing unit 1321.

  The virtual reality object placement position determination unit 1340 determines the placement position of the virtual reality object in the virtual space based on the surrounding situation information indicating the surrounding situation of the user. The virtual reality object arrangement position determination unit 1340 is based on the positional relationship of the objects around the user specified by the object position specifying unit 2361 based on the image acquired by each camera input as the surrounding state information. The virtual reality object read by the real object reading unit 1330 is arranged at a position where it does not interfere with objects around the user.

  Preferably, in a virtual space in which the physical environment around the user in the real world generated by the object position specifying unit 2361 is modeled with 3D polygon data, The virtual reality object read by the virtual reality object reading unit 1330 at a position where interference does not occur (for example, a position where an object existing in the real space and the virtual reality object do not overlap with each other by a predetermined amount but touch an area of a predetermined amount or more) Place.

  The display video generation processing unit 1350 displays the video displayed on the display unit 150 of the virtual reality presentation device 200 worn by the user on the head based on the user status information indicating the user status and the arrangement position of the virtual reality object. Generate. In the virtual reality service providing server 2300, the display video generation processing unit 1350 includes a viewpoint control unit 1351, a light source control unit 1352, a drawing processing unit 1353, and a video encoding processing unit 1354.

  The viewpoint control unit 1351 controls the position and direction of the viewpoint when drawing the virtual reality object based on the user state information indicating the user state. Specifically, based on the tilt information indicating the tilt of the user's head included in the user state information and the acceleration information indicating the acceleration of the user's head, a virtual space corresponding to the position of the left eye and the position of the right eye of the user The viewpoint position at is calculated. Further, the line-of-sight direction in the virtual space corresponding to the direction of the user's line of sight is calculated based on the acceleration information. Here, the line-of-sight direction corresponds to the direction of the user's head, not the direction of the line of sight when the user moves only the eyeball.

  It is more preferable that the user status information includes azimuth information indicating the direction of the user, movement speed information indicating the movement speed of the user, and the like. The viewpoint control unit 1351 calculates the posture of the user's head based on each piece of information included in the user state information, and specifies the positions of the user's left eye and right eye and the direction in which the user's head is facing. The viewpoint control unit 1351 calculates the position of the viewpoint and the direction of the line of sight in the virtual space corresponding to the position of the user's eyes and the direction of the head.

  The light source control unit 1352 calculates the position of the light source and the intensity (brightness) of the light source in the virtual space based on the optical component extracted by the optical component extraction unit 2362 of the image analysis processing unit 2360. For example, the light source control unit 1352 calculates the intensity of the light source at infinity by calculating the average value of the luminance of each pixel of the image extracted by the optical component extraction unit 2362, and the luminance between the pixels of the image. Based on the difference, the position and intensity of another light source are calculated. Further, the light source control unit 1352 may calculate the color of the light source based on the color difference signal of the pixel in the image subjected to the analysis process.

  The drawing processing unit 1353 generates a two-dimensional image by rendering the virtual reality object in real time. As an example, the rendering processing unit 1353 performs rendering by combining a part or a plurality of processes such as a projection process, a clipping process, a hidden surface process, a shading process, and a texture mapping process shown below to generate a video of a virtual reality object. To do.

  The drawing processing unit 1353 sets an image plane at a predetermined position in the virtual space based on the viewpoint position and the line-of-sight direction obtained by the viewpoint control unit 1351, and projects the virtual reality object. Get the projected image of the object. The drawing processing unit 1353 performs clipping processing of the image plane on which the virtual reality object is projected, and selects a range displayed on the display unit 150 in the virtual reality presentation device 200.

  The drawing processing unit 1353 performs hidden surface processing based on the model of the real world object specified by the object position specifying unit 2361. The drawing processing unit 1353 is a pixel of the real world located between the viewpoint position calculated by the viewpoint control unit 1351 and the virtual reality object, out of the pixels constituting the virtual reality object image projected onto the image plane. Hidden surface processing is performed by deleting pixel data that overlaps the projected image of the object model.

  The drawing processing unit 1353 performs shading processing based on the position and intensity of the light source calculated by the light source control unit 1352, and adjusts the brightness of the surface of the virtual reality object, the degree of light rise, and the light transmission state. The rendering processing unit 1353 performs texture mapping processing using texture data pasted on the surface of the virtual reality object, appropriately performs correction processing such as anti-aliasing processing, and finally is performed by the display unit 150 of the virtual reality presentation device 200. Generate video for display.

  Note that the drawing processing unit 1353 generates two images, a left-eye image and a right-eye image, based on the viewpoint corresponding to the left eye and the viewpoint corresponding to the right eye controlled by the viewpoint control unit 1351. The drawing processing unit 1353 outputs the generated video to the video encoding processing unit 1354.

  The video encoding processing unit 1354 performs compression encoding processing on videos sequentially generated by the real-time rendering processing in the drawing processing unit 1353 based on a predetermined encoding method. Video encoded data after the video encoding processing unit 1354 encoding process is output to the communication unit 1310. The communication unit 1310 transmits the video encoded data output from the video encoding processing unit 1354 to the virtual reality presentation device 200, and the decoding control is performed by the display control unit 140 of the virtual reality presentation device 200. The video is displayed on the display unit 150.

  As described above, according to the virtual reality service providing system 2000 according to the second embodiment, the virtual reality service is provided based on the determination result of whether the service use condition is satisfied. It is possible to prevent service provision when the relationship is not appropriate.

  The virtual reality service presentation server 2300 is more preferably configured as shown in FIG. In the virtual reality service providing server 2300 illustrated in FIG. 7, the virtual reality service provision management unit 1320 includes a virtual reality service interruption processing unit 1323, and the image analysis processing unit 2360 includes a movement speed calculation unit 2363.

  The moving speed calculation unit 2363 calculates the moving speed of each object based on the temporal change in the position of each object existing around the user specified by the object position specifying unit 2361. The movement speed calculation unit 2363 outputs the calculated movement speed of each object to the virtual reality service interruption processing unit 1323. Further, the object position specifying unit 2361 outputs the calculated distance of each object to the virtual reality service interruption processing unit 1323.

  The virtual reality service interruption processing unit 1323 performs a process of forcibly interrupting the provision of the virtual reality service when a virtual reality service is provided to the user and a predetermined interruption condition is satisfied. The virtual reality service usage condition storage DB 2400 further stores a virtual reality service provision interruption condition that is a condition for interrupting provision of the virtual reality service, and the virtual reality service interruption processing unit 1323 stores the virtual reality service provision interruption condition. By referring, it is determined whether or not the provision of the virtual reality service is interrupted.

  For example, the virtual reality service interruption processing unit 1323 suspends provision of the virtual reality service being provided when a virtual reality service interruption request for requesting interruption of use of the virtual reality service is received from the user by the communication unit 1310. I do.

  Further, for example, the virtual reality service interruption processing unit 1323 has the distance L to the object within a predetermined range among the objects specified by the object position specifying unit 2361, and the moving speed V of the object is a predetermined value. When there is an object whose speed is greater than or equal to, the virtual reality service is interrupted. For example, as a virtual reality service provision interruption condition, a condition such as 0 m <L ≦ 20 m, 5 m / s ≦ V ≦ 200 m / s may be set, or a condition such as 0 <L / V ≦ 4 may be set. It may be set.

  Further, for example, the virtual reality service interruption processing unit 1323 is a case where the user is moving and an object that becomes an obstacle within a predetermined distance in the moving direction of the user is specified by the object position specifying unit 2361. The virtual reality service is interrupted.

  The virtual reality service interruption processing unit 1323 generates virtual reality service interruption notification information for notifying that provision of the virtual reality service is interrupted, and outputs the generated virtual reality service interruption notification information to the communication unit 1310. The virtual reality service interruption notification information is transmitted from the communication unit 1310 to the virtual reality presentation device 200. When the virtual reality presentation control unit 260 of the virtual reality presentation device 200 receives the virtual reality service interruption notification information, the virtual reality service interruption notification information is displayed. The control unit 140 is instructed to interrupt the display of the image being displayed on the display unit 150. The display control unit 140 performs control to interrupt display of the video of the virtual reality object displayed on the display unit 150 based on the interruption instruction. In this case, if the display control unit 140 is configured to display an image indicating that the display of the virtual reality object is forcibly interrupted on the display unit 150, the provision of the virtual reality service is compulsory due to the occurrence of a failure or the like. It is preferable because the user can appropriately recognize that the operation has been interrupted.

  When resuming the interrupted virtual reality service, the virtual reality service interruption processing unit 1323 generates virtual reality service restart notification information and transmits the virtual reality service restart notification information to the virtual reality presentation device 200 from the communication unit 1310. The virtual reality presentation control unit 260 that has received the virtual reality service resumption notification information resumes the display control unit 140 to decode the video encoded data received by the communication unit 130 and display the video on the display unit 150. Let

  Note that the virtual reality service interruption processing unit 1323 may stop the rendering process by instructing the drawing processing unit 1353 to interrupt the rendering process while the virtual reality service is interrupted.

  The virtual reality service interruption processing unit 1323 and the image analysis processing unit 2360 described above may be included in the virtual reality presentation device 200. In this case, the image analysis processing unit 2360 performs the above-described predetermined analysis processing on the video imaged by the surrounding state detection unit 120, so that the positional relationship between the objects around the user and the relative relationship between each object and the user. Detect movement speed. The virtual reality service interruption processing unit 1323 displays information related to the positional relationship of the object acquired based on the analysis processing in the image analysis processing unit 2360, information related to the moving speed, and the user status detected by the user status detection unit 110. Based on this, a process for determining whether to interrupt the presentation of the virtual reality service currently being presented to the user is performed. Specifically, an instruction to interrupt the video display of the virtual reality object being performed on the display unit 150 is output to the display control unit 140. The display control unit 140 performs control to interrupt the display processing of the virtual reality object video on the display unit 150 based on the suspension instruction from the virtual reality service suspension processing unit 1323. In this case, instead of the video of the interrupted virtual reality object, the display control unit 140 displays a screen that warns the user that the virtual reality service will be interrupted due to the situation around the user corresponding to the interrupt condition. It is good also as a structure which performs control displayed on the part 150. FIG.

(Embodiment 3)
FIG. 8 is an external perspective view of a head-mounted display (HMD) device 300 that is a virtual reality presentation device according to the third embodiment from obliquely forward and obliquely rearward. The HMD device 300 includes at least a frame 301 and a front panel 302.

  A frame 301 is a housing of the HMD device 300, and includes an information processing processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read only memory), a wireless communication processing processor, a battery, and the like. Is arranged. In addition, a dedicated video processor or the like that draws video as necessary is also arranged. The frame 301 is roughly composed of a right frame 301a, a left frame 301b, and a center frame 301c.

  The light frames 301a and 301b are housing members that are connected to the left and right ears of the user, respectively, and speakers 391a and 391b that convert analog electric signals into air vibrations and output sound are arranged inside one end, respectively. The end is connected to the center frame 301c.

  In addition, surrounding microphones 372a and 372b that collect surrounding sounds are arranged outside one ends of the right frame 301a and the left frame 301b. In addition, a user microphone 371 that collects a voice uttered by the user is provided below the front end inside the light frame 301a. The sound collected by the user microphone 371 and the surrounding microphones 372a and 372b is converted into a digital sound signal by AD conversion and output to the information processing processor.

  At the ends of the light frames 301a and 301b, inclination sensors 111a and 111b for detecting inclination and acceleration sensors 112a and 112b for detecting acceleration are arranged, respectively. In addition, the surrounding cameras 122a and 122b for photographing the surrounding scenery are arranged outside the right frame 301a and the left frame 301b, respectively.

  Both ends of the center frame 301c are connected to the right frame 301a and the left frame 301b, respectively, and the front panel 302 is connected below the center frame 301c.

  Near the center and near both ends of the center frame 301c, tilt sensors 111c, 111d, and 111e that detect tilt and acceleration sensors 112c, 112d, and 112e that detect acceleration are arranged, respectively. In addition, line-of-sight cameras 121a and 121b for photographing a scene corresponding to the user's field of view are arranged on the front side of the center frame 301c and at the positions of the left and right eyes of the user. The line-of-sight cameras 121a and 121b are arranged on the front side of the center frame 301c at positions separated by a distance corresponding to the distance between the left and right eyes of the user.

  The front panel 302 is disposed at a position in front of the user's eyes with the HMD device 300 mounted on the head. In the HMD device 300 according to the third embodiment, the front panel 302 is a transmissive display, and includes a light display panel (first video display unit) 151a that is a display area for displaying images in front of the right eye and in front of the left eye, respectively. A left display panel (second video display unit) 151b is arranged.

  In the HMD device 300 shown in FIG. 8, the right display panel 151a and the left display panel 151b have a structure in which a light guide plate and a liquid crystal panel (LCD) are overlapped. The liquid crystal panel has a structure in which a transparent electrode (transparent conductive film), a polarizing plate (polarizing film), a liquid crystal layer, a glass substrate, a color filter, and the like are laminated.

  FIG. 9 is a cross-sectional view of the vicinity of the light display panel 151 a of the HMD device 300. A front panel 302 is disposed below the center frame 301c, and a light source 352 is disposed below the center frame 301c.

  Visible light emitted from the light source 352 is irradiated to the front panel 302 side. The front panel 302 has a configuration in which a light guide plate 353 and an LCD 354 are combined. Visible light emitted from the light source 352 is reflected to the user side by the light guide plate 353 and transmitted through the display-controlled LCD 354 to display an image. Is done.

  FIG. 10 is a block diagram illustrating a functional configuration of the virtual reality service providing system 3000 according to the third embodiment. The virtual reality service providing system 3000 includes a contractor information storage DB 3100, a virtual reality content storage DB 3200, a virtual reality service providing server 3300, and a save data storage DB 3500, and includes an HMD device 300 that is a virtual reality presentation device and the Internet. Connected via a communication network.

  The HMD device 300 includes a user state detection unit 110, an ambient state detection unit 120, a communication unit 130, a display control unit 140, a display unit 150, a virtual reality presentation control unit 260, a voice input unit 370, and a voice. A playback control unit 380 and an audio output unit 390 are provided.

  The user state detection unit 110 detects the state of a user who is using the HMD device 300 that is a virtual reality presentation device. As the user state detection unit 110, the HMD device 300 includes an inclination sensor 111 that detects inclination and an acceleration sensor 112 that detects acceleration. The inclination information generated by the inclination sensor 111 and the acceleration information generated by the acceleration sensor 112 are transmitted from the communication unit 130 to the virtual reality service providing server 3300 as user state information indicating the state of the user.

  The surrounding situation detection unit 120 detects the surrounding situation of the user who is using the HMD device 300. Specifically, the surrounding situation detection unit 120 detects the surrounding situation by acquiring an image of the external environment. As the surrounding state detection unit 120, the HMD device 300 includes a line-of-sight camera 121 that captures a scene corresponding to the user's line of sight, a surrounding camera 122 that captures a range of scenes that cannot be captured by the line-of-sight camera 121, such as both sides and behind the user, Is provided. The image of the scenery in front of the user photographed by the line-of-sight camera 121 and the image of the user's surroundings photographed by the peripheral camera 122 are each appropriately encoded such as compression, and then transmitted from the communication unit 130 as ambient state information. It is transmitted to the virtual reality service providing server 3300.

  In accordance with control from the virtual reality presentation control unit 260, the communication unit 130 includes user status information indicating the user status detected by the user status detection unit 110, and a surrounding status indicating the user's surrounding status detected by the ambient status detection unit 120. The situation information and the voice information indicating the voice input by the voice input unit 370 are transmitted to the virtual reality service providing server 3300, respectively. Further, the communication unit 130 outputs the video data and audio data received from the virtual reality service providing server 3300 to the display control unit 140 and the audio reproduction processing unit 380, respectively.

  The display control unit 140 receives encoded data of a video generated by the virtual reality service providing server 3300 among various data received by the communication unit 130, and decodes the input encoded data. Control to display the video of the real object on the display unit 150 is performed.

  The display unit 150 displays a predetermined video based on the control from the display control unit 140. The video displayed on the display unit 150 is a video such as a still image or a moving image of a virtual reality object for providing virtual reality. As the display unit 150, the HMD device 100 is provided with a right display panel 151a and a left display panel 151b, and each displays an image with added parallax.

  The virtual reality presentation control unit 260 controls the process of presenting virtual reality to the user with the HMD device 300. The virtual reality presentation control unit 260 starts control for presenting virtual reality when the communication unit 130 receives virtual reality service start notification information instructing the start of the virtual reality service.

  Specifically, the virtual reality presentation control unit 260 acquires the user state information, the surrounding state information, and the sound information from the user state detecting unit 110, the surrounding state detecting unit 120, and the voice input unit 370 based on the start notification information. The start is instructed and the communication unit 130 is instructed to periodically transmit the information to the virtual reality service providing server 2100. In addition, the virtual reality presentation control unit 260 instructs the display control unit 140 to start control of displaying video on the display unit 150, and starts the playback control of audio output from the audio output unit 390 to the audio playback control unit 380. Instruct.

  The voice input unit 370 collects voice and converts it into an analog electric signal, performs AD conversion processing and converts it into a digital voice signal, and inputs voice. As the voice input unit 370, the HMD device 300 includes a user microphone 371 that collects voices uttered by the user and a surrounding microphone 372 that collects surrounding voices.

  The user microphone 371 is arranged on the inner side (user side) of the HMD device 300 so that the user's voice can be collected efficiently, while the surrounding microphone 372 can efficiently collect the voice from the user's external environment. , Arranged outside the HMD device 300. The surrounding microphone 372 is preferably arranged in the vicinity of the position of the ear in the HMD device 300 so as to collect sound that can be heard directly by the user's ear when the user is not wearing the HMD device 300. The user microphone 371 and the surrounding microphone 372 collect sound and convert it into an analog electric signal, and perform AD conversion processing to convert it into a digital sound signal, thereby inputting the sound. The input speech is subjected to speech coding as necessary, and then transmitted to the virtual reality service providing server 3300 from the communication unit 130 as user speech information and ambient speech information, respectively.

  The audio reproduction control unit 380 inputs audio data generated by the virtual reality service providing server 3300 among various data received by the communication unit 130, and generates audio generated by the virtual reality object based on the input audio data. Control to output from the audio output unit 390 is performed.

  Here, in order to make the user recognize the sound caused by the virtual phenomenon as the sound caused by the real phenomenon, the user is recognized as if the sound is transmitted from the position of the virtual reality object that is the sound generation source. This is preferable in attracting users to the virtual reality world. Therefore, it is preferable that the audio reproduction control unit 380 reproduces audio data in a form corresponding to the three-dimensional sound such as 5.1Ch, 6.1Ch, or 7.1Ch.

  The audio output unit 390 outputs audio according to the reproduction control from the audio reproduction control unit 370. As the audio output unit 390, the HMD device 300 includes speakers 391 a and 391 b arranged near the rear ends of the right frame 301 a and the left frame 301 b, and the audio data of each channel reproduced by the audio reproduction control unit 380 It performs DA conversion, converts it into an analog electrical signal, converts it into air vibration, and emits the sound to the outside.

  Next, the virtual reality service providing system 3000 will be described. The contractor information storage database unit 3100 stores contractor information for managing contractors who are users of the virtual reality service. Each user has a virtual reality service use contract with a virtual reality service provider in order to use the virtual reality service.

  FIG. 11 shows an example of contractor information stored in the contractor information storage database unit 3100. In FIG. 11, the contractor information includes the personal information (contractor personal information) of the user who has concluded the virtual reality service use contract, the plan contracted by the user and the service content included in the plan. include.

  The virtual reality content storage database 3200 stores virtual reality content. Here, virtual reality content is a general term for data used to create a virtual reality world, model data defining the appearance of a virtual reality object, audio data emitted by the virtual reality object, virtual data A virtual reality service control program for controlling the real service. FIG. 12 shows the structure of virtual reality content.

  The model data of the virtual reality object includes polygon data that defines the shape of the virtual reality object and texture data that is attached to the surface of the shape that is defined by the polygon data.

  The sound data of the virtual reality object includes sentence sound data that records in advance the sound of a predetermined sentence read out by a person or the like represented by the virtual reality object, sound basic data that is a source of sound generated by the virtual reality object, and including. For example, when the virtual reality object is a human object created based on a real person, the voice basic data is data such as voice quality, pitch, voice waveform, and phoneme extracted from the real voice of the real person. These voice basic data are synthesized by voice synthesis processing and presented to the user, so that the user senses that the person object is actually speaking. Note that the basic voice data extraction method (recording method) can be extracted at various angles, and may be extracted in the form of voice pronunciation data and voice expression data, or other classification methods. It may be extracted with. The extraction method is determined by a speech synthesis algorithm to be performed.

  The virtual reality control program is a program executed to control a virtual reality service provided to the user, and is largely a virtual reality main control program, a virtual reality sub-control program, a virtual reality object personality control program, And a virtual reality object behavior control program. FIG. 13 shows the hierarchical structure of the virtual reality control program.

  The virtual reality object behavior control program is a program for controlling the behavior of the virtual reality object arranged in the virtual space, and is read by the virtual reality object personality control program. For example, when the virtual reality object is a human object, the virtual reality object behavior control program A performs a control such that the joint point of the right knee is slightly bent and the texture data of the mouth is gradually changed to the smile texture data. It is a program. On the other hand, the virtual reality object behavior control program N performs control such that the hip joint point is bent so as to slide, and the right arm and the right elbow joint point are rotated by a predetermined amount to project the right arm forward. By executing the behavior control program, the appearance shape of the object arranged in the virtual space changes as the frame progresses.

  The virtual reality object personality control program is a program read by the virtual reality sub-control program, and controls the correspondence between the input and output for the virtual reality object arranged in the virtual space.

  For example, it is assumed that the context “Good morning” is acquired by the context acquisition unit 3381 of the voice analysis unit 3380. In this case, when the virtual reality object personality control program X is executed, the virtual reality object behavior control program A is read and executed in response to the input “Good morning”. On the other hand, when the virtual reality object personality control program Z is executed, the virtual reality object behavior control program N, not the virtual reality object behavior control program A, reads even if the same “Good morning” input is made. It is issued and executed. In this way, the virtual reality object personality control program executed for each virtual reality object placed in the virtual space can correspond to the input from the outside world and the output from the virtual reality object. The relationship is different. Accordingly, when the virtual reality object is regarded as a real person, the virtual reality object personality control program can be regarded as representing the personality of the virtual reality object.

The virtual reality object behavior control program and the virtual reality object personality control program can be programmed as a method for the corresponding virtual reality object.

  The virtual reality sub-control program is a program that is appropriately called by the virtual reality main control program, and corresponds to the provided virtual reality service. The virtual reality sub-control program reads the virtual reality object used for the virtual reality service being provided in the virtual space, or directly reads out the virtual reality object behavior control program according to a preset order, Control is performed to perform a predetermined behavior.

  The virtual reality main control program is a program corresponding to each virtual reality service, and is a program executed first when the provision of the virtual reality service is started. By executing the virtual reality main control program, the virtual space setting and environment setting for the virtual reality service corresponding to the program are performed. Based on information such as user state information, ambient state information, and audio information that is appropriately transmitted from the HMD device 300, the virtual reality main control program reads an appropriate virtual reality sub-control program and executes it by the virtual reality sub-control program The virtual reality world is presented to the user via the HMD device 300 in accordance with the content to be performed.

  The save data storage database 3500 stores user save data. Save data is data for starting a virtual reality service halfway. Some virtual reality services have story characteristics, some change in service contents as the virtual reality service is used, and others can acquire items, experience values, and the like in the virtual reality service. Therefore, the saved data is stored so that the user can store the portion that has been used so far and can restart the virtual reality service from the middle point where it was previously terminated.

  FIG. 14 is an example of a save data management file stored in the save data storage database 3500. The save data management file includes a save data management number uniquely assigned to each save data, a user name related to the save data, a user identification number identifying the user, a virtual reality service name related to the save data, the virtual data A virtual reality service identification number for uniquely identifying a real service, a storage address of the save data, and the like are stored in association with each other.

  The virtual reality service providing server 3300 includes a communication unit 1310, a virtual reality service provision management unit 1320, a display image generation processing unit 1350, an image analysis processing unit 2360, a virtual reality service control unit 3330, and a virtual reality object behavior control. Unit 3370, audio analysis processing unit 3380, and reproduction audio generation processing unit 3390.

  The virtual reality service provision management unit 1320 refers to the contractor information stored in the contractor information storage DB 3100 and manages the virtual reality service provided to the user. For example, the virtual reality service provision management unit 1320 performs a virtual reality virtual reality service start process and an end process based on a virtual reality service start request, a virtual reality service end request, and the like issued by the user.

  The contractor information includes conditions for the user to use the virtual reality service, information on the virtual reality service available to the user, and the like. That is, the contractor information includes the virtual reality service usage conditions. Based on these pieces of information, the virtual reality service provision management unit 1320 determines whether or not the virtual reality service can be provided, and performs a process of determining the virtual reality service to be provided.

  The virtual reality service provision management unit 1320 outputs to the virtual reality service control unit 3330 information identifying the virtual reality service to be provided and the user who uses the virtual reality service. In addition, the virtual reality service provision management unit 1320 notifies the user that the provision of the virtual reality service is started.

  The virtual reality service control unit 3330 controls the virtual reality service. Specifically, when the virtual reality service control unit 1320 determines to start providing the virtual reality service, the virtual reality service control unit 3330 displays the virtual reality content used for the virtual reality service in the virtual reality content storage database. Read from 3200.

  When the save data of the virtual reality service is stored in the save data storage database 3500, the virtual reality service control unit 3330 reads the save data of the user who provides the virtual reality service and indicates the save data as the save data. The virtual reality content of the part related to the service restart position to be read is read.

  Specifically, the virtual reality service control unit 3330 reads and executes a virtual reality main control program that controls the entire virtual reality service, thereby setting the virtual space and providing the HMD of the user who provides the virtual reality service. A port number used for communication with the apparatus 300 is set.

  Also, the virtual reality service control unit 3330 reads out and executes the virtual reality sub-control program, thereby reading out the virtual reality object from the virtual reality content storage database 3200 and placing the read virtual reality object at a predetermined position in the virtual space. Deploy.

  Here, the virtual reality service control unit 3330 may arrange a virtual reality object at a predetermined position in the virtual space, or an object around the user in the real space obtained by the image analysis processing unit 2360. Considering this arrangement relationship, the virtual reality object may be arranged at a position where no interference occurs. In this case, the virtual reality service control unit 3330 arranges model data such as an object existing around the user generated by the modeling processing in the image analysis processing unit 2360 in the virtual space, and the object indicated by the model data The arrangement position of the virtual reality object is determined so that interference does not occur due to the overlap. The virtual reality service control unit 3330 arranges the virtual reality object at the determined arrangement position in the virtual space.

  The virtual reality object behavior control unit 3370 controls the behavior of the virtual reality object arranged in the virtual space. When the virtual reality object arranged in the virtual space is an operable object, the virtual reality object behavior control unit 3370 executes a virtual reality object behavior control program for controlling the behavior of the virtual reality object, Causes the virtual reality object to perform a behavior corresponding to the program.

  In addition, the virtual reality service control unit 3330, when the virtual reality object arranged in the virtual space is a dynamic virtual reality object that returns a predetermined output in response to an input such as sound, The virtual reality object behavior control unit 3370 is caused to execute a virtual reality object personality control program that defines a causal relationship between the input and the behavior of the virtual reality object based on the input. The virtual reality object behavior control unit 3370 executes the virtual reality object personality control program with the analysis results in the image analysis processing unit 2360 and the voice analysis processing unit 3380 as inputs, thereby executing the virtual reality object behavior control program corresponding to the input. Identify. The virtual reality object behavior control unit 3370 reads the identified virtual reality object behavior control program from the virtual reality content storage DB 3200 and executes it, thereby controlling the behavior of the virtual reality object. That is, the virtual reality object behavior control unit 3370 deforms the shape of each virtual reality object arranged in the virtual space according to the virtual reality object to be executed.

  The display video generation processing unit 1350 generates a video of the virtual reality object arranged in the virtual space from the viewpoint position indicated by the user state information. When the virtual reality object is a dynamic virtual reality object, the display video generation processing unit 1350 performs the above-described operation at each frame timing of the virtual reality object whose appearance shape sequentially changes according to the control by the virtual reality object behavior control unit 3370. Video from the viewpoint position is generated sequentially. When the virtual reality object is a static virtual reality object, the display video generation processing unit 1350 causes the virtual reality service control unit 3330 to execute the virtual reality sub-control program and place the virtual reality object in the virtual space. The video from the viewpoint position at each frame timing is sequentially generated. In this case, if there is no change in the viewpoint of the user, there is no change in the video displayed on the display panels 151a and 151b of the HMD device 300, so the video generated at the previous frame timing is duplicated and the current timing is reproduced. Video.

  The voice analysis processing unit 3380 performs processing for analyzing the voice input by the voice input unit 370 of the HMD device 300. The voice analysis processing unit 3380 includes a voice level extraction unit 3381, an emotion element extraction unit 3382, and a context acquisition unit 3383.

  The voice level extraction unit 3381 identifies the voice level (volume level) of noise generated in the user's external environment based on the input ambient voice information. The voice level extraction unit 3381 identifies the voice level of the voice uttered by the user based on the input user voice information. The audio level extraction unit 3381 outputs the audio level of the external environment audio extracted from the ambient audio information to the reproduction audio generation processing unit 3390, and obtains the user's audio level extracted from the user audio information from the emotion element extraction unit 3382 and the context. The data are output to the unit 3383.

  The emotion element extraction unit 3382 extracts a user's emotion element based on the input user voice information. Specifically, the emotion element extraction unit 3382 performs a matching process with a user's voice waveform registered in advance to identify a state such as the user's emotions and extract it as an emotion element. Here, the emotion element extraction unit 3382 extracts an emotion element after performing amplification processing or the like on the voice included in the user voice information input based on the user voice level extracted by the voice level extraction unit 3381. Also good. The emotion element extraction unit 3382 outputs the extracted emotion element data to the virtual reality object behavior control unit 3370.

  The context acquisition unit 3383 acquires the context of the voice uttered by the user based on the input user voice information. The context acquisition unit 3383 performs amplification processing based on the audio level extracted by the audio level extraction unit 3381 on the audio data included in the input user audio information, and then the audio data is converted according to a predetermined audio analysis algorithm. Performs processing to extract the indicated context.

  For example, the context acquisition unit 3383 divides the sound data into unit sounds by performing a matching process between the sound unit data included in the sound database storing a plurality of sound units and the input sound data. The voice unit data included in the voice database is assigned a code such that the unit voice “I” is code No004 and the unit voice “ko” is code No021, for example. The context acquisition unit 3383 divides the input voice data into unit voices, thereby converting the input voice data into a code enumeration. The voice database is not shown.

  The context acquisition unit 3383 accesses a word / grammar database storing a plurality of word data and grammar data defining a grammatical structure, and stores some of a series of codes obtained by the speech analysis processing and the database. It extracts meaningful parts by matching with word data. The context acquisition unit 3383 acquires the context indicated by the input audio data based on the series of processes.

  For example, when the input voice data is a voice saying “I want to play with cats”, the context acquisition unit 3383 has “ne”, “ko”, “to”, “ta”, “wa”, “ “Mu”, “Re”, “Ta”, “I”, “N” and divided into unit sounds. Since a code is assigned to each unit voice, the context acquisition unit 3383 converts the voice data into a code enumeration such as “072/02/041/036/124/085/099/036/04043”. To do.

  The context acquisition unit 3383 performs a matching process between the code converted from the input voice data and a code indicating a word in the word / grammar database to convert the code into a list of words. For example, by performing the matching process, the context acquisition unit 3383 extracts “072021”, “034”, “036124085099”, and “0360004043” for each group, and “Ob: Cat, Of: and Va: Deflection. , Vc: taina "and organize it into a subject, object, predicate, particle, etc. The context acquisition unit 3383 determines that the input voice indicates a context of “I want to play with a cat” by the processing.

  Thus, the context acquired by the context acquisition unit 3383 is output to the virtual reality service provision management unit 1320, the virtual reality service control unit 3330, and the virtual reality object behavior control unit 3370, respectively.

  When the context of the audio data acquired by the context acquisition unit 3383 is a context requesting the start of a virtual reality service saying “I want to start Gaga Frontier”, the virtual reality service provision management unit 1320 indicates the context. To start providing virtual reality services. As described above, the virtual reality service provision management unit 1320 inputs a request from the user in the form of voice data, such as a virtual reality service start request, a virtual reality service end request, and a virtual reality service interruption request.

  Based on the audio data context acquired by the context acquisition unit 3383, the virtual reality service control unit 3330 reads the virtual reality service control program corresponding to the audio input from the virtual reality content storage DB 3200 and executes it.

  The virtual reality object behavior control unit 3370 is based on the context of the voice data acquired by the context acquisition unit 3383 and the emotion element data extracted by the emotion element extraction unit 3382. Control behavior.

  The playback sound generation processing unit 3390 generates sound data that is played back by the sound playback control unit 380 of the HMD device 300. The playback audio generation processing unit 3390 performs the processing on the audio data read from the virtual reality content storage DB 3200 by the virtual reality service control unit 3330 and the audio data generated by the virtual reality object controlled by the virtual reality object behavior control unit 3370. Then, gain adjustment processing based on the voice level extracted by the voice level extraction unit 3381 is performed. Based on the posture of the user's head indicated by the user status information received by the communication unit 1310, the playback audio generation processing unit 3390 identifies the position of the user's left and right ears, and the input audio data is a plurality of The audio data of each channel is generated by dividing into channels. The playback audio generation processing unit 3390 encodes the audio data divided into the plurality of channels by an encoding method such as CELP, generates audio encoded data, and outputs the audio encoded data to the communication unit 1310. The encoded speech data generated by the reproduced speech generation processing unit 3390 is transmitted from the communication unit 1310 to the HMD device 300 and subjected to speech decoding processing by the speech reproduction control unit 380 of the HMD device 300, and then the speech output unit Audio is output from each speaker 170.

  Next, the operation of the virtual reality service providing system 3000 will be described. FIG. 15 is a flowchart illustrating an example of a flow of a virtual reality service provision start process.

  The virtual reality service provision management unit 1320 performs a determination process for determining to provide the virtual reality service to the user (steps S301 to S304). Specifically, it is determined whether the virtual reality service start request is received by the communication unit 1310 (step S301). If the virtual reality service start request is not received by the communication unit 1320, the communication unit 1320 waits until the service start request is received.

  When the virtual reality service start request is received by the communication unit 1310, the virtual reality service provision management unit 1320 is designated by the requesting user in the virtual reality service start request based on the virtual reality service start request. It is determined whether provision of the virtual reality service can be started (step S302). Specifically, the virtual reality service provision management unit 1320 sets a virtual reality service usage condition that is a condition for using the virtual reality service stored in the virtual reality service usage condition storage DB 2400 or the contractor information storage DB 3100. With reference to this, it is determined whether it is possible to start providing the virtual reality service specified by the virtual reality service start request.

  As a result of the determination in step S302, when it is determined that the virtual reality service usage condition is not satisfied, the virtual reality service provision management unit 1320 provides the virtual reality service that is requested to start by the virtual reality service start request. A notification of refusal is made (step S303). Specifically, the virtual reality service provision management unit 1320 generates virtual reality service start rejection notification information indicating that the start of provision of the virtual reality service is rejected, and the request source of the virtual reality service start request from the communication unit 1310 Send to user. It is preferable that the virtual reality service start rejection notification information includes a virtual reality service provision rejection reason indicating a reason why provision of the virtual reality service cannot be started.

  On the other hand, as a result of the determination in step S302, when it is determined that the virtual reality service usage condition is satisfied, the virtual reality service provision management unit 1320 determines the start of provision of the virtual reality service based on the request (step S304). Specifically, the virtual reality service provision management unit 1320 instructs the virtual reality service control unit 3330 to start control of the requested virtual reality service.

  When the virtual reality service provision management unit 1320 determines to start providing the virtual reality service, the virtual reality service control unit 3330 reads the virtual reality content related to the virtual reality service from the virtual reality content storage DB 3200 (step S305). .

  The virtual reality service control unit 3330 sets a virtual space for the provided virtual reality service (step S306). Usually, since the virtual reality world presented to the user is within the range of the user's field of view, it is preferable to set the size to about 10 m square. However, when the user moves and appreciates the virtual reality world, it is preferable to reserve a virtual space of about 100 mm square in advance. In this way, the required virtual space size varies depending on the contents of the virtual reality service. Therefore, the virtual reality service control unit 3330 is preferably configured to set the virtual space based on the virtual reality service determined by the virtual reality service provision management unit 1320. For example, when providing a virtual reality service of S001, a virtual space of 50 m square is set, and when providing a virtual reality service of S025, a virtual space of 1 km in the X direction, 1 km in the Y direction, and 20 m in the Z direction is set. It can be set as the structure to do. The size of the virtual space required may be stored in the virtual reality content storage DB 3200 as a part of the virtual reality content control data. The virtual reality service control unit 3330 reads the virtual reality content and sets the virtual space with reference to the control data included in the virtual reality content. Setting a virtual space that is unnecessarily large consumes unnecessary resources, but in a virtual space that is too small compared to the virtual reality service, the virtual reality object used to provide the virtual reality service may not be properly placed Remains. From such a viewpoint, it is preferable that the size of the virtual space is determined with reference to control data stored in advance.

  Next, the virtual reality service control unit 3330 sets environment parameters used for providing the virtual reality service (step S307). The environment parameter is information that defines control of communication performed between the HMD device (virtual reality presentation device) 300 and the virtual reality service providing server 3300 in order to present the virtual reality world to the user.

  For example, the virtual reality service control unit 3330 sets an encryption key for encrypting communication between the HMD device 300 and the virtual reality service providing server 3300 as the environment parameter. In the HMD device 300, since the surrounding state information includes image data obtained by photographing a scene around the user, there is a possibility that a privacy related matter is captured. Therefore, in order to encrypt communication between the two devices, the virtual reality service control unit 3330 generates or reads out an encryption key for communication from the virtual reality content storage DB 3200. The encryption key may be a common key or a public key.

  Further, for example, the virtual reality service control unit 3330 sets a port used for communication between the HMD device 300 and the virtual reality service providing server 3300 as an environment parameter. As the port, a plurality of different ports may be assigned according to information to be transmitted / received, or one port may be used in common.

  Further, for example, the virtual reality service control unit 3330 sets, as environment parameters, a transmission interval, a data format, and the like of user status information and ambient status information that are periodically transmitted from the HMD device 300 to the virtual reality service providing server 3300. . In a virtual reality service in which a virtual reality object moves greatly in a virtual space, it is necessary to frequently update the real world situation so that the moving virtual reality object does not overlap with an object in the real world. . Further, when the movement of the user's head is small, past information can be reused for the user state information and the surrounding state information. In this way, the acquisition interval (generation interval), transmission interval, data format, and the like of information periodically transmitted from the HMD device 300 to the virtual reality service presentation server 3300 are set as environment parameters.

  Further, for example, the virtual reality service control unit 3330 uses, as an environment parameter, information for identifying a frame rate of a video generated by the virtual reality service providing server 3300, a coding method used in the video coding process, The number of audio channels to be used, information for identifying the audio encoding method, and the like are set. In this way, the virtual reality service control unit 3330 sets parameters regarding information periodically transmitted from the virtual reality service providing server 3300 to the HMD device 300 as environment parameters.

  The virtual reality service control unit 3330 is provided for each processing unit performed to provide a virtual reality service such as a video generation processing unit 1350, an image analysis processing unit 2360, an audio analysis processing unit 3380, and an audio reproduction processing unit 3390. Then, control for executing the responsible process is performed according to the set environmental parameter (step S308). For example, based on the control, the video generation processing unit 1350 obtains a two-dimensional image by rendering the virtual reality object arranged in the virtual space according to the frame rate set by the virtual reality service control unit 3330.

  When the setting of the environment parameter is completed, the virtual reality service provision management unit 1320 generates virtual reality service start notification information and transmits the virtual reality service start notification information to the HMD device 300 from the communication unit 1320 to notify the start of the virtual reality service. (Step S309). The virtual reality service start notification information includes the set environmental parameters.

  Note that the service start determination process is not limited to the case where a service start request is received as a trigger, but scheduling based on resources available to the server among users who can currently use the service. It is also possible to adopt a configuration in which Steps S306 and S307 may be out of order.

  FIG. 16 shows main control processing of the virtual reality service when provision of the virtual reality service is determined by the virtual reality service provision management unit 1320. When the virtual reality service provision management unit 1320 notifies the HMD device 300 of the start of the virtual reality service, the HMD device 300 periodically transmits user status information and ambient status information according to the environmental parameters. The communication unit 1310 receives the user state information and the surrounding state information (step S310). Normally, the reception frequency of the user status information is set by the environmental parameter so as to be higher than the reception frequency of the surrounding state information. In addition, the voice input by the voice input unit 370 of the HMD device 300 is also transmitted as voice information periodically or when voice of a predetermined voice level or higher is input, and the communication unit 1310 transmits the voice information. Receive.

  The image analysis processing unit 2360 performs a predetermined analysis process based on the forward view image and the image data of the surrounding view image transmitted as the surrounding state information, and identifies the positional relationship of the actual objects around the user. Then, a modeling process for modeling the identified object and generating model data is performed (step S311). For example, since the line-of-sight camera that acquires the forward scenery image is arranged at a position corresponding to the left eye and the right eye, it is possible to specify the distance to the object by comparing the images captured by these two cameras Become. The model data includes polygon data that defines the appearance shape of an actual object and coordinate data that indicates the position of the object, and more preferably is represented by model label information that identifies the model data and the model data. It includes object label information for identifying an actual object, object type information indicating the type of the actual object, and the like. The model data of the real object generated by the image analysis processing unit 2360 is output to the virtual reality service control unit 3330.

  The virtual reality service control unit 3330 places the model data of the real object around the user output from the image analysis processing unit 2360 in the virtual space set in step S306 (step S312). Since the model data includes coordinate data indicating a relative position from the user's position, the image analysis processing unit 2360 has a position obtained by adding the coordinate data to the user position coordinate indicating the user's position in the virtual space. Arrange the model data of the real object obtained by the modeling process. Note that the user position coordinates in the virtual space are appropriately changed based on the user state information. In addition, as the user moves, the shooting range of the image input to the image analysis processing unit 2360 changes, so that model data of an actual object in the range that has not been captured in the image until now is sequentially generated. The virtual reality service control unit 3330 sequentially updates the model data of the real objects arranged in the virtual space based on the model data sequentially generated by the image analysis processing unit 2360.

  The virtual reality service control unit 3330 determines whether to arrange a new virtual reality object in the virtual space (step S313). The virtual reality service control unit 3330 determines whether to arrange the virtual reality object in the virtual space based on various control programs included in the virtual reality content read from the virtual reality content storage DB 3200.

  When it is determined to arrange a new virtual reality object, the virtual reality service control unit 3330 reads the virtual reality object from the virtual reality content storage DB 3200 and arranges it in the virtual space (step S314). Here, the virtual reality service control unit 3330 arranges the virtual reality object at a position that does not overlap the model data of the real object arranged in the virtual space in the virtual space. More preferably, an arrangement condition is set for the virtual reality object, and the virtual reality service control unit 3330 arranges the virtual reality object in the virtual space according to the arrangement condition. For example, the virtual reality object is a human object, and the “placement condition 1: the area overlapping with other objects is 1% or less, the placement condition 2: place in contact with the floor”. Is set, the virtual reality service control unit 3330 contacts the upper part of the floor model data with an overlap state of 1% or less among the model data of the real objects arranged in the virtual space. Arrange virtual reality objects in state.

  On the other hand, if it is determined in step S313 that no new virtual reality object is to be arranged, step S314 is skipped and the process proceeds to the next step S315.

  The virtual reality service control unit 3330 determines whether to control the behavior of the virtual reality object arranged in the virtual space (step S315). The virtual reality service control unit 3330 executes the virtual reality sub-control program, inputs the processing results in the image analysis processing unit 2360 and the audio analysis processing unit 3380, and determines the virtual reality object being arranged based on the processing results. Process to determine whether to control the behavior. Note that the virtual reality service control unit 3330 may sequentially control the behavior of the virtual reality object in accordance with the currently executed virtual reality sub-control program, regardless of the input of the processing result.

  If it is determined in step S315 that the behavior of the virtual reality object is to be controlled, the virtual reality object control unit 3370 controls the behavior of the virtual reality object (step S316). Specifically, when the virtual reality service control unit 3330 determines that the behavior of the virtual reality object is to be controlled in step S316, the virtual reality service control unit 3330 stores the virtual reality object personality control program from the virtual reality content storage DB 3200, The virtual reality object behavior control program read by the virtual reality object personality control program is read and sent to the virtual reality object control unit 3370. The virtual reality object control unit 3370 executes the virtual reality object personality control program, and selects the virtual reality object behavior control program to be executed by using the processing results in the image analysis processing unit 2360 and the sound analysis processing unit 3380 as inputs. The virtual reality object control unit 3370 executes the selected virtual reality object behavior control program so that the position of the virtual reality object in the virtual space, the position and angle of the joint point of the virtual reality object, and the surface of the virtual reality object are displayed. Change the pasted texture. As described above, the virtual reality object control unit 3370 controls the behavior of the virtual reality object according to the virtual reality object behavior control program.

  If it is determined in step S315 that the behavior of the virtual reality object is not to be controlled, the process proceeds to the next step S317 without performing the behavior control process in step S316.

  Next, the virtual reality service control unit 3330 determines whether or not to delete the model data of the real object being arranged in the virtual space from the virtual space (step S317). The virtual reality service control unit 3330 determines whether or not the model data of an object that does not affect the presentation of virtual reality or the model data of another object because the relative distance from the user is greater than or equal to a predetermined reference value. It is determined whether there is model data that satisfies a predetermined deletion criterion, such as model data that has caused interference, or model data of an object that has disappeared from the user's surroundings due to movement.

  The virtual reality service control unit 3330 deletes the model data of the real object determined to satisfy the predetermined deletion criterion from the virtual space (step S318). On the other hand, if there is no model data of an actual object determined to have a predetermined deletion criterion, the process proceeds to the next step S319 without performing the deletion process.

  Next, the virtual reality service control unit 3330 determines whether or not to delete the virtual reality object being arranged in the virtual space from the virtual space (step S319). The virtual reality service control unit 3330 does not affect the presentation of the virtual reality because the virtual reality object that has become unnecessary or the relative distance from the user is more than a predetermined reference value as the virtual reality service progresses. It is determined whether there is a virtual reality object that satisfies a predetermined deletion criterion, such as a virtual reality object, a model data of a real object, or a virtual reality object that has caused interference with another virtual reality object.

  The virtual reality service control unit 3330 deletes the virtual reality object determined to satisfy the predetermined deletion criterion from the virtual space, and returns to step S310 (step S320). On the other hand, if there is no virtual reality object determined to have the predetermined deletion criterion, the process returns to the next step S310 without performing the deletion process.

  As described above, the virtual reality service control unit 3330 updates the virtual space by repeatedly adding, transforming, and deleting the model data of the virtual reality object and the real object to be arranged in the set virtual space.

  Next, a process for generating a video displayed on the HMD device 300 will be described. FIG. 17 is a flowchart showing the flow of the video generation process.

  When the user status information is newly received by the communication unit 1310 (step S321), the display video generation processing unit 1350 controls the viewpoint in the virtual space based on the received latest user status information (step S322). . Specifically, the display video generation processing unit 1350 uses the coordinates in the virtual space corresponding to the position of the left eye and the position of the right eye of the user as the viewpoint position based on the acceleration information and the tilt information included in the user state information. Further, the direction in which the user's head is facing is obtained as the line-of-sight direction. On the other hand, when the user status information is not received, the viewpoint control at this timing is skipped.

  On the other hand, when the ambient condition information is newly received by the communication unit 1310 (step S323), the display video generation processing unit 1350 controls the light source in the virtual space based on the received latest ambient condition information (step S323). S324). Specifically, the display video generation processing unit 1350 sets the position of the light source and the light intensity arranged in the virtual space. On the other hand, when the ambient condition information is not received, the light source control at this timing is skipped.

  Next, the display video generation processing unit 1350 generates a video from the viewpoint controlled in step S321 of the virtual reality object arranged in the virtual space in the irradiation environment from the light source controlled in step S323 (step S321). S325). Specifically, the display video generation processing unit 1350 projects the virtual reality object in the virtual space onto the image plane arranged in front of the visual line direction of the viewpoint position controlled in step S321. Here, the display video generation processing unit 1350 projects only the virtual reality object, not the model data of the real object on the image plane. Here, when there is model data of a real object between the virtual reality object and the viewpoint position, the display video generation processing unit 1350 does not project the portion hidden in the model data in the virtual reality object. Process. Further, the display video generation processing unit 1350 performs a shading process on the projected virtual reality object based on the irradiation environment from the light source controlled in step S324. By performing these processes, the display image generation processing unit 1350 generates two images to be displayed on the display panels 151a and 151b of the HMD device 300.

  Next, the display video generation processing unit 1350 generates video encoded data by encoding the generated video using a predetermined compression encoding method (step S326), and the communication unit 1310 generates the generated video code. The converted data is transmitted to the HMD device 300 (step S327). When communication between the HMD device 300 and the virtual reality service providing server 3300 is encrypted, the encoded video data generated in step S326 is encrypted and transmitted.

  By repeating the processes in steps S321 to S327 as one frame process, video images of virtual reality objects are sequentially displayed on the display panels 151a and 151b.

  Next, a process for generating sound played back by the HMD device 300 will be described. FIG. 18 is a flowchart showing the flow of the sound generation processing.

  When the user state information is newly received by the communication unit 1310 (step S331), the reproduction sound generation processing unit 3390 controls the position of the ear in the virtual space based on the received latest user state information (step S331). S332). Specifically, the reproduction sound generation processing unit 3390, based on the acceleration information and the inclination information included in the user state information, the sound collection position (sound collection position coordinates) in the virtual space corresponding to the position of the user's ear. Ask for. On the other hand, when the user status information is not received, the control of the ear position (sound collection position) at this timing is skipped.

  On the other hand, when ambient audio information is newly received by the communication unit 1310 (step S333), the playback audio generation processing unit 3390 determines the audio level such as noise around the user based on the received latest ambient audio information. Is calculated, and the sound level in the sound reproduced by the sound reproduction control unit 380 of the HMD device 300 is set (step S334). On the other hand, when ambient audio information is not received, audio level setting at this timing is skipped.

  The virtual reality service control unit 3330 determines whether to reproduce the sound based on the processing results in the image analysis processing unit 2360 and the sound analysis processing unit 3380 (step S335). If it is determined that playback is not performed at this timing, the process returns to step S331 and waits until the next audio playback determination timing.

  On the other hand, when it is determined that the voice is to be reproduced, the virtual reality service control unit 3330 determines whether to generate the voice to be reproduced by voice synthesis (step S336). When speech synthesis is not performed, the virtual reality service control unit 3330 reads audio data stored in advance in the virtual reality content storage DB 3200 as a part of the virtual reality content and outputs the audio data to the reproduction audio generation processing unit 3390 (Step S390). S337).

  On the other hand, when it is determined to synthesize the voice, the virtual reality service control unit 3330 reads the voice basic data stored in advance as part of the virtual reality content in the virtual reality content storage DB 3200 and sends it to the playback voice generation processing unit 3390. The sending / reproducing voice generation processing unit 339 generates synthesized voice data by synthesizing the voice basic data in accordance with the context obtained by itself or the context generated by another context generation processing unit (step S338).

  The playback sound generation processing unit 3390 performs gain adjustment processing on the sound data output from the virtual reality service control unit 3330 or the sound data generated by the sound synthesis process based on the sound level set in step S334 ( Step S339).

  The reproduction sound generation processing unit 3390 performs a distribution process of distributing the gain-adjusted sound data to a plurality of channels based on the ear position (sound collection position) controlled in step S332 (step S340). Through the distribution process, the reproduction audio generation processing unit 3390 generates channel-specific audio data that is reproduced on each of the plurality of audio channels included in the HMD device 300.

  The reproduction audio generation processing unit 3390 converts the audio data distributed to each channel to H.264. Encoding processing is performed in accordance with a speech encoding scheme such as 729, speech encoded data is generated, and output to the communication unit 1310 (step S341).

  The communication unit 1310 transmits the audio encoded data output from the reproduction audio generation processing unit 3390 to the HMD device 300, and returns to step S331 (step S342). The transmitted encoded audio data is decoded by the audio reproduction control unit 380 of the HMD device 300, and the audio output unit 390 converts the audio data of each channel into an analog audio signal by DA conversion processing. It is converted and output as audio to the outside.

  Next, the flow of the virtual reality service interruption process will be described. FIG. 19 is a flowchart showing the flow of the virtual reality service interruption process.

  The virtual reality service provision management unit 1320 determines whether to interrupt the currently provided virtual reality service (step S351). For example, the virtual reality service provision management unit 1320 provides the current provision when a virtual reality service suspension request is received or when a virtual reality service provision suspension condition preset as a condition for suspending the virtual reality service is satisfied. It is determined that the virtual reality service is interrupted.

  If the virtual reality service provision management unit 1320 determines in step S351 that the provision of the virtual reality service is to be suspended, the virtual reality service provision management unit 1320 generates virtual reality service suspension notification information for notifying that the provision of the virtual reality service is to be suspended, and transmits the virtual reality service suspension notification information to the communication unit 1310. Output (step S352).

  The communication unit 1310 transmits the virtual reality service interruption notification information to the HMD device 300 (step S353). The virtual reality service interruption notification information is received by the communication unit 130 of the HMD device 300 and sent to the virtual reality presentation control unit 260. The virtual reality presentation control unit 260 performs control to stop the function of each unit based on the interruption notification information. For example, the display control unit 140 interrupts the video display control, and the audio reproduction control unit 380 interrupts the audio reproduction control. However, the virtual reality service providing server 3300 does not interrupt the functions of the user status detection unit 110 and the ambient status detection unit 120 so that the virtual status service providing server 3300 can track the status of the user who is interrupting the service and the surrounding status. A configuration is preferable. Even in the case where provision of the virtual reality service is interrupted, in the HMD device 300, the user status detection unit 110 detects the user status and generates user status information, and the virtual reality service provision server from the communication unit 130 3300. Even when provision of the virtual reality service is interrupted, in the HMD device 300, the surrounding state detection unit 120 detects the surrounding state of the user and generates the surrounding state information. It is transmitted to the virtual reality service providing server 3300.

  The virtual reality service provision management unit 1320 sends a processing interruption instruction to each processing unit in its own server (step S354). The display video generation process and the playback audio generation process are stopped based on the instruction, thereby suppressing the resource consumption of the own server. However, step S354 may be skipped and each process continued so that the subsequent virtual reality service can be resumed without causing a time lag. In this case, although video data and audio data are transmitted to the HMD device 300, video display and audio reproduction are not performed by control performed by the virtual reality presentation control unit 260 of the HMD device 300 based on the service interruption notification information. Therefore, the virtual reality world is not presented to the user.

  In a state where the provision of the virtual reality service is suspended, the virtual reality service provision management unit 1320 determines whether or not to resume the provision of the suspended virtual reality service (step S355). The virtual reality service provision management unit 1320 determines to resume provision of the virtual reality service when the virtual reality service resumption request is received by the communication unit 1310 or when the virtual reality service provision interruption condition is resolved.

  The virtual reality service provision management unit 1320 instructs each processing unit in its own server to resume processing, thereby resuming the processing stopped in step S354 (step S356).

  The virtual reality service provision management unit 1320 generates virtual reality service restart notification information indicating that provision of the virtual reality service is resumed, and outputs the virtual reality service restart notification information to the communication unit 1310 (step S357).

  The communication unit 1310 transmits the virtual reality service restart notification information to the HMD device 300, and returns to Step S351 (Step S358). The virtual reality service restart notification information is received by the communication unit 130 of the HMD device 300 and sent to the virtual reality presentation control unit 260. The virtual reality presentation control unit 260 restarts the function of each unit based on the restart notification information. For example, the display control unit 140 is caused to resume video display control, and the audio reproduction control unit 380 is caused to resume audio reproduction control.

  Next, the flow of the virtual reality service end process will be described. FIG. 20 is a flowchart showing the flow of the virtual reality service end process.

  The virtual reality service provision management unit 1320 determines whether to end the currently provided virtual reality service (step S361). For example, the virtual reality service provision management unit 1320 terminates the currently provided virtual reality service when a virtual reality service termination request is received or when the current state of the user does not satisfy the virtual reality service usage conditions. Judge that.

  If the virtual reality service provision management unit 1320 determines in step S361 that the provision of the virtual reality service is to be terminated, the virtual reality service provision management unit 1320 generates virtual reality service termination notification information for notifying that the provision of the virtual reality service is to be terminated, and sends it to the communication unit 1310. Output (step S362).

  The communication unit 1310 transmits the virtual reality service end notification information to the HMD device 300 (step S363). The virtual reality service end notification information is received by the communication unit 130 of the HMD device 300 and sent to the virtual reality presentation control unit 260. Based on the end notification information, the virtual reality presentation control unit 260 performs control to stop the function of each unit related to the virtual reality presentation and end the virtual reality presentation. For example, the display control unit 140 interrupts the video display control, and the audio reproduction control unit 380 ends the audio reproduction control. Also, the virtual reality presentation control unit 260 causes the user state detection unit 110 to end the user state detection process, and causes the surrounding state detection unit 120 to end the surrounding state detection process. Specifically, power supply to the tilt sensor 111 and the acceleration sensor 112 is cut off, and power supply to the front camera 121 and the surrounding camera 122 is cut off.

  The virtual reality service provision management unit 1320 sends a processing end instruction to each processing unit in its own server (step S364). Based on the termination instruction, processing of each unit related to provision of the virtual reality service is terminated, and the resources of the own server are released. Further, the virtual space set for the virtual reality service is also released.

  The virtual reality service provision management unit 1320 stores the progress of the virtual reality service so far in the save data storage DB 3500 as save data, and ends the provision of the virtual reality service (step S365). For example, information regarding virtual items and experience values acquired by the user in the virtual reality world, progress on the story of the virtual reality service, and the like are stored as saved data.

  21 shows a data structure of various requests transmitted from the HMD device 300 to the virtual reality service providing server 3300, and FIG. 22 shows a data structure of various notification information transmitted from the virtual reality service providing server 3300 to the HMD device 300. Each is shown.

  Each request (information) includes the address of the device 300 as a transmission source address and the address of the virtual reality service providing server 3300 as a transmission destination address. On the other hand, each notification information includes the address of the own server 3300 as a transmission source address and the address of the HMD device 300 as a transmission destination address.

  Each request includes request type information indicating the request content and request identification information which is a management number assigned to identify the request. For example, the request type information is 2-bit information, “00” is a virtual reality service start request, “01” is a virtual reality service suspension request, “10” is a virtual reality service restart request, and “11” is virtual. Each real service end request is shown.

  On the other hand, each notification information includes notification type information indicating the content of notification and the request identification information for identifying which request is a response. For example, the notification type information is above 3 bits, “000” is virtual reality service start notification information, “001” is virtual reality service start rejection notification information, “010” is virtual reality service interruption notification information, “011” represents virtual reality service restart notification information, and “100” represents virtual reality service end notification information. Note that when the virtual reality service providing server 3300 voluntarily transmits not the information notified as a response to the request sent from the HMD device 300, the request identification information is not included in each notification information.

  The virtual reality service start request includes a user ID for identifying the user associated with the request, a password for starting the virtual reality service, and virtual reality service identification information for identifying the virtual reality service that requests the start. Upon receiving the service start request, the virtual reality service provision management unit 1320 compares the user ID and password set included in the contractor information stored in the contractor information storage DB 3100 with a request from an appropriate user. Determine if there is. Also, the virtual reality service provision management unit 1320 determines whether the user can use the virtual reality service identified by the virtual reality service identification information included in the service start request. When these determinations are properly passed, provision start processing of the virtual reality service proceeds.

  The virtual reality service start notification information includes virtual reality service identification information for identifying a virtual reality service to be provided, and each communication generated between the server 3300 and the HMD device 300 in the virtual reality service. A virtual reality service management number being provided for use in distinguishing it from the environment setting information for setting various environment parameters. The environment setting information includes an encryption key for encrypting communication between the local server 3300 and the HMD device 300, a used port number for designating a port number used in the communication, and the local server 3300 to the HMD device 300. Frame rate information for specifying the frame rate of video data to be transmitted, HMD side information transmission frequency designation information for designating the transmission frequency of user status information and surrounding situation information to be transmitted from the HMD device 300 to the own server 3300, and the own server Video encoding method specifying information and audio encoding method specifying information for specifying video and audio encoding methods to be transmitted from 3300 to the HMD device 300 are included.

  In the HMD device 300 that has received the virtual reality service start notification information, the virtual reality presentation control unit 260 sets each parameter based on the environment setting information included in the service start notification information, and the HMD device 300 side Each process is started.

  In addition to the transmission source address, transmission destination address, notification type information, and request identification information described above, the virtual reality service start rejection notification information may include a reason for rejecting the provision start of the virtual reality service as additional information. good.

  The virtual reality service interruption request, the virtual reality service restart request, and the virtual reality service termination request include the above-described virtual reality service management numbers that are allocated to the virtual reality services related to the suspension, restart, and termination, respectively. The virtual reality service provision management unit 1320 that has received the virtual reality service interruption request, the virtual reality service restart request, and the virtual reality service termination request, respectively, manages the virtual reality managed by the virtual reality service management number being provided included in the request. Executes service interruption processing, resumption processing, and termination processing.

  The virtual reality service interruption notification information, the virtual reality service resumption notification information, and the virtual reality service end notification information include provided virtual reality service identification information for identifying the virtual reality service related to interruption, resumption, and termination, and for the virtual reality service, respectively. And the virtual reality service management number being provided assigned to the. In the HMD device 300 that has received the service interruption notification information, service resumption notification information, and service end notification information, the virtual reality presentation control unit 260 suspends, resumes, and ends the presentation of the virtual reality related to the designated virtual reality service. Control.

  As described above, according to the virtual reality service providing system 3000 according to the third embodiment, the virtual reality object behavior control unit that controls the behavior of the virtual reality object arranged in the virtual space is provided to the user. Since the virtual reality object represented in the virtual reality world is operated, a richer virtual reality world can be presented to the user.

  In the above description, the HMD device 300 has been described with respect to the configuration in which the light emitted from the light source 352 is reflected toward the user side and passes through a display such as an LCD, but the HMD device of the present invention is described. Is not limited to this configuration. For example, the display unit 150 may include a projection mechanism instead of the LCD 154.

  FIG. 23 shows a cross-sectional view of the front panel 102 in the HMD device 300 having the display unit 150 of the projection mechanism. A projection mechanism is arranged as a display unit 150 inside a center frame 101 c located on the upper part of the front panel 102.

  After the white light emitted from the light source 152 disposed above the center frame 101c is collected by the lens group 155a, the polarization component is adjusted in one direction by the polarizing converter (PLC) 155b. The light is reflected in the vertical direction by the first reflecting mirror 155c. In the white light reflected by the first reflecting mirror 155c, only the red light component (R component) is reflected in the horizontal direction by the R reflecting dichroic mirror 155d, and the light of the other components is transmitted through the R reflecting dichroic mirror 155d. .

  From the light transmitted through the R reflecting dichroic mirror 155d, only the green light component (G component) is reflected in the horizontal direction by the G reflecting dichroic mirror 155e, and the remaining blue light component (B component) passes through the G reflecting dichroic mirror 155e. To Penetrate.

  The B component light transmitted through the G reflecting dichroic mirror 155e is reflected in the horizontal direction by the second reflecting mirror 155f, further reflected vertically upward by the third reflecting mirror 155g, and then transmitted through the first liquid crystal panel 155h. The light enters the prism 155j.

  On the other hand, the R component light reflected by the R reflecting dichroic mirror 155d is reflected in the vertical direction by the fourth reflecting mirror 155k, passes through the second liquid crystal panel 155m, and then enters the cross dichroic prism 155j. The G component light reflected by the G reflecting dichroic mirror 155e passes through the third liquid crystal panel 155n and then enters the cross dichroic prism 155j.

  The cross dichroic prism 155j combines the three light components R, G, and B that have been separated and passed through the liquid crystal panel, and reflects them in the horizontal direction. The combined light output from the cross dichroic prism 155j is reflected in the vertical direction by the galvano mirror 155p, and is irradiated to the front panel 102 from the irradiation window 155q. The galvanometer mirror 155p reflects the combined light output from the cross dichroic prism 155j while scanning in the horizontal direction (y direction).

  A transmissive light guide plate 153 is disposed below the irradiation window 155q in the front panel 102, and light reflected by the light guide plate 153 in the horizontal direction again enters the user's eyes. Since the light is operated in the horizontal direction by the galvanometer mirror 155p, the light reflected by the light guide plate 153 is scanned in the vertical direction (z direction).

  Note that each member in FIG. 23 has a configuration that is long in the x direction, that is, the direction perpendicular to the paper surface, and collectively displays a line in the x direction in the image for the right eye. The row of images is scanned in the y direction by the galvano mirror 155p and sequentially displayed, so that a two-dimensional image in the xz plane is displayed on the light guide plate 153 of the front panel 102. Similarly, the same projection mechanism is also arranged at the upper part of the left eye, and an image for the left eye is displayed on the light guide plate 153. As described above, the display unit is formed by the right-eye projection mechanism and the left-eye projection mechanism stored in the light guide plate 153 and the center frame 101c, and the light guide plate 153 functions as a display panel. Note that the left-eye projection mechanism and the right-eye projection mechanism may be shared.

  In addition, line-of-sight cameras 121a and 121b are arranged on the center frame 101c. The line-of-sight camera 121a includes a lens group 121a-1 that collects external light, a diaphragm mechanism 121a-2 that restricts the collected light, and a solid-state imaging device 121a-3 such as a CCD or a CMOS. After being converted from light to an analog electrical signal by the solid-state imaging device 121a-3, after being converted to a digital signal by an AD converter, a predetermined correction processing and encoding processing are performed by a subsequent DSP (Digital Signal Processor) Then, it is transmitted from the communication unit 130 to the virtual reality service providing server 3300 as the surrounding state information.

  In the above description, the case where the HMD device 300 is a virtual reality presentation device including a transmissive display is described, but the present invention is not limited to this. For example, like the HMD device 300b shown in FIG. 24, the front panel 102 is a shielded panel, and the line-of-sight cameras 121a and 121b are arranged outside and the display panels 151a and 151b are arranged inside, respectively. Also good. However, in this case, since the user's field of view is shielded by the front panel 302, the images taken by the line-of-sight cameras 121a and 121b are displayed on the display panels 151a and 151b, so that the user sees the front view. Take a configuration that can.

  FIG. 25 is a block diagram showing a configuration of the HMD device 300b shown in FIG. In the HMD device 300b, the display control unit 140 includes a video decoding processing unit 141 and a video composition processing unit 142.

  The video decoding processing unit 141 acquires a video of the virtual reality object by performing a decoding process on the encoded video data generated by the virtual reality service providing server 3300.

  The video synthesis processing unit 142 generates a virtual reality video by synthesizing the video of the virtual reality object acquired by the video decoding processing unit 141 with the forward scene image captured by the line-of-sight camera 121. Specifically, the video composition processing unit 142 synthesizes the front scene image captured by the right gaze camera 121a and the video of the virtual reality object generated from the right viewpoint, and the front scene captured by the left gaze camera 121b. Two images are generated by combining the image and the image of the virtual reality object generated from the left viewpoint. The video composition processing unit 142 displays the two generated videos on the right display panel 151a and the left display panel 151b, respectively.

  In addition, when taking the said form, it is good also as a structure where ambient condition information is not transmitted to the virtual reality service provision server 3300. In this case, the HMD device 300b may be configured as shown in FIG. In the HMD device 300 b shown in FIG. 26, the surrounding state detection unit 120 includes a line-of-sight camera 121 and a line-of-sight auxiliary camera 123.

  When the thickness of the front panel 102 is increased, the mounting property is lowered. Therefore, the front panel 102 is preferably as thin as possible. On the other hand, if the thickness of the front panel 102 is thin, a sufficient optical path cannot be secured in the visual line camera 121, so that a relatively large distortion occurs in the captured image. Therefore, a line-of-sight auxiliary camera 123 is separately arranged in front of the front panel 102.

  The display control unit 140 includes a video decoding processing unit 141, a video composition processing unit 142, a visual field video generation processing unit 143, and a boundary specification processing unit 144. The video decoding processing unit 141 acquires the video of the virtual reality object by performing a decoding process on the encoded data of the video of the virtual reality object generated by the virtual reality service providing server 3300.

  The visual field image generation processing unit 143 corrects distortion by combining the image captured by the visual line camera 121 and the video captured by the visual axis auxiliary camera 123, and generates a visual field image that is an image serving as the user's visual field. The field-of-view image is an image of a scene that is substantially the same as the field of view that the user can see with the naked eye when the HMD device 300 is not attached. The view image generation processing unit 143 generates a view image for the left eye and a view image for the right eye.

  The boundary identification processing unit 144 performs boundary determination processing on the user's visual field image generated by the visual field image generation processing unit 143 to identify the boundary and distance of each object shown in the video. The boundary identification processing unit 144 determines that the portion is the boundary of the object when the difference between the color difference signals of adjacent pixels or blocks exceeds a predetermined reference value, and each of the two left and right view images The distance to the object is calculated from the difference between the positions where the same object is displayed.

  The video composition processing unit 142 includes a left-eye virtual reality object video and a right-eye virtual image acquired by the video decoding processing unit 141 for the left-eye visual image and the right-eye visual image generated by the visual image generation processing unit 143, respectively. A process of synthesizing the video of the real object is performed.

  Here, the video composition processing unit 142 performs correction processing for deleting a part of the virtual reality object along the boundary based on the boundary position of the object included in the visual field image specified by the boundary specifying processing unit 144, A video for display is generated by combining the video of the virtual reality object on which the correction processing has been performed with the visual field video. This is because a video of a virtual reality object is generated by the virtual reality service providing server 3300, whereas a visual field image is generated by the HMD device 300, and thus there may be a shift in the composition.

  In the case of this configuration, it is more preferable that the distance information indicating the distance to the virtual reality object is transmitted from the virtual reality service providing server 3300 together with the video data. In this case, the video composition processing unit 142 performs correction processing for deleting a part of the video of the virtual reality object at the boundary of the object on the user side with respect to the distance of the virtual reality object indicated by the distance information. The view image and the virtual reality object image are combined.

  For example, assuming that the virtual reality objects A, B, and C are included in the video generated by the virtual reality service providing server 3300, the distance from the user to the virtual reality objects A, B, and C is 2.1 m, respectively. Suppose that it was 4.0 m and 4.8 m. On the other hand, by the boundary identification processing in the boundary identification processing unit 144, the objects P, Q, and R at the distances of 1.0 m, 3.1 m, 4.5 m, 5.2 m, and 8 m from the position of the user in the view image, respectively. , S, and T are identified. In this case, since only the object P is closer to the user than the virtual reality object A, a process of deleting the image of the portion overlapping the object P from the image of the virtual reality object A is performed. Similarly, since the objects P and Q are closer to the user than the virtual reality object B, a process of deleting the image of the portion overlapping the objects P and Q from the image of the virtual reality object B is performed. Similarly, a process of deleting a video of a portion that overlaps the objects P, Q, and R from the video of the virtual reality object C is performed.

  As described above, the display control unit 140 corrects the video of the virtual reality object generated by the virtual reality service providing server 3300 based on the object that enters the user's field of view, and displays the video of the virtual reality object after the correction process. By performing control to display each on the display panels 151a and 151b, the virtual reality object is assimilated with the scenery around the user in a more natural state, so that a virtual reality world with high reality can be presented to the user. It becomes possible.

  Note that the configuration of an HMD device including a non-transmissive display may be an HMD device 300c shown in FIG. The HMD device 300c has the same configuration as the HMD device 300 shown in FIG. 8 except that it is a non-transmissive display.

  On the other hand, in the virtual reality service providing server 3300, the display video generation processing unit 1350 includes a viewpoint control unit 1351, a light source control unit 1352, a drawing processing unit 1353, an encoding processing unit 1354, a video composition processing unit 1355, Is provided.

  The viewpoint control unit 1351 controls the viewpoint position and the line-of-sight direction in the virtual space based on the position of the user's eyes (the position of the line-of-sight camera) indicated by the user state information. The light source control unit 1352 controls the light source position and the intensity (illuminance) of the light source based on the image analysis result in the image analysis processing unit 2360.

  The drawing processing unit 1353 generates an image of the virtual reality object arranged in the virtual space based on the viewpoint and the light source controlled by the viewpoint control unit 1351 and the light source control unit 1352, respectively.

  The video composition processing unit 1355 synthesizes the video of the virtual reality object generated by the drawing processing unit 1353 with the forward scene video captured by the line-of-sight camera 121, and is displayed on the display panels 151a and 151b of the HMD device 300c, respectively. Generate video.

  Here, the composition means that the video of the virtual reality object is synthesized with the video of the virtual reality object on the front scenery video. That is, by replacing the pixels corresponding to the virtual reality object in the front scene video with the pixels of the video of the virtual reality object, these two videos are synthesized. However, when the virtual reality object is displayed as semi-transparent, the pixel value is the average value of the pixel value of the front view image and the pixel value of the image of the virtual reality object or a value obtained by weighting and adding one of the values. The two images may be combined by replacing.

  In addition, by determining the boundary between the front scene video and the position of the object in the front scene video and the position of the virtual reality object, priority is given to the object located ahead of the virtual reality object, and a part of the virtual reality object is determined. You may perform the process to delete.

  The video generated by the synthesis process in the video synthesis processing unit 1355 is converted into video encoded data by the video encoding processing unit 1354 and transmitted from the communication unit 1310 to the HMD device 300c.

  Each process performed by the virtual reality service providing server 3300 described above may be performed by a single server, or may be configured by a plurality of servers working together. For example, in the virtual reality service providing system 3000 shown in FIG. 28, a virtual reality service providing server 3300 that provides a virtual reality service, an image analysis server 3600 that analyzes an image acquired as ambient state information in the HMD device 300, and an HMD A speech analysis server 3700 that analyzes speech input in the device 300, a response context generation server 3800 that generates a context of a response to the user, and a speech synthesis server 3900 that synthesizes speech are provided.

  The image analysis processing server 3600 receives an image captured by the HMD device 300, performs boundary identification processing, object identification processing, object distance calculation processing, object modeling processing, and the like, and performs object types, object positions, and objects. Distance, object model data, etc. are generated.

  In the boundary specifying process, it is determined whether the boundary is an object boundary by comparing adjacent blocks and adjacent pixels in the same image. In the object specifying process, a pattern matching process with an image of each object stored in advance in the reference object database is performed to determine that the object is an image having a correlation value equal to or greater than a predetermined reference value. In the object distance calculation process, a distance is calculated based on a difference in pixel position of an object shown in an image photographed by a plurality of cameras. In the modeling process, each object is extracted along the boundary specified in the boundary specifying process, the vertex coordinates of the shape of the object are obtained, and 3D polygon data is generated to acquire the model data of the object.

  The voice analysis processing server 3700 receives the voice data input by the HMD device 300, and divides the voice data into a plurality of phoneme data in time series based on the feature amount included in the voice data. Each divided phoneme data is subjected to matching processing with reference phoneme data stored in the reference phoneme database, thereby specifying which sound the divided phoneme data is. Since each reference phoneme data is associated with a code indicating the sound of the phoneme data, the input speech data is converted into a code string. The speech analysis processing server 3700 performs matching processing between the code associated with the word stored in the word database and the code string, thereby converting the converted code string into a sequence of meaningful words. Convert and get input audio context.

  The voice analysis processing server 3700 extracts emotion elements based on the voice level and pitch of the input voice. For example, the emotional elements in the input voice are classified into a five-step evaluation of “feeling emotional”. For example, the voice analysis processing server 3700 extracts emotion elements of high pitch voice as “Raku 5”. Further, for example, the voice analysis processing server 3700 extracts a voice emotion element having a low pitch and a low voice level as “sorrow 2”.

  The voice analysis processing server 3700 outputs the acquired context and the extracted emotion element to the response context generation server 3800.

  The response context generation server 3800 receives the context and emotional elements acquired by the voice analysis processing server 3700 and generates a response context for the voice input by the HMD device 300. Here, the response context generation server 3800 reads out and executes the virtual reality object personality control program stored in the virtual reality content storage DB 3200, and based on the context and emotion elements input from the speech analysis processing server 3700. To generate a response context. The response context generation server 3800 outputs the generated response context to the speech synthesis processing server 3900.

  The speech synthesis processing server 3900 combines the synthesized speech data by combining the basic speech data associated with the virtual reality object stored in the virtual reality content storage DB 3200 according to the response context generated by the response context generation server 3800. Is generated.

  The virtual reality service providing server 3300 arranges the virtual reality object in the virtual space, and generates a video of the virtual reality object from the viewpoint controlled based on the user state information. The virtual reality service providing server 3300 encodes the generated video and the synthesized voice data generated by the voice synthesis processing server 3900 and transmits the encoded voice data to the HMD device 300. Also, the virtual reality service providing server 3300 encodes the audio data read from the virtual reality content storage DB 3200 and transmits the encoded audio data to the HMD device 300. The HMD device 300 presents the virtual reality world to the user by displaying video images of these virtual reality objects or reproducing audio data. It may be configured in this way.

(Embodiment 4)
The present invention presents the virtual reality world to the user. Although humans sense events occurring in the outside world using the five senses, the amount of information acquired by visual sense is the largest in normal times, so that the virtual reality worn by the user to present a virtual reality world The video of the virtual reality object is displayed on the display unit of the presentation device. Here, each process required for the display may be performed by the virtual reality presentation device or may be performed by an external server having a high processing capability. Where the processing is performed is determined based on the processing capacity, storage capacity, power consumption, etc. of the virtual reality presentation device, and does not significantly affect the virtual reality world presented to the user. Absent. In other words, how the processing is shared between the virtual reality presentation device and the external information processing server can be determined according to the capability of the virtual reality presentation device as appropriate, and the virtual reality is presented. As a virtual reality presentation system, the same system is used regardless of the processing sharing.

  FIG. 29 is a block diagram illustrating a configuration of a virtual reality presentation system 4000 according to the fourth embodiment. The virtual reality presentation system 4000 includes at least a user state detection unit 410, a virtual reality content storage unit 420, a viewpoint control unit 430, a drawing processing unit 440, and a display unit 450.

  The user state detection unit 410 corresponds to the above-described user state detection unit 110 and detects the current state of the user. The user state detection unit 410 uses a tilt sensor that detects the tilt of the user's head as the user state, an acceleration sensor that detects the acceleration of the user's head, a speed sensor that detects the user's speed, a GPS, and the like. It is configured by combining part or all of a position detection unit that detects the current position of the user.

  The virtual reality content storage unit 420 corresponds to the above-described virtual reality content storage DB 3200 and stores virtual reality content including at least a virtual reality object.

  The viewpoint control unit 430 corresponds to the viewpoint control unit 1351, and controls the viewpoint in the virtual reality object drawing process based on the user state detected by the user state detection unit 410. The viewpoint control unit 430 calculates the position of the viewpoint and the direction of the viewpoint (line of sight) based on the user state in the user state detection unit 410. The viewpoint control unit 430 controls at least two viewpoints of a first viewpoint corresponding to the user's right eye and a second viewpoint corresponding to the user's left eye. Here, the positions and lines of sight of the first viewpoint and the second viewpoint are linked. The first viewpoint and the second viewpoint are separated by a preset distance, and the directions (line of sight) of the first viewpoint and the second viewpoint are the same direction.

  The drawing processing unit 440 corresponds to the drawing processing unit 1353, and performs a process of drawing a virtual reality object read from the virtual reality content storage unit 420 based on the viewpoint controlled by the viewpoint control unit 430. The rendering processing unit 440 generates two images of the same virtual reality object by rendering the virtual reality object based on the two viewpoints of the first viewpoint and the second viewpoint controlled by the viewpoint control unit 430. . However, since the drawing processing unit 440 is linked to the first viewpoint and the second viewpoint, it is not necessary to perform the drawing process independently, and the correction process is performed on the first video drawn based on the first viewpoint. The second video may be generated by doing so. The drawing processing unit 440 may be referred to as a video generation processing unit because it performs processing to generate a video including a virtual reality object.

  The display unit 450 corresponds to the display unit 150 described above, and displays the video drawn by the drawing processing unit 440. The display unit 450 is specifically two display panels arranged in front of the user's left eye and right eye, and displays two images of the virtual reality object generated by the drawing processing unit 440, respectively.

  Note that the viewpoint control performed by the viewpoint control unit 430 also controls the position of the virtual reality object displayed on the display unit 450, and thus may be referred to as a virtual reality object display position control unit.

  FIG. 30 is a flowchart showing an operation flow of the virtual reality presentation system 4000. When the user state detection unit 410 newly detects the user state (Yes in step S401), the viewpoint control unit 430 controls the viewpoint based on the detected latest user state (step S402). When the next frame timing has arrived (Yes in step S403), the drawing processing unit 440 generates a video of the virtual reality object (step S404). The display unit 450 displays the video generated in step S404 (step S405).

  According to this configuration, it is possible to present a virtual reality world corresponding to the behavior of the real world to the user.

(Embodiment 5)
The viewpoint control performed in the fourth embodiment is essential for presenting the virtual reality world. On the other hand, if the brightness of the virtual reality object is different from the brightness of the object surface in the real world, such as when the brightness of the displayed virtual reality object is too high even though the real world is dark at night, the user feels uncomfortable. Will be generated. The fifth embodiment is characterized by having a function of correcting the luminance shift.

  FIG. 31 is a block diagram illustrating a configuration of a virtual reality presentation system 5000 according to the fifth embodiment. The virtual reality presentation system 5000 includes at least a user state detection unit 410, a virtual reality content storage unit 420, a viewpoint control unit 430, a drawing processing unit 440, a display unit 450, an ambient state detection unit 510, and light source control. Part 520.

  The surrounding situation detection unit 510 corresponds to the surrounding situation detection unit 120 described above, and detects the surrounding situation of the user. Specifically, the surrounding state detection unit 510 detects the brightness (illuminance) around the user. The ambient condition detection unit 510 includes a line-of-sight camera that captures the user's line-of-sight direction, a peripheral camera that captures the user's surroundings such as the user's left, right, up, down, and rear, and other optical sensors that detect the brightness of the outside world.

  The light source control unit 520 corresponds to the light source control unit 1352 described above, and controls the light source in the virtual reality object drawing process based on the user's surroundings detected by the surrounding state detection unit 420. Specifically, the light source control unit 520 calculates the luminance of the light source based on the brightness (illuminance) around the user detected by the ambient condition detection unit 510. Note that when the ambient condition detection unit 510 detects an environment around the user by capturing an image having a positional resolution such as a line-of-sight camera, the light source control unit 520 is based on the image captured by the ambient condition detection unit 510. Then, the luminance of the light source and the position of the light source are calculated.

  The rendering processing unit 440 performs processing for rendering the virtual reality object read from the virtual reality content storage unit 420 based on the viewpoint controlled by the viewpoint control unit 430 and the light source controlled by the light source control unit 520. . The drawing processing unit 440 performs a shading process for calculating the luminance of the surface of the virtual reality object based on the light source controlled by the light source control unit 520. The drawing processing unit 440 generates a video of the virtual reality object whose surface brightness is adjusted by the shading process.

  FIG. 32 is a flowchart showing an operation flow of the virtual reality presentation system 5000. When the user state detection unit 410 newly detects the user state (Yes in step S401), the viewpoint control unit 430 controls the viewpoint based on the detected latest user state (step S402).

  The light source control unit 510 determines whether to newly control the light source, that is, whether to update the arrangement relationship and luminance of the light source (step S501). If it is determined that the light source is to be controlled, the light source control unit 510 controls the light source based on the user's surroundings detected by the surrounding state detection unit 510, that is, updates the settings relating to the light source (step S502). Note that the light source control unit 510 may perform the update at a predetermined time interval, or when the surrounding situation detected by the surrounding situation detection unit 510 has changed more than a predetermined reference, It is also possible to update the settings relating to the light source such as the arrangement relationship of the light sources and the luminance of the light source.

  When the next frame timing has arrived (Yes in step S403), the drawing processing unit 440 generates a video of the virtual reality object (step S404). Specifically, the drawing processing unit 440 performs a shading process, calculates the luminance of the surface of the virtual reality object, and generates a video of the virtual reality object represented by the calculated luminance. The display unit 450 displays the video generated in step S404 (step S405).

  As described above, in the virtual reality presentation system 5000 according to the fifth embodiment, by performing light source control in addition to viewpoint control, it is possible to present a more realistic virtual reality world to the user.

(Embodiment 6)
In the virtual reality presentation system, the video of the virtual reality object is displayed on the display unit so as to be at a predetermined position in the real world. However, when a user's arm or the like enters between the predetermined position and the user, if a virtual reality object that should be hidden by the arm and disappear from the display is continuously displayed on the display unit, Deviation from the virtual reality world will occur, and the reality will be reduced. In the sixth embodiment, when a real object enters between the virtual reality object and the position of the user as described above, the control is performed so that the behavior of the virtual reality world matches the behavior of the real world. It is characterized by performing.

  FIG. 33 is a block diagram illustrating a configuration of a virtual reality presentation system 6000 according to the sixth embodiment. The virtual reality presentation system 6000 includes at least a user state detection unit 410, a virtual reality content storage unit 420, a viewpoint control unit 430, a drawing processing unit 440, a display unit 450, an ambient condition detection unit 510, and light source control. A unit 520 and an object specifying unit 610.

  The object position specifying unit 610 corresponds to the object position specifying unit 2361 described above, and specifies at least the position of the object in front of the user based on the surroundings of the user detected by the surroundings detection unit 510. .

  The drawing processing unit 440 is based on the virtual reality content storage unit 420 based on the viewpoint controlled by the viewpoint control unit 430, the light source controlled by the light source control unit 520, and the position of the object specified by the object position specifying unit 610. A process of drawing the read virtual reality object is performed. The drawing processing unit 440 performs hidden surface processing for hiding a part or all of the virtual reality object based on the position of the object specified by the object position specifying unit 610. The drawing processing unit 440 generates a video of a virtual reality object that is partially or entirely hidden by hidden surface processing.

  FIG. 34 is a flowchart showing an operation flow of the virtual reality presentation system 6000. When the user state detection unit 410 newly detects the user state (Yes in step S401), the viewpoint control unit 430 controls the viewpoint based on the detected latest user state (step S402).

  The light source control unit 510 determines whether to newly control the light source, that is, whether to update the arrangement relationship and luminance of the light source (step S501). If it is determined that the light source is to be controlled, the light source control unit 510 controls the light source based on the user's surroundings detected by the surrounding state detection unit 510, that is, updates the settings relating to the light source (step S502).

  The object position specifying unit 610 specifies the position of an object around the user based on the situation around the user detected by the surrounding state detection unit 510 (step S601). Specifically, the object position specifying unit 610 specifies the position of an object around the user based on the image taken by the surrounding state detection unit 510. It is even better if the object position specifying unit 610 is configured to specify the type and arrangement relationship of objects around the user.

  When the next frame timing has arrived (Yes in step S403), the drawing processing unit 440 generates a video of the virtual reality object (step S404). Specifically, the rendering processing unit 440 performs hidden surface processing for hiding part or all of the virtual reality object based on the position of the object specified in step S601, and part or all of the virtual surface object is hidden by the hidden surface processing. A video of the virtual reality object that is present. The display unit 450 displays the video generated in step S404 (step S405).

  With this configuration, the behavior of the virtual reality world and the behavior of the real world can be matched, and a virtual reality world with a higher sense of reality can be presented to the user.

(Embodiment 7)
In the virtual reality presentation system, the video of the virtual reality object is displayed on the display unit so as to be at a predetermined position in the real world. If the virtual reality object is a static object such as a vase or a building, there is no problem. However, if the virtual reality object is a human being or the like, a simple doll can be used if it cannot operate in the virtual reality world. As a result, the reality of the virtual reality world decreases. The seventh embodiment is characterized in that control is performed so that a virtual reality object is represented in the virtual reality world in the same manner as a human or the like corresponding in the real world.

  FIG. 35 is a block diagram illustrating a configuration of a virtual reality presentation system 7000 according to the seventh embodiment. The virtual reality presentation system 7000 includes at least a user state detection unit 410, a virtual reality content storage unit 420, a viewpoint control unit 430, a drawing processing unit 440, a display unit 450, an ambient condition detection unit 510, and light source control. Unit 520, object specifying unit 610, virtual reality management unit 710, virtual space setting unit 720, virtual reality object placement unit 730, and virtual reality object behavior control unit 740.

  The virtual reality management unit 710 corresponds to the virtual reality service provision management unit 1320 described above, and manages the presentation of the virtual reality world to the user by performing a virtual reality presentation start process and an end process.

  The virtual space setting unit 720 corresponds to the virtual reality service control unit 3330 described above, and sets a virtual space in which a virtual reality object is placed. The virtual space setting unit 720 generates a new virtual space when it is determined by the virtual reality management unit 710 to start presentation of virtual reality.

  The virtual reality object placement unit 730 corresponds to the virtual reality object placement position determination unit 1340 and the virtual reality service control unit 3330 described above, and is read from the virtual reality content storage unit 420 in the virtual space set by the virtual space setting unit 720. Arrange virtual reality objects. The virtual reality object placement unit 730 determines the placement position of the virtual reality object in the virtual space based on the user's surroundings detected by the surrounding state detection unit 510 and the user's state detected by the user state detection unit 410. It is preferable that the configuration is arranged.

  The virtual reality object behavior control unit 740 corresponds to the above-described virtual reality object behavior control unit 3370, and displays the behavior of the virtual reality object placed by the virtual reality object placement unit 730 in the virtual space set by the virtual space setting unit 720. Control. The virtual reality object behavior control unit 740 changes the appearance and shape of the virtual reality object by changing the angle and position of the joint point set in the virtual reality object, or creates a texture attached to the surface of the virtual reality object. The behavior of the virtual reality object is controlled by changing it.

  The rendering processing unit 440 renders the virtual reality object whose behavior is controlled by the virtual reality object behavior control unit 740 based on the viewpoint controlled by the viewpoint control unit 430, thereby controlling the virtual reality object whose behavior is controlled. Generate video for

  FIG. 36 is a flowchart showing an operation flow of the virtual reality presentation system 7000. The virtual reality management unit 710 determines whether to start presenting virtual reality anew (step S701). When it is determined in step S701 that presentation of virtual reality is newly started, the virtual space setting unit 720 newly sets a virtual space (step S702).

  Next, the virtual reality object placement unit 730 places the virtual reality object read from the virtual reality content storage unit 420 in the virtual space set in step S702 (step S703). Even while the virtual reality world is being presented, the virtual reality object placement unit 730 adds a new virtual reality object and places it in the virtual space as necessary.

  The virtual reality object behavior control unit 730 determines whether to control the behavior of the virtual reality object arranged in the virtual space in step S703, that is, whether to deform the appearance shape of the virtual reality object (step S704). When it is determined that the behavior of the virtual reality object is to be controlled, the behavior of the virtual reality object is controlled, that is, the appearance shape of the virtual reality object is changed (step S705).

  When the user state detection unit 410 newly detects the user state (Yes in step S401), the viewpoint control unit 430 controls the viewpoint based on the detected latest user state (step S402).

  The light source control unit 510 determines whether to newly control the light source, that is, whether to update the arrangement relationship and luminance of the light source (step S501). If it is determined that the light source is to be controlled, the light source control unit 510 controls the light source based on the user's surroundings detected by the surrounding state detection unit 510, that is, updates the settings relating to the light source (step S502).

  The object position specifying unit 610 specifies the position of an object around the user based on the situation around the user detected by the surrounding state detection unit 510 (step S601). Specifically, the object position specifying unit 610 specifies the position of an object around the user based on the image taken by the surrounding state detection unit 510. It is even better if the object position specifying unit 610 is configured to specify the type and arrangement relationship of objects around the user.

  When the next frame timing has arrived (Yes in step S403), the drawing processing unit 440 generates a video of the virtual reality object (step S404). Specifically, the rendering processing unit 440 performs hidden surface processing for hiding part or all of the virtual reality object based on the position of the object specified in step S601, and part or all of the virtual surface object is hidden by the hidden surface processing. A video of the virtual reality object that is present. The display unit 450 displays the video generated in step S404 (step S405).

  The virtual reality management unit 710 determines whether to end the presentation of virtual reality (step S707). When it is determined that the presentation of the virtual reality is to be ended, an instruction to end the processing and control is issued to each part of the system, and the presentation of the virtual reality is ended. On the other hand, if the presentation of virtual reality is not terminated, the process returns to step S703.

  With this configuration, it is possible to present a more realistic virtual reality world by moving a virtual reality object in the virtual reality world.

  In addition, as the virtual reality object behavior control processing in steps S704 and S705, a plurality of methods for the virtual reality object are prepared in advance, and the user status and ambient condition detection unit 420 detected by the user status detection unit 410 are prepared. It is possible to adopt a configuration in which a predetermined method is executed based on the surroundings of the user detected in step S1, other voices uttered by the user, surrounding voices, and the like.

  In addition, some or all of the functions in the virtual reality presentation system are realized by the virtual reality presentation device, and the remaining part is realized by the virtual reality service providing server.

(Embodiment 8)
We humans live in this real world by sensing various events that occur in the real world using the sensory organs we have born and acting in the real world. That is, humans can be said to be organisms that input information from outside using the five senses, process the input information in the brain, and output the processing results in the form of actions.

  The virtual reality presentation device is a device that allows a user to sense a virtual reality world different from reality by deceiving the user's senses. That is, as illustrated in FIG. 37, it is different from the real world by shielding part or all of the information input based on an event that occurs in the real world and complementing it with artificially created information. It is a device that makes the human brain sense the artificially created virtual reality world. This means that virtual reality worlds with different numbers of virtual reality presentation devices can be created.

  The user using the virtual reality presentation device senses an event that occurs in the created virtual reality world, and the user's behavior is also captured and reflected in the virtual reality world by the virtual reality presentation device. Therefore, the user can be regarded as living in the virtual reality world. On the other hand, the user senses an event in the real world, and the action performed by the user is reflected in the real world. That is, as shown in FIG. 38, it can be said that the user using the virtual reality presentation apparatus lives in a world where the virtual reality world and the real world overlap.

  As shown in FIG. 39, if it is possible to create a different dimension world that does not interact with the real world as a virtual reality world and send the user in a returnable state, each user's Since input and output are closed within the same world, the actions of people living in each world do not affect people living in other worlds. Even if there is a world called heaven or hell, there is no point of contact with us who live in this world, so people living in this world are worried that their actions will affect people in heaven. It is not necessary, and conversely, human beings living in this world are not affected by the actions of human beings in hell.

  However, in the form shown in FIG. 38, the person who uses the virtual reality presentation device and the person who does not use it straddle the real world that is equally common. May affect the real world.

  For example, it is assumed that the user a is in a virtual reality world A created by a virtual reality service having a concept of “hunting a dragon in the city”. It is assumed that the user has punched the dragon represented in front of him. The virtual reality presentation device worn by the user a captures the user's action by image analysis processing or the like, and displays an image of the dragon falling down. Therefore, the user defeats the dragon in the virtual reality world.

  On the other hand, when another person X stands in front of the user a in the real world, the punch that the user starts out will hit the other person X. In this case, the user's action in the virtual reality world is not a problem just by defeating the dragon, but in the real world, it leads to a problem of scolding others.

  As described above, the subject of the user a who uses the virtual reality presentation device is in the virtual reality world, but the user a who is objectively viewed by others is in the real world. Therefore, it is necessary to present the virtual reality world to the user in such a state that the action of the user a in the virtual reality world straddling the real world does not become a problem behavior with respect to the real world. However, the conventional virtual reality presentation device has not been considered from such a viewpoint, and no adjustment has been made between the virtual reality world and the real world.

  In the eighth embodiment, in view of the above problems, a virtual reality presentation system, a virtual reality presentation device, and a virtual reality presentation method are provided in consideration of the influence of the action of a user in the virtual reality world on the real world. Hereinafter, it will be described in detail with reference to the drawings.

FIG. 40 is a block diagram showing a configuration of a virtual reality presentation system 10000 according to Embodiment 8 of the present invention. The virtual reality presentation system 10000 includes an ambient condition detection unit 11010, a virtual reality presentation unit 11020, and a virtual reality presentation interruption control unit 11030.

  The surrounding situation detection unit 11010 detects the surrounding situation of the user who receives the presentation of virtual reality. For example, the surrounding state detection unit 11010 detects the surroundings of the user by photographing the scenery around the user.

  The virtual reality presentation unit 11020 presents virtual reality to the user. Here, virtual reality (VR) refers to a world with reality created artificially by combining computer graphics (CG) and sound effects.

  Humans sense events that occur around them using the five senses, and process the sensed information in the brain to grasp the situation. Therefore, it is possible to present a virtual reality world by displaying images and outputting sounds so as to deceive human senses. In this specification, the virtual reality presentation unit 11020 presents the virtual reality to the user by displaying at least a virtual reality video, and more preferably outputs the virtual reality sound to output the virtual reality. The presentation shall be made to the user.

  FIG. 41 shows a correspondence relationship between the real world, the displayed image, and the virtual reality world sensed by the user. The virtual reality presenting unit 11020 displays the image of the dog, which is a virtual reality object, to the user in the room where the sofa is placed, so that the world that the user sees (the world that the user perceives) is on the sofa. It becomes a virtual reality world different from the real world where the dog is sleeping.

  Here, the virtual reality object refers to a virtual person or a building represented in the virtual reality world by being arranged in a virtual space which is a virtual space artificially set. By setting the virtual space so as to correspond to the real space around the user, the virtual reality world is presented to the user.

  Each type of virtual reality object includes 3D polygon data that defines an appearance shape by 3D computer graphics, texture data pasted on the surface of the shape formed by the polygon data, and the like. Moreover, it is more preferable that the configuration includes joint point data, skeleton data, and the like for operating the virtual reality object. The joint point data and the vertex of the polygon data are associated, and changing the position and angle of the joint point changes the position of the vertex of the associated polygon data, and the appearance of the virtual reality object The shape changes.

  Virtual reality objects can be classified into human objects, animal objects, non-animal objects, and other objects according to the types of objects arranged in the virtual reality world. Each virtual reality object is assigned an object ID for uniquely identifying the virtual reality object.

  A person object is an object such as a real person, an imaginary / imaginary person, a historical person, a creative person, etc., and is at least operable, and preferably speaks (outputs sound). It has the feature that it is possible.

  The animal object is an object such as a real animal, an imaginary / fantasy animal, or a creative animal, and has a feature that it can operate. In addition, a part of the animal object can be configured to speak (output a sound). Since it is a virtual animal placed in the virtual reality world, it is possible to adopt a configuration that speaks like a human.

  A non-animal object is a static object such as a plant, furniture, or building and is basically an object that does not move. However, the plant may be configured to partially move in conjunction with the wind blowing in the real world.

  Other objects are, for example, background objects. Other objects may be used alone to create a virtual reality world, or may be used in combination with a person object, an animal object, or the like.

  FIG. 42 shows an example of a virtual reality object. The virtual reality object is composed of 3D polygon data defining an appearance shape, texture data pasted on the surface, and the like.

  The virtual reality presentation interruption control unit 11030 performs control to interrupt the presentation of the virtual reality based on the surroundings of the user detected by the surroundings state detection unit 11010. For example, when the surrounding state detection unit 11010 detects the presence of another person within a predetermined distance from the user, control is performed to interrupt the presentation of virtual reality. Based on the control from the virtual reality presentation interruption control unit 11030, the virtual reality presentation unit 11020 interrupts the presenting virtual reality presentation.

  As described above, the virtual reality presentation system 10000 according to the eighth embodiment presents virtual reality to the user, while sensing the situation in the user's real world, and the situation in the user's real world. The presentation of virtual reality is interrupted accordingly. Therefore, based on the situation of the real world shared with others, it is possible to switch whether the user senses the virtual reality world or the real world, and the behavior of the user in the virtual reality world Can take into account the impact of the real world.

(Embodiment 9)
FIG. 43 is a block diagram illustrating a configuration of a virtual reality presentation system 12000 according to the ninth embodiment. The virtual reality presentation system 12000 includes an ambient condition detection unit 11010, a virtual reality presentation unit 11020, a virtual reality presentation interruption control unit 11030, and a user state detection unit 12010. The virtual reality presentation unit 11020 includes a video generation processing unit 11021 and a video display unit 11022.

  The user state detection unit 12010 detects the current state of the user and generates user state information indicating the current state of the user. In the virtual reality presentation system 12000 according to the ninth embodiment, the user state detection unit 12010 includes an inclination sensor 12011 that detects the inclination of the user's head and an acceleration sensor 12012 that detects the acceleration of the user's head. The detection result in the inclination sensor 12011 and the detection result in the acceleration sensor 12012 are output to the video generation processing unit 11021 as user state information.

  The video generation processing unit 11021 generates a video for virtual reality presentation. The video generation processing unit 11021 generates a virtual reality presentation video based on the current state of the user detected by the user status detection unit 12010, and outputs the generated video to the video display unit 11022. The virtual reality presentation video includes, for example, a video in which a virtual reality object is drawn from a viewpoint position and a viewpoint direction obtained based on the user state detected by the user state detection unit 12010.

  The video display unit 11022 presents the virtual reality by displaying the video generated by the video generation processing unit 11021. For example, the video display unit 11022 corresponds to a display display disposed in front of the user's eyes in a head mounted display device worn by the user as a virtual reality presentation device.

  The virtual reality presentation interruption control unit 11030 issues an instruction to interrupt the video display of the virtual reality object to the video display unit 11022 as control for interrupting the virtual reality presentation. The video display unit 11022 that has received the instruction interrupts the display of the video generated by the video generation processing unit 11021.

  Note that the virtual reality presentation interruption control unit 11030 may be configured to issue an instruction to interrupt the generation of the virtual reality object video to the video generation processing unit 11021 as control to interrupt the virtual reality presentation. Receiving the instruction, the video generation processing unit 11021 interrupts the generation of the virtual reality presentation video.

  As described above, according to the virtual reality presentation system 12000 according to the ninth embodiment, the virtual reality world is presented to the user by displaying the virtual reality presentation video. The amount of information that humans obtain through vision is much larger than the amount of information perceived by other senses. Therefore, it is possible to draw the user into the virtual reality world by displaying an image controlled based on the state of the user so as to cause an illusion to human vision.

  Note that the virtual reality presentation system 12000 according to the ninth embodiment is more favorable when configured as shown in FIG. The virtual reality presentation system 12000 shown in FIG. 44 further includes a virtual reality object storage unit 12020 and a virtual reality object placement unit 12030. The video generation processing unit 11021 includes a viewpoint control unit 11021-1, a drawing processing unit 11021. -2 is a feature.

  The virtual reality object storage unit 12020 stores a virtual reality object. The virtual reality object includes, for example, polygon data that defines an appearance shape and texture data that is pasted on the surface.

  The virtual reality object placement unit 12030 reads the virtual reality object stored in the virtual reality object storage unit 12020 and places it in the virtual space.

  The viewpoint control unit 11021-1 controls the position and direction of the viewpoint when drawing the virtual reality object based on the user state information indicating the user state. Specifically, based on the tilt information indicating the tilt of the user's head included in the user state information and the acceleration information indicating the acceleration of the user's head, a virtual space corresponding to the position of the left eye and the position of the right eye of the user The viewpoint position at is calculated. Further, the line-of-sight direction in the virtual space corresponding to the direction of the user's line of sight is calculated based on the acceleration information. Here, the line-of-sight direction does not correspond to the direction of the line of sight when the user moves only the eyeball, but corresponds to the direction of the user's head, that is, the direction of the HMD device worn by the user. However, when the virtual reality presentation device is not an HMD device but a contact lens type device or an implant type device, it is preferable that the eyeball direction of the user is the line-of-sight direction. In this case, it is preferable that the user state detection unit 12010 further includes a gaze direction sensor that detects the gaze direction of the user. The viewpoint control unit 11021-1 calculates the position of the viewpoint and the direction of the line of sight in the virtual space corresponding to the position of the user's eyes and the direction of the head based on the user state information.

  The drawing processing unit 11021-2 renders the virtual reality object in real time to generate a video. The video generated by the drawing processing unit 11021-2 is displayed on the video display unit 11022.

  For example, the drawing processing unit 11021-2 sets an image plane at a predetermined position in the virtual space based on the viewpoint position and the line-of-sight direction obtained by the viewpoint control unit 11021-1 and projects a virtual reality object. As a result, a projection image of the virtual reality object is acquired. The drawing processing unit 11021-2 performs clipping processing of the image plane onto which the virtual reality object is projected, and selects a range to be displayed on the video display unit 11022.

  The drawing processing unit 11021-2 performs texture mapping processing using texture data pasted on the surface of the virtual reality object, appropriately performs correction processing such as anti-aliasing processing, and is finally displayed on the video display unit 11022. Generate video.

  With this configuration, a virtual reality world in which a virtual reality object created artificially exists can be presented to the user.

  In addition, the virtual reality presentation system 12000 according to the ninth embodiment is more favorable when configured as shown in FIG. In the virtual reality presentation system 12000 shown in FIG. 45, the virtual reality presentation unit 11020 includes an audio output unit 11023 in addition to the video generation processing unit 11021 and the video display unit 11022. The virtual reality presentation system 12000 shown in FIG. 45 includes an interruption video data storage unit 12040, a virtual reality presentation audio data storage unit 12050, and an interruption audio data storage unit 12060.

  The video display unit 11022 includes a video display control unit 11022-1 and a display panel 11022-2. The video display control unit 11022-1 receives the video of the virtual reality object generated by the video generation processing unit 11021 and performs control to display it on the display panel 11022-2.

  Also, the video display control unit 11022-1 displays the video of the virtual reality object on the display panel 11022-2 when the instruction signal instructing the virtual reality presentation suspension control unit 11030 to interrupt the virtual reality presentation is input. Control to interrupt that.

  When receiving the instruction to interrupt the virtual reality presentation, the video display control unit 11022-1 reads the video data for interruption stored in the video data storage unit 12040 for interruption, and displays the interruption data on the display panel 11022-2. Control to display video.

  FIG. 46 shows an example of the interruption video displayed based on the virtual reality presentation interruption control. The video display control unit 11022-1 presents virtual reality by displaying the video of the dog, which is a virtual reality object generated by the video generation processing unit 11021, on the display panel 11022-2 so as to overlap the real world. doing. When a virtual reality presentation suspension instruction is issued from the virtual reality presentation suspension control unit 11030, the video display control unit 11022-1 reads the suspension video data from the suspension video data storage unit 12040 and displays the display panel 11022. -2 displays a video for interruption.

  In FIG. 46, image data indicating the content that “virtual reality presentation has been interrupted due to approach of a vehicle” is used as video data for interruption. The video display control unit 11022-1 switches the video input source to be displayed on the display panel 11022-2 from the video generation processing unit 11021 to the video data storage unit for interruption 12040, and displays the video of the virtual reality object based on the interruption instruction. Control to interrupt.

  The video of the virtual reality object generated by the video generation processing unit 11021 is controlled based on the current state of the user detected by the user state detection unit 12010. That is, since the video generation processing unit 11021 generates a video of the virtual reality object based on the current state of the user, the generated video is changed in conjunction with the operation that the user moves or tilts the head. I will do it.

  FIG. 47 shows the real world scenery around the user when the user's state changes, the video displayed on the display panel 11022-2, that is, the video generated by the video generation processing unit 11021, and the virtual reality. The time change with the scenery which the user shown is perceived is shown.

  In FIG. 47, the user rotates his neck 120 degrees clockwise in the room where the sofa is placed. The virtual reality object of the dog is arranged in the virtual space by the virtual reality object arrangement unit 12030, and the video generation processing unit 11021 corresponds to the position of the user's eyes and the direction of the head detected by the user state detection unit 12010. Based on the viewpoint position and the line-of-sight direction, the virtual reality object arranged in the virtual space is drawn. Accordingly, the video of the virtual reality object generated by the video generation processing unit 11021 changes in conjunction with the movement of the user's neck. For this reason, a user who is viewing the video displayed on the display panel 11022-2 overlaid on the real world scenery lies a dog represented by a virtual reality object on a sofa placed in the real world. A virtual reality world is perceived.

  On the other hand, FIG. 48 shows a temporal change of the scenery sensed by the user when the virtual reality presentation is interrupted and the interrupting video is displayed on the display panel 11022-2. The scenery that enters the user's field of view changes as the user rotates his neck. On the other hand, the video display control unit 11022-1 continues to display the interruption video at the same position on the display panel 11022-2. Since the interruption video is displayed in order for the user to recognize that the presentation of the virtual reality has been interrupted, the interruption video continues to enter the user's field of view regardless of a change in the state of the user.

  As described above, the video of the virtual reality object related to the presentation of the virtual reality is such that the display position on the display panel 11022-2 and the shape of the virtual reality object change in conjunction with the change of the user state. The interruption video is displayed on the display panel 11022-2 without being interlocked with the change of the state of the user for a certain period or until the reason for the interruption is resolved.

  The virtual reality presentation sound data storage unit 12050 stores sound data for virtual reality presentation. The virtual reality presentation sound data is associated with a virtual reality object stored in the virtual reality object storage unit 12020.

  FIG. 49 is an example of a management file that manages the virtual reality object and the virtual reality presentation sound data associated with the virtual reality object. Each virtual reality object is assigned an object ID that uniquely identifies the object. Each virtual reality object is classified according to a plurality of types. Type 1 indicates a rough classification of virtual reality objects, and indicates whether the object is a human object representing a human or an animal object representing an animal. Type 2 indicates whether the object represents an object in the real world or an object representing an imaginary object or person. The type N indicates a more detailed classification.

  An object representing a person or animal having the ability to emit sound, such as a human object or animal object, is associated with the sound emitted by the object. For example, the virtual reality object with the object ID 000001 has M001. mp3, M003. mp3, M034. The audio data of mp3 is associated.

  The interruption voice data storage unit 12060 stores interruption voice data to be reproduced when the presentation of virtual reality is interrupted. The interruption voice data corresponds to, for example, a voice that reads out a sentence saying “virtual reality presentation has been interrupted due to vehicle approach”.

  The audio output unit 11023 reads the virtual reality presentation audio data and the interruption audio data and outputs them as audio. The audio output unit 11023 includes an audio reproduction control unit 11023-1 and a speaker 11023-2.

  When presenting virtual reality to the user, the audio reproduction control unit 11023-1 reads the virtual reality presentation audio data from the virtual reality presentation audio data storage unit 12050 and reproduces it. Specifically, the audio reproduction control unit 11023-1 distributes the virtual reality presentation audio data to the audio data of a plurality of channels based on the user status detected by the user status detection unit 12010. The audio reproduction control unit 11023-1 performs DA conversion processing and amplification processing on each distributed audio data to convert it into an analog audio signal, and outputs the analog audio signal to the speaker 11023-2.

  The speaker 11023-2 converts the analog audio signal of each channel output from the audio reproduction control unit 11023-1 into air vibration and emits the sound to the outside.

  According to the said structure, the user can recognize clearly that the presentation of virtual reality was interrupted | interrupted and the audio | voice for interruption. In addition, by presenting virtual reality through hearing, in which human beings usually obtain information from outside after vision, the user can more naturally enter the virtual reality world.

  In addition, it is preferable that the audio can be played back on many channels such as 6.1Ch and 7.1Ch so that the played back sound becomes natural.

  Also, the virtual reality presentation audio data reproduced by the audio reproduction control unit 11023-1 is not limited to the configuration stored in advance in the virtual reality presentation audio data storage unit 12050, and is appropriately generated by speech synthesis. It is good also as a structure to be performed. For example, the virtual reality presentation voice data storage unit 12050 is configured to store voice basic data that is a source of voice synthesis, and a voice synthesis processing unit provided separately synthesizes the voice basic data according to a voice output context. It is even better if it is configured to generate synthesized speech data for virtual reality presentation. The sound basic data is data that is a unit of sound associated with the virtual reality object, and is sound data that records a real voice such as 50 sounds uttered by a person represented by the virtual reality object. The synthesized voice data is output to the voice reproduction control unit 11023-1, and the voice reproduction control unit 11023-1 reproduces the synthesized voice data.

  Further, the sound reproduction control unit 11023-1 reproduces sound so that the sound generation source comes from the direction of the virtual reality object so that a person or an animal represented by the virtual reality object actually emits sound. It is preferable that the control be performed. Therefore, as in the virtual reality presentation system 12000 shown in FIG. 50, the virtual reality object placement unit 12030 instructs the audio reproduction control unit 11023-1 to indicate the position of each virtual reality object placed in the virtual space. It is even better when the position information is output.

  The audio reproduction control unit 11023-1 is obtained based on the position coordinates of the virtual reality object indicated by the virtual reality object position information and the user state information indicating the user state generated by the user state detection unit 12010. Based on (the position coordinates of the user's left ear and the position coordinates of the right ear), the direction and distance of the virtual reality object viewed from the user position are specified. The audio reproduction control unit 11023-1 performs distribution processing for distributing audio data to be reproduced to a plurality of channels based on the direction of the identified virtual reality object, and generates audio data of each channel. Further, based on the distance to the identified virtual reality object, amplification processing for amplifying the audio data to be reproduced is performed, and audio output from each channel to the speaker 11023-2 is generated.

  In this way, by distributing the audio data, the user can hear the sound from the direction of the person or animal represented by the virtual reality object. Sounds like it sounds. Further, by adjusting the sound level based on the distance to the virtual reality object, the user senses that sound is emitted from the virtual reality object that the user actually sees. Therefore, the user is drawn deeper into the virtual reality world where persons, animals, and the like represented by virtual reality objects that do not originally exist in the real world exist.

  Note that the virtual reality presentation interruption control unit 11030 may perform control for presenting virtual reality based on various reasons for interruption. In this case, it is preferable that the user can be notified of the reason why the presentation of virtual reality is forcibly interrupted. This is because the user who confirms the reason for the interruption can receive the presentation of the virtual reality again by acting himself so that the reason for the interruption is resolved.

  Therefore, the virtual reality presentation system 12000 according to the ninth embodiment is more favorable when configured as shown in FIG. The virtual reality presentation system 12000 shown in FIG. 51 is characterized by newly including an interruption video data selection processing unit 12070 and an interruption audio data selection processing unit 12080, as compared with the case of FIG. .

  The interruption video data storage unit 12040 and the interruption audio data storage unit 12060 each store a plurality of types of interruption video data and interruption audio data. FIG. 52 is an example of a management file for managing the video data for interruption and the audio data for interruption.

  The interruption video data storage unit 12040 and the interruption audio data storage unit 12060 each store a plurality of interruption video data and a plurality of interruption audio data according to the reason for interruption.

  The virtual reality presentation interruption control unit 11030 is predetermined when virtual reality presentation is performed by the virtual reality presentation unit 11020 displaying a video of virtual reality content or outputting a virtual reality presentation sound. If the cause of interruption occurs, control to interrupt the presentation of virtual reality is performed. Specifically, the virtual reality presentation interruption control unit 11030 issues a virtual reality presentation interruption instruction to the interruption video data selection processing unit 12070 and the interruption audio data selection processing unit 12080.

  Here, the virtual reality presentation interruption control unit 11030 performs the instruction by outputting an interruption instruction signal including information indicating the interruption reason to the interruption video data selection processing unit 12070 and the interruption audio data selection processing unit 12080.

  The interruption video data selection processing unit 12070 selects and reads out the interruption video data corresponding to the interruption cause indicated by the interruption instruction from the virtual reality presentation interruption control unit 11030 from the interruption video data storage unit 12040, and reads the video. The data is output to the display control unit 11022-1.

  When the interruption video data selection processing unit 12070 reads the interruption video data, the video display control unit 11022-1 displays the interruption video data on the display panel 11022-2 in place of the virtual reality object video. Control the display.

  Similarly, the interruption audio data selection processing unit 12080 selects the interruption audio data corresponding to the interruption reason indicated by the interruption instruction from the virtual reality presentation interruption control unit 11030 from the interruption audio data storage unit 12060. Read and output to the audio reproduction control unit 11023-1.

  When the interruption audio data selection processing unit 12080 reads the interruption audio data, the audio reproduction control unit 11023-1 reproduces the interruption audio data instead of reproducing the virtual reality presentation audio data. Control to output from the speaker 11023-2 is performed.

  According to this configuration, since it is possible to appropriately recognize whether the presentation of virtual reality has been interrupted for a reason such as a user, a user who wishes to continue to enjoy the virtual reality world should resolve the reason for the interruption. By acting, you can quickly return to the virtual reality world.

  In this case, the virtual reality presentation interruption control unit 11030 performs control to cause the virtual reality presentation unit 11020 to resume presentation of virtual reality when the interruption reason is resolved. For example, the virtual reality presentation interruption control unit 11030 stops the display of the interruption video to the video display control unit 11022-1, and displays the video of the virtual reality object generated by the video generation processing unit 11021 on the display panel 11022-2. A configuration in which an instruction to display is performed is preferable.

(Embodiment 10)
When presenting virtual reality to the user, the user perceives a virtual reality world that is different from the real world that others are sensing, while the user's behavior is not only the virtual reality world, Also affects the world. Therefore, it is preferable that the virtual reality presentation system that presents the virtual reality performs overall control regarding the presentation of the virtual reality while considering the influence of the user's action in the real world.

  FIG. 53 is a block diagram illustrating a configuration of a virtual reality presentation system 13000 according to the tenth embodiment. The virtual reality presentation system 13000, the virtual reality presentation system 10000, the ambient state detection unit 11010, the virtual reality presentation unit 11020, the virtual reality presentation interruption control unit 11030, the virtual reality presentation interruption condition storage unit 13010, and an image analysis process Unit 13020.

  In the virtual reality presentation system 13000 according to the tenth embodiment, the surrounding state detection unit 11010 includes a surrounding photographing unit 1011 that photographs at least the surroundings of the user and acquires a surrounding scenery image. Specifically, the ambient photographing unit 1011 corresponds to a camera attached to a virtual reality presentation device such as an HMD device worn by a user. In the virtual reality presentation device, at least a plurality of cameras are installed as the surrounding photographing unit 1011. The surrounding scenery image acquired by the surrounding photographing unit 1011 is output to the image analysis processing unit 13020 as surrounding state information.

  The image analysis processing unit 13020 performs a process of extracting predetermined information by analyzing the surrounding scenery image in which the surrounding scenery of the user taken by the surrounding photographing unit 1011 is captured. In the virtual reality presentation system 13000 according to the tenth embodiment, the image analysis processing unit 13020 includes an object distance calculation unit 13021 and an object moving speed calculation unit 13022.

  The object distance calculation unit 13021 specifies the position of the object shown in the surrounding scenery image. Specifically, the object distance calculation unit 13021 identifies boundaries of a plurality of objects shown in the surrounding scene image by comparing color difference signals, luminance signals, and the like of adjacent pixels or blocks in the surrounding scene image. Label the identified object. The object distance calculation unit 13021 performs the boundary determination process and the labeling process on the surrounding scene image acquired by the plurality of surrounding photographing units 1011 installed in the virtual reality presentation device. The object distance calculation unit 13021 calculates the distance to the object based on the difference between the positions in the image of the object with the same label. Here, the object distance calculation unit 13021 is more preferably configured to calculate the position coordinates of the object. The object distance calculation unit 13021 outputs information related to the calculated distance to the object to the object moving speed calculation unit 13022 and the virtual reality presentation interruption determination processing unit 11031 of the virtual reality presentation interruption control unit 11030.

  The object moving speed calculation unit 13022 calculates the moving speed of the object in the user direction based on the temporal change in the distance to the object calculated by the object distance calculation unit 13021. When the object distance calculation unit 13021 functions as an object position specifying unit that obtains the position coordinates of each object that appears in the surrounding scenery image, the object moving speed calculation unit 13022 is based on the time change of the position coordinates of the object. It is even better if the movement vector of the object is calculated. The object moving speed calculation unit 13022 calculates the moving speed and moving direction of the object shown in the surrounding scenery image. The object moving speed calculation unit 13022 outputs the calculated moving speed of the object to the virtual reality presentation interruption determination processing unit 11031 of the virtual reality presentation interruption control unit 11030. When the object movement speed calculation unit 13022 is configured to calculate the movement vector of the object, the object movement speed calculation unit 13022 calculates the movement speed calculated by the virtual reality presentation interruption determination processing unit 11031 of the virtual reality presentation interruption control unit 11030. And information indicating the moving direction is output.

  The virtual reality presentation interruption condition storage unit 13010 stores a virtual reality presentation interruption condition that is a condition for interrupting the presentation of virtual reality. FIG. 54 shows an example of the virtual reality presentation interruption condition. A plurality of conditions are stored in the virtual reality presentation interruption condition storage unit 13010 as virtual reality presentation interruption conditions.

  The virtual reality presentation interruption control unit 11030 includes a virtual reality presentation interruption determination processing unit 11031 and a virtual reality presentation interruption instruction unit 11032.

  The virtual reality presentation interruption determination processing unit 11031 determines whether or not the situation around the user detected by the surrounding situation detection unit 11010 satisfies the virtual reality presentation interruption condition. The virtual reality presentation interruption determination processing unit 11031 reads out the virtual reality presentation interruption condition stored in the virtual reality presentation interruption condition storage unit 13010 when a new detection result is obtained by the surrounding state detection unit 11010 and reads the detection result. To determine whether the current surrounding situation of the user satisfies the virtual reality presentation interruption condition.

  Specifically, the virtual reality presentation interruption determination processing unit 11031 satisfies the virtual reality presentation interruption condition when the information obtained from the image analysis result of the surrounding scene image captured by the surrounding photographing unit 1011 which is the surrounding state detection unit 11010 satisfies the virtual reality presentation interruption condition. It is determined whether or not. The virtual reality presentation interruption determination processing unit 11031 indicates information indicating the distance to each object around the user from the object distance calculation unit 13021, and indicates the moving speed of the moving object from the object moving speed calculation unit 13022. Each information is input and compared with the virtual reality presentation interruption condition to determine whether the condition for interrupting the virtual reality presentation is satisfied.

  For example, in the sixth condition of the virtual reality presentation interruption condition shown in FIG. 5415, it is set as an interruption condition that there is an object whose moving speed component in the user direction is 8 m / s or more. The virtual reality presentation interruption determination processing unit 11031 calculates an object moving speed as to whether or not there is an object whose moving speed in the direction of the user is 8 m / s or more among the objects for which the distance to the user is obtained by the object distance calculating unit 13021. The determination is made based on the moving speed obtained by the unit 13022.

  The virtual reality presentation interruption determination processing unit 11031 receives each condition set as a virtual reality presentation interruption condition when various information is input from a predetermined interval or from the surrounding state detection unit 11010 or the image analysis processing unit 13020. The comparison judgment process is performed.

  The virtual reality presentation interruption instruction unit 11032 issues an instruction to interrupt the virtual reality presentation to the virtual reality presentation unit 11020 based on the determination result in the virtual reality presentation interruption determination processing unit 11031. For example, if there is an object that moves at a speed of 14 m / sec among the objects calculated by the object moving speed calculation unit 13022, the virtual reality presentation interruption instruction unit is satisfied because the condition for interrupting the virtual reality presentation is satisfied. 11032 outputs an instruction signal instructing the virtual reality presentation unit 11020 to interrupt the presentation of the virtual reality.

  The virtual reality presentation unit 11020 that has received the interruption instruction during the presentation of the virtual reality interrupts the presentation of the virtual reality. For example, when the virtual reality presentation unit 11020 displays a video for virtual reality presentation, the display of the video is interrupted. Further, when the virtual reality presentation unit 11020 outputs a sound for virtual reality presentation, the output of the sound is interrupted.

  As described above, according to the virtual reality presentation system 13000 according to the tenth embodiment, various interruption conditions are stored in advance, and the situation around the user detected in real time by the surrounding situation detection unit 11010 and the interruption. By comparing the conditions, presentation of virtual reality can be appropriately controlled. Therefore, it is possible to switch whether the user senses the virtual reality world or the real world, and it is possible to consider the influence of the action of the user in the virtual reality world on the real world.

  Note that the virtual reality presentation interruption condition is not limited to the above-described conditions. For example, the virtual reality presentation system 13000 according to the tenth embodiment can be configured as shown in FIG. In the virtual reality presentation system 13000 shown in FIG. 55, the image analysis processing unit 13020 includes an object distance calculating unit 13021, an object moving speed calculating unit 13022, an object specifying unit 13023, an object type specifying unit 13024, and an object position specifying unit. 13025 and a user movement speed calculation unit 13026.

  The object specifying unit 13023 specifies each object included in the surrounding scenery image acquired by the surrounding photographing unit 1011 by a boundary determination process or the like in the image, and labels each object appearing in the specified surrounding scenery image. Process.

  The object type specifying unit 13024 specifies the type of each object specified by the object specifying unit 13023. Specifically, the object type specifying unit 13024 stores the feature patterns of each object in the real world in advance, and performs pattern matching that detects a feature pattern having a high correlation value with the object specified by the object specifying unit 13023. Process. The object type identification unit 13024 identifies the type associated with the feature pattern whose correlation value is equal to or greater than a predetermined reference value in the pattern matching process as the type of the object.

  Here, the object specified by the object type specifying unit 13024 also specifies a floor, a wall, or a person as an object. In addition, the object type identification unit 13024 is preferably configured to detect a step on the floor together. The object type identification unit 13024 outputs information indicating the type of each identified object to the virtual reality presentation interruption determination processing unit 11031.

  The object position specifying unit 13025 specifies the position coordinates of each object specified by the object specifying unit 13023. The object position specifying unit 13025 specifies the position of the object based on the difference in the position in the image of the same object that appears in the plurality of surrounding scenery images.

  The object distance calculation unit 13021 calculates a distance to the object by calculating a norm of a difference vector between the position coordinates of the object obtained by the object position specifying unit 13025 and the user position. The object distance calculation unit 13021 outputs information indicating the calculated distance to each object to the virtual reality presentation interruption determination processing unit 11031.

  The object moving speed calculation unit 13022 calculates the moving speed of the object based on the time change of the position coordinates of the object specified by the object position specifying unit 13025. The object movement speed calculation unit 13022 outputs the calculated movement speed of each object to the user movement speed calculation unit 13026 and the virtual reality presentation interruption determination processing unit 11031.

  The user movement speed calculation unit 13026 specifies whether the user is moving based on the movement speeds of a plurality of objects calculated by the object movement speed calculation unit 13022, and determines the user movement speed when the user is moving. calculate. When the user is moving, all the surrounding objects move relatively to the user. Therefore, it can be determined whether the object is moving or the user is moving based on the moving speeds of the plurality of objects. The user movement speed calculation unit 13026 outputs information indicating the calculated user movement speed to the virtual reality presentation interruption determination processing unit 11031.

  The virtual reality presentation interruption determination processing unit 11031 determines whether each information output from each unit of the image analysis processing unit 13020 satisfies a virtual reality presentation interruption condition stored in the virtual reality presentation interruption condition storage unit 13010, Based on the determination result, the virtual reality presentation unit 11020 is instructed to interrupt the virtual reality presentation.

  For example, the virtual reality presentation interruption determination processing unit 11031 follows the first interruption condition shown in FIG. 54 to obtain the user moving speed calculated by the user moving speed calculation unit 13026 and the objects calculated by the object distance calculation unit 13021. Based on the distance, it is determined whether there is an object at a position within 5 m of the user's moving direction. As a result of the determination, when there is an object at a position within 5 m of the user's moving direction, it is determined that the virtual reality presentation is interrupted.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 follows the second interruption condition of FIG. 54 to determine the type of each object specified by the object type specifying unit 13024 and the objects obtained by the object distance calculation unit 13021. Based on the distance, it is determined whether a person is present at a position within 3 m from the user. As a result of the determination, it is determined that the presentation of the virtual reality is interrupted when a person is present at a position within 3 m from the user.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 moves at a speed of 10 m / s or more based on the moving speed of the object calculated by the object moving speed calculation unit 13022 according to the third interruption condition of FIG. It is determined whether a certain object exists around the user. As a result of the determination, when an object moving at a speed of 10 m / s or more is present around the user, it is determined that the presentation of the virtual reality is interrupted.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 follows the fourth interruption condition of FIG. 54, and the user moves at a speed of 3 m / s or more based on the user movement speed calculated by the user movement speed calculation unit 13026. It is determined whether or not. As a result of the determination, it is determined that the virtual reality presentation is interrupted when the user is moving at a moving speed of 3 m / s or more.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 follows the fifth interruption condition of FIG. 54, and the user moving speed obtained by the user moving speed calculating part 13026 and each object obtained by the object distance calculating part 13021. Whether or not there is a step at a position within 5 m in the moving direction of the user. Since the object specifying unit 13023 recognizes a step as an object, the determination can be performed. As a result of the determination, when there is a step at a position within 5 m of the user's moving direction, it is determined that the presentation of virtual reality is interrupted.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 has a velocity component in the user direction of 8 m / s based on the object movement speed calculated by the object movement speed calculation unit 13022 in accordance with the sixth interruption condition of FIG. It is determined whether an object moving at the above speed exists around the user. For example, even if the speed is high, there is a low possibility of a problem even if there is a car that runs at a high speed parallel to the user at a distance from the user. As a result of the determination, it is determined that the presentation of the virtual reality is interrupted when there is an object around the user whose velocity component in the user direction is moving at a speed of 8 m / s or more.

  For example, the virtual reality presentation interruption determination processing unit 11031 follows the seventh interruption condition of FIG. 54 until the user's movement speed calculated by the user movement speed calculation unit 13026 and the object calculated by the object distance calculation unit 13021. Based on the distance, it is determined whether there is an object that reaches the user movement direction within 5 seconds. For example, even if there is a step near the front of the user, if the user's moving speed is sufficiently low, it is unlikely that it will cause an immediate obstacle. This is because it may become an obstacle at the earliest. As a result of the determination, when there is an object that reaches the user movement direction within 5 seconds, it is determined that the presentation of the virtual reality is interrupted.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 follows the eighth interruption condition of FIG. 54, and the object movement speed obtained by the object movement speed calculation part 13022 and each object obtained by the object distance calculation part 13021. Is determined whether there is an object whose moving speed component in the user direction is 3 m or more per second and whose distance from the user is within 20 m. In this way, it may be determined whether or not to stop presenting virtual reality using a plurality of conditions in combination. As a result of the determination, if there is an object that satisfies these conditions, it is determined that the presentation of the virtual reality is interrupted.

  Note that the above-described determination conditions and types of information to be input are examples, and can be changed as appropriate. For example, the virtual reality presentation system 13000 can be configured as shown in FIG. The virtual reality presentation system 13000 shown in FIG. 56 includes a user state detection unit 12010 that detects a user state, a voice input unit 13030 that inputs voice, and a user behavior specification unit 13040 that specifies user behavior. To do.

  In the virtual reality presentation system 13000 illustrated in FIG. 56, the user state detection unit 12010 includes a user position detection unit 12013, a user movement speed detection unit 12014, and a user orientation detection unit 12015.

  The user position detection unit 12013 is a GPS receiver installed in a virtual reality presentation device worn by the user, for example, and periodically detects the current position of the user and generates current position information. The user position detection unit 12013 outputs current position information indicating the current position of the user to the virtual reality presentation interruption determination processing unit 11031.

  The user moving speed detection unit 12014 detects the moving speed of the user. The user movement speed detection unit 12014 may detect the movement speed based on acceleration information obtained by an acceleration sensor or the like installed in a virtual reality presentation device worn by the user, or may be detected by the user position detection unit 12014. You may detect a moving speed from the difference of a user's present position.

  As described above, the virtual reality presentation system 13000 illustrated in FIG. 56 acquires the moving speed of the user by a method different from the image analysis processing as compared with the virtual reality presentation system 13000 illustrated in FIG. The user moving speed detection unit 12014 outputs information indicating the detected moving speed of the user to the virtual reality presentation interruption determination processing unit 11031.

  The user orientation detection unit 12015 detects the direction (azimuth) that the user is facing. The user orientation detection unit 12015 may be configured by, for example, an orientation magnet attached to a virtual reality presentation device worn by the user, or detects the user direction based on acceleration information detected by an acceleration sensor. May be. The user orientation detection unit 12015 outputs information indicating the detected user direction to the virtual reality presentation interruption determination processing unit 11031.

  The voice input unit 13030 inputs ambient voice. Specifically, the voice input unit 13030 can be a microphone installed in a virtual reality presentation device worn by the user.

  The user action specifying unit 13040 is a user's surroundings detected by the surrounding state detection unit 11010, a voice input by the voice input unit 13030, and a user state detected by the user state detection unit 12010 as necessary. Based on the state or the like, the action that the user is currently performing is specified.

  For example, the user action specifying unit 13040 includes a car handle in the surrounding scene image acquired by the surrounding state detection unit 11010, and the voice input by the voice input unit 13030 includes a car engine sound. In addition, the user specifies that he is driving a car. In addition, the user's behavior such as the user commuting, sleeping, resting, or working is specified. The user action specifying unit 13040 outputs information indicating the specified user action to the virtual reality presentation interruption determination processing unit 11031.

  The virtual reality presentation interruption determination processing unit 11031 determines whether or not the current position detected by the user position detection unit 12013 is a position range where virtual reality cannot be presented according to the ninth interruption condition of FIG. For example, when virtual reality presentation is prohibited at a station platform, the location range of the station platform is stored as an interruption condition. In FIG. 54, when the GPS coordinates of the current position of the user are within the ranges of latitudes 38.121 to 38.253 and longitudes 135.124 to 135.131, it is determined that the presentation of the virtual reality is interrupted.

  Further, for example, the virtual reality presentation interruption determination processing unit 11031 is an action that the current action of the user specified by the user action specifying part 13040 is driving a car according to the tenth interruption condition of FIG. In this case, it is determined that the presentation of the virtual reality is interrupted.

  The virtual reality presentation interruption condition stored in the virtual reality presentation interruption condition storage unit 13010 may be appropriately selected according to the type of virtual reality presented to the user.

  For example, a user who is presented with a virtual reality of the content of “fighting a dragon” is assumed to move his body, so there may be a problem if there is a person nearby. Therefore, the virtual reality presentation interruption determination processing unit 11031 performs determination based on the second interruption condition. On the other hand, it is unlikely that a user who is presented with a virtual reality of the content “decorate a vase in the house” will be a problem even if there are others nearby. Therefore, the virtual reality presentation interruption determination processing unit 11031 does not perform determination based on the second interruption condition.

  Thus, it is even better if each interruption condition is determined according to the type of virtual reality presented to the user. In this case, the virtual reality presentation system 13000 preferably has the configuration shown in FIG. The virtual reality presentation system 13000 shown in FIG. 57 includes a virtual reality management control unit 13050.

  The virtual reality management control unit 13050 performs management and control of the virtual reality presented by the virtual reality presenting unit 11020. Based on a request from the user, the virtual reality management control unit 13050 determines a virtual reality service to be provided to the user, and instructs the virtual reality presenting unit 11020 to present the virtual reality related to the determined virtual reality service. .

  In addition, the virtual reality management control unit 13050 notifies the virtual reality presentation interruption determination processing unit 11031 of a virtual reality service identification number for identifying the virtual reality service related to the virtual reality presented by the virtual reality presenting unit 11020.

  Each virtual reality service is associated with an interruption condition used for determination. FIG. 58 is a management file for managing the correspondence between each virtual reality service and the interruption condition. When the virtual reality presentation interruption determination processing unit 11031 receives information identifying the virtual reality service related to the virtual reality presented by the virtual reality presentation unit 11020 from the virtual reality management control unit 13050, the virtual reality presentation interruption determination processing unit 11031 makes a determination with reference to the management file. Identify the interruption conditions used for processing. For example, when the virtual reality service is a virtual reality service identified by No: VRS0001, a determination process is performed to determine whether the first interruption condition and the third interruption condition are satisfied. As described above, by adopting a configuration in which the interruption condition used for the determination is appropriately selected according to the presented virtual reality, it is possible to perform appropriate interruption processing without causing unnecessary interruption.

  Further, it is even better if each interruption condition used in the determination process is determined according to the user receiving the virtual reality presentation. Each user sets an interruption condition in advance, and the virtual reality presentation interruption determination processing unit 11031 determines whether to interrupt the virtual reality presented to the user by using an interruption condition determined for each user. It may be configured in this way.

(Embodiment 11)
A user using the virtual reality presentation system senses a virtual reality world different from the real world. At this time, the user may be suspicious that the world that he or she perceives is an artificially created virtual reality world, or the user may have the complete illusion that it is a real world There is also.

This introduces the concept of an illusion degree that indicates the degree to which a user receiving virtual reality is perceived as to the extent that the world they perceive is perceiving the real world. Can do.

  In the case where the user completely believes that the world that the user senses is the real world even though the virtual reality is presented, the illusion degree is 100. On the other hand, the user receiving the presentation of virtual reality understands that the world that he / she senses is a virtual reality world created artificially, and what part he / she senses is artificial. When it is recognized in a state where it can be clearly discriminated which part is a real part, the illusion degree is zero. As shown in FIG. 59, the user who is receiving the presentation of virtual reality is placed in any state of this continuous illusion.

  The virtual reality interruption function shown in the eighth to tenth embodiments performs processing to instantaneously return the user's illusion to 0 and pull the user back from the virtual reality world to the real world when a predetermined event occurs. It can be said that it is equivalent to

FIG. 60 shows temporal changes in the illusion degree of the user. The virtual reality presentation is started at T0, the virtual reality presentation is interrupted at T1, and the virtual reality presentation is resumed at T2. In the vicinity of T0 where the presentation of virtual reality is started, the user has a sense of incongruity in the presence of an object or a person represented by a newly appearing virtual reality object, so the user's illusion is low. However, with the passage of time, the person represented by the virtual reality object is controlled so as to be interlocked with the user's action and behaves so as to merge with the real world, so that the user is represented by the virtual reality object. Forgetting to say that a person or the like is a virtual reality object that has been artificially created, it becomes as if it is a real person. That is, the illusion degree increases.

  Here, if the video display of the virtual reality object is interrupted at the timing of T1 due to the interruption reason, the user immediately represented by the virtual reality object that is currently viewed is not in the real world, Recognizing that the object was an artificially created virtual reality object, the illusion degree falls to zero. Even if the presentation of virtual reality is resumed at T2, in addition to the user's illusion being reduced, the person who will perceive again is clearly recognized as a virtual reality object. Therefore, it will be said that the degree of illusion does not go up easily.

  The main purpose of the virtual reality presentation system is to allow the user to enter a virtual reality world that is different from the real world, and to enjoy a world that cannot be realized in the real world due to physical and social constraints. For this reason, it is preferable that a state where the illusion degree of the user is high can be maintained. However, if the virtual reality is frequently interrupted, the illusion of the user does not increase at all, and there is a problem that it cannot be said that the user enters the virtual reality world.

  In view of such a problem, in the virtual reality presentation system according to the eleventh embodiment, the user moves the virtual reality world while taking into consideration the influence of the action of the user sensing the virtual reality world on the real world. It is characterized by being able to continue to experience it.

  FIG. 61 is a block diagram illustrating a configuration of a virtual reality presentation system 14000 according to the eleventh embodiment. The virtual reality presentation system 14000 includes an ambient condition detection unit 11010, a virtual reality presentation unit 11020, a user state detection unit 12010, and an alerting control unit 14010. The surrounding state detection unit 11010 detects a state around the user, and the user state detection unit 12010 detects the current state of the user.

  The virtual reality presentation unit 11020 includes a video generation processing unit 11021, a video display unit 11022, and an audio output unit 11023. The video generation processing unit 11021 generates a virtual reality presentation video based on the current state of the user detected by the user state detection unit 12010. The video display unit 11022 displays the video generated by the video generation processing unit 11021. The audio output unit 11023 outputs audio corresponding to the virtual reality video generated by the video generation processing unit 11021.

  The alerting control unit 14010 alerts the user using the video display unit 11022 or the audio output unit 11023 based on the surroundings of the user detected by the surroundings state detection unit 11010.

  For example, the alerting control unit 14010 alerts the user by displaying an alerting video on the video display unit 11022. At this time, the video display unit 11022 multiplexes and displays the virtual reality video generated by the video generation processing unit 11021 and the video for alerting to the user without interrupting the presentation of the virtual reality. Call attention.

  Further, for example, the alerting control unit 14010 performs alerting by causing the audio output unit 11023 to output alerting sound. At this time, the voice output unit 11023 alerts the user without interrupting the presentation of the virtual reality by multiplexing and outputting the voice for virtual reality presentation and the voice for alerting.

  As described above, according to the virtual reality presentation system 14000 according to the eleventh embodiment, the user is alerted based on the situation in the real world without interrupting the presentation of the virtual reality. Accordingly, it is possible to control the influence of the user's behavior on the real world without greatly reducing the illusion of the user.

  FIG. 62 shows a temporal change between a real world landscape, a video displayed on the video display unit 11022, and a landscape perceived by the user in a case where attention is given. As the user's head rotates, the video of the dog, which is a virtual reality object, starts to be displayed at timing T3. Attention is performed by displaying a video for alerting at the timing of T5, but the video of the virtual reality object continues to be displayed. Therefore, even in the state where the alert is performed, the user continues to sense the virtual reality world where the dog indicated by the virtual reality object is lying on the sofa.

  Note that the virtual reality presentation system 14000 according to the eleventh embodiment is more favorable when configured as shown in FIG. A virtual reality presentation system 14000 shown in FIG. 63 includes an ambient state detection unit 11010, a virtual reality presentation unit 11020, a user state detection unit 12010, a virtual reality object storage unit 12020, a virtual reality object placement unit 12030, a virtual reality The audio data storage unit for presentation 12050, the alerting control unit 14010, the video data storage unit for alerting 4020, and the alerting audio data storage unit 4030 are provided.

  The alerting video data storage unit 4020 stores alerting video data. The alerting voice data storage unit 4030 stores alerting voice data.

  The virtual reality presentation unit 11020 includes a video generation processing unit 11021, a video display unit 11022, and an audio output unit 11023. The video display unit 11022 includes a video display control unit 11022-1 and a display panel 11022-2. The audio output unit 11023 includes an audio reproduction control unit 11023-1 and a speaker 11023-2. The video display control unit 11022-1 and the audio reproduction control unit 11023-1 are for alerting from the alerting video data storage unit 4020 and the alerting voice data storage unit 4030, respectively, in accordance with instructions from the alerting control unit 14010. The video data and audio data are read out and displayed and output.

  FIG. 64 is a block diagram showing a specific configuration of the virtual reality presentation unit 11020. In the virtual reality presenting unit 11020, the video display control unit 11022-1 includes a video data reading unit 11022-1a and a video multiplex processing unit 11022-1b.

  When the video data reading unit 11022-1a receives a warning instruction from the warning control unit 14010, the video data reading unit 11022-1a reads the video data for calling attention from the video data storage unit 4020 for calling attention and outputs it to the video multiplexing processing unit 11022-1b.

  The video multiplex processing unit 11022-1b performs a multiplexing process of superimposing the video of the virtual reality object generated by the video generation processing unit 11021 and the alerting video read by the video data reading unit 11022-1a. The multiplexed video is displayed on the display panel 11022-2.

  The audio reproduction control unit 11023-1 includes an audio data reading unit 11023-1a, a DA conversion processing unit 11023-1b, and an audio multiplex processing unit 11023-1c.

  The audio data reading unit 11023-1a reads the virtual reality presentation audio data stored in the virtual reality presentation audio data storage unit 12050. The audio data reading unit 11023-1a reads the virtual reality presentation audio data associated with the virtual reality object arranged in the virtual space by the virtual reality object arrangement unit 12030, and outputs it to the DA conversion processing unit 11023-1b.

  In addition, when the voice data reading unit 11023-1a receives a warning instruction from the warning control unit 14010, the voice data reading unit 11023-1a reads the warning data from the warning voice data storage unit 4030 and outputs it to the DA conversion processing unit 11023-1b. To do.

  The DA conversion processing unit 11023-1b performs processing for converting the virtual reality presentation audio data read by the audio data reading unit 11023-1a and the attention environment audio data from digital audio data to analog audio signals, respectively. . The converted analog audio signal is output to the audio multiplex processing unit 11023-1c.

  The sound multiplex processing unit 11023-1c performs a multiplexing process in which the virtual reality sound converted into the analog sound signal and the alerting sound are overlapped. The sound superimposed by the sound multiplex processing unit 11023-1c is emitted to the outside from the speaker 11023-2.

  More preferably, based on the placement position coordinates of the virtual reality object placed in the virtual space by the virtual reality object placement unit 12030 and the user position obtained based on the user state detected by the user state detection unit 12010, It is preferable to include a voice distribution processing unit that weights and distributes the actual presentation voice to a plurality of channels. In this case, the alerting sound can be evenly distributed to a plurality of channels regardless of the user state and the arrangement position of the virtual reality object.

Embodiment 12
When presenting virtual reality to the user, the user perceives a virtual reality world that is different from the real world that others are sensing, while the user's behavior is not only the virtual reality world, Also affects the world. Therefore, it is preferable that the virtual reality presentation system that presents the virtual reality performs overall control regarding the presentation of the virtual reality while considering the influence of the user's action in the real world.

  FIG. 65 is a block diagram illustrating a configuration of a virtual reality presentation system 15000 according to the twelfth embodiment. The virtual reality presentation system 15000 is the same as the virtual reality presentation system 10000, the surrounding state detection unit 11010, the virtual reality presentation unit 11020, the image analysis processing unit 13020, the alerting control unit 14010, the alerting condition storage unit 15010, Is provided.

  In the virtual reality presentation system 15000 according to the twelfth embodiment, the surrounding state detection unit 11010 includes a surrounding photographing unit 1011 that photographs at least the surroundings of the user and acquires a surrounding scenery image. Specifically, the ambient photographing unit 1011 corresponds to a camera attached to a virtual reality presentation device such as an HMD device worn by a user. In the virtual reality presentation device, at least a plurality of cameras are installed as the surrounding photographing unit 1011. The surrounding scenery image acquired by the surrounding photographing unit 1011 is output to the image analysis processing unit 13020 as surrounding state information.

  The image analysis processing unit 13020 performs a process of extracting predetermined information by analyzing the surrounding scenery image in which the surrounding scenery of the user taken by the surrounding photographing unit 1011 is captured. In the virtual reality presentation system 15000 according to the twelfth embodiment, the image analysis processing unit 13020 includes an object distance calculation unit 13021 and an object moving speed calculation unit 13022.

  The object distance calculation unit 13021 specifies the position of the object shown in the surrounding scenery image. Specifically, the object distance calculation unit 13021 identifies boundaries of a plurality of objects shown in the surrounding scene image by comparing color difference signals, luminance signals, and the like of adjacent pixels or blocks in the surrounding scene image. Label the identified object. The object distance calculation unit 13021 performs the boundary determination process and the labeling process on the surrounding scene image acquired by the plurality of surrounding photographing units 1011 installed in the virtual reality presentation device. The object distance calculation unit 13021 calculates the distance to the object based on the difference between the positions in the image of the object with the same label. Here, the object distance calculation unit 13021 is more preferably configured to calculate the position coordinates of the object. The object distance calculation unit 13021 outputs information related to the calculated distance to the object to the object moving speed calculation unit 13022 and the attention determination processing unit 14011 of the attention control unit 14010.

  The object moving speed calculation unit 13022 calculates the moving speed of the object in the user direction based on the temporal change in the distance to the object calculated by the object distance calculation unit 13021. When the object distance calculation unit 13021 functions as an object position specifying unit that obtains the position coordinates of each object that appears in the surrounding scenery image, the object moving speed calculation unit 13022 is based on the time change of the position coordinates of the object. It is even better if the movement vector of the object is calculated. The object moving speed calculation unit 13022 calculates the moving speed and moving direction of the object shown in the surrounding scenery image. The object moving speed calculation unit 13022 outputs the calculated moving speed of the object to the attention determination processing unit 14011 of the attention control unit 14010. When the object movement speed calculation unit 13022 is configured to calculate the movement vector of the object, the object movement speed calculation unit 13022 uses the movement speed and movement direction calculated by the attention determination processing unit 14011 of the attention control unit 14010. The information shown is output.

  The alerting condition storage unit 15010 stores alerting conditions that are conditions for alerting the presentation of virtual reality. FIG. 66 shows an example of an alerting condition. A plurality of conditions are stored in the alert condition storage unit 15010 as alert conditions.

  Here, in the alerting condition stored in the alerting condition storage unit 15010, a looser condition is set as compared with the virtual reality presentation suspension condition stored in the virtual reality presentation suspension condition storage unit 13010 described above.

  The alerting control unit 14010 includes an alerting determination processing unit 14011 and an alerting instruction unit 14012.

  The alert determination processing unit 14011 determines whether the user's surroundings detected by the ambient state detection unit 11010 satisfy the alerting condition. When a new detection result is obtained by the surroundings state detection unit 11010, the attention determination processing unit 14011 reads out the warning condition stored in the attention condition storage unit 15010 and compares it with the detection result. It is determined whether the current surrounding situation of the user satisfies the alert condition.

  Specifically, the alert determination processing unit 14011 determines whether the information obtained from the image analysis result of the surrounding scene image captured by the ambient photographing unit 1011 that is the ambient condition detection unit 11010 satisfies the alert condition. . The alert determination processing unit 14011 receives information indicating the distance to each object around the user from the object distance calculating unit 13021 and information indicating the moving speed of the moving object from the object moving speed calculating unit 13022. Each is input and compared with the alerting condition to determine whether the alerting condition is satisfied.

  For example, in the sixth condition of the alert condition shown in FIG. 66, it is set as an alert condition that there is an object whose moving speed component in the user direction is 6 m / s or more. The attention determination processing unit 14011 determines whether there is an object whose moving speed in the direction of the user is 6 m / s or more among the objects whose distance to the user is obtained by the object distance calculating unit 13021. The determination is made based on the moving speed obtained in step (1).

  The attention determination processing unit 14011 performs comparison determination processing with each condition set as the alerting condition when various information is input from a predetermined interval or from the surrounding state detection unit 11010 or the image analysis processing unit 13020 I do.

  The attention instructing unit 14012 issues an instruction to alert the user to the virtual reality presenting unit 11020 based on the determination result in the attention determining unit 14011. For example, when there is an object moving at a speed of 7 m / sec among the objects calculated by the object moving speed calculation unit 13022, the condition for performing the alert is satisfied, and therefore the alert instruction unit 14012 An instruction signal for instructing the presentation unit 11020 to execute alerting is output.

  The virtual reality presenting unit 11020 that has received the above-mentioned alerting instruction during the presentation of the virtual reality alerts the user. For example, when the virtual reality presentation unit 11020 displays a virtual reality presentation video, the virtual reality presentation unit 11020 instructs the virtual reality presentation video to be overlaid with a warning video. Further, when the virtual reality presentation unit 11020 outputs a virtual reality presentation sound, the virtual reality presentation unit 11020 instructs to multiplex and reproduce the output of the virtual reality presentation sound.

As described above, according to the virtual reality presentation system 15000 according to the twelfth embodiment, various alerting conditions are stored in advance, and the situation around the user detected in real time by the ambient condition detection unit 11010 By comparing the alerting conditions, it is possible to appropriately alert the user. Therefore, by continuing to present the virtual reality to the user, the user can be alerted in consideration of the influence of the user on the real world while the user stays in the virtual reality world. It is possible to consider the influence of the actions of users in the world on the real world.

  Note that the alerting conditions are not limited to the above-described conditions. For example, the virtual reality presentation system 15000 according to the twelfth embodiment can be configured as shown in FIG. In the virtual reality presentation system 15000 shown in FIG. 67, the image analysis processing unit 13020 includes an object distance calculating unit 13021, an object moving speed calculating unit 13022, an object specifying unit 13023, an object type specifying unit 13024, and an object position specifying unit. 13025 and a user movement speed calculation unit 13026.

  The object specifying unit 13023 specifies each object included in the surrounding scenery image acquired by the surrounding photographing unit 1011 by boundary determination processing in the image, and performs a labeling process on each object appearing in the specified surrounding scenery image. Do.

  The object type specifying unit 13024 specifies the type of each object specified by the object specifying unit 13023. Specifically, the object type specifying unit 13024 stores the feature patterns of each object in the real world in advance, and performs pattern matching that detects a feature pattern having a high correlation value with the object specified by the object specifying unit 13023. Process. The object type identification unit 13024 identifies the type associated with the feature pattern whose correlation value is equal to or greater than a predetermined reference value in the pattern matching process as the type of the object.

  Here, the object specified by the object type specifying unit 13024 also specifies a floor, a wall, or a person as an object. In addition, the object type identification unit 13024 is preferably configured to detect a step on the floor together. The object type identification unit 13024 outputs information indicating the type of each identified object to the alert determination processing unit 14011.

  The object position specifying unit 13025 specifies the position coordinates of each object specified by the object specifying unit 13023. The object position specifying unit 13025 specifies the position of the object based on the difference in the position in the image of the same object that appears in the plurality of surrounding scenery images.

  The object distance calculation unit 13021 calculates a distance to the object by calculating a norm of a difference vector between the position coordinates of the object obtained by the object position specifying unit 13025 and the user position. The object distance calculation unit 13021 outputs information indicating the calculated distance to each object to the attention determination processing unit 14011.

  The object moving speed calculation unit 13021 calculates the moving speed of the object based on the time change of the position coordinates of the object specified by the object position specifying unit 13025. The object movement speed calculation unit 13021 outputs the calculated movement speed of each object to the user movement speed calculation unit 13026 and the attention determination processing unit 14011.

  The user movement speed calculation unit 13026 specifies whether the user is moving based on the movement speeds of a plurality of objects calculated by the object movement speed calculation unit 13021, and determines the user movement speed when the user is moving. calculate. When the user is moving, all the surrounding objects move relatively to the user. Therefore, it can be determined whether the object is moving or the user is moving based on the moving speeds of the plurality of objects. The user movement speed calculation unit 13026 outputs information indicating the calculated user movement speed to the alert determination processing unit 14011.

  The alert determination processing unit 14011 determines whether each information output from each unit of the image analysis processing unit 13020 satisfies the alert condition stored in the alert condition storage unit 15010, and based on the determination result. The virtual reality presentation unit 11020 is instructed to alert the virtual reality presentation.

  For example, the alert determination processing unit 14011, according to the first alert condition of FIG. 66, the user's moving speed obtained by the user moving speed calculating unit 13026 and the distance to each object obtained by the object distance calculating unit 13021. Based on the above, it is determined whether there is an object at a position within 7 m in the moving direction of the user. As a result of the determination, if there is an object at a position within 7 m of the user's moving direction, it is determined to call attention.

  Further, for example, the attention determination processing unit 14011, according to the second alert condition in FIG. 66, the type of each object specified by the object type specifying unit 13024 and the distance to each object obtained by the object distance calculation unit 13021. Based on the above, it is determined whether there is a person within 5 m from the user. As a result of the determination, if there is a person at a position within 5 m from the user, it is determined to call attention.

  For example, the attention determination processing unit 14011 is moving at a speed of 7 m / s or more based on the moving speed of the object calculated by the object moving speed calculating unit 13022 in accordance with the third warning condition of FIG. It is determined whether an object exists around the user. As a result of the determination, when an object moving at a speed of 7 m / s or more is present around the user, it is determined to call attention.

  Further, for example, whether the user is moving at a speed of 2 m or more per second based on the user's moving speed calculated by the user moving speed calculating unit 13026 according to the fourth warning condition shown in FIG. Determine. As a result of the determination, when the user is moving at a moving speed of 2 m / s or more, it is determined to call attention.

  In addition, for example, the attention determination processing unit 14011 determines the user's movement speed obtained by the user movement speed calculation unit 13026 and each object obtained by the object distance calculation unit 13021 according to the fifth attention calling condition of FIG. Based on the distance, it is determined whether or not there is a step at a position within 7 m in the moving direction of the user. Since the object specifying unit 13023 recognizes a step as an object, the determination can be performed. As a result of the determination, when there is a step at a position within 7 m of the user's moving direction, it is determined to call attention.

  In addition, for example, the alert determination processing unit 14011 has a speed component in the user direction of 6 m or more per second based on the moving speed of the object calculated by the object moving speed calculating unit 13022 according to the sixth alerting condition of FIG. It is determined whether an object moving at a speed of is present around the user. For example, even if the speed is high, there is a low possibility of a problem even if there is a car that runs at a high speed parallel to the user at a distance from the user. As a result of the determination, if there is an object around the user whose speed component in the direction of the user is moving at a speed of 6 m / s or more, it is determined to call attention.

  Also, for example, the alert determination processing unit 14011 follows the seventh alert condition in FIG. 66 to the user moving speed calculated by the user moving speed calculating unit 13026 and the object calculated by the object distance calculating unit 13021. Based on the distance, it is determined whether there is an object that reaches the user movement direction within 8 seconds. For example, even if there is a step near the front of the user, if the user's moving speed is sufficiently low, it is unlikely that it will cause an immediate obstacle. On the other hand, even if there is a step far in front of the user, the user's moving speed is sufficient. This is because it may become an obstacle at the earliest. As a result of the determination, when there is an object that arrives within 8 seconds in the user movement direction, it is determined to call attention.

  In addition, for example, the attention determination processing unit 14011 performs the object movement speed obtained by the object movement speed calculation unit 13022 and each object obtained by the object distance calculation unit 13021 in accordance with the eighth attention calling condition of FIG. Is determined whether there is an object whose moving speed component in the user direction is 2 m or more per second and whose distance from the user is within 30 m. In this way, it may be determined whether or not to alert the presentation of virtual reality using a plurality of conditions together. As a result of the determination, if there is an object that satisfies these conditions, it is determined to call attention.

  Note that the above-described determination conditions and types of information to be input are examples, and can be changed as appropriate. For example, the virtual reality presentation system 15000 can be configured as shown in FIG. The virtual reality presentation system 15000 shown in FIG. 68 includes a user state detection unit 12010 that detects a user state, a voice input unit 13030 that inputs a voice, and a user behavior identification unit 13040 that identifies a user behavior. To do.

  In the virtual reality presentation system 15000 shown in FIG. 68, the user state detection unit 12010 includes a user position detection unit 12013, a user movement speed detection unit 12014, and a user orientation detection unit 12015.

  The user position detection unit 12013 is a GPS receiver installed in a virtual reality presentation device worn by the user, for example, and periodically detects the current position of the user and generates current position information. The user position detection unit 12013 outputs current position information indicating the current position of the user to the alert determination processing unit 14011.

  The user moving speed detection unit 12014 detects the moving speed of the user. The user movement speed detection unit 12014 may detect the movement speed based on acceleration information obtained by an acceleration sensor or the like installed in a virtual reality presentation device worn by the user, or may be detected by the user position detection unit 12014. You may detect a moving speed from the difference of a user's present position.

  As described above, the virtual reality presentation system 15000 shown in FIG. 68 acquires the moving speed of the user by a method different from the image analysis processing as compared with the virtual reality presentation system 15000 shown in FIG. The user movement speed detection unit 12014 outputs information indicating the detected movement speed of the user to the alert determination processing unit 14011.

  The user orientation detection unit 12015 detects the orientation that the user is facing. The user orientation detection unit 12015 may be configured with, for example, an orientation magnet attached to a virtual reality presentation device worn by the user, or detects the user direction based on acceleration information detected by an acceleration sensor. May be. The user orientation detection unit 12015 outputs information indicating the detected user orientation to the alert determination processing unit 14011.

  The voice input unit 13030 inputs ambient voice. Specifically, the voice input unit 13030 can be a microphone installed in a virtual reality presentation device worn by the user.

The user action specifying unit 13040 is a user's surroundings detected by the surrounding state detection unit 11010, a voice input by the voice input unit 13030, and a user state detected by the user state detection unit 12010 as necessary. Based on the state or the like, the action that the user is currently performing is specified.

  For example, the user action specifying unit 13040 includes a car handle in the surrounding scene image acquired by the surrounding state detection unit 11010, and the voice input by the voice input unit 13030 includes a car engine sound. In addition, the user specifies that he is driving a car. In addition, the user's behavior such as the user commuting, sleeping, resting, or working is specified. The user behavior specifying unit 13040 outputs information indicating the specified user behavior to the alert determination processing unit 14011.

  The attention determination processing unit 14011 determines whether or not the current position detected by the user position detection unit 12013 is a position range to be alerted according to the ninth alert condition in FIG. For example, when it is assumed that attention is required for presentation of virtual reality at a station platform, the location range of the station platform is stored as an alert condition. In FIG. 66, when the GPS coordinates of the current position of the user are within the range of latitudes 38.10 to 38.253 and longitudes 135.120 to 135.140, it is determined to call attention.

  In addition, for example, the alert determination processing unit 14011, according to the tenth alert condition in FIG. 66, is the action that the user's current action specified by the user action specifying unit 13040 is driving a car. It is determined to call attention to.

  Note that the alerting condition stored in the alerting condition storage unit 15010 may be appropriately selected according to the type of virtual reality presented to the user.

  For example, a user who is presented with a virtual reality of the content of “fighting a dragon” is assumed to move his body, so there may be a problem if there is a person nearby. Therefore, the alert determination processing unit 14011 performs determination based on the second alert condition. On the other hand, it is unlikely that a user who is presented with a virtual reality of the content “decorate a vase in the house” will be a problem even if there are others nearby. Therefore, the alert determination processing unit 14011 does not perform determination based on the second alert condition.

  Thus, it is even better if each alerting condition is determined according to the type of virtual reality presented to the user. In this case, the virtual reality presentation system 15000 preferably has the configuration shown in FIG. A virtual reality presentation system 15000 shown in FIG. 69 includes a virtual reality management control unit 13050.

  The virtual reality management control unit 13050 performs management and control of the virtual reality presented by the virtual reality presenting unit 11020. Based on a request from the user, the virtual reality management control unit 13050 determines a virtual reality service to be provided to the user, and instructs the virtual reality presenting unit 11020 to present the virtual reality related to the determined virtual reality service. .

  In addition, the virtual reality management control unit 13050 notifies the alert determination processing unit 14011 of a virtual reality service identification number for identifying the virtual reality service related to the virtual reality presented by the virtual reality presenting unit 11020.

  Each virtual reality service is associated with an alert condition used for determination. For example, when the information for identifying the virtual reality service related to the virtual reality presented by the virtual reality presenting unit 11020 is input from the virtual reality management control unit 13050, the alert determination processing unit 14011 sets the alert condition used for the determination. Identify the alert condition used for the judgment process by referring to the management file to be managed. For example, if the virtual reality service is a virtual reality service identified by No: VRS0001, a determination process is performed to determine whether the second alert condition and the fifth alert condition are satisfied. In this way, it is possible to perform appropriate alerting processing without causing unnecessary alerting by adopting a configuration in which the alerting condition used for determination is appropriately selected according to the virtual reality presented Become.

  In addition, it is even better if each alerting condition used in the determination process is determined according to the user receiving the virtual reality presentation. Each user sets an alerting condition in advance, and the alerting determination processing unit 14011 determines whether to alert the virtual reality presented to the user using the alerting condition determined for each user. It may be configured in this way.

  In addition, it is more preferable that the alerting condition used for the determination is associated with the alerting method. For example, as shown in FIG. 70, the attention instructing unit 14012 refers to a management file that summarizes the correspondence between the alerting conditions hit by the attention determination processing unit 14011 and the alerting method performed by the virtual reality presenting unit 11020. Then, the virtual reality presentation unit 11020 is instructed to call attention.

  In FIG. 70, when it is determined by the attention determination processing unit 14011 that “there is an object at a position within 7 m of the moving direction of the user” which is the first attention condition shown in FIG. 14012 instructs the video display control unit 11022-1 to read the alerting video data Wm023.jpg stored in the storage address (x0000000001) and display it on the display panel 11022-2 for 3 seconds.

  When the alert determination processing unit 14011 determines that the user is moving at a speed of 2 m / s or more as the fourth alert condition shown in FIG. 66, the alert instruction unit 14012 stores the storage address The video display control unit 11022-1 is instructed to read the video data Wm059.jpg stored in (x0000010001) and display it on the display panel 11022-2 until the fourth warning condition is canceled. At the same time, the audio data Wm59.mp3 stored in the storage address (x1100010001) is read, and the audio reproduction control unit 11023-1 is continuously reproduced until the fourth alerting condition is canceled. Instruct. As a result of the determination in the alert determination processing unit 14011, if the alert condition is eliminated based on the result of the image analysis process, the alert instructing unit 14012 displays the video display control unit 11022-1, audio playback control unit, and the like. 11023-1 is instructed to stop alerting.

  With this configuration, instead of simply issuing a message saying “Please be careful” to the user, if it is determined that there is a failure ahead, “Be careful because there is a failure ahead. If you find that a moving object such as a car is approaching from behind, a car with a speed of 40 km / h is approaching from behind. If you decide that the user is driving a car, use the virtual reality service while driving a car is recommended. It is possible not to call it. ” Therefore, based on the specific content of the alert, the user can avoid the virtual world by taking avoidance action so that the impact in the real world is reduced. Can be managed.

(Embodiment 13)
The virtual reality presentation system according to the eleventh and twelfth embodiments is compared with the virtual reality presentation system according to the eighth to tenth embodiments, without returning the user who has entered the virtual reality world to the real world. The virtual reality world and the real world are harmonized by managing the impact on

  Here, from the viewpoint of the user in the virtual reality world, it is preferable that the influence on the real world is managed without reducing the illusion of the virtual reality world as much as possible. However, if an alert window that is not related to the virtual reality world is displayed, or an alert such as an electronic sound or guide voice that is not related to the virtual reality world is played, When compared with the case where the user's consciousness deviates from the virtual reality world and the presentation of the virtual reality is interrupted, the decrease in the illusion degree is suppressed, but some decrease in the illusion degree is unavoidable Will remain. Therefore, the virtual reality presentation system according to the thirteenth embodiment appropriately alerts the user to suppress the influence on the real world while suppressing a decrease in the illusion degree of the user experiencing the virtual reality world. It is characterized by that.

FIG. 71 is a block diagram illustrating a configuration of a virtual reality presentation system 16000 according to the thirteenth embodiment. The virtual reality presentation system 16000 includes an ambient condition detection unit 11010, a virtual reality presentation unit 11020, a user state detection unit 12010, a virtual reality object placement unit 12030, a virtual reality management control unit 13050, and a virtual reality content storage unit 16010. And a virtual reality object behavior control unit 16020.

The virtual reality management control unit 13050 manages the virtual reality presented to each user by performing a virtual reality presentation start process and an end process for the user. The virtual reality management control unit 13050 executes the control program read from the virtual reality content storage unit 16010 to start virtual reality when starting presentation of virtual reality to the user based on a virtual reality presentation request from the user. Instructions necessary for starting the presentation of virtual reality are issued to the real object placement unit 12030 and the virtual reality object behavior control unit 16020, respectively.

The virtual reality content storage unit 16010 stores virtual reality content used for presentation of virtual reality. As shown in FIG. 72, the virtual reality content includes a virtual reality control program executed for presentation of virtual reality in addition to model data and audio data of the virtual reality object.

Virtual reality control programs are roughly classified into a virtual reality main control program, a virtual reality sub-control program, a virtual reality object personality control program, and a virtual reality object behavior control program. FIG. 73 shows the hierarchical structure of the virtual reality control program.

  The virtual reality object behavior control program is a program for controlling the behavior of the virtual reality object arranged in the virtual space, and is read by the virtual reality object personality control program. For example, when the virtual reality object is a human object, the virtual reality object behavior control program A performs a control such that the joint point of the right knee is slightly bent and the texture data of the mouth is gradually changed to the smile texture data. It is a program. On the other hand, the virtual reality object behavior control program N performs control such that the hip joint point is bent so as to slide, and the right arm and the right elbow joint point are rotated by a predetermined amount to project the right arm forward. By executing the behavior control program, the appearance shape of the object arranged in the virtual space changes as the frame progresses.

  The virtual reality object behavior control program further includes a normal-time virtual reality object behavior control program that is executed to present virtual reality at a normal time, and a caution virtual reality object behavior control program that is executed at the time of alerting .

  The virtual reality object personality control program is a program read by the virtual reality sub-control program, and controls the correspondence between the input and output for the virtual reality object arranged in the virtual space.

  The virtual reality sub-control program is a program that is appropriately called by the virtual reality main control program, and corresponds to the provided virtual reality service. The virtual reality sub-control program reads the virtual reality object used for the virtual reality service being provided in the virtual space, or directly reads out the virtual reality object behavior control program according to a preset order, Control is performed to perform a predetermined behavior.

  The virtual reality main control program is a program corresponding to each virtual reality service, and is a program executed first when the provision of the virtual reality service is started. By executing the virtual reality main control program, the virtual space setting and environment setting for the virtual reality service corresponding to the program are performed.

  The virtual reality object placement unit 12030 reads out a virtual reality object used for the virtual reality to be presented from the virtual reality content storage unit 16010 based on the virtual reality presentation start instruction in the virtual reality management control unit 13050, and performs predetermined processing in the virtual space. Place at the position.

  The virtual reality object behavior control unit 16020 controls the behavior of the virtual reality object arranged in the virtual space. The virtual reality object behavior control unit 16020 executes the virtual reality object behavior control program from the virtual reality content storage unit 16010, thereby controlling the behavior of the virtual reality object arranged in the virtual space. The virtual reality object behavior control unit 16020 generates a video of the virtual reality object whose appearance shape changes in the virtual space, and the video display processing unit 11022 displays the video.

  Also, the virtual reality object behavior control unit 16020 reads the audio data of the virtual reality object from the virtual reality content storage unit 16010 and outputs the audio data to the audio reproduction control unit 11023-1, thereby reproducing the audio related to the presentation of the virtual reality.

  The virtual reality object behavior control unit 16020 executes the normal-time virtual reality object behavior control program based on an instruction from the virtual reality management control unit 13050, thereby changing the behavior of the virtual reality object arranged in the virtual space. Control.

  On the other hand, when the virtual reality object behavior control unit 16020 receives an instruction to perform attention using a virtual reality object from the attention control unit 14010, the virtual reality object behavior control unit 16020 generates a virtual reality object control program for alerting based on the instruction. The program is read from the storage unit 16010 and executed. As described above, the virtual reality object behavior control unit 16020 alerts the user using the virtual reality object used in the currently presented virtual reality.

FIG. 74 is a flowchart showing the flow of the alerting control process executed by the alerting control unit 14010. When the surrounding situation is updated (Yes in step S1101), the alerting control unit 14010 executes determination processing for determining whether or not alerting is performed. Note that the alert control unit 14010 may determine that the surrounding situation has been updated, for example, when the surrounding situation information is newly generated by the surrounding situation detection unit 11010, or when a predetermined time interval has elapsed. It may be determined that the surrounding situation has been updated.

  The alerting control unit 14010 executes determination processing for determining whether or not alerting is performed based on the ambient state information input from the ambient state detection unit 11010 (step S1102). For example, the alert control unit 14010 determines whether or not the current surrounding situation of the user satisfies the alert condition by comparing the input ambient condition information with a preset alert condition.

  As a result of the determination in step S1102, if the alert condition is not satisfied (No in step S1103), the process returns to step S1101 and waits for the next update of the surrounding situation.

  On the other hand, if the result of determination in step S1102 is that the alert condition is satisfied (Yes in step S1103), the alert controller 14010 determines whether alert has already been performed (step S1104).

  As a result of the determination, if an alert has already been issued, for example, if it is determined that an alert condition has been satisfied at a past timing and an alert has been issued (Yes in step S1104), step S1101 Return to and wait until the next update timing.

  On the other hand, when alerting is not currently performed (No in step S1104), the alerting control unit 14010 instructs execution of alerting using a virtual reality object (step S1105).

  FIG. 75 is a flowchart showing the flow of the virtual reality object placement process. The virtual reality object placement unit 12030 determines whether to newly place a virtual reality object in the virtual space (step S1201). For example, the virtual reality object placement unit 12030 determines whether to place a virtual reality object based on whether a virtual reality object placement instruction has been received from the virtual reality management control unit 13050. The virtual reality management control unit 13050 issues a virtual reality object placement instruction by advancing the virtual reality service in accordance with, for example, the passage of time, the user state, and the change in the surrounding situation.

  When it is determined in step S1201 that a virtual reality object is newly arranged, the virtual reality object placement unit 12030 reads the virtual reality object from the virtual reality content storage unit 16010 and places it in the virtual space (step S1202).

  Next, the virtual reality object arrangement unit 12030 determines whether or not to delete the virtual reality object currently arranged in the virtual space from the virtual space (step S1203).

  If it is determined in step S1203 that the virtual reality object is to be deleted from the virtual space, the virtual reality object placement unit 12030 deletes the virtual reality object determined to be deleted from the virtual space, and returns to step S1201 (step S1204).

  In this way, by deleting virtual reality objects that no longer need to be placed in the virtual space as appropriate, only the virtual reality objects used for presenting the virtual reality remain in the virtual space, and the video related to the virtual reality object Display and audio playback are performed.

  FIG. 76 is a flowchart showing the flow of a virtual reality object behavior control process. The virtual reality object behavior control unit 16020 determines whether or not a virtual reality object behavior control instruction has been issued (step S1301). The instruction is issued to the virtual reality object behavior control unit 16020 based on the virtual reality main control program and the virtual reality sub-control program executed in the virtual reality management control unit 13050.

  When receiving the instruction, the virtual reality object behavior control unit 16020 reads the instructed normal-time virtual reality object behavior control program from the virtual reality object storage unit 12020 and executes it (step S1302). Here, the normal-time virtual reality object behavior control program is a program used to execute a virtual reality service provided to the user.

  Here, when an interrupt instruction is input from the alerting control unit 14010 (Yes in step S1303), the virtual reality object behavior control unit 16020 indicates the alerting virtual reality object instructed from the virtual reality content storage unit 16010. The behavior control program is read and executed (step S1304).

  Here, the virtual reality object behavior control program for alerting is a normal time based on the virtual reality service being provided to the user when the alert controller 14010 determines that the alerting condition set in advance is satisfied. It is a program that is executed preferentially by interrupting the virtual reality object behavior control program.

  For example, it is assumed that the user is provided with a virtual reality service in which “Great Greek philosopher Aristotle gives a philosophy lecture by gesturing for 30 minutes”. In this case, the virtual reality content storage unit 16010 stores virtual reality object model data imitating the Aristotle, audio data of the lecture content imitating Aristotle's voice, and virtual reality that operates the virtual reality object of the Aristotle. A control program is stored.

  Based on the use request of the virtual reality service from the user, the virtual reality management control unit 13050 sets the virtual space by reading and executing the virtual reality main control program and the sub control program related to the virtual reality service, The virtual reality object placement unit 12030 reads and places the Aristotle virtual reality object in the virtual space.

  The virtual reality object behavior control unit 16020 executes the virtual reality object behavior control program from the virtual reality content storage unit 16010 as appropriate according to an instruction issued based on the sub-control program executed by the virtual reality management control unit 13050. The appearance shape of the virtual reality object arranged in the space is changed. A video of the virtual reality object is generated by the video generation processing unit 11021. Also, the virtual reality object behavior control unit 16020 reads the audio data of the lecture content that the virtual reality object of the Aristotle reads from the virtual reality content storage unit 16010 and sends it to the audio reproduction control unit 11023-1. The audio reproduction control unit 11023-1 has the relative position relationship between the user position detected by the user state detection unit 12010 and the virtual reality object position arranged by the virtual reality object arrangement unit 12030 for the transmitted audio data. Based on this, it is distributed to a plurality of audio channels, DA conversion is performed, and sound is emitted from the speaker 11023-2.

  By continuing such processing, the user senses the video and audio in which Aristotle, who is represented by the virtual reality object represented in the user's field of view, lectures in front of his / her eyes.

  In the situation where the virtual reality service is provided in this way, when the alert control unit 14010 determines that the alert condition is satisfied, the virtual reality object behavior control program for lecture is temporarily stopped and alerted instead. A virtual reality object behavior control program is executed.

  For example, if the alert control unit 14010 determines that there is a failure ahead, the virtual reality object behavior control unit 16020 indicates that the currently presented Aristotle virtual reality object is “There is a failure ahead. Please read and execute the virtual reality object control program for alerting that makes a gesture and sound like "Please do".

  Further, for example, when the alert control unit 14010 determines that a moving body such as a car is approaching from behind, the virtual reality object behavior control unit 16020 displays the virtual reality object of Aristotle that is currently presented. Read and execute a virtual reality object control program for alerting that makes a gesture and voice saying “A car at 40 km / h is approaching from behind.

When the alerting is finished (Yes in step S1305), the virtual reality object behavior control unit 16020 resumes execution of the stopped virtual reality object behavior control program for normal time (step S1306). On the other hand, when the alerting is not finished (No in step S1305), the alerting virtual reality object behavior program is continuously executed.

  As described above, according to the virtual reality presentation system according to the thirteenth embodiment, the user is alerted using the virtual reality object used for virtual reality presentation. That is, control is performed so that alerting to suppress the influence of the action of the user who senses the virtual reality world on the real world is performed in the virtual reality world instead of the real world. For example, when the user wants to move the user in the right direction in the real world, the user's behavior is managed by talking to the user so that the virtual reality object moves in the right direction.

  According to the said structure, since the user is acting so that the influence in the real world may be suppressed based on the alerting performed in the virtual reality world, the degree of illusion of the user in the virtual reality world decreases. Can be prevented.

  In this way, by executing the alert using the virtual reality object used in the virtual reality service while interrupting the virtual reality service provided to the user, the illusion degree of the user is increased or decreased by the alert. The user can stably enjoy the virtual reality world.

  Note that if the influence by the user's action is not improved even if the alert using the virtual reality object is performed, the alert controller 14010 does not use the virtual reality object shown in the eleventh and twelfth embodiments. It is good also as a structure which performs. If the influence of the user's action is not improved, it is even better if the virtual reality presentation shown in Embodiments 8 to 10 is interrupted.

  That is, the alert determination processing unit 14011 determines whether the surrounding state detected by the ambient state detection unit 11010 and the user state detected by the user state detection unit 12010 satisfy the first type of alert condition. Do. The first type of alert is an alert with a low degree of urgency. If it is determined that the alert condition is satisfied as a result of the determination, the virtual reality object is executed by executing a method that is a predetermined alert program that changes the behavior of the virtual reality object currently placed in the virtual space. Raise attention on the world side. At this time, it is preferable that a plurality of alert conditions are stored in the alert condition storage unit 15010 according to the type of virtual reality service currently provided to the user. The alert determination processing unit 14011 is set for the virtual reality service currently provided to the user by the surrounding state detected by the surrounding state detection unit 11010 and the user state detected by the user state detection unit 12010. It is configured to determine whether or not the alerting condition is satisfied. Note that the alerting condition may be set for the virtual reality service itself or may be set for the class to which the virtual reality service belongs. In addition, the method corresponding to the alerting program executed for the virtual reality object is appropriate based on the surrounding state detected by the surrounding state detection unit 11010 and the user state detected by the user state detection unit 12010. It is recommended to select a method that selects the correct method. That is, the virtual reality object behavior control unit 16020 uses the surrounding state detected by the surrounding state detection unit 11010 and the user state detection unit 12010 when the attention calling unit 14010 determines that the virtual reality object is to call attention. Attention using the virtual reality object is performed by executing a method corresponding to the attention determined based on the detected user state.

  Further, the alert determination processing unit 14011 determines whether the surrounding state detected by the ambient state detection unit 11010 and the user state detected by the user state detection unit 12010 satisfy the second type of alert condition. Do. The second type of alert is an alert with a high degree of urgency. As a result of the determination, if it is determined that the alerting condition is satisfied, the video display control unit 11022-1 reads the alerting video data from the alerting video data storage unit 4020, and the video generation processing unit 11021 A process of multiplexing the video data of the generated virtual reality object is performed. The virtual reality world side is alerted by executing a method that is a predetermined alerting program for changing the behavior of the virtual reality object currently arranged in the virtual space. The alerting video is not a video obtained by drawing a virtual reality object arranged in the virtual space, and thus is a non-virtual reality video. Therefore, when the user's head or line of sight changes, the virtual reality video generated by the video generation processing unit 11021 changes by drawing the virtual reality object, but the alerting video is displayed on the display panel 11022-2. Will continue to be displayed at the same position.

  In addition, the virtual reality presentation interruption determination processing unit 11031 determines whether the surrounding state detected by the surrounding state detection unit 11010 and the user state detected by the user state detection unit 12010 satisfy the virtual reality presentation interruption condition. Do. The virtual reality presentation interruption condition is the condition with the highest degree of urgency. If it is determined that the virtual reality presentation interruption condition is satisfied as a result of the determination, the video display control unit 11022-1 displays the virtual reality video generated by the video generation processing unit 11021 on the display panel 11022-2. Stop displaying.

  In this way, by adopting a configuration in which alerting and virtual reality presentation are interrupted according to a plurality of stages, it is possible to realize appropriate virtual reality presentation.

  Also, as a method of alerting, for example, when a vehicle is approaching from the right front and the video of the virtual reality object is displayed on the right side of the display panel 11022-2, the virtual reality object is moved to the left side. The behavior of the virtual reality object may be controlled so that the right field of view of the user can be secured by walking.

(Embodiment 14)
In the virtual reality presentation systems shown in the eighth to thirteenth embodiments, the influence of the user's action on the real world is managed by interrupting or alerting the presentation of the virtual reality. That is, in Embodiments 8 to 10, the user's consciousness is returned to the real world by reducing the user's illusion degree to 0, whereas in Embodiments 11 and 12, the user's consciousness is reduced without reducing the illusion degree. It tries to indirectly reach the real world.

  As can be seen from FIG. 37, the presentation of virtual reality is the replacement of part or all of the information perceived by the user with artificial information, thereby replacing the virtual reality world created artificially with the real world. It is as if the user has an illusion.

  Therefore, as the amount of information from the real world is reduced and the information from the virtual reality presentation device is increased, the virtual reality in the world sensed by the user increases. For example, in (a) shown in FIG. 77, the information presented to the user from the virtual reality presentation device is 0, and therefore the world sensed by the user is the real world itself. On the other hand, in (b), about 30% of the information from the real world is blocked and supplemented with information from the virtual reality presentation device. Therefore, the world perceived by the user is a virtual reality world. However, since 70% of information uses information in the real world as it is, the virtual reality in the virtual reality world is relatively low. On the other hand, in (c), 90% of the information is replaced, and most of the world that the user senses is artificially created, and the virtual reality is high.

  Alerting the user means directing the user's consciousness in the virtual reality world to the real world. By performing this control, the user's action in the virtual reality world gives to the real world. The purpose is to mitigate the impact. For this purpose, it is preferable that the configuration is such that the user's attention is directed to the real world by controlling the virtual reality instead of controlling the illusion. As described above, in the virtual reality presentation system according to the fourteenth embodiment, by controlling the virtual reality degree in the virtual reality presented to the user, the action of the user who senses the virtual reality world is given to the real world. It is characterized by managing impacts.

FIG. 78 is a block diagram illustrating a configuration of a virtual reality presentation system 17000 according to the fourteenth embodiment. The virtual reality presentation system 17000 includes an ambient condition detection unit 11010, a virtual reality presentation unit 11020, and a virtual reality management unit 17010.

  The virtual reality degree indicates the magnitude of deviation between the virtual reality world presented to the user and the real world. For example, the virtual reality world B shown in FIG. 38 has a large overlap with the real world, and most of the virtual reality world sensed by the user as shown in FIG. 77B is information from the real world. In other words, the virtual reality in such a virtual reality world is low. On the other hand, the virtual reality world F shown in FIG. 38 has a small overlap with the real world, and the virtual reality presentation device creates most of the virtual reality world sensed by the user as shown in FIG. 77C. This indicates that the information is artificial information, and the virtual reality in such a virtual reality world is high.

  The virtual reality level Rd is Rd = {INA / (INR + INA)} when the amount of information received by the user from the real world is INR and the amount of artificial information presented by the virtual reality presentation device is INA. It is defined as an amount represented by x100.

  It should be noted that a more accurate calculation formula for the virtual reality Rd can be defined by performing correction based on a correlation value between the user's real-world environment and artificially presented virtual reality video and audio. preferable. The reason why it is better to perform the above correction is that when the user is currently in the zoo, the virtual reality world felt by the user when displaying the video of the virtual reality object of the giraffe and when the user is at home This is because the virtual reality world felt by the user when displaying the video of the virtual reality object is greatly different from the viewpoint of unusualness, and the latter has a low illusion of the user. Users with low illusions recognize that even if a virtual reality world is presented, it is a false image, and the user's attention is strongly directed toward the real world, so virtual reality The degree will go down. Here, this point of view is omitted, and the virtual reality Rd is simplified by the above formula, and the value displayed in 256 levels as represented by 8 bits is the virtual reality as referred to in this specification. explain.

  The virtual reality management unit 17010 sets the virtual reality of the virtual reality that can be presented by the virtual reality presenting unit 11020 based on each information detected by the surrounding situation detection unit 11010 and the user state detection unit 12010.

  For example, when the user is alone at home, even if the user perceives a virtual reality world that is significantly different from the real world, there is no problem because the influence of the user's action on the real world is small. . On the other hand, when the user is in a public place, if the difference between the virtual reality world perceived by the user and the real world where there is another person is large, the user has made it to fit the virtual reality world The possibility that the content of the action will have a greater impact on the real world increases. In addition, when the user is stopped, even if it does not matter if the virtual reality world sensed by the user is significantly different from the real world, if the user is moving at high speed, Even if they act to fit the virtual reality world, it can be a problem in the real world.

  Therefore, the virtual reality management unit 17010 manages the virtual reality indicating the degree of divergence between the virtual reality world presented by the virtual reality presenting unit 11020 and the real world based on the user state and the surrounding situation. The virtual reality is expressed in, for example, 256 levels. The virtual reality degree 0 indicates that all of the information sensed by the user is information in the real world, and the virtual reality presentation device does not display video or play audio at all. On the other hand, the virtual reality 256 is an image in which all the images displayed on the display panel 11022-2 arranged in front of the user's eyes are artificially created, and the sound in the real world is blocked. It shows that the sound created artificially by the speaker 11023-2 is emitted.

  Note that since humans have a higher rate of visual information acquisition than auditory sense, for example, out of 256 levels, the virtual reality level by video is 192 levels, and the virtual reality level by sound is expressed by 64 levels. For example, the virtual reality is 192 in the case where an artificial image is displayed in a state where light from the real world is blocked on the entire display panel 11022-2 without playing any sound, whereas the display panel 11022 is displayed. -2 does not display any artificial video, and the virtual reality is 64 when the real world voice is cut off and the artificial voice is reproduced. Thus, the virtual reality degree indicates the size of the proportion of artificial information in the information received by the user from the virtual reality presentation device.

  The virtual reality presentation unit 11020 presents virtual reality so as to match the virtual reality managed by the virtual reality management unit 17010. For example, when the virtual reality degree that can be presented is set as 100 in the virtual reality degree management unit 17010, the virtual reality presentation unit 11020 presents the virtual reality so as to be less than or equal to the virtual reality degree. Therefore, in this case, the virtual reality presentation unit 11020 cannot display the video of the virtual reality object on the entire display panel 11022-2. The virtual reality presentation unit 11020 presents virtual reality while controlling the information amount of video and audio related to the virtual reality object.

  As described above, the virtual reality presentation system 17000 according to the fourteenth embodiment is characterized in that the virtual reality degree of the virtual reality to be presented is controlled based on the state of the user or the situation around the user. Therefore, when it is desired to direct the user's attention to the real world based on the surrounding situation or the like, control is performed to lower the virtual reality and lower the relative amount of information presented from the virtual reality presentation unit 11020 to the user. That is, control is performed to relatively increase the amount of information sensed from the real world compared to artificial information. With this configuration, it is possible to consider the influence of the user's action on the real world while continuing to present the virtual reality.

  Note that the virtual reality presentation system 17000 is even better when configured as shown in FIG. The virtual reality presentation system 17000 shown in FIG. 79 includes an ambient state detection unit 11010, a virtual reality presentation unit 11020, a user state detection unit 12010, a virtual reality object placement unit 12030, a voice input unit 13030, and virtual reality management control. Unit 13050, virtual reality content storage unit 16010, virtual reality object behavior control unit 16020, and virtual reality management unit 17010.

  The virtual reality management unit 17010 includes a presentable virtual reality calculation processing unit 17011, a scheduled virtual reality calculation processing unit 17012, a presentation virtual reality distribution processing unit 17013, and a virtual reality reduction ratio instruction unit 17014. .

  The presentable virtual reality calculation processing unit 17011 is a virtual reality that can be presented to the user based on the user's surroundings detected by the surrounding state detection unit 11010 and the user's state detected by the user state detection unit 12010. Calculate virtual reality. The presentable virtual reality calculation processing unit 17011 may be configured to calculate the virtual reality of the virtual reality that can be provided according to the virtual reality service provided to the user. For example, the presentable virtual reality calculation processing unit 17011 includes the virtual reality service provided to the user, the user's surrounding state detected by the surrounding state detection unit 11010, and the user state detected by the user state detection unit 12010. Based on the state, the virtual reality of virtual reality that can be presented to the user is calculated.

  The scheduled presentation virtual reality calculation processing unit 17012 receives the video data generated by the video generation processing unit 11021 and the audio data sent from the virtual reality object behavior control unit 16020 to the audio reproduction control unit 11023-1. The virtual reality degree of the virtual reality scheduled to be presented is calculated.

  The presentable virtual reality distribution processing unit 17013 uses the virtual reality calculated by the presentable virtual reality calculation processing unit 17011 based on the virtual reality expected to be presented obtained by the scheduled presentation virtual reality calculation processing unit 17012. Process to distribute the reality.

  For example, the presentable virtual reality calculated by the presentable virtual reality calculation processing unit 17011 is 120, the scheduled presentation virtual reality related to the video calculated by the planned presentation virtual reality calculation processing unit 17012 is 80, audio It is assumed that the presentation planned virtual reality related to is 30. In this case, since the total of the presentation-scheduled virtual reality levels is lower than the presentable virtual reality level, the presentable virtual reality distribution processing unit 17013 determines that the present status can be presented as it is.

  On the other hand, for example, the presentable virtual reality calculated by the presentable virtual reality calculation processing unit 17011 is 90, and the scheduled presentation virtual reality related to the video calculated by the planned presentation virtual reality calculation processing unit 17012 is 70. Suppose that the virtual reality degree to be presented related to the voice is 30. In this case, since the total of the virtual realities scheduled to be presented exceeds the virtual reality that can be presented, the presentable virtual reality distribution processing unit 17013 determines that virtual reality cannot be presented as it is.

  In this case, the presentable virtual reality distribution processing unit 17013 sets the presentable virtual reality to be for video so that the virtual reality of 70 + 30 = 100 currently scheduled to be presented is equal to or less than the presentable virtual reality 90. Distribute for audio.

  For example, the presentable virtual reality distribution processing unit 17013 sets a virtual reality for video to 63 and a virtual reality for audio to 27 by multiplying a ratio of 90/100.

  As another method, the presentable virtual reality distribution processing unit 17013 distributes the virtual reality so that more virtual reality is allocated for video. For example, the presentable virtual reality distribution processing unit 17013 maintains the virtual reality for video 70 as it is, and assigns the virtual reality 20 for audio.

  The virtual reality reduction ratio instructing unit 17014 includes the video display control unit 11022-1 and the audio reproduction control unit as virtual reality reduction ratios according to the distribution of the presentable virtual reality determined by the presentable virtual reality distribution processing unit 17013. 11023-1. For example, in the first example, it is instructed that both the reduction ratios are 10%. In the second example, the video display control unit 11022-1 has a reduction ratio of 0%, and the audio reproduction control unit 11023- 1 is instructed that the reduction ratio is 33.3%.

  The video display control unit 11022-1 performs a correction process based on the virtual reality reduction ratio instructed by the virtual reality reduction ratio instruction unit 17014 on the virtual reality object video generated by the video generation processing unit 11021. Do. Further, the audio reproduction control unit 11023-1 corrects the audio data sent from the virtual reality object behavior control unit 16020 based on the virtual reality reduction ratio instructed by the virtual reality reduction ratio instruction unit 17014. Process. Here, the audio reproduction control unit 11023-1 performs a process of lowering the audio level as a specific correction process.

  FIG. 80 is a block diagram showing a specific configuration of the virtual reality presenting unit 11020. The video display control unit 11022-1 includes a video correction processing unit 11022-1c. The video correction processing unit 11022-1c performs correction processing on the video of the virtual reality object generated by the video generation processing unit 11021, and performs control to display the video after the correction processing on the display panel 11022-2. Specifically, the video correction processing unit 11022-1c performs a correction process for reducing the luminance of the video of the virtual reality object generated by the video generation processing unit 11021, or the importance level of the virtual reality objects included in the video is high. Correction processing such as removing a low virtual reality object from the video is performed.

  The audio reproduction control unit 11023-1 includes a first DA conversion processing unit 11023-1d, a first distribution processing unit 11023-1e, a first amplification processing unit 11023-1f, a second DA conversion processing unit 11023-1g, A two-distribution processing unit 11023-1h, a second amplification processing unit 11023-1i, and a reproduction speech synthesis processing unit 11023-1j.

  The first DA conversion processing unit 11023-1d converts the audio data for virtual reality presentation sent from the virtual reality object behavior control unit 16020 into an analog audio signal.

  The first distribution processing unit 11023-1 e converts the analog audio converted based on the current position of the user detected by the user state detection unit 12010 and the position of the virtual reality object arranged by the virtual reality object arrangement unit 12030. Distribute the signal to multiple channels.

The first amplification processing unit 11023-1 f performs gain adjustment on the distributed audio of each channel based on the reduction ratio instructed from the virtual reality reduction ratio instruction unit 17014. For example, when the virtual reality level for audio reproduction is small, adjustment is performed to lower the audio level according to the reduction ratio.

  The second DA conversion processing unit 11023-1 g converts audio data obtained by collecting real world audio sent from the audio input unit 13030 into an analog audio signal. The virtual reality presentation device is provided with a microphone that collects the voice around the user as the voice input unit 13030. The digital audio data collected by the microphone and generated by AD conversion is subjected to processing for returning to an analog audio signal by performing DA conversion.

  The second distribution processing unit 11023-1h distributes the converted analog audio signal to a plurality of channels. When there is only one microphone provided as the audio input unit 13030, since there is no position resolution, the audio input by the audio input unit 13030 is equally distributed to all channels. On the other hand, when a plurality of microphones are provided as the voice input unit 13030, the voice input by the voice input unit 13030 is equally distributed to each channel in accordance with the arrival direction of each voice.

  The second amplification processor 110233-1i performs gain adjustment on the distributed audio of each channel. For example, the voice input by the voice input unit 13030 is amplified with an amplification factor determined by a gain set by the user himself / herself and an amplification factor in the first amplification processing unit 11023-1f.

  The reproduction speech synthesis processing unit 11023-1j synthesizes the audio of each channel amplified by the first amplification processing unit 11023-f and the second amplification processing unit 11023-j, and outputs the synthesized speech to the speaker 11023-2. The speaker 11023-2 emits sound in a state where the sound for virtual reality presentation that has been adjusted for virtual reality is superimposed on the surrounding sound in the real world of the user.

  According to this configuration, since the amount of information presented to the user is adjusted based on the situation around the user, it is possible to control whether the user's attention is directed to the virtual reality world or the real world. .

  FIG. 81 shows a video obtained by the video correction processing in the video correction processing unit 11022-1c and a scene sensed by the user. In FIG. 81 (a), the virtual reality object is zero because the virtual reality object is not included in the video generated by the video generation processing unit 11021, and the user sees the real world as it is. is there. In the real world, sofas are arranged on the left and right sides of the user.

  In (b) of FIG. 81, a virtual reality object of a lion and a virtual reality object of a dog are arranged in the virtual space, and the video generation processing unit 11021 generates videos of these virtual reality objects. Since the drawing viewpoint position and line-of-sight direction in the virtual space are controlled based on the user's state, the user senses that the lion and the dog represented by the virtual reality object are lying on the sofa, respectively. . It is assumed that the virtual reality degree of the virtual reality world where the lion and the dog lie is 120.

  Based on the user status and the surrounding situation, when the virtual reality distribution processing unit 17013 that can be presented for video exceeds 120, it can be displayed without performing correction processing. The video correction processing unit 11022-1c of the unit 11022-1 does not perform the correction process based on the virtual reality, and the video shown in FIG. 81B is displayed on the display panel 11022-2 as it is.

  On the other hand, if the virtual reality allocated by the presentable virtual reality distribution processing unit 17013 for video is lower than 120 based on the user state and the surrounding situation, it cannot be displayed on the display panel 11022-2 as it is. Therefore, the video correction processing unit 11022-1c of the video display control unit 11022-1 performs a correction process for reducing the virtual reality degree for the video of the virtual reality object generated by the video generation processing unit 11021.

  In FIG. 81 (c), the video correction processing unit 11022-1c performs correction processing for deleting the video part of the virtual reality object of the dog among the two virtual reality objects included in the video. By the correction process, the virtual reality of the corrected video is lowered from 120 to 100. If the virtual reality allocated by the presentable virtual reality distribution processing unit 17013 for the video is 100 or more, the corrected video from which the video of the dog is deleted is sent to the display panel 11022-2 and displayed. .

  Here, if the virtual reality allocated by the presentable virtual reality distribution processing unit 17013 for video is less than 100, the correction processing still does not satisfy the condition of virtual reality. Therefore, the video correction processing unit 11022-1c performs a correction process for reducing the luminance of the video of the virtual reality object.

  In FIG. 81 (d), since the luminance in the video of the remaining lion virtual reality object is lowered, the real world behind the portion where the video of the virtual reality object is displayed on the display panel 11022-2. The object is visible. That is, the information acquired by the user is increasing from the real world. In FIG. 81 (d), the virtual reality level has dropped to 60 due to the correction processing by the luminance reduction, and display is possible.

  Note that the correction processing performed when the virtual reality in the video generated by the video generation processing unit 11021 exceeds the reference value is limited to a configuration in which a part of the virtual reality object is first deleted. Instead, the brightness of the entire virtual reality object is lowered first so that the light from the real world can enter the user's eyes in a state where it overlaps the virtual reality object. It may be controlled to go to.

  In addition, it is better if the configuration in which the brightness of the virtual reality object arranged at which position is lowered or deleted from the video is determined based on the situation around the user and the user state.

  For example, if the vehicle approaches from the left front of the user, and the virtual reality presentation system wants to direct the user's attention to the vehicle, the video of the dog virtual reality object displayed on the right side of the user is deleted. Even if the brightness is reduced, the user's attention is not directed to the direction of the car in question. In this case, it is preferable to reduce the brightness in the image of the virtual reality object of the lion arranged on the left side of the user so that the user can appropriately see the car passing through the lion and approaching from behind.

  Therefore, the virtual reality management unit 17010 does not calculate the virtual reality that can be presented for video, but divides the video into a plurality of areas, and displays video display area 1, video display area 2,. It is further preferable that the virtual reality that can be presented to each video display area in the form of the area N is calculated and notified to the video correction processing unit 11022-1c.

In this case, for example, even when the vehicle is approaching from the left front of the user, if the video panel of the virtual reality object is not included in the left half of the display panel 11022-2, the virtual reality degree is displayed in the right half, for example. Even if a video of a high virtual reality object is displayed, since the virtual reality that can be provided in each video display area is satisfied, the video can be displayed without performing correction processing as it is.

FIG. 82 shows the relationship among the presentable virtual reality, the presentation-scheduled virtual reality, and the corrected virtual reality that are divided by region. The presentable virtual reality calculation processing unit 17011 calculates the virtual reality that can be presented for each display area in the nine-divided display panel 11022-2 based on the surrounding situation, the user state, and the like. In FIG. 82, since there is an obstacle near the user's lower left foot, the virtual reality that can be presented around the lower left is smaller than the virtual reality that can be presented around the lower right.

The scheduled presentation virtual reality calculation unit 17012 calculates the virtual reality for each display area as the presentation scheduled virtual reality for the video of the virtual reality object generated by the video generation processing unit 11021. In FIG. 82, in the generated video, the video of the lion's virtual reality object is included on the left side, and the video of the dog's virtual reality object is included on the right side, and the video image of the virtual reality object is included. Then, the virtual reality degree is high.

Here, the presentable virtual reality degree in the lower left column and the lower middle column is set to 0 by the presentable virtual reality calculation unit 17011, and the virtual reality object generated by the video generation processing unit 11021 in the region Cannot be displayed as they are because the virtual reality levels are 12 and 5, respectively. Therefore, the video correction processing unit 11022-1c corrects the virtual reality level of the area to be 0, which is the presentable virtual reality level.

  Here, since the virtual reality degree 0 means that no presentation regarding the virtual reality is made, the video correction processing unit 11022-1c sets the luminance of the virtual reality object in the area to 0.

In addition, while the virtual reality degree that can be presented in the left middle stage is 10, the virtual reality degree of the video generated by the video generation processing unit 11021 is 33 and cannot be displayed as it is. Therefore, the video correction processing unit 11022-1c reduces the brightness of the video of the lion, which is a virtual reality object included in the left middle region, until the virtual reality level becomes 10, thereby transmitting the lion behind. You can see the objects in the real world.

On the other hand, since the virtual reality object of the dog arranged on the right side is lower than the presentable virtual reality level, the video generated by the video generation processing unit 11021 is displayed without correction based on the virtual reality level. It is possible.

  In the above description, the video correction processing unit 11022-1c is configured to reduce the virtual reality of the video presented to the user by reducing the luminance of the virtual reality object, but the present invention is not limited to this. Changes can be made as appropriate according to the configuration of the virtual reality presentation device that presents virtual reality to the user.

  For example, when the virtual reality presentation device including at least the display panel 11022-2 is a transmissive HMD device and a liquid crystal panel or the like that controls the amount of light emitted from the outside world is disposed, the video correction processing unit 11022- First, by increasing the amount of light emitted from the outside world, the information amount is relatively increased compared to the information from the image of the virtual reality object that has been created artificially. You may control to.

  In the case where the virtual reality presentation device is not a transmissive display, it is preferable that the video captured by the video imaging unit 1011 and the video of the virtual reality object are combined and displayed on the display panel. In this case, it is preferable to change the weighting in the synthesis of the video shot by the video shooting unit 1011 and the video of the virtual reality object so that the video of the real world in the portion hidden in the video of the virtual reality object can be seen through. .

  In the above description, a case has been described in which the virtual reality level to be presented to the user is controlled to fall within the virtual reality level that can be presented by controlling the luminance of the video of the virtual reality object. However, the present invention is not limited to this. . For example, the virtual reality reduction ratio instructing unit 17014 gives an instruction to delete the number or capacity of virtual reality objects arranged in the virtual space to the virtual reality object arranging unit 12030, so that the virtual reality presented to the user is displayed. The virtual reality may be controlled so as to be within the presentable virtual reality.

(Embodiment 15)
While various functions constituting the virtual reality presentation system according to the eighth to fourteenth embodiments are assumed to be configured to be provided independently in the virtual reality presentation device worn by the user, as described in the first to seventh embodiments. In some cases, it may be preferable to have an external server in charge of part of the processing.

  FIG. 83 is a block diagram illustrating a configuration of the virtual reality presentation system 18000 according to the fifteenth embodiment. The virtual reality presentation system 18000 includes a virtual reality presentation device 18100, a virtual reality service providing server 18200, a contractor information storage database 18300, and a virtual reality content storage database 18400.

The virtual reality presentation device 18100 is a device worn by the user on the head, a head mounted display (HMD) device in which a display panel is arranged in front of the user's eyes, a contact lens type device worn by the user's eyes, a user An implantable device that is implanted in the human eyeball, or an implantable device that is implanted in the user's brain in the form of an electrode.

  The virtual reality presentation device 18100 may be a non-transparent HMD device. In this case, a video obtained by synthesizing a video for presenting virtual reality with a video taken by the line-of-sight camera is displayed on the display panel.

  In this case, in the normal time when no alert is given, the video is displayed by combining the two videos with the video captured by the line-of-sight camera as the lower layer and the video of the virtual reality object as the upper layer at the virtual reality object display position. Displayed in a panel. On the other hand, at the time of alerting, pixel data obtained by synthesizing the pixel data in the video of the virtual reality object and the pixel data of the video taken by the line-of-sight camera is displayed at the display position of the virtual reality object.

  For example, at predetermined coordinates (x, y) at which the virtual reality object is displayed, the pixel data of the video of the virtual reality object generated by the video generation processing unit is (Y1, U1, V1), and the line of sight at the coordinates It is assumed that the pixel data of the video imaged by the camera is (Y2, U2, V2). In a normal time without alerting, when the placement position of the virtual reality object is closer to the user side than the object shown in the video captured by the line-of-sight camera, the pixel data represented by the coordinates on the display panel is (Y1, On the other hand, when the object shown in the video captured by the line-of-sight camera is closer to the user side than the arrangement position of the virtual reality object, the pixel data represented by the coordinates on the display panel is (Y2, U2, V2).

  Here, when alerting is performed by the alerting process in the alerting control unit, even when the virtual reality object is on the user side, the pixel data represented by the coordinates is (Y1, U1, V1). (([Alpha] Y1 + (1- [alpha]) Y2, [beta] U1 + (1- [beta]) U2), [gamma] V1 + (1- [gamma]) V2). Here, α, β, and γ are 0 ≦ α, β, and γ ≦ 1, respectively.

  FIG. 84 shows an image displayed on the display panel in accordance with the level of alerting in the non-transmissive HMD device. As shown in FIG. 84, the user is in the room where the sofa is placed, and the line-of-sight camera (front camera) arranged so as to correspond to the line of sight of the user in front of the non-transparent HMD device. Scenery is taken. Also, an image of the elephant virtual reality object is generated in the image generation processing unit.

  The video display control unit included in the non-transmissive HMD device generates a display video by combining the video captured by the line-of-sight camera and the video of the virtual reality object generated by the video generation processing unit. Control the display. Here, the elephant, which is a virtual reality object, is arranged at a controlled position so as to stand on the sofa.

  Here, when it is determined that the weak alert level is to be issued as a result of the determination of necessity of alerting performed by the alert controller, the video display controller changes the α from 1 to 0.8. Two images are combined to generate a display image. As can be seen from FIG. 84, at the pixel position of the virtual reality object, the value obtained by mixing the components of the image captured by the line-of-sight camera becomes pixel data at the position, so that it can be seen through the image of the elephant that is the virtual reality object. In addition, the tree that is the object of the real world behind is faintly visible.

  On the other hand, as a result of determining whether or not the alerting control is required, the video display control unit changes the α to 0.6 and changes the two videos when it is determined that the middle level alerting is performed. Are combined to generate a display video. In this case, since images of the real world scene are mixed at a rate of 1−α = 0.4, the appearance of trees and the like behind the virtual reality object that is an elephant enters the user's eyes more clearly. Therefore, even when the user is normally looking forward, the user has acquired more information from the real world, and the user's attention is directed to the real world.

  On the other hand, as a result of the determination of necessity of alerting performed by the alerting control unit, when it is determined that the strong alerting is performed, the video display control unit changes the above α to 0.4 and changes the two videos. Are combined to generate a display video. Therefore, an image of a tree or the like behind the virtual reality object that is an elephant is more impressed by the user, and attention to the real world is stronger.

  In this way, the alerting control unit performs control to change the values of α, β, and γ based on the strength of alerting, and in the image composition processing unit included in the video display control unit, the video of the virtual reality object And the video imaged by the line-of-sight camera are combined according to the above formula at each coordinate.

  The alerting control unit increases the required alerting strength when the user's moving speed is fast, when there is an obstacle near the user, or when the moving body approaching the user's direction is fast, etc. Set. In other words, the attention control unit sets the values of α, β, and γ to be larger, and emphasizes and displays the real world video captured by the line-of-sight camera on the display panel than the virtual reality object video. Take control.

  It should be noted that the coefficient controlled by the video composition performed in conjunction with the above alerting may be only the coefficient α of the luminance signal Y, and the coefficients β and γ for the color difference signals U and V are also controlled. Also good.

  The virtual reality presentation device 18100 is connected to the virtual reality service providing server 18200 via the communication network 10 such as the Internet.

  The virtual reality service providing server 18200 provides a virtual reality service to the virtual reality presentation device 18100 via the communication network 10.

  In the contractor information storage database 18300, information of each user who receives the provision of the virtual reality service is stored as contractor information, and the virtual reality service available to the user indicated by the contractor information is the virtual reality service. The providing server 18200 provides the user.

  The virtual reality content storage database 18400 stores virtual reality content used for the virtual reality service. The virtual reality service providing server 18200 presents virtual reality to the user wearing the virtual reality presentation device 18100 by reading out and executing the virtual reality content related to the virtual reality service provided to the user from the virtual reality content storage database 18400. To do.

  As described above, the various functions included in the virtual reality presentation system described in the eighth to fourteenth embodiments may be configured to be arranged in either the virtual reality presentation device 18100 or the virtual reality service providing server 18200 worn by the user. it can. Here, the virtual reality presentation device 18100 corresponds to the virtual reality presentation device 100 described in the first to seventh embodiments, and the virtual reality service providing server 18200 includes the virtual reality service providing server 1300 described in the first to seventh embodiments. 2300, 4300. For example, various functions such as the ambient state detection unit 11010, the user state detection unit 12010, the video display unit 11022, the audio output unit 11023, and the audio input unit 3030 described above are implemented on the virtual reality presentation device 18100 side, and virtual reality video generation processing is performed. Unit 11021, virtual reality presentation suspension control unit 11030, virtual reality object storage unit 12020, virtual reality object placement unit 12030, video data storage unit for suspension 12040, audio data storage unit for virtual reality presentation 12050, audio data storage unit for suspension 12060 , Video data selection processing unit 12070 for interruption, audio data selection processing unit 12080 for interruption, virtual reality presentation interruption condition storage unit 13010, image analysis processing unit 13020, user action specifying unit 13040, virtual reality management control unit 13050, alert control Part 14010, note Functions such as the alerting video data storage unit 14020, the alerting voice data storage unit 14030, the alerting condition storage unit 15010, the virtual reality content storage unit 16010, the virtual reality object behavior control unit 16020, and the virtual reality management unit 17010 The configuration may be implemented on the virtual reality service providing server 18200 side.

  In the case of a configuration including an LCD that controls the incident state of external light, the virtual reality correction may be performed by controlling the amount of light incident from the LCD instead of controlling the luminance using the LCD.

  For example, the virtual reality presentation device 18100 is an HMD device that includes a transmissive display including an LCD that controls external light, and emits external light at a position corresponding to the video of the virtual reality object generated by the video generation processing unit 11021. Control the LCD to shut off.

  In other words, the virtual reality presentation device 18100 generates a video by controlling the transmission amount of natural light incident from the front of the user and generating a video by controlling the transmission amount of light incident from the light source lamp. A second display mechanism that combines the video generated by the first display mechanism and the video generated by the second display mechanism, and the images generated by the two display mechanisms are combined. The video is presented to the user.

  In the virtual reality presentation device 18100, the user state detection unit detects the moving speed of the user, and based on the detected moving speed, the first state at a position corresponding to the video of the virtual reality object generated by the second display mechanism. Control the video of the display mechanism. If the moving speed is fast, control is performed to increase the light transmission amount in the first display mechanism at the position corresponding to the video of the virtual reality object, and control is performed to decrease the light transmission amount when the user's moving speed is slow. .

(Embodiment 16)
As described in the above-described embodiment, in the virtual reality presentation system, the user in the virtual reality field is alerted based on the situation in the real world and the provision of the virtual reality service is interrupted. It is preferable to have a configuration. Here, when a high-level process is required as a determination regarding alerting or interruption, it is preferable to perform the process on the virtual reality service providing server side because it requires a large amount of processing resources. In some cases, it is preferable to perform the operation on the attached virtual reality presentation device. Therefore, it is preferable to mount the function on both sides so that it can be used properly according to the situation.

 FIG. 85 is a block diagram showing the overall configuration of the virtual reality presentation system of the sixteenth embodiment. The virtual reality presentation system includes a virtual reality presentation device 800, a contractor information storage database 3100, a virtual reality content storage database 3200, a virtual reality service providing server 3300, and an interruption / warning condition storage database 3600.

  The virtual reality presentation device 800 includes a user state detection unit 110, an ambient state detection unit 120, a communication unit 130, a display control unit 140, a virtual reality presentation control unit 260, a display unit 150, and a voice input unit 370. , An audio reproduction control unit 380, an audio output unit 390, an interruption / attention calling condition storage unit 810, and an interruption / attention calling content storage unit 820.

  The user state detection unit 110 detects the state of the user who is using the virtual reality presentation device 800 and generates user state information. Specifically, the user state detection unit 110 includes an inclination sensor 111 that detects the inclination of the user's head, an acceleration sensor that detects acceleration of the user's head, an electroencephalogram sensor 114 that detects the user's brain wave state, A user detection unit 115 that detects the current position, a user movement speed detection unit 116 that detects the movement speed of the user, and the like. Note that the user state detection unit 110 may include other biosensors that detect the amount of sweating, blood pressure, body temperature, heart rate, etc. of the user, eyeball sensors that detect the movement of the user's eyeball, and the like.

  The surrounding state detection unit 120 detects the surrounding state of the user who is using the virtual reality presentation device 800 and generates the surrounding state information. Specifically, the surrounding state detection unit 120 is installed in the direction in which the user's head is facing, and the line-of-sight camera 121 that captures an image corresponding to the user's line of sight, on both sides and the back of the virtual reality presentation device 800 An ambient camera that is installed and captures images of both sides and the rear of the user, an air temperature sensor 123 that measures the ambient air temperature, and the like.

  The voice input unit 370 inputs voices uttered by the user and surrounding voices.

  The communication unit 130 transmits the user status information generated by the user status detection unit 110, the ambient status information generated by the ambient status detection unit 120, and the voice input by the voice input unit 370 to the virtual reality service providing server 3300. . In addition, the communication unit 130 receives video data and audio data of a virtual reality object transmitted from the virtual reality service providing server, and other virtual reality presentation interruption instructions and alerting instructions.

  The display control unit 140 performs control to decode the video data of the virtual reality object received by the communication unit 130 and display the video of the virtual reality object on the display unit 150.

  The display unit 150 presents the virtual reality to the user by displaying the video of the virtual reality object according to the control from the display control unit 140.

  The sound reproduction control unit 380 performs control to decode the sound data of the virtual reality object received by the communication unit 130 and output the sound of the virtual reality object from the sound output unit 390.

  The audio output unit 390 presents the virtual reality to the user by outputting the sound of the virtual reality object according to the control from the audio reproduction control unit 380.

  The interruption / warning condition setting unit 810 sets a condition for interrupting the presentation of virtual reality to the user or performing alerting to the user while presenting the virtual reality. The interruption / warning condition setting unit 810 may perform the setting based on an operation from the user, or may perform the setting based on an instruction from the virtual reality service providing server 3300 side.

  When the interruption / attention calling condition storage unit 820 interrupts the presentation of virtual reality to the user set by the interruption / attention calling condition setting unit 810 or alerts the user while presenting the virtual reality Memorize the conditions. The interruption / attention condition storage unit 820 does not execute the setting process by the interruption / attention condition setting unit 810, and the user is interrupted by presenting the virtual reality or presenting the virtual reality as a default. You may memorize the conditions to call attention to.

  The interruption / attention content storage unit 830 stores, as interruption / attention content, images and sounds to be displayed when the virtual reality presentation to the user is interrupted, and images and sounds to be displayed at the time of alerting. .

  The virtual reality presentation control unit 260 controls the virtual reality presented to the user by the display unit 150 and the audio output unit 390. The virtual reality presentation control unit 260 includes a virtual reality interruption / attention determination processing unit 261 and a virtual reality presentation interruption / attention control processing unit 262.

  The virtual reality presentation interruption / attention calling determination processing unit 261 performs a process of determining whether to interrupt the virtual reality currently being presented to the user or to perform alerting while presenting the virtual reality. Specifically, the user status information generated by the user status detection unit 110, the ambient status information generated by the ambient status detection unit 120, the voice input by the voice input unit 370, and the interruption / attention condition storage unit By comparing the interruption / attention calling conditions stored at 820, a process for determining whether to interrupt or alert the virtual reality presentation is performed.

  Note that the determination process performed in the virtual reality presentation interruption / attention determination processing unit 261 is the determination process performed at the virtual reality presentation interruption / attention determination processing unit 1324 provided on the virtual reality service providing server 3300 described later. Different information and conditions are used. The virtual reality presentation interruption / attention determination processing unit 261 preferably performs a determination process for highly urgent conditions. For example, the virtual reality presentation interruption / attention alert determination processing unit 261 detects the virtual reality when the acceleration sensor 112 detects an acceleration that is equal to or higher than a predetermined reference value or when the user's moving speed exceeds a predetermined reference value. It is determined that the presentation will be interrupted or alerted. Predetermined reference values for the acceleration and the moving speed are stored in the interruption / attention condition storage unit 820 as interruption / attention conditions.

  Based on the determination result in the virtual reality presentation interruption / attention call determination processing unit 261 and the interruption instruction or attention instruction received by the communication unit 130, the virtual reality presentation interruption / attention control processing unit 262 Control to interrupt the presentation or to call attention.

  Specifically, the virtual reality presentation interruption / attention control processing unit 262 determines whether the virtual reality presentation interruption / attention determination processing unit 261 interrupts the virtual reality presentation or the communication unit 130 presents the virtual reality. When the interruption instruction is received, the virtual reality presentation is controlled to be interrupted. More specifically, the virtual reality presentation interruption / attention control processing unit 262 instructs the display control unit 140 to interrupt the display of the video of the virtual reality object on the display unit 150, or performs sound reproduction control. The unit 380 is controlled to interrupt the output of the sound of the virtual reality object from the sound output unit 390. In addition, the virtual reality presentation interruption / attention calling control processing unit 262 reads out the interruption image and sound from the interruption / attention calling content storage unit 830 and displays the image or sound on the display unit 150 or output from the audio output unit 390. Thus, the display control unit 140 and the audio reproduction control unit 380 are instructed.

  Also, the virtual reality presentation interruption / warning control processing unit 262 determines that the virtual reality presentation suspension / warning determination processing unit 261 determines to call attention or when the communication unit 130 receives a warning instruction. Control for alerting the user is performed.

  More specifically, the virtual reality presentation interruption / warning control processing unit 262 displays the video of the virtual reality object on the display unit 150 while displaying the video of the virtual reality object on the display unit 150 from the interrupt / warning content storage unit 830. An instruction for reading the attention image is read out, and a control to multiplex the image for the attention image on the video of the virtual reality object and display it on the display unit 150 is instructed. Based on the instruction, the display control unit 140 performs control to multiplex and display the virtual reality object video and the alert image on the display unit 150.

  Further, the virtual reality presentation interruption / warning control processing unit 262 alerts the audio reproduction control unit 380 from the interruption / warning content storage unit 830 while outputting the sound of the virtual reality object from the audio output unit 390. And a control to multiplex the alerting sound with the sound of the virtual reality object and output from the sound output unit 390 is instructed. Based on the instruction, the sound reproduction control unit 380 performs control to multiplex the sound of the virtual reality object and the alerting sound and output from the sound output unit 390.

  Next, the virtual reality service providing system side will be described. The virtual reality service providing server 3300 includes a communication unit 1310, a virtual reality service provision management unit 1320, a display video generation processing unit 1350, an image analysis processing unit 2360, an audio analysis processing unit 3380, and a virtual reality service control unit 3330. And a virtual reality object behavior control unit 3370 and a reproduction sound generation processing unit 3390.

  The virtual reality service provision management unit 1320 includes a virtual reality service start processing unit 1321, a virtual reality service provision end processing unit 1322, a virtual reality presentation interruption / attention determination processing unit 1324, and a virtual reality presentation interruption / attention instruction unit. 1325.

  The virtual reality presentation interruption / attention calling determination processing unit 1324 receives the user status information and the ambient situation information received by the communication unit 1310, and the result of the image analysis processing performed by the image analysis processing unit 2360 on the ambient status information and the communication unit 1310. Based on the result of the voice analysis processing performed by the voice analysis processing unit 3380 on the voice received in step S3, the process of determining whether to interrupt the presentation of virtual reality or to call attention is performed. Specifically, the virtual reality presentation interruption / attention calling determination processing unit 1324 compares the conditions stored in the interruption / attention calling condition storage database 3600 with these input information, so that the interruption condition and the attention attraction are obtained. Processing to determine whether the condition is satisfied is performed.

  When the virtual reality presentation suspension / attention calling instruction unit 1325 determines that the virtual reality presentation suspension / attention calling determination processing unit 1324 interrupts or alerts the virtual reality presentation, the virtual reality presentation suspension / attention calling instruction unit 1325 transmits a suspension instruction or attention attraction instruction to the communication unit. This is performed to the virtual reality presentation control unit 260 via 1310.

  Further, when the virtual reality presentation interruption / attention calling instruction unit 1325 determines that the virtual reality presentation interruption / attention calling determination processing unit 1324 performs the attention using the virtual reality object, the virtual reality service control unit 3300 and the virtual reality service control unit 3300 An instruction for alerting the real object behavior control unit 3370 is issued. Upon receiving the instruction, the virtual reality object behavior control unit 3370 executes a process for causing the virtual reality object currently arranged in the virtual space to be alerted, thereby allowing the user to perform the behavior through the behavior of the virtual reality object. Call attention to it. That is, in the display video generation processing unit 1350, video data of the virtual reality object that behaves for the attention is generated, so that the display control unit 140 decodes the video data and the virtual reality object is displayed on the display unit 150. The user is alerted by displaying a video that is performing a behavior for alerting. When a virtual reality object that can be used for alerting is not arranged in the virtual space, the virtual reality service control unit 3330 that has received the alerting instruction receives virtual reality for alerting from the virtual reality content storage database 3200. A configuration may be adopted in which an object is read out and placed in a virtual space, and alerting is performed using the virtual reality object newly placed in the virtual space. The display video generation processing unit 1350 generates video data of the virtual reality object for alerting, and the reproduction sound generation processing unit 3390 generates audio data for alerting.

  The predetermined virtual reality object used for alerting, the alerting virtual reality object behavior control program (method or subroutine, etc.) and alerting voice data are set and set for each provided virtual reality service. It may be stored, or the system may be uniquely set and stored. Preferably, the virtual reality object corresponding to each alert condition determined to be applicable by the virtual reality presentation interruption / attention determination processing unit 1324 is read from the virtual reality content storage database 3200 and arranged in the virtual space. Or by executing the corresponding virtual reality object behavior control program for alerting, the display video generation processing unit 1350 generates alerting video data using the virtual reality object. It is preferable that the audio data for calling is read from the virtual reality content storage database 3200 and the audio data for calling is generated by the reproduction audio generation processing unit 3390.

  The virtual reality presentation interruption / attention determination processing unit 1324 displays the type (class) / content of the virtual reality service provided to the user and the setting information set for interruption / attention by the user. Based on the interruption / attention condition storage database 3600, the corresponding interruption / attention conditions are read out and compared with the input user status information, surrounding situation information, image analysis processing result, and voice analysis processing result. It is even better if the construction is performed.

  Here, the virtual reality service provision interruption processing unit 1323 described above performs processing for interrupting provision of the virtual reality service itself that is currently being provided, whereas the virtual reality presentation suspension / attention instruction unit 1325 is a virtual reality service. However, it is different in that an instruction to temporarily interrupt the presentation of virtual reality is given.

  Note that the virtual reality presentation interruption / attention calling determination processing unit 1324 has the same functions as the virtual reality presentation interruption determination processing unit 11031 and the attention attraction determination processing unit 14011 described above. The virtual reality presentation interruption / attention instruction unit 1325 has the same functions as the virtual reality presentation interruption instruction unit 11032 and the attention instruction unit 14012. The interruption / attention calling condition storage database 3600 has the same functions as the virtual reality interruption condition storage unit 13010 and the attention calling condition storage unit 15010.

  As described above, the virtual reality presentation system according to the sixteenth embodiment includes at least a virtual reality presentation device and a virtual reality service providing server, and the virtual reality presentation device includes the user state or the surrounding situation and the first type of interruption condition. Alternatively, based on the first type of alerting condition, a process for determining whether to interrupt the presentation of virtual reality to the user or a process for determining whether to alert the user who is presented with virtual reality is performed. And the virtual reality presentation to the user is interrupted based on the state of the user or the surrounding situation and the second type interruption condition or the second type alert condition. A determination process for determining whether or not to alert a user who is presented with virtual reality, or a second type determination processing unit . In addition, when the virtual reality presentation device determines that the virtual reality presentation is interrupted by either the first type determination processing unit provided in the virtual reality presentation device or the second type determination processing unit provided in the virtual reality service providing server. Then, control for interrupting presentation of virtual reality is performed by stopping video display of the virtual reality object or stopping audio output of the virtual reality object. In addition, the virtual reality presentation device is cautioned when it is determined to alert one of the first type determination processing unit provided in the virtual reality presentation device and the second type determination processing unit provided in the virtual reality service providing server. Control is performed to alert the user while displaying virtual reality by displaying an image for alerting or outputting alerting sound. In addition, the virtual reality service providing server is a virtual reality object arranged in the virtual space when it is determined that the second type determination processing unit included in the virtual reality service providing server performs alerting using the virtual reality object. A video or sound for alerting using a virtual reality object is generated by performing a behavior for alerting or arranging a virtual reality object for alerting in a virtual space. The virtual reality presentation device performs control for alerting the user while presenting the virtual reality by displaying or outputting a video or sound for alerting using the virtual reality object.

For example, as the first type determination process performed by the virtual reality presentation interruption / attention calling determination processing unit 261 on the virtual reality presentation device 800 side, for example, it is detected by the user movement speed detection unit 116 included in the user state detection unit 110. 54 is compared with the interruption condition “No user movement speed is 3 m / s or more” of No: C004 in FIG. 54 to determine whether the interruption condition is satisfied, and the virtual reality service providing server In the virtual reality presentation interruption / attention calling determination processing unit 1324 on the 3300 side, the user state information, the surrounding state information, the image analysis result in the image analysis processing unit 2360, the voice analysis result in the voice analysis processing unit 3380, and other types shown in FIG. It is good also as a structure which determines whether the said interruption condition is not satisfied by comparing with interruption conditions.

For example, as a first type of determination process performed by the virtual reality presentation interruption / attention determination processing unit 261 on the virtual reality presentation device 800 side, for example, the user movement speed detection unit 116 included in the user state detection unit 110 may be used. The detected moving speed of the user is not satisfied by comparing it with the warning condition “No user position is latitude 38... Longitude 135... In the virtual reality presentation interruption / attention calling determination processing unit 1324 on the virtual reality service providing server 3300 side, the user state information, the surrounding state information, the image analysis result in the image analysis processing unit 2360, and the voice analysis processing unit 3380 A structure for determining whether or not the alert condition is satisfied by comparing the voice analysis result with another alert condition shown in FIG. It may be.

In these cases, the corresponding interruption condition and alerting condition are stored in the interrupting / noticeing condition storage unit 820 and the interrupting / noticeing condition storage unit database 3600, respectively.

In this way, the virtual reality presentation device 800 side performs the comparison determination process with the interruption condition / warning condition having a high degree of urgency and the interruption condition / warning condition that can be performed with relatively small processing, and based on the determination result. The virtual reality presentation is interrupted and alerted, and the virtual reality service providing server 3300 performs comparison determination processing with the interrupting condition / alerting condition that requires relatively large processing such as image analysis processing and voice analysis processing. It is good to have a configuration to do. Depending on the amount of processing required and the speed with which interruption / warning is required, the decision processing is shared between the virtual reality presentation device side and the virtual reality service providing server side for more appropriate interruption / warning It can be performed.

  As described above in Embodiments 8 to 16, the virtual reality system according to the present invention is a virtual reality presentation system that presents a virtual reality world to the user, and the user acts by sensing the virtual reality world. In this case, the influence of the user's action on the real world can be controlled.

Moreover, each embodiment mentioned above shows one Example of this invention, and can be changed suitably. For example, the above-described embodiments can be combined.

In the virtual reality presentation system described above, the video generation processing unit may be provided on the virtual reality presentation device side worn by the user, or may be provided on the virtual reality service providing server side. In this case, the virtual reality presentation device wirelessly transmits the acquired user state information and surrounding state information to the virtual reality service providing server side from the communication unit, and encodes the video of the virtual reality object generated by the virtual reality service providing server. The data is wirelessly received by the communication unit. In the virtual reality presentation device, the video display control unit performs a decoding process on the video encoded data of the virtual reality object wirelessly received by the communication unit, and displays the decoded video on the display panel.

In the virtual reality presentation system described above, the alerting control unit and the virtual reality presentation interruption control unit may be incorporated in the virtual reality presentation device or may be arranged on the virtual reality service providing server side. In this case, a determination process for determining whether to interrupt the presentation of virtual reality based on user state information and ambient condition information wirelessly transmitted from the communication unit of the virtual reality presentation device and a determination process for determining whether to perform alerting are performed. It is executed on the virtual reality service providing server side.

In the above description, a case where an LCD (Liquid Crystal Display) is used as a display panel for displaying an image has been described. However, the present invention is not limited to this. For example, OELD (Organic Electro-Luminescence Display) may be used, and other video display displays may be used.

Further, the virtual reality presentation device is preferably provided with two display panels respectively arranged in front of the user's left eye and right eye, and the viewpoint control unit described above corresponds to the user's left eye and right eye, respectively. It is preferable to control the position and direction of the viewpoint and generate a video to be displayed on each of the two display panels in the video generation processing unit.

Further, although the above-described color space is described as being a YUV series, control or the like may be performed in an RGB series, a CMY series, or other color spaces. In this case, the luminance control performed by the display control unit or the like can naturally be substituted by brightness control, R component control, or the like.

Each unit described above may be realized by hardware, or may be realized by cooperation of hardware and software by causing a processor such as a CPU (Central Processing Unit) to execute a program.

The virtual reality presentation device described above can also be used as an HMD device that displays information other than virtual reality.

  Note that the above-described user state detection unit and surrounding state detection unit are not limited to those arranged in the virtual reality presentation device worn by the user. For example, as shown in FIG. 86, a user who wants to receive a virtual reality presentation can enter the virtual reality experience dome with the virtual reality presentation device and receive the virtual reality presentation. The virtual reality experience dome is provided with a plurality of sensors for detecting the state of the user and sensors for detecting the situation around the user, and information detected by the sensor is sent to the virtual reality service providing server.

  Further, the virtual reality presentation device can be configured such that the HMD device worn on the head and the control processing device worn on the upper body other than the head are connected by a connection cable. Only as few functions as possible are mounted on the HMD device mounted on the head, and a battery, a CPU, and the like are disposed on the control processing device side. The control processing device may be configured to be worn around the waist or may be configured to be supported from the shoulder. With this configuration, it is possible to enjoy the virtual reality world for a long time without causing the user to feel tired. However, if a small battery that can be used for a long time can be mounted on the HMD device due to the advancement of the battery technology or the semiconductor integration technology, the form in which the small battery is mounted on the HMD device is more easily mounted. preferable.

  Moreover, each embodiment mentioned above shows one Example of this invention, and can be changed suitably. For example, the above-described embodiments can be combined.

  Further, in the above description, the case where the virtual reality is presented by displaying the video of the virtual reality object on at least the display unit (display panel) arranged in front of the user's eyes is described, but the present invention is not limited to this. Absent. For example, by irradiating a user's brain with a magnetic field generator having position resolution, a specific region in the brain can be stimulated so that the user can view an image without passing through the eyeball. However, in this case, it is necessary to grasp in advance at least the current neural network connection configuration in the brain. However, ultimately, it is considered that the virtual reality presentation device reaches a configuration that directly stimulates the user's brain.

  In addition, as the above-described ambient condition detection unit, an acoustic sonar or an ultrasonic sensor can be used. Further, the surrounding state detection unit may be configured to detect the presence of a human being around the user as the surrounding state of the user by acquiring position information indicating the current position of another person. For example, the position information of another person may be obtained by acquiring position information acquired by a mobile terminal owned by the other person. In addition, the presence of a vehicle running around the vehicle may be detected based on an electromagnetic wave radiated from an electromagnetic wave generator attached to the vehicle.

  The virtual reality presentation device described above can also be used as an HMD device that displays information other than virtual reality.

The present invention can take the following forms.
(Appendix 1) (Basic system design)
Virtual reality object storage means for storing a plurality of virtual reality objects;
Available virtual reality service information storage means for storing information indicating virtual reality services available to the user;
Virtual reality object reading means for reading out a virtual reality object used for a virtual reality service used by the user from the virtual reality object storage means;
Virtual reality object placement position determining means for determining the placement position of the virtual reality object in the virtual reality space, based on ambient situation information indicating a situation around the user;
Display image generation means for generating an image to be displayed on a display panel of a head mounted display device worn by a user on the head based on user state information indicating a user state and an arrangement position of the virtual reality object;
A virtual reality service providing system.
(Appendix 2) (Basic system design)
Receiving means for receiving the ambient status information and the user status information from the head-mounted display device;
Video encoding means for encoding the video generated by the display video generation means;
Transmitting means for transmitting the encoded video to the head mounted display device;
The virtual reality service providing system according to appendix 1, further comprising:
(Appendix 3) (Gaze Camera)
As the surrounding state information, the head-mounted display device includes a line-of-sight camera that acquires a front scene image by photographing the front.
The virtual reality service providing system according to appendix 1 or 2.
(Appendix 4) (Ambient camera)
As the surrounding state information, the head mounted display device includes a surrounding camera that obtains a surrounding scene image by photographing a direction outside the photographing range of the line-of-sight camera.
The virtual reality service providing system according to Appendix 3.
(Appendix 5) (Tilt sensor)
As the user status information, the head mounted display device includes an inclination sensor that acquires inclination information by detecting the inclination of the user's head,
The virtual reality service providing system according to any one of appendices 1 to 4.
(Appendix 6) (Acceleration sensor)
As the user state information, the head mounted display device includes an acceleration sensor that acquires acceleration information by detecting acceleration accompanying movement of the user's head,
The virtual reality service providing system according to any one of appendices 1 to 5.
(Appendix 7) (Direction sensor)
As the user status information, the head mounted display device includes an orientation sensor that acquires orientation information by detecting an orientation in which a user's head is facing,
The virtual reality service providing system according to any one of appendices 1 to 6.
(Appendix 8) (Location information: GPS)
As the user status information, the head mounted display device includes a position information acquisition unit that acquires position information by detecting the position of the user.
The virtual reality service providing system according to any one of appendices 1 to 7.
(Appendix 9) (User voice input)
The head mounted display device includes a user voice input unit that inputs a voice uttered by a user and generates user voice information.
The virtual reality service providing system according to any one of appendices 1 to 8.
(Appendix 10) (User voice input)
The head mounted display device includes an ambient audio input unit that inputs ambient audio of a user and generates ambient audio information.
The virtual reality service providing system according to any one of appendices 1 to 9.
(Appendix 11) (Virtual reality service provision start)
Comprising virtual reality service start determining means for determining to start providing virtual reality service to the user,
When the virtual reality service start determining means determines to start providing the virtual reality service, the virtual reality object reading means stores the virtual reality object used for the virtual reality service provided to the user as the virtual reality object storage means. The virtual reality service providing system according to any one of supplementary notes 1 to 10, wherein the virtual reality service providing system is further read out.
(Supplementary note 12) (Service starts in response to user request)
The receiving means receives a service start request for requesting to start providing virtual reality service;
The virtual reality service start determining means determines that the provision of the virtual reality service to the requesting user of the service start request is started when the receiving means receives the service start request;
The virtual reality service providing system according to attachment 11.
(Supplementary note 13) (Start request input by voice)
The receiving means receives user voice information including a user voice input by a head-mounted display device,
Further comprising context acquisition means for analyzing a user's voice included in the user voice information and acquiring a context;
The virtual reality service start determination means determines to start providing the virtual reality service to the user when the context acquired by the context acquisition means is a request to start a virtual reality service;
The virtual reality service providing system according to attachment 11.
(Supplementary Note 14) (Determining availability of provision of virtual reality service)
Service provision start condition storage means for storing a service provision start condition as a condition for starting provision of the virtual reality service;
The virtual reality service start determining means starts providing the virtual reality service to the user when the surrounding situation of the user indicated by the surrounding situation information received by the receiving means satisfies the service provision start condition. decide,
The virtual reality service providing system according to any one of appendices 11 to 13.
(Supplementary note 15) (Service startable position)
The service provision start condition storage means stores a service available position range indicating a position range where the provision of the virtual reality service can be received as a condition for starting the provision of the virtual reality service,
The receiving means receives at least position information indicating the current position of the user as the ambient state information,
The virtual reality service start determination unit is configured to notify the user of the virtual reality service when the current position of the user indicated by the position information, which is the surrounding state information received by the receiving unit, is within the service available position range. Decide to start offering,
The virtual reality service providing system according to appendix 14.
(Supplementary Note 16) (Virtual Reality Service Provision Start Notification)
The transmission means sends service start notification information for notifying the start of provision of the virtual reality service to the user when the virtual reality service start determining means determines to start providing the virtual reality service to the user. Send to the mount display device,
The virtual reality service providing system according to any one of appendices 11 to 15.
(Supplementary note 17) (End of provision of virtual reality service)
A virtual reality service end determining means for determining to end the virtual reality service being provided to the user;
When it is determined by the virtual reality service end determining means that the provision of the virtual reality service is ended, the virtual reality object reading means stores the virtual reality object used for the virtual reality service available to the user in the virtual reality object storage. The virtual reality service providing system according to any one of appendices 1 to 16, wherein the virtual reality service providing system is read out from the means.
(Supplementary note 18) (Virtual reality service provision end notification)
When the virtual reality service termination determining unit determines to terminate the provision of the virtual reality service being provided to the user, the transmitting unit transmits service termination notification information for notifying the termination of the provision of the virtual reality service. Send to the user's head-mounted display device,
The virtual reality service providing system according to appendix 17.
(Supplementary note 19) (Service terminated upon user request)
The receiving means receives a service termination request for requesting termination of provision of the virtual reality service;
The virtual reality service termination determining means determines that the provision of the virtual reality service being provided to the requesting user of the service termination request is terminated when the service termination request is received by the receiving means;
The virtual reality service providing system according to appendix 17 or 18.
(Supplementary note 20) (Enter end request by voice)
The receiving means receives user voice information including a user voice input by a head-mounted display device,
Further comprising context acquisition means for analyzing a user's voice included in the user voice information and acquiring a context;
The virtual reality service termination determination means determines to terminate provision of the virtual reality service being provided to the user when the context acquired by the context acquisition means is a request for termination of the virtual reality service;
The virtual reality service providing system according to any one of appendices 17 to 19.
(Supplementary Note 21) (Determining whether or not virtual reality service provision is complete)
Service provision end condition storage means for storing a service provision end condition that is a condition for terminating provision of the virtual reality service being provided to the user;
The virtual reality service termination determining means terminates the provision of the virtual reality service to the user when the surrounding environment of the user indicated by the ambient situation information received by the receiving means satisfies the service provision termination condition. decide,
The virtual reality service providing system according to any one of appendices 17 to 20.
(Appendix 22) (Service end position)
The service provision end condition storage means stores a service available position range indicating a position range where the provision of the virtual reality service can be received as the service provision end condition,
The receiving means receives at least position information indicating the current position of the user as the ambient state information,
The virtual reality service termination determining means is configured to provide the user with a virtual reality service when the current position of the user indicated by the position information, which is ambient information received by the receiving means, is outside the service available position range. To end providing
The virtual reality service providing system according to attachment 21.
(Supplementary note 23) (Suspension of virtual reality service provision)
Service provision interruption condition storage means for storing a service provision interruption condition which is a condition for interrupting provision of the virtual reality service being provided to the user;
Service provision interruption for performing a process of interrupting provision of the virtual reality service being provided to the user when a situation around the user indicated by the ambient condition information received by the receiving unit satisfies the service provision interruption condition Processing means;
The virtual reality service providing system according to any one of appendices 1 to 22, comprising:
(Supplementary Note 24) (Video generation interruption instruction)
The service provision interruption processing means generates video generation interruption instruction information instructing the display video generation means to interrupt generation of the video,
The display video generation means interrupts generation of the video based on the video generation interruption instruction information;
The virtual reality service providing system according to attachment 23.
(Appendix 25) (Video transmission interruption instruction)
The service provision interruption processing means generates video transmission interruption instruction information instructing transmission interruption of the video generated by the display video generation means,
The transmission means interrupts transmission of the video generated by the display video generation means to the head mounted display device based on the video transmission interruption instruction information.
The virtual reality service providing system according to attachment 23.
(Supplementary note 26) (Virtual reality service provision interruption instruction)
The service provision interruption processing means generates video display interruption instruction information for instructing interruption of display of the video generated by the display video generation means,
The transmission means transmits the video display interruption instruction information generated by the service provision interruption processing means to the head mounted display device;
The virtual reality service providing system according to attachment 23.
(Appendix 27) (Interruption by position)
The service provision interruption condition storage means stores a service available position range indicating a position range where the provision of the virtual reality service can be received as the service provision interruption condition,
The receiving means receives at least position information indicating the current position of the user as the ambient state information,
The virtual reality service interruption processing means provides the user with a virtual reality service when the current position of the user indicated by the position information, which is the surrounding information received by the receiving means, is outside the service available position range. Process to interrupt the provision of
27. The virtual reality service providing system according to any one of supplementary notes 23 to 26.
(Appendix 28) (Interruption due to object approach)
The service provision interruption condition storage means stores, as the service provision interruption condition, a condition relating to the speed and distance of an object approaching the user,
The receiving means receives, as the ambient state information, an ambient landscape image obtained by capturing at least a landscape around the user,
The service provision interruption processing means is configured such that the speed of the object included in the surrounding scenery image and the distance to the user are faster than the speed stored as the service provision interruption condition and closer to the distance, respectively. A process of interrupting provision of the virtual reality service being provided to the user,
27. The virtual reality service providing system according to any one of appendices 23 to 26.
(Supplementary Note 29) (Dynamic Virtual Reality Object)
Virtual reality object control means for controlling the behavior of the virtual reality object arranged in the virtual reality space,
The display video generation means has the virtual reality whose behavior is controlled by the virtual reality object control means at a display position determined based on user state information indicating the state of the user and an arrangement position of the virtual reality object. Generate a video that displays the object,
The virtual reality service providing system according to any one of appendices 1 to 27.
(Appendix 30) (Behavior control according to ambient conditions)
The virtual reality object control means controls the behavior of the virtual reality object based on the surrounding situation information.
The virtual reality service providing system according to attachment 29.
(Appendix 31) (Behavior control by voice)
The receiving means receives audio information including audio input by a head mounted display device,
Further comprising context acquisition means for analyzing a voice included in the voice information and acquiring a context;
The virtual reality object control means controls the behavior of the virtual reality object arranged in the virtual reality space based on the context acquired by the context acquisition means;
The virtual reality service providing system according to attachment 29.
(Supplementary Note 32) (Optical correction by illuminance)
Optical correction means for correcting the brightness of the virtual reality object arranged in the virtual reality space based on ambient illuminance information indicating the brightness of the user's surroundings,
The display video generation means is a virtual display in which brightness correction is performed by the optical correction means at a display position determined based on user state information indicating the state of the user and an arrangement position of the virtual reality object. Generate video that displays real objects,
The virtual reality service providing system according to any one of appendices 1 to 31.
(Supplementary Note 33) (Optical Correction by Illuminance at Arrangement Position)
As the ambient state information, the light irradiation state at a position corresponding to the position of the virtual reality space where the virtual reality object is arranged is specified based on an image of a landscape around the user taken by the head mounted display device Means for identifying the light irradiation status,
Optical correction means for correcting the brightness of the virtual reality object arranged in the virtual reality space based on the specified light irradiation state;
With
The display video generation means is a virtual display in which brightness correction is performed by the optical correction means at a display position determined based on user state information indicating the state of the user and an arrangement position of the virtual reality object. Generate video that displays real objects,
The virtual reality service providing system according to any one of appendices 1 to 31.
(Supplementary Note 34) (Speech synthesis of virtual reality object)
Audio element storage means for storing a plurality of audio element data associated with the virtual reality object;
Sound determining means for determining the content of sound emitted by the virtual reality object based on the ambient information or sound information input by the head-mounted display device;
Sound generating means for generating sound emitted from the virtual reality object by combining the sound element data according to the determined sound content;
The virtual reality service providing system according to any one of appendices 1 to 33, comprising:
(Supplementary Note 35) (Sound reproduction virtual reality object)
Audio conversion means for converting the generated audio into audio of a plurality of channels output from a speaker of the head-mounted display device based on an arrangement position of the virtual reality object in the virtual reality space;
34. The virtual reality service providing system according to supplementary note 34.
(Appendix 36) (Basic method design)
Virtual used in a virtual reality service providing system comprising: a virtual reality object storage database that stores a plurality of virtual reality objects; and an available virtual reality service information storage database that stores information indicating virtual reality services that can be used by a user. A real service providing method,
Based on the information indicating the virtual reality service available to the user stored in the available virtual reality service information storage database, the virtual reality object used for the virtual reality service used by the user is retrieved from the virtual reality object storage database. A virtual reality object reading step for reading;
A virtual reality object placement position determining step for determining a placement position of the read virtual reality object in the virtual reality space based on ambient situation information indicating a situation around the user;
A display image generation step for generating an image to be displayed on a display panel of a head mounted display device worn by the user based on user state information indicating a user state and an arrangement position of the virtual reality object; A virtual reality service providing method.
(Appendix 101) (System basic design)
Ambient condition detection means for detecting the situation of the user's surroundings,
Virtual reality presentation means for presenting virtual reality;
Virtual reality presentation interruption control means for performing control to interrupt the presentation of virtual reality based on the detected situation around the user;
A virtual reality presentation system comprising:
(Appendix 102) (Basic system design)
An interruption condition storage means for storing an interruption condition that is a condition for interrupting the presentation of virtual reality;
The virtual reality presentation interruption control means includes
Virtual reality presentation interruption determination means for determining whether the detected situation around the user satisfies the interruption condition;
Based on the determination result, virtual reality presentation interruption instruction means for instructing the virtual reality presentation means to interrupt the presentation of the virtual reality;
With
The virtual reality presenting means interrupts the presentation of virtual reality when receiving the instruction;
The virtual reality presentation system according to attachment 101.
(Supplementary Note 103) (Virtual Reality Object)
Virtual reality object storage means for storing the virtual reality object,
The virtual reality presentation means presents virtual reality by displaying a video of the virtual reality object.
The virtual reality presentation system according to attachment 101 or 102.
(Appendix 104) (Rendering video generation)
User status detection means for detecting the user status;
Virtual reality object placement means for reading a virtual reality object stored in the virtual reality object storage means and placing it in a virtual space;
A two-dimensional image of the virtual reality object is drawn by drawing the virtual reality object arranged in the virtual space based on the viewpoint position and the line-of-sight direction obtained based on the user state obtained by the user state detection means. Video generation processing means for generating
With
The virtual reality presentation means presents virtual reality by displaying the generated video.
The virtual reality presentation system according to attachment 103.
(Appendix 105) (Interrupted video)
Further comprising interrupted video data storage means for storing interrupted video data indicating that the presentation of virtual reality is interrupted,
The virtual reality presentation interruption means performs control to interrupt the presentation of virtual reality by instructing the virtual reality presentation means to display the interruption video data.
The virtual reality presentation system according to attachment 103 or 104.
(Appendix 106) (Voice)
Voice data storage means for storing voice data associated with the virtual reality object;
The virtual reality presentation means presents virtual reality by reproducing the audio data.
The virtual reality presentation system according to any one of appendices 101 to 105.
(Supplementary note 107) (Interruption voice)
Further comprising interrupted video data storage means for storing interrupted audio data indicating that the virtual reality presentation is interrupted,
The virtual reality presentation interruption means performs control to interrupt the presentation of virtual reality by instructing the virtual reality presentation means to reproduce the interruption voice data.
The virtual reality presentation system according to attachment 106.
(Supplementary Note 108) (Object Movement Speed)
The ambient condition detection unit includes a plurality of ambient imaging units that capture at least a user's surroundings and acquire a surrounding image,
An object moving speed calculating means for calculating a moving speed of an object shown in the surrounding image;
Further comprising
The virtual reality presentation interruption control means includes
When the moving speed of the object exceeds a predetermined reference speed, control to interrupt the presentation of virtual reality is performed.
108. A virtual reality presentation system according to any one of appendices 101 to 107.
(Appendix 109) (Object relative movement speed)
The object moving speed calculating means calculates a moving speed component of the object in the user direction,
The virtual reality presentation interruption control means includes
When the moving speed component in the user direction of the object exceeds a predetermined reference speed, control to interrupt the presentation of virtual reality is performed.
The virtual reality presentation system according to attachment 108.
(Supplementary Note 110) (Object Movement Speed)
An object distance calculating means for calculating a distance to an object shown in the surrounding image;
The virtual reality presentation interruption control means performs control to interrupt the presentation of virtual reality when the moving speed of an object within a predetermined distance or the moving speed component in the direction of the user exceeds a predetermined reference speed,
The virtual reality presentation system according to attachment 109.
(Appendix 111) (Collision time)
The ambient condition detection unit includes a plurality of ambient imaging units that capture at least a user's surroundings and acquire a surrounding image,
An object distance calculating means for calculating a distance to an object shown in the surrounding image;
Object moving speed calculating means for calculating the moving speed of the object based on a change in the distance to the object;
Arrival time estimation means for estimating the arrival time to the user position based on the distance to the object and the moving speed of the object;
Further comprising
The virtual reality presentation interruption control means includes
When the arrival time is within a predetermined time, control to interrupt the presentation of virtual reality,
108. A virtual reality presentation system according to any one of appendices 101 to 107.
(Supplementary note 112) (Advancing direction obstacle)
The ambient condition detection unit includes a plurality of ambient imaging units that capture at least a user's surroundings and acquire a surrounding image,
User movement detection means for detecting user movement;
A failure determining means for determining whether a failure exists in the moving direction of the user based on the surrounding image;
The virtual reality presentation interruption control means performs control to interrupt the presentation of virtual reality when it is determined that there is a failure in the movement direction of the user.
108. A virtual reality presentation system according to any one of appendices 101 to 107.
(Supplementary note 113) (Advancing direction obstacle)
User moving speed calculating means for calculating the moving speed of the user;
An object / step distance calculating means for calculating a distance to the object or step existing in the moving direction of the user;
Further comprising
The obstacle determination means determines whether a failure exists in the movement direction of the user based on the moving speed of the user and a distance to an object or a step existing in the movement direction of the user.
The virtual reality presentation system according to attachment 112.
(Appendix 114) (Advancing direction obstacle)
The obstacle determination means determines that there is an obstacle in the user's moving direction when a value obtained by dividing a distance to an object or a step existing in the user's moving direction by the moving speed is equal to or less than a predetermined reference value. ,
The virtual reality presentation system according to attachment 113.
(Appendix 115) (Basic method design)
An ambient condition detection step for detecting an ambient condition of the user;
A virtual reality presentation step for presenting virtual reality;
A virtual reality presentation interruption control step for performing control to interrupt the presentation of virtual reality based on the detected situation around the user;
A virtual reality presentation method comprising:
(Appendix 116) (System Basic Design)
Ambient condition detection means for detecting the situation of the user's surroundings,
User status detection means for detecting the user status;
Video generation means for generating a video of a virtual reality object based on the state of the user;
Video display means for displaying the generated video;
Audio output means for outputting audio associated with the virtual reality object;
Based on the detected situation around the user, attention calling means for calling attention to the user using the video display means or the audio output means;
A virtual reality presentation system comprising:
(Appendix 117) (Awareness judgment)
It further includes a warning condition storage means for storing a warning condition that is a condition for performing a warning,
The alerting means is:
Attention determination means for determining whether or not the detected situation around the user satisfies the alert condition;
Based on the determination result, attention instructing instruction means for performing an instruction to display a warning image on the image display means or to output a sound for attention to the sound output means,
The virtual reality presentation system according to attachment 101, comprising:
(Appendix 118) (Awareness determination)
The alert condition storage means stores a plurality of alert conditions,
The alert determination unit determines whether the detected situation around the user satisfies any of the plurality of alert conditions,
The attention instructing means corresponds to an instruction to display on the display means an image for alerting that is associated with the alerting condition when it is determined that the alerting condition is satisfied, or to the alerting condition Instructing the voice output means to output a warning sound attached to the voice output means;
The virtual reality presentation system according to attachment 117.
(Appendix 119) (Awareness judgment)
The attention instructing means instructs the sound output means to output a sound for attention associated with the virtual reality object based on the determination result.
The virtual reality presentation system according to attachment 117.
(Appendix 20) (Awareness judgment)
A reminder data storage means for storing alert sound data associated with the virtual reality object and alert sound data not associated with the virtual reality object;
When the alert data storage means stores the alert data associated with the virtual reality object of the video generated by the video generation means, the alert control means generates the video The control unit reads out the alert sound data associated with the virtual reality object of the video generated by the means, and causes the sound output means to output the alert sound associated with the virtual reality object. When the alerting sound data associated with the virtual reality object of the video generated by the video generation means is not stored in the warning data storage means, the video generation means generates The alerting sound data associated with the virtual reality object of the video is read, and the temporary output is sent to the sound output means. Performs control to output the sound for alerting that is not associated with a real object,
The virtual reality presentation system according to attachment 118.
(Supplementary Note 121) (Caution to call attention)
The alerting means performs the alerting by performing correction to reduce the luminance or brightness of the video of the virtual reality object generated by the video generating means.
120. The virtual reality system according to any one of appendices 116 to 119.
(Supplementary Note 122) (Basic design of equipment)
Ambient condition detection means for detecting the situation of the user's surroundings,
User status detection means for detecting the user status;
Video display means for displaying a video of a virtual reality object generated based on the state of the user;
Audio output means for outputting audio associated with the virtual reality object;
Based on the detected situation around the user, attention calling means for calling attention to the user using the video display means or the audio output means;
A virtual reality presentation device.
(Supplementary Note 123) (Basic design of equipment)
Transmitting means for transmitting user status information indicating the status of the user and ambient status information indicating the status of the user;
Receiving means for receiving video encoded data of the virtual reality object generated based on the transmitted user state information and ambient state information;
Further comprising
The video display means displays a video of a virtual reality object obtained by decoding the video encoded data received by the receiving means;
The virtual reality presentation device according to attachment 116.
(Appendix 124) (Awareness determination)
An ambient condition detection step for detecting an ambient condition of the user;
A user status detection step for detecting a user status;
A video generation step of generating a video of a virtual reality object based on the state of the user;
A video display step for displaying the generated video;
An audio output step for outputting audio associated with the virtual reality object;
An alerting step for alerting based on the detected situation around the user;
A virtual reality presentation method comprising:

  The present invention can be widely used to provide a virtual reality world with high creativity.

DESCRIPTION OF SYMBOLS 100 Virtual reality presentation apparatus 110 User state detection part 111 Inclination sensor 112 Acceleration sensor 113 Current position detection part 114 Electroencephalogram sensor 120 Ambient condition detection part 121 Gaze camera 121-1 Lens group 121-2 Aperture mechanism 121-3 Solid-state image sensor 122 Ambient Camera 123 Temperature sensor 130 Communication unit 131 Wireless transmission unit 132 Wireless reception unit 140 Display control unit 141 Video decoding processing unit 142 Video composition processing unit 143 View image generation processing unit 144 Boundary specification processing unit 150 Display unit 151a Right display panel 151b Left display Panel 152 Light source 153 Light guide plate 154 LCD
155a Lens group 155b PLC
155c first reflecting mirror 155d R reflecting dichroic mirror 155e G reflecting dichroic mirror 155f second reflecting mirror 155g third reflecting mirror 155h first liquid crystal panel 155j cross dichroic prism 155k fourth reflecting mirror 155m second liquid crystal panel 155n third liquid crystal panel 155p Galvano mirror 155q Irradiation window 200 Virtual reality presentation device 260 Virtual reality presentation control unit 261 Virtual reality presentation interruption / attention determination processing unit 262 Virtual reality presentation interruption / attention control processing unit 300 HMD device (virtual reality presentation device) 301a Light frame 301b Left frame 301c Center frame 302 Front panel 370 Audio input unit 371 User microphone 372 Peripheral microphone 380 Audio reproduction control unit 390 Audio output Unit 391a right speaker 391b left speaker 410 user state detection unit 420 virtual reality content storage unit 430 viewpoint control unit 440 drawing processing unit 450 display unit 510 ambient state detection unit 520 light source control unit 610 object position specifying unit 710 virtual reality management unit 720 virtual Space setting unit 730 Virtual reality object placement unit 740 Virtual reality object behavior control unit 800 Virtual reality presentation device 810 Interruption / attention condition setting unit 820 Interruption / attention condition storage unit 830 Interruption / attention content storage unit 1000 Virtual reality service Provision System 1100 Available Virtual Reality Service Information Storage DB
1200 Virtual reality object storage DB
1300 Virtual reality service provision server 1310 Communication unit 1311 Reception unit 1312 Transmission unit 1320 Virtual reality service provision management unit 1321 Virtual reality service provision start processing unit 1322 Virtual reality service provision end processing unit 1323 Virtual reality service provision interruption processing unit 1324 Virtual reality presentation Suspension / attention calling determination processing unit 1325 Virtual reality presentation suspension / attention calling instruction unit 1330 Virtual reality object reading unit 1340 Virtual reality object placement position determination unit 1350 Display video generation processing unit 1351 Viewpoint control unit 1352 Light source control unit 1353 Drawing processing unit 1354 Video encoding processor
2000 Virtual reality service providing system 2300 Virtual reality service providing server 2360 Image analysis processing unit 2361 Object position specifying unit 2362 Optical component extracting unit 2363 Movement speed calculating unit 2400 Virtual reality service use condition storage DB
3000 Virtual Reality Service Providing System 3100 Contractor Information Storage DB
3200 Virtual reality content storage DB
3300 Virtual reality service provision server
3330 Virtual Reality Service Control Unit 3370 Virtual Reality Object Behavior Control Unit 3380 Audio Analysis Processing Unit 3381 Audio Level Extraction Unit 3382 Emotion Element Extraction Unit 3383 Context Acquisition Unit 3390 Playback Audio Generation Processing Unit 3500 Save Data Storage DB
3600 Suspension / Warning Condition Memory DB
4000 Virtual reality presenting system 5000 Virtual reality presenting system 6000 Virtual reality presenting system 7000 Virtual reality presenting system 10000 Virtual reality presenting system 11010 Ambient condition detecting unit 11020 Virtual reality presenting unit 11021 Video generation processing unit 11021-1 Viewpoint controlling unit 11021-2 Drawing Processing unit 11021-3 Light source control unit 11022 Video display unit 11022-1 Video display control unit 11022-1a Video data reading unit 11022-1b Video multiplex processing unit 11022-1c Video correction processing unit 11022-2 Display panel 11023 Audio output unit 11023 -1 audio reproduction control unit 11023-1a audio data reading unit 11023-1b DA conversion processing unit 11023-1c audio multiplex processing unit 11023-1d first DA conversion processing unit 11023- 1e 1st distribution process part 11023-1f 1st amplification process part 11023-1g 2nd DA conversion process part 11023-1h 1st distribution process part 11023-1i 2nd amplification process part 11023-1j reproduction | regeneration voice synthesizing process part 11023-2 speaker 11030 Virtual reality presentation interruption control unit 11031 Virtual reality presentation interruption determination processing unit 11032 Virtual reality presentation interruption instruction unit 12000 Virtual reality presentation system 12010 User state detection unit 12011 Inclination sensor 12012 Acceleration sensor 12013 User position detection unit 12014 User movement speed detection unit 12015 User orientation detection unit 12020 Virtual reality object storage unit 12030 Virtual reality object placement unit 12040 Video data storage unit for interruption
12050 Virtual reality presentation audio data storage unit 12060 Interruption audio data storage unit 12070 Interruption video data selection processing unit 12080 Interruption audio data selection processing unit 13000 Virtual reality presentation system 13010 Virtual reality presentation interruption condition storage unit 13020 Image analysis processing unit 13021 Object distance calculating unit 13022 Object moving speed calculating unit 13023 Object specifying unit 13024 Object type specifying unit 13025 Object position specifying unit 13026 User moving speed calculating unit 13030 Voice input unit 13040 User action specifying unit 13050 Virtual reality management control unit 14000 Virtual reality presentation System 14010 Attention Awareness Control Unit 14011 Attention Awareness Determination Processing Unit 14012 Attention Awareness Instructing Unit 15000 Virtual Reality Presenting System 15010 Attention Awareness Condition Storage Unit 16000 Virtual Reality Presenting System 16010 Virtual reality content storage unit 16020 Virtual reality object behavior control unit 17000 Virtual reality presentation system 17010 Virtual reality management unit 17011 Presentable virtual reality calculation processing unit 17012 Presented virtual reality calculation processing unit 17013 Presentable virtual reality distribution Processing unit 17014 Virtual reality reduction ratio instruction unit (allocation virtual reality notification unit)
18000 Virtual reality presentation system 18100 Virtual reality presentation device 18200 Virtual reality service providing server 18300 Contractor information storage database 18400 Virtual reality content storage database

Claims (20)

Virtual reality object storage means for storing a plurality of virtual reality objects;
Available virtual reality service information storage means for storing information indicating virtual reality services available to the user;
Virtual reality object reading means for reading out a virtual reality object used for a virtual reality service used by the user from the virtual reality object storage means;
Virtual reality object placement position determining means for determining the placement position of the virtual reality object in the virtual reality space, based on ambient situation information indicating a situation around the user;
Display image generation means for generating an image to be displayed on a display panel of a head mounted display device worn by a user on the head based on user state information indicating a user state and an arrangement position of the virtual reality object;
A virtual reality service providing system. Receiving means for receiving the ambient status information and the user status information from the head-mounted display device;
Video encoding means for encoding the video generated by the display video generation means;
Transmitting means for transmitting the encoded video to the head mounted display device;
The virtual reality service providing system according to claim 1, further comprising: Comprising virtual reality service start determining means for determining to start providing virtual reality service to the user,
When the virtual reality service start determining means determines to start providing the virtual reality service, the virtual reality object reading means stores the virtual reality object used for the virtual reality service provided to the user as the virtual reality object storage means. The virtual reality service providing system according to claim 2, wherein the virtual reality service providing system is further read out. The receiving means receives user voice information including a user voice input by a head-mounted display device,
Further comprising context acquisition means for analyzing a user's voice included in the user voice information and acquiring a context;
The virtual reality service start determination means determines to start providing the virtual reality service to the user when the context acquired by the context acquisition means is a request to start a virtual reality service;
The virtual reality service providing system according to claim 3. Service provision start condition storage means for storing a service provision start condition as a condition for starting provision of the virtual reality service;
The virtual reality service start determining means starts providing the virtual reality service to the user when the surrounding situation of the user indicated by the surrounding situation information received by the receiving means satisfies the service provision start condition. decide,
The virtual reality service providing system according to claim 3 or 4. The service provision start condition storage means stores a service provisionable position range indicating a position range where the provision of the virtual reality service can be received as a condition for starting the provision of the virtual reality service,
The receiving means receives at least position information indicating the current position of the user as the ambient state information,
The virtual reality service start determining unit is configured to notify the user of the virtual reality service when the current position of the user indicated by the position information, which is the surrounding state information received by the receiving unit, is within the service available position range. Decide to start offering,
The virtual reality service providing system according to claim 5. (Dynamic virtual reality object)
Virtual reality object control means for controlling the behavior of the virtual reality object arranged in a virtual space,
The display video generation means has the virtual reality whose behavior is controlled by the virtual reality object control means at a display position determined based on user state information indicating the state of the user and an arrangement position of the virtual reality object. Generate a video that displays the object,
The virtual reality service providing system according to any one of claims 1 to 6. (Behavior control by voice)
The receiving means receives audio information including audio input by a head mounted display device,
Further comprising context acquisition means for analyzing a voice included in the voice information and acquiring a context;
The virtual reality object control means controls the behavior of the virtual reality object arranged in the virtual reality space based on the context acquired by the context acquisition means;
The virtual reality service providing system according to claim 7. (Method basic design)
Virtual used in a virtual reality service providing system comprising: a virtual reality object storage database that stores a plurality of virtual reality objects; and an available virtual reality service information storage database that stores information indicating virtual reality services that can be used by a user. A real service providing method,
Based on the information indicating the virtual reality service available to the user stored in the available virtual reality service information storage database, the virtual reality object used for the virtual reality service used by the user is retrieved from the virtual reality object storage database. A virtual reality object reading step for reading;
A virtual reality object placement position determining step for determining a placement position of the read virtual reality object in the virtual reality space based on ambient situation information indicating a situation around the user;
A display image generation step for generating an image to be displayed on a display panel of a head mounted display device worn by the user based on user state information indicating a user state and an arrangement position of the virtual reality object; A virtual reality service providing method. Ambient condition detection means for detecting the situation of the user's surroundings,
User status detection means for detecting the user status;
Video generation means for generating a video of a virtual reality object based on the state of the user;
Video display means for displaying the generated video;
Audio output means for outputting audio associated with the virtual reality object;
Alerting means for alerting the user using the video display means or the audio output means based on at least the detected situation of the surroundings of the user;
A virtual reality presentation system comprising: It further includes a warning condition storage means for storing a warning condition that is a condition for performing a warning,
The alerting means is:
Attention determination means for determining whether or not the detected situation around the user satisfies the alert condition;
Based on the determination result, attention instructing instruction means for performing an instruction to display a warning image on the image display means or to output a sound for attention to the sound output means,
The virtual reality presentation system according to claim 10, comprising: The alert condition storage means stores a plurality of alert conditions,
The alert determination unit determines whether the detected situation around the user satisfies any of the plurality of alert conditions,
The attention instructing means corresponds to an instruction to display on the display means an image for alerting that is associated with the alerting condition when it is determined that the alerting condition is satisfied, or to the alerting condition Instructing the voice output means to output a warning sound attached to the voice output means;
The virtual reality presentation system according to claim 11. The attention instructing means instructs the sound output means to output a sound for attention associated with the virtual reality object based on the determination result.
The virtual reality presentation system according to claim 11 or 12. The alerting means controls the behavior of the virtual reality object so that the displayed virtual reality object alerts the user,
The virtual reality presentation system according to any one of claims 10 to 13. The alerting means performs the alerting by performing correction to reduce the luminance or brightness of the video of the virtual reality object generated by the video generating means.
The virtual reality system according to any one of claims 10 to 13. The alerting means performs the alerting by performing control to increase the amount of light transmitted from a position corresponding to the display position of the virtual reality object in the video display means.
The virtual reality system according to any one of claims 10 to 13. The ambient condition detection means captures at least the front of the user and obtains a front view image,
The alerting means determines whether there is an obstacle in front of the user based on the forward scenery image acquired by the surrounding condition detection means, and if the result of the determination is that there is an obstacle Alert the user to
The virtual reality presentation system according to any one of claims 10 to 16. The ambient condition detection means captures a surrounding scene image by photographing the surroundings of the user,
The alerting means determines whether a person exists within a predetermined distance from the user based on the surrounding scenery image acquired by the surrounding state detection means, and as a result of the determination, it is determined that a person exists. Alert the user in case
The virtual reality presentation system according to any one of claims 10 to 16. Ambient condition detection means for detecting the situation of the user's surroundings,
User status detection means for detecting the user status;
Video display means for displaying a video of a virtual reality object generated based on the state of the user;
Audio output means for outputting audio associated with the virtual reality object;
Alerting means for alerting the user using the video display means or the audio output means based on at least the detected situation of the surroundings of the user;
A virtual reality presentation device. An ambient condition detection step for detecting an ambient condition of the user;
A user status detection step for detecting a user status;
A video generation step of generating a video of a virtual reality object based on the state of the user;
A video display step for displaying the generated video;
An audio output step for outputting audio associated with the virtual reality object;
An alerting step for alerting at least based on the detected surroundings of the user;
A virtual reality presentation method comprising: