JP7111848B2 - Program, Information Processing Apparatus, and Method - Google Patents

Description

The present invention relates to programs, information processing apparatuses, and methods.

In Patent Document 1, when a second camera and a mirror object are arranged in a virtual space, and a player character is arranged in the field of view of the second camera, the appearance of the player character is transferred to the mirror object as an image. A HMD system for displaying is disclosed.

Japanese Patent No. 6220937

Conventional techniques have room for providing useful information to the user without reducing the user's visibility in the virtual space.

An object of one aspect of the present disclosure is to provide a user with useful information without reducing the visibility of the user in a virtual space.

According to one aspect of the invention, a program is provided that is executed by a computer having a processor. The program instructs the processor to define a virtual space including a first avatar associated with a first user and a second avatar associated with a second user; controlling a first field of view from a first avatar in response to movement of the head; placing an object in a first field of view; displaying first information on the first object; displaying a first field of view image corresponding to the first field of view on a first head-mounted device; to run.

According to one aspect of the present disclosure, it is possible to provide a user with useful information without reducing the user's visibility in the virtual space.

1 is a schematic representation of the configuration of an HMD system according to an embodiment; FIG. It is a block diagram showing an example of the hardware constitutions of the computer according to an embodiment. FIG. 4 is a diagram conceptually representing a uvw viewing coordinate system set in an HMD according to an embodiment; FIG. 2 is a diagram conceptually representing one aspect of representing a virtual space according to an embodiment; FIG. 2 is a top view of a user's head wearing an HMD according to an embodiment; It is a figure showing the YZ cross section which looked at the field-of-view area from the X direction in virtual space. It is a figure showing the XZ cross section which looked at the field-of-view area from the Y direction in virtual space. FIG. 2 is a diagram representing a schematic configuration of a controller according to an embodiment; FIG. FIG. 4 illustrates an example of yaw, roll, and pitch directions defined for a user's right hand according to an embodiment; It is a block diagram showing an example of a hardware configuration of a server according to an embodiment. 1 is a block diagram representing a computer as a module configuration according to an embodiment; FIG. 4 is a sequence chart representing part of the processing performed in the HMD set according to one embodiment; FIG. 2 is a schematic diagram showing a situation in which each HMD provides a virtual space to a user in a network; It is a figure which shows the user's 5A view image in FIG. 12(A). FIG. 4 is a sequence diagram showing processing performed in the HMD system according to one embodiment; 3 is a block diagram showing the detailed configuration of the modules of the computer according to one embodiment; FIG. FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 is a sequence diagram showing an example of processing executed in the HMD system according to one embodiment; FIG. 4 is a diagram for explaining a method of obtaining dimension data according to an embodiment; FIG. FIG. 4 is a diagram showing an example data structure of location information according to an embodiment; FIG. 4 is a diagram showing an example data structure of dimension data according to an embodiment; 4 is a flow chart representing a process for obtaining dimensional data according to one embodiment. FIG. 4 illustrates an example rotation data structure according to one embodiment. 4 is a sequence chart representing part of the processing performed in the HMD set according to one embodiment; FIG. 10 is a diagram showing virtual space and field of view images in one embodiment; FIG. 4 is a diagram showing virtual space and field-of-view images according to an embodiment; FIG. 4 is a diagram representing an example of a user's posture according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment; FIG. 4 illustrates virtual space and field of view images according to an embodiment;

[Embodiment 1]
Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. In one or more of the embodiments presented in this disclosure, elements of each embodiment may be combined with each other and the combined result shall also form part of the embodiments presented in this disclosure.

[Configuration of HMD system]
A configuration of an HMD (Head-Mounted Device) system 100 will be described with reference to FIG. FIG. 1 is a diagram showing a schematic configuration of an HMD system 100 according to this embodiment. The HMD system 100 is provided as a system for home use or as a system for business use.

HMD system 100 includes server 600 , HMD sets 110 A, 110 B, 110 C, and 110 D, external device 700 , and network 2 . Each of HMD sets 110A, 110B, 110C, and 110D is configured to communicate with server 600 and external device 700 via network 2 . Hereinafter, the HMD sets 110A, 110B, 110C, and 110D are collectively referred to as the HMD set 110 as well. The number of HMD sets 110 configuring the HMD system 100 is not limited to four, and may be three or less or five or more. HMD set 110 includes HMD 120 , computer 200 , HMD sensor 410 , display 430 and controller 300 . HMD 120 includes monitor 130 , gaze sensor 140 , first camera 150 , second camera 160 , microphone 170 , and speaker 180 . Controller 300 may include motion sensor 420 .

In one aspect, computer 200 is connectable to network 2 such as the Internet, and is capable of communicating with server 600 and other computers connected to network 2 . Other computers include, for example, computers of other HMD sets 110 and external devices 700 . In another aspect, HMD 120 may include sensor 190 instead of HMD sensor 410 .

The HMD 120 can be worn on the head of the user 5 and provide the user 5 with a virtual space during operation. More specifically, HMD 120 displays a right-eye image and a left-eye image on monitor 130 . When each eye of the user 5 views the respective image, the user 5 can perceive the image as a three-dimensional image based on the parallax of both eyes. The HMD 120 may include both a so-called head-mounted display having a monitor and a head-mounted device on which a smartphone or other terminal having a monitor can be attached.

The monitor 130 is implemented as, for example, a non-transmissive display device. In one aspect, the monitor 130 is arranged on the main body of the HMD 120 so as to be positioned in front of both eyes of the user 5 . Therefore, the user 5 can immerse himself in the virtual space by viewing the three-dimensional image displayed on the monitor 130 . In one aspect, the virtual space includes images of, for example, a background, objects that the user 5 can manipulate, and menus that the user 5 can select. In one aspect, the monitor 130 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smart phone or other information display terminal.

In another aspect, monitor 130 may be implemented as a transmissive display. In this case, the HMD 120 may be an open type, such as a glasses type, instead of a closed type that covers the eyes of the user 5 as shown in FIG. Transmissive monitor 130 may be temporarily configurable as a non-transmissive display by adjusting its transmittance. The monitor 130 may include a configuration for simultaneously displaying a portion of the image forming the virtual space and the real space. For example, the monitor 130 may display an image of the real space captured by a camera mounted on the HMD 120, or may make the real space visible by setting a partial transmittance high.

In one aspect, monitor 130 may include a sub-monitor for displaying images for the right eye and a sub-monitor for displaying images for the left eye. In another aspect, the monitor 130 may be configured to display the image for the right eye and the image for the left eye as one. In this case, monitor 130 includes a high speed shutter. The high speed shutter operates to alternately display the right eye image and the left eye image so that the image is perceived by only one eye.

In one aspect, the HMD 120 includes multiple light sources (not shown). Each light source is implemented by, for example, an LED (Light Emitting Diode) that emits infrared rays. HMD sensor 410 has a position tracking function for detecting movement of HMD 120 . More specifically, HMD sensor 410 reads a plurality of infrared rays emitted by HMD 120 and detects the position and tilt of HMD 120 in the physical space.

In another aspect, HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the HMD 120 by executing image analysis processing using the image information of the HMD 120 output from the camera.

In another aspect, HMD 120 may include sensor 190 instead of HMD sensor 410 or in addition to HMD sensor 410 as a position detector. HMD 120 can detect the position and tilt of HMD 120 using sensor 190 . For example, if sensor 190 is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, HMD 120 can detect its own position and tilt using any of these sensors instead of HMD sensor 410 . As an example, when the sensor 190 is an angular velocity sensor, the angular velocity sensor temporally detects angular velocities around three axes of the HMD 120 in real space. The HMD 120 calculates temporal changes in angles around the three axes of the HMD 120 based on the angular velocities, and further calculates the tilt of the HMD 120 based on the temporal changes in the angles.

Gaze sensor 140 detects the directions in which the user's 5 right and left eyes are directed.
That is, the gaze sensor 140 detects the line of sight of the user 5 . Detection of the line-of-sight direction is achieved by, for example, a known eye-tracking function. Gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right and left eyes of the user 5 with infrared rays and receives reflected light from the cornea and iris of the irradiated light, thereby detecting the rotation angle of each eyeball. The gaze sensor 140 can detect the line of sight of the user 5 based on each detected rotation angle.

The first camera 150 photographs the lower part of the user's 5 face. More specifically, first camera 150 captures the nose, mouth, and the like of user 5 . The second camera 160 photographs the eyes, eyebrows, etc. of the user 5 . The housing of the HMD 120 on the side of the user 5 is defined as the inside of the HMD 120 , and the housing of the HMD 120 on the side opposite to the user 5 is defined as the outside of the HMD 120 . In one aspect, first camera 150 may be positioned outside HMD 120 and second camera 160 may be positioned inside HMD 120 . Images generated by first camera 150 and second camera 160 are input to computer 200 . In another aspect, the first camera 150 and the second camera 160 may be realized as one camera, and the face of the user 5 may be photographed with this one camera.

The microphone 170 converts the speech of the user 5 into an audio signal (electrical signal) and outputs it to the computer 200 . The speaker 180 converts the audio signal into audio and outputs it to the user 5 . In another aspect, HMD 120 may include earphones instead of speaker 180 .

The controller 300 is connected to the computer 200 by wire or wirelessly. The controller 300 accepts command input from the user 5 to the computer 200 . In one aspect, controller 300 is configured to be grippable by user 5 . In another aspect, controller 300 is configured to be attachable to part of user's 5 body or clothing. In yet another aspect, controller 300 may be configured to output at least one of vibration, sound, and light based on signals transmitted from computer 200 . In yet another aspect, controller 300 receives an operation from user 5 for controlling the position and movement of an object placed in the virtual space.

In one aspect, controller 300 includes multiple light sources. Each light source is implemented, for example, by an LED that emits infrared rays. The HMD sensor 410 has a position tracking function. In this case, the HMD sensor 410 reads multiple infrared rays emitted by the controller 300 and detects the position and tilt of the controller 300 in the physical space. In another aspect, HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the controller 300 by executing image analysis processing using the image information of the controller 300 output from the camera.

Motion sensor 420 , in one aspect, is attached to the hand of user 5 to detect movement of the hand of user 5 . For example, the motion sensor 420 detects hand rotation speed, number of rotations, and the like. The detected signal is sent to computer 200 . Motion sensor 420 is provided in controller 300, for example. In one aspect, motion sensor 420 is provided, for example, in controller 300 configured to be grippable by user 5 . In another aspect, for safety in the real space, the controller 300 is attached to an object such as a glove that is attached to the hand of the user 5 so as not to fly off easily. In yet another aspect, sensors not worn by user 5 may detect movement of user's 5 hand. For example, a signal from a camera that takes an image of user 5 may be input to computer 200 as a signal representing the motion of user 5 . Motion sensor 420 and computer 200 are, for example, wirelessly connected to each other. In the case of wireless communication, the form of communication is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

Display 430 displays an image similar to the image displayed on monitor 130 . This allows users other than the user 5 wearing the HMD 120 to view the same image as that of the user 5 . The image displayed on display 430 does not have to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 430 include a liquid crystal display and an organic EL monitor.

Server 600 may transmit programs to computer 200 . In another aspect, server 600 may communicate with other computers 200 to provide virtual reality to HMDs 120 used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's action with the other computers 200 via the server 600 to generate a plurality of games in the same virtual space. users to enjoy common games. Each computer 200 may communicate signals based on each user's actions with other computers 200 without going through the server 600 .

The external device 700 may be any device as long as it can communicate with the computer 200 . The external device 700 may be, for example, a device capable of communicating with the computer 200 via the network 2, or may be a device capable of directly communicating with the computer 200 through short-range wireless communication or wired connection. Examples of the external device 700 include smart devices, PCs (Personal Computers), and peripheral devices of the computer 200, but are not limited to these.

[Computer hardware configuration]
A computer 200 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of computer 200 according to this embodiment. The computer 200 includes a processor 210, a memory 220, a storage 230, an input/output interface 240, and a communication interface 250 as main components. Each component is connected to bus 260 respectively.

Processor 210 executes a series of instructions contained in a program stored in memory 220 or storage 230 based on a signal given to computer 200 or based on the establishment of a predetermined condition. In one aspect, processor 210 is implemented as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), MPU (Micro Processor Unit), FPGA (Field-Programmable Gate Array), or other device.

Memory 220 temporarily stores programs and data. The program is loaded from storage 230, for example. Data includes data input to computer 200 and data generated by processor 210 . In one aspect, memory 220 is implemented as random access memory (RAM) or other volatile memory.

Storage 230 permanently holds programs and data. The storage 230 is implemented as, for example, a ROM (Read-Only Memory), hard disk device, flash memory, or other non-volatile storage device. The programs stored in the storage 230 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 230 includes data and objects for defining the virtual space.

In another aspect, storage 230 may be implemented as a removable storage device such as a memory card. In still another aspect, instead of storage 230 built into computer 200, a configuration using programs and data stored in an external storage device may be used. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used, such as an amusement facility, it is possible to collectively update programs and data.

Input/output interface 240 communicates signals between HMD 120 , HMD sensor 410 , motion sensor 420 and display 430 . Monitor 130 , gaze sensor 140 , first camera 150 , second camera 160 , microphone 170 and speaker 180 included in HMD 120 can communicate with computer 200 via input/output interface 240 of HMD 120 . In one aspect, the input/output interface 240 is implemented using a USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or other terminals. Input/output interface 240 is not limited to the one described above.

In some aspects, input/output interface 240 may also communicate with controller 300 . For example, input/output interface 240 receives signals output from controller 300 and motion sensor 420 . In another aspect, input/output interface 240 sends instructions output from processor 210 to controller 300 . The command instructs the controller 300 to vibrate, output sound, emit light, or the like. Upon receiving the command, the controller 300 performs any one of vibration, sound output, or light emission according to the command.

The communication interface 250 is connected to the network 2 and communicates with other computers connected to the network 2 (for example, the server 600). In one aspect, the communication interface 250 is implemented as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication) or other wireless communication interface. be done. Communication interface 250 is not limited to the one described above.

In one aspect, processor 210 accesses storage 230, loads one or more programs stored in storage 230 into memory 220, and executes the sequences of instructions contained in the programs. The one or more programs may include an operating system of computer 200, an application program for providing virtual space, game software executable in virtual space, and the like. Processor 210 sends a signal for providing virtual space to HMD 120 via input/output interface 240 . HMD 120 displays an image on monitor 130 based on the signal.

In the example shown in FIG. 2, computer 200 is provided outside HMD 120, but computer 200 may be built into HMD 120 in another aspect. As an example, a portable information communication terminal (for example, a smart phone) including monitor 130 may function as computer 200 .

The computer 200 may have a configuration that is commonly used by a plurality of HMDs 120 . According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so each user can enjoy the same application as other users in the same virtual space.

In one embodiment, in the HMD system 100, a real coordinate system, which is a coordinate system in real space, is set in advance. The real coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction perpendicular to the vertical direction, and the front-rear direction perpendicular to both the vertical and horizontal directions. The horizontal direction, vertical direction (vertical direction), and front-rear direction in the real coordinate system are defined as x-axis, y-axis, and z-axis, respectively. More specifically, in the real coordinate system, the x-axis is parallel to the horizontal direction of the real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the physical space.

In one aspect, HMD sensor 410 includes an infrared sensor. The presence of HMD 120 is detected when the infrared sensor detects infrared rays emitted from each light source of HMD 120 . The HMD sensor 410 further detects the position and inclination (orientation) of the HMD 120 in the physical space according to the movement of the user 5 wearing the HMD 120, based on the value of each point (each coordinate value in the real coordinate system). do. More specifically, the HMD sensor 410 can detect temporal changes in the position and tilt of the HMD 120 using each value detected over time.

Each tilt of HMD 120 detected by HMD sensor 410 corresponds to each tilt of HMD 120 around three axes in the real coordinate system. The HMD sensor 410 sets the uvw visual field coordinate system to the HMD 120 based on the tilt of the HMD 120 in the real coordinate system. The uvw visual field coordinate system set in the HMD 120 corresponds to the viewpoint coordinate system when the user 5 wearing the HMD 120 sees an object in virtual space.

[uvw visual field coordinate system]
The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw viewing coordinate system set in the HMD 120 according to one embodiment. HMD sensor 410 detects the position and tilt of HMD 120 in the real coordinate system when HMD 120 is activated. Processor 210 sets the uvw viewing coordinate system to HMD 120 based on the detected values.

As shown in FIG. 3 , the HMD 120 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 5 wearing the HMD 120 . More specifically, the HMD 120 rotates the horizontal direction, the vertical direction, and the front-rear direction (x-axis, y-axis, and z-axis) defining the real coordinate system by tilts around each axis of the HMD 120 within the real coordinate system. The three directions newly obtained by tilting around the axes are set as the pitch axis (u-axis), yaw-axis (v-axis), and roll-axis (w-axis) of the uvw visual field coordinate system in the HMD 120 .

In one aspect, when user 5 wearing HMD 120 is standing upright and looking straight ahead, processor 210 sets HMD 120 to a uvw viewing coordinate system that is parallel to the real coordinate system. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the real coordinate system are the pitch axis (u-axis) and yaw-axis (v-axis) of the uvw visual field coordinate system in the HMD 120. , and the roll axis (w-axis).

After the uvw visual field coordinate system is set on the HMD 120 , the HMD sensor 410 can detect the tilt of the HMD 120 in the set uvw visual field coordinate system based on the movement of the HMD 120 . In this case, the HMD sensor 410 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 120 in the uvw visual field coordinate system as the tilt of the HMD 120 . The pitch angle (θu) represents the tilt angle of the HMD 120 around the pitch axis in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 120 around the yaw axis in the uvw visual field coordinate system. A roll angle (θw) represents the tilt angle of the HMD 120 around the roll axis in the uvw visual field coordinate system.

The HMD sensor 410 sets the uvw visual field coordinate system in the HMD 120 after the HMD 120 moves based on the detected inclination of the HMD 120 to the HMD 120 . The relationship between the HMD 120 and the uvw visual field coordinate system of the HMD 120 is always constant regardless of the position and tilt of the HMD 120 . When the position and tilt of the HMD 120 change, the position and tilt of the uvw visual field coordinate system of the HMD 120 in the real coordinate system change in conjunction with the change in the position and tilt.

In one aspect, HMD sensor 410 detects HMD 120 based on the infrared light intensity and the relative positional relationship between a plurality of points (for example, the distance between points) obtained based on the output from the infrared sensor. position in the physical space may be specified as a relative position with respect to the HMD sensor 410 . Processor 210 may determine the origin of the uvw visual field coordinate system of HMD 120 in physical space (real coordinate system) based on the specified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually showing one aspect of expressing virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure covering the entire 360-degree direction of the center 12 . In FIG. 4, the celestial sphere in the upper half of the virtual space 11 is illustrated in order not to complicate the explanation. Each mesh is defined in the virtual space 11 . The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system, which is the global coordinate system defined in the virtual space 11 . The computer 200 associates each partial image that constitutes the panorama image 13 (still image, moving image, etc.) that can be developed in the virtual space 11 with each corresponding mesh in the virtual space 11 .

In one aspect, the virtual space 11 defines an XYZ coordinate system with the center 12 as the origin. The XYZ coordinate system is parallel to the real coordinate system, for example. The horizontal direction, vertical direction (vertical direction), and front-rear direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the The Z-axis (front-rear direction) is parallel to the z-axis of the real coordinate system.

When the HMD 120 is activated, that is, in the initial state of the HMD 120 , the virtual camera 14 is arranged at the center 12 of the virtual space 11 . In one aspect, processor 210 displays an image captured by virtual camera 14 on monitor 130 of HMD 120 . The virtual camera 14 similarly moves in the virtual space 11 in conjunction with the movement of the HMD 120 in the real space. Thereby, changes in the position and tilt of the HMD 120 in the real space can be similarly reproduced in the virtual space 11 .

A uvw field-of-view coordinate system is defined in the virtual camera 14 as in the case of the HMD 120 . The uvw visual field coordinate system of the virtual camera 14 in the virtual space 11 is defined to interlock with the uvw visual field coordinate system of the HMD 120 in the real space (real coordinate system). Therefore, when the tilt of HMD 120 changes, the tilt of virtual camera 14 also changes accordingly. The virtual camera 14 can also move in the virtual space 11 in conjunction with the movement of the user 5 wearing the HMD 120 in the real space.

The processor 210 of the computer 200 defines the field of view area 15 in the virtual space 11 based on the position and tilt of the virtual camera 14 (reference line of sight 16). The field of view area 15 corresponds to an area of the virtual space 11 that is viewed by the user 5 wearing the HMD 120 . That is, the position of the virtual camera 14 can be said to be the viewpoint of the user 5 in the virtual space 11 .

The line of sight of the user 5 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 5 visually recognizes an object. The uvw visual field coordinate system of the HMD 120 is equal to the viewpoint coordinate system when the user 5 visually recognizes the monitor 130 . The uvw visual field coordinate system of the virtual camera 14 is interlocked with the uvw visual field coordinate system of the HMD 120 . Therefore, the HMD system 100 according to one aspect can regard the line of sight of the user 5 detected by the gaze sensor 140 as the line of sight of the user 5 in the uvw field coordinate system of the virtual camera 14 .

[User's line of sight]
Determination of the line of sight of the user 5 will be described with reference to FIG. FIG. 5 is a top view of the head of user 5 wearing HMD 120 according to an embodiment.

In one aspect, gaze sensor 140 detects the line of sight of user 5's right and left eyes. In one aspect, when user 5 is looking near, gaze sensor 140 detects lines of sight R1 and L1. In another aspect, when user 5 is looking far away, gaze sensor 140 detects lines of sight R2 and L2. In this case, the angle between the lines of sight R2 and L2 with respect to the roll axis w is smaller than the angle between the lines of sight R1 and L1 with respect to the roll axis w. Gaze sensor 140 transmits the detection result to computer 200 .

When computer 200 receives detection values of lines of sight R1 and L1 from gaze sensor 140 as a detection result of lines of sight, computer 200 identifies point of gaze N1, which is the intersection of lines of sight R1 and L1, based on the detected values. On the other hand, when computer 200 receives detection values of lines of sight R2 and L2 from gaze sensor 140, computer 200 identifies the intersection of lines of sight R2 and L2 as the point of gaze. The computer 200 identifies the line of sight N0 of the user 5 based on the identified position of the gaze point N1. The computer 200 detects, for example, the extending direction of a straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 5 and the gaze point N1 as the line of sight N0. The line of sight N0 is the direction in which the user 5 is actually looking with both eyes. The line of sight N<b>0 corresponds to the direction in which the user 5 actually looks toward the field of view area 15 .

In another aspect, HMD system 100 may include a television broadcast reception tuner. With such a configuration, the HMD system 100 can display television programs in the virtual space 11 .

In still another aspect, the HMD system 100 may have a communication circuit for connecting to the Internet or a calling function for connecting to a telephone line.

[Vision area]
The field of view area 15 will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a diagram showing a YZ cross section of the visual field area 15 viewed from the X direction in the virtual space 11. As shown in FIG. FIG. 7 is a diagram showing an XZ cross section of the visual field area 15 viewed from the Y direction in the virtual space 11. As shown in FIG.

As shown in FIG. 6 , the field of view area 15 in the YZ cross section includes area 18 . A region 18 is defined by the position of the virtual camera 14 , the reference line of sight 16 and the YZ section of the virtual space 11 . The processor 210 defines an area 18 centered on the reference line of sight 16 in the virtual space and including the polar angle α.

As shown in FIG. 7 , the viewing area 15 in the XZ cross section includes area 19 . area 19
is defined by the position of the virtual camera 14 , the reference line of sight 16 and the XZ section of the virtual space 11 . The processor 210 defines a range including the azimuth angle β around the reference line of sight 16 in the virtual space 11 as the region 19 . The polar angles α and β are determined according to the position of the virtual camera 14 and the tilt (orientation) of the virtual camera 14 .

In one aspect, HMD system 100 provides user 5 with a view in virtual space 11 by displaying view image 17 on monitor 130 based on a signal from computer 200 . The field-of-view image 17 is an image corresponding to a portion of the panoramic image 13 corresponding to the field-of-view area 15 . When the user 5 moves the HMD 120 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the viewing area 15 in the virtual space 11 changes. As a result, the field-of-view image 17 displayed on the monitor 130 is updated to an image of the panorama image 13 that is superimposed on the field-of-view area 15 in the direction the user 5 faces in the virtual space 11 . The user 5 can visually recognize a desired direction in the virtual space 11 .

Thus, the tilt of the virtual camera 14 corresponds to the line of sight (reference line of sight 16 ) of the user 5 in the virtual space 11 , and the position where the virtual camera 14 is arranged corresponds to the viewpoint of the user 5 in the virtual space 11 . Therefore, by changing the position or tilt of the virtual camera 14, the image displayed on the monitor 130 is updated and the field of view of the user 5 is moved.

While wearing the HMD 120, the user 5 can visually recognize only the panoramic image 13 developed in the virtual space 11 without visually recognizing the real world. Therefore, the HMD system 100 can give the user 5 a high sense of immersion in the virtual space 11 .

In one aspect, processor 210 can move virtual camera 14 in virtual space 11 in conjunction with movement of user 5 wearing HMD 120 in the physical space. In this case, processor 210 identifies an image area (viewing area 15 ) projected on monitor 130 of HMD 120 based on the position and tilt of virtual camera 14 in virtual space 11 .

In one aspect, virtual camera 14 may include two virtual cameras, a virtual camera for providing images for the right eye and a virtual camera for providing images for the left eye. Appropriate parallax is set for the two virtual cameras so that the user 5 can recognize the three-dimensional virtual space 11 . In another aspect, virtual camera 14 may be realized by one virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by one virtual camera. In the present embodiment, the virtual camera 14 includes two virtual cameras, and the roll axis (w) generated by synthesizing the roll axes of the two virtual cameras is adapted to the roll axis (w) of the HMD 120. The technical idea according to the present disclosure is exemplified as configured as follows.

[controller]
An example of the controller 300 will be described with reference to FIG. FIG. 8 is a diagram showing a schematic configuration of controller 300 according to an embodiment.

As shown in FIG. 8, in one aspect, controller 300 may include a right controller 300R and a left controller (not shown). Right controller 300R is operated by user 5's right hand. The left controller is operated with the user's 5 left hand. In one aspect, right controller 300R and left controller are symmetrically configured as separate devices. Therefore, the user 5 can freely move the right hand holding the right controller 300R and the left hand holding the left controller. In another aspect, controller 300 may be an integrated controller that accepts two-handed operation. The right controller 300R will be described below.

Right controller 300R includes grip 310 , frame 320 , and top surface 330 . The grip 310 is configured to be gripped by the user's 5 right hand. For example, grip 310 may be held by user 5's right palm and three fingers (middle finger, ring finger, little finger).

Grip 310 includes buttons 340 and 350 and motion sensor 420 . Button 340 is arranged on the side surface of grip 310 and is operated by the middle finger of the right hand. Button 350 is arranged on the front surface of grip 310 and is operated by the index finger of the right hand. In one aspect, buttons 340, 350 are configured as trigger buttons. Motion sensor 420 is built into the housing of grip 310 . The grip 310 may not include the motion sensor 420 if the motion of the user 5 can be detected from around the user 5 by a camera or other device.

Frame 320 includes a plurality of infrared LEDs 360 arranged along its circumference. Infrared LED 360 emits infrared light in accordance with the progress of a program using controller 300 during execution of the program. Infrared rays emitted from infrared LED 360 can be used to detect the positions and orientations (inclination and orientation) of right controller 300R and left controller. Although the example shown in FIG. 8 shows infrared LEDs 360 arranged in two rows, the number of arrays is not limited to that shown in FIG. A single row or three or more row arrays may be used.

Top surface 330 includes buttons 370 and 380 and analog stick 390 . Buttons 370 and 380 are configured as push buttons. Buttons 370 and 380 accept operations by user 5's right thumb. In a certain aspect, the analog stick 390 accepts an operation in any direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object placed in the virtual space 11 .

In one aspect, right controller 300R and left controller include batteries for powering infrared LEDs 360 and other components. Batteries include, but are not limited to, rechargeable, button-type, dry cell-type, and the like. In another aspect, right controller 300R and left controller may be connected to a USB interface of computer 200, for example. In this case, right controller 300R and left controller do not require batteries.

As shown in states (A) and (B) of FIG. 8, for example, the yaw, roll, and pitch directions are defined for the user's 5 right hand. When the user 5 extends the thumb and index finger, the direction in which the thumb extends is the yaw direction, the direction in which the index finger extends is the roll direction, and the direction perpendicular to the plane defined by the yaw direction axis and the roll direction axis is the pitch direction. defined as

[Server hardware configuration]
Server 600 according to the present embodiment will be described with reference to FIG. FIG. 9 is a block diagram showing an example hardware configuration of server 600 according to an embodiment. The server 600 includes a processor 610, a memory 620, a storage 630, an input/output interface 640, and a communication interface 650 as main components. Each component is respectively connected to bus 660 .

Processor 610 executes a series of instructions contained in a program stored in memory 620 or storage 630 based on a signal provided to server 600 or based on the establishment of a predetermined condition. In one aspect, processor 610 is implemented as a CPU, GPU, MPU, FPGA, or other device.

Memory 620 temporarily stores programs and data. The program is loaded from storage 630, for example. Data includes data input to server 600 and data generated by processor 610 . In one aspect, memory 620 is implemented as RAM or other volatile memory.

Storage 630 permanently holds programs and data. The storage 630 is implemented as, for example, a ROM, hard disk device, flash memory, or other non-volatile memory device. The programs stored in the storage 630 may include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with the computer 200. The data stored in the storage 630 may include data and objects for defining the virtual space.

In another aspect, storage 630 may be implemented as a removable storage device such as a memory card. In still another aspect, instead of storage 630 built into server 600, a configuration using programs and data stored in an external storage device may be used. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used, such as an amusement facility, it is possible to collectively update programs and data.

Input/output interface 640 communicates signals with input/output devices. In one aspect, input/output interface 640 is implemented using a USB, DVI, HDMI (registered trademark) or other terminal. Input/output interface 640 is not limited to the one described above.

Communication interface 650 is connected to network 2 and communicates with computer 200 connected to network 2 . In one aspect, communication interface 650 is implemented as, for example, a LAN or other wired communication interface, or a WiFi, Bluetooth, NFC or other wireless communication interface. Communication interface 650 is not limited to those described above.

In one aspect, processor 610 accesses storage 630, loads one or more programs stored in storage 630 into memory 620, and executes the sequences of instructions contained in the programs. The one or more programs may include an operating system of server 600, an application program for providing virtual space, game software executable in virtual space, and the like. Processor 610 may send signals to computer 200 via input/output interface 640 to provide the virtual space.

[HMD control device]
A control device for the HMD 120 will be described with reference to FIG. In one embodiment, the controller is implemented by computer 200 having a well-known configuration. FIG. 10 is a block diagram representing a modular configuration of computer 200 according to one embodiment.

As shown in FIG. 10, computer 200 comprises control module 510 , rendering module 520 , memory module 530 and communication control module 540 . In one aspect, control module 510 and rendering module 520 are implemented by processor 210 . In another aspect, multiple processors 210 may act as control module 510 and rendering module 520 . Memory module 530 is implemented by memory 220 or storage 230 . Communication control module 540 is implemented by communication interface 250 .

A control module 510 controls the virtual space 11 provided to the user 5 . The control module 510 defines the virtual space 11 in the HMD system 100 using virtual space data representing the virtual space 11 . Virtual space data is stored, for example, in memory module 530 . The control module 510 may generate virtual space data or acquire virtual space data from the server 600 or the like.

The control module 510 places objects in the virtual space 11 using object data representing the objects. Object data is stored, for example, in memory module 530 . The control module 510 may generate object data or acquire object data from the server 600 or the like. The objects include, for example, an avatar object that is the alter ego of the user 5, a character object, an operation object such as a virtual hand operated by the controller 300, landscapes including forests, mountains, etc. arranged according to the progress of the story of the game, townscapes, and animals. etc.

The control module 510 places the avatar object of the user 5 of the other computer 200 connected via the network 2 in the virtual space 11 . In one aspect, control module 510 places user 5's avatar object in virtual space 11 . In one aspect, the control module 510 places an avatar object that resembles the user 5 in the virtual space 11 based on the image containing the user 5 . In another aspect, the control module 510 arranges, in the virtual space 11, an avatar object selected by the user 5 from among multiple types of avatar objects (for example, an object imitating an animal or a deformed human object). do.

Control module 510 identifies the tilt of HMD 120 based on the output of HMD sensor 410 . In another aspect, control module 510 identifies the tilt of HMD 120 based on the output of sensor 190, which functions as a motion sensor. The control module 510 detects the facial organs of the user 5 (eg, mouth, eyes, eyebrows) from the facial images of the user 5 generated by the first camera 150 and the second camera 160 . The control module 510 detects the motion (shape) of each detected organ.

The control module 510 detects the line of sight of the user 5 in the virtual space 11 based on the signal from the gaze sensor 140 . The control module 510 detects the viewpoint position (coordinate values in the XYZ coordinate system) where the detected line of sight of the user 5 and the celestial sphere of the virtual space 11 intersect. More specifically, the control module 510 detects the viewpoint position based on the line of sight of the user 5 defined by the uvw coordinate system and the position and tilt of the virtual camera 14 . Control module 510 transmits the detected viewpoint position to server 600 . In another aspect, control module 510 may be configured to transmit gaze information representing the gaze of user 5 to server 600 . In this case, the viewpoint position can be calculated based on the line-of-sight information received by the server 600 .

The control module 510 reflects the movement of the HMD 120 detected by the HMD sensor 410 on the avatar object. For example, the control module 510 detects that the HMD 120 is tilted, and tilts and arranges the avatar object. The control module 510 reflects the motion of the detected facial features on the face of the avatar object placed in the virtual space 11 . The control module 510 receives line-of-sight information of the other user 5 from the server 600 and reflects the line-of-sight information of the avatar object of the other user 5 . In one aspect, control module 510 reflects movements of controller 300 on avatar objects and operation objects. In this case, controller 300 includes a motion sensor, an acceleration sensor, a plurality of light emitting elements (for example, infrared LEDs), or the like for detecting movement of controller 300 .

The control module 510 arranges in the virtual space 11 an operation object for receiving an operation by the user 5 in the virtual space 11 . The user 5 operates an object placed in the virtual space 11, for example, by operating the operation object. In one aspect, the manipulation object may include, for example, a hand object, which is a virtual hand corresponding to the user's 5 hand. In one aspect, the control module 510 moves the hand object in the virtual space 11 based on the output of the motion sensor 420 in conjunction with the hand movement of the user 5 in the real space. In one aspect, the manipulation object may correspond to a hand portion of the avatar object.

The control module 510 detects a collision when each of the objects placed in the virtual space 11 collides with another object. The control module 510 can detect, for example, the timing at which the collision area of a certain object touches the collision area of another object, and performs predetermined processing when the detection is made. The control module 510 can detect the timing when the objects are separated from the touching state, and performs predetermined processing when the detection is made. The control module 510 can detect that objects are touching. For example, when the operating object touches another object, the control module 510 detects that the operating object touches the other object, and performs predetermined processing.

In one aspect, control module 510 controls image display on monitor 130 of HMD 120 . For example, control module 510 places virtual camera 14 in virtual space 11 . The control module 510 controls the position of the virtual camera 14 in the virtual space 11 and the tilt (orientation) of the virtual camera 14 . The control module 510 defines the field of view area 15 according to the tilt of the head of the user 5 wearing the HMD 120 and the position of the virtual camera 14 . Rendering module 520 generates view image 17 displayed on monitor 130 based on determined view area 15 . The field-of-view image 17 generated by the rendering module 520 is output to the HMD 120 by the communication control module 540 .

When the control module 510 detects an utterance by the user 5 using the microphone 170 from the HMD 120, the control module 510 specifies the computer 200 to which the audio data corresponding to the utterance is to be transmitted. The audio data is sent to the computer 200 specified by control module 510 . When the control module 510 receives voice data from another user's computer 200 via the network 2 , the control module 510 outputs voice (utterance) corresponding to the voice data from the speaker 180 .

Memory module 530 holds data used by computer 200 to provide virtual space 11 to user 5 . In one aspect, memory module 530 holds spatial information, object information, and user information.

The spatial information holds one or more templates defined to provide the virtual space 11. FIG.

The object information includes a plurality of panoramic images 13 forming the virtual space 11 and object data for arranging objects in the virtual space 11 . Panorama image 13 may include still images and moving images. The panorama image 13 may include an image of unreal space and an image of real space. Images of unreal space include, for example, images generated by computer graphics.

User information holds a user ID that identifies the user 5 . The user ID can be, for example, an IP (Internet Protocol) address or a MAC (Media Access Control) address set in the computer 200 used by the user. In another aspect, the user ID can be set by the user. The user information includes programs and the like for causing the computer 200 to function as a control device for the HMD system 100 .

Data and programs stored in memory module 530 are input by user 5 of HMD 120 . Alternatively, processor 210 downloads a program or data from a computer (for example, server 600 ) operated by the content provider, and stores the downloaded program or data in memory module 530 .

Communication control module 540 can communicate with server 600 and other information communication devices via network 2 .

In one aspect, control module 510 and rendering module 520 may be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, the control module 510 and the rendering module 520 can also be implemented as a combination of circuit elements that implement each process.

Processing in computer 200 is realized by hardware and software executed by processor 210 . Such software may be pre-stored on a hard disk or other memory module 530 . The software may be distributed as a program product stored in a CD-ROM or other computer-readable non-volatile data recording medium. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other network. Such software is read from a data recording medium by an optical disk drive or other data reading device, or downloaded from the server 600 or other computer via the communication control module 540, and then temporarily stored in the storage module. . The software is read from the storage module by processor 210 and stored in RAM in the form of executable programs. Processor 210 executes the program.

[Control Structure of HMD System]
The control structure of the HMD set 110 will be described with reference to FIG. FIG. 11 is a sequence chart representing part of the processing performed in HMD set 110 according to one embodiment.

As shown in FIG. 11, in step S1110, processor 210 of computer 200, as control module 510, specifies virtual space data and defines virtual space 11. As shown in FIG.

At step S1120, processor 210 initializes virtual camera . For example, the processor 210 places the virtual camera 14 at the predefined center 12 in the virtual space 11 in the work area of the memory, and directs the line of sight of the virtual camera 14 in the direction the user 5 is facing.

In step S1130, processor 210, as rendering module 520, generates view image data for displaying an initial view image. The generated visual field image data is output to HMD 120 by communication control module 540 .

In step S<b>1132 , monitor 130 of HMD 120 displays a field-of-view image based on the field-of-view image data received from computer 200 . The user 5 wearing the HMD 120 can recognize the virtual space 11 by viewing the visual field image.

In step S<b>1134 , HMD sensor 410 detects the position and tilt of HMD 120 based on a plurality of infrared rays emitted from HMD 120 . The detection result is output to the computer 200 as motion detection data.

In step S<b>1140 , processor 210 identifies the viewing direction of user 5 wearing HMD 120 based on the position and tilt included in the motion detection data of HMD 120 .

In step S1150, processor 210 executes the application program and places objects in virtual space 11 based on instructions included in the application program.

In step S<b>1160 , controller 300 detects the operation of user 5 based on the signal output from motion sensor 420 and outputs detection data representing the detected operation to computer 200 . In another aspect, manipulation of controller 300 by user 5 may be detected based on images from cameras placed around user 5 .

In step S<b>1170 , processor 210 detects an operation of controller 300 by user 5 based on the detection data acquired from controller 300 .

In step S<b>1180 , processor 210 generates view image data based on the operation of controller 300 by user 5 . The generated visual field image data is output to HMD 120 by communication control module 540 .

In step S<b>1190 , HMD 120 updates the field-of-view image based on the received field-of-view image data, and displays the updated field-of-view image on monitor 130 .

[Avatar object]
An avatar object according to this embodiment will be described with reference to FIGS. Hereafter, it is a figure explaining the avatar object of each user 5 of HMD set 110A, 110B. Hereinafter, the user of the HMD set 110A will be referred to as the user 5A, the user of the HMD set 110B as the user 5B, the user of the HMD set 110C as the user 5C, and the user of the HMD set 110D as the user 5D. A is attached to the reference sign of each component relating to the HMD set 110A, B is attached to the reference sign of each component relating to the HMD set 110B, C is attached to the reference sign of each component relating to the HMD set 110C, and the HMD set A D is attached to the reference number of each component related to 110D. For example, HMD 120A is included in HMD set 110A.

FIG. 12A is a schematic diagram showing a situation in which each HMD 120 provides the virtual space 11 to the user 5 in the network 2. FIG. Computers 200A-200D provide users 5A-5D with virtual spaces 11A-11D via HMDs 120A-120D, respectively. In the example shown in FIG. 12A, virtual space 11A and virtual space 11B are configured with the same data. In other words, the computers 200A and 200B share the same virtual space. An avatar object 6A of the user 5A and an avatar object 6B of the user 5B exist in the virtual space 11A and the virtual space 11B. Although the avatar object 6A in the virtual space 11A and the avatar object 6B in the virtual space 11B are each wearing the HMD 120, this is for the sake of clarity of explanation, and actually these objects are not wearing the HMDs 120. not

In one aspect, processor 210A may position virtual camera 14A capturing view image 17A of user 5A at the eye position of avatar object 6A.

FIG. 12(B) is a diagram showing a field-of-view image 17A of the user 5A in FIG. 12(A). The field-of-view image 17A is an image displayed on the monitor 130A of the HMD 120A. This field-of-view image 17A is an image generated by the virtual camera 14A. An avatar object 6B of the user 5B is displayed in the field-of-view image 17A. Although not particularly illustrated, the avatar object 6A of the user 5A is similarly displayed in the visual field image of the user 5B.

In the state of FIG. 12B, the user 5A can communicate with the user 5B through dialogue through the virtual space 11A. More specifically, the voice of user 5A acquired by microphone 170A is transmitted to HMD 120B of user 5B via server 600 and output from speaker 180B provided in HMD 120B. The voice of user 5B is transmitted to HMD 120A of user 5A via server 600, and is output from speaker 180A provided in HMD 120A.

Actions of the user 5B (actions of the HMD 120B and actions of the controller 300B) are reflected in the avatar object 6B arranged in the virtual space 11A by the processor 210A. Thereby, the user 5A can recognize the action of the user 5B through the avatar object 6B.

FIG. 13 is a sequence chart showing part of the processing executed in HMD system 100 according to the present embodiment. Although the HMD set 110D is not shown in FIG. 13, the HMD set 110D operates similarly to the HMD sets 110A, 110B, and 110C. In the following description, A is attached to the reference sign of each component regarding the HMD set 110A, B is attached to the reference sign of each component regarding the HMD set 110B, and C is attached to the reference sign of each component regarding the HMD set 110C. , and D is added to the reference numeral of each component of the HMD set 110D.

In step S1310A, the processor 210A in the HMD set 110A acquires avatar information for determining the motion of the avatar object 6A in the virtual space 11A. This avatar information includes, for example, information about the avatar such as motion information, face tracking data, and voice data. The movement information includes information indicating temporal changes in the position and tilt of the HMD 120A, information indicating hand movements of the user 5A detected by the motion sensor 420A, and the like. The face tracking data includes data specifying the position and size of each part of the user 5A's face. The face tracking data includes data indicating the movement of each organ that constitutes the face of the user 5A and line-of-sight data. The audio data includes data indicating the audio of the user 5A acquired by the microphone 170A of the HMD 120A. The avatar information may include information specifying the avatar object 6A or the user 5A associated with the avatar object 6A, information specifying the virtual space 11A in which the avatar object 6A exists, and the like. A user ID is mentioned as information which specifies the avatar object 6A and the user 5A. Room ID is mentioned as information which specifies virtual space 11A in which avatar object 6A exists. Processor 210A transmits the avatar information obtained as described above to server 600 via network 2 .

In step S1310B, processor 210B in HMD set 110B acquires avatar information for determining the motion of avatar object 6B in virtual space 11B and transmits it to server 600, as in the process in step S1310A. Similarly, in step S1310C, processor 210C in HMD set 110C acquires avatar information for determining the motion of avatar object 6C in virtual space 11C and transmits it to server 600. FIG.

In step S1320, server 600 temporarily stores the player information received from each of HMD set 110A, HMD set 110B, and HMD set 110C. Server 600 integrates avatar information of all users (users 5A to 5C in this example) associated with common virtual space 11 based on user IDs, room IDs, and the like included in each avatar information. Then, the server 600 transmits the integrated avatar information to all users associated with the virtual space 11 at a predetermined timing. Accordingly, synchronization processing is executed. Such synchronization processing enables the HMD set 110A, the HMD set 110B, and the HMD set 110C to share each other's avatar information at approximately the same timing.

Subsequently, based on the avatar information transmitted from server 600 to each HMD set 110A-110C, each HMD set 110A-110C executes the process of steps S1330A-S1330C. The processing of step S1330A corresponds to the processing of step S1180 in FIG.

In step S1330A, the processor 210A in the HMD set 110A updates information on the avatar objects 6B and 6C of the other users 5B and 5C in the virtual space 11A. Specifically, the processor 210A updates the position, orientation, etc. of the avatar object 6B in the virtual space 11 based on the motion information included in the avatar information transmitted from the HMD set 110B. For example, processor 210A updates the information (such as position and orientation) of avatar object 6B contained in the object information stored in memory module 530. FIG. Similarly, the processor 210A updates the information (position, orientation, etc.) of the avatar object 6C in the virtual space 11 based on the motion information included in the avatar information transmitted from the HMD set 110C.

In step S1330B, processor 210B in HMD set 110B updates information on avatar objects 6A and 6C of users 5A and 5C in virtual space 11B, similar to the process in step S1330A. Similarly, in step S1330C, the processor 210C in the HMD set 110C updates information on the avatar objects 6A, 6B of the users 5A, 5B in the virtual space 11C.

[Detailed module configuration]
Details of the module configuration of the computer 200 will be described with reference to FIG. FIG. 14 is a block diagram showing the detailed configuration of the modules of computer 200 according to one embodiment. As shown in FIG. 14, the control module 510 includes a virtual object generation module 1421, a virtual camera control module 1422, a manipulated object control module 1423, an avatar object control module 1424, a motion detection module 1425, a collision detection module 1426, and a virtual object module. A control module 1427 is provided.

A virtual object generation module 1421 generates various virtual objects in the virtual space 11 . In one aspect, the virtual objects may include, for example, landscapes including forests, mountains, etc., animals, etc. arranged according to the progress of the story of the game. In one aspect, the virtual object may include an avatar object, an operation object, a stage object, and a UI (User Interface) object.

The virtual camera control module 1422 controls behavior of the virtual camera 14 in the virtual space 11 . The virtual camera control module 1422 controls, for example, the arrangement position of the virtual camera 14 in the virtual space 11 and the orientation (inclination) of the virtual camera 14 .

The manipulation object control module 1423 controls manipulation objects for receiving manipulations by the user 5 in the virtual space 11 . The user 5 operates, for example, a virtual object placed in the virtual space 11 by operating the operation object. In one aspect, the operation object may include, for example, a hand object (virtual hand) corresponding to the hand of user 5 wearing HMD 120 . In one aspect, the manipulation object can correspond to a hand portion of an avatar object, which will be described later.

Avatar object control module 1424 reflects the movement of HMD 120 detected by HMD sensor 410 on the avatar object. For example, the avatar object control module 1424 detects that the HMD 120 is tilted and generates data for tilting and arranging the avatar object. In one aspect, the avatar object control module 1424 reflects movements of the controller 300 on the avatar object. In this case, controller 300 includes a motion sensor, an acceleration sensor, a plurality of light emitting elements (for example, infrared LEDs), or the like for detecting movement of controller 300 . The avatar object control module 1424 reflects the motion of facial organs detected by the motion detection module 1425 on the face of the avatar object placed in the virtual space 11 . That is, the avatar object control module 1424 reflects the motion of the user 5's face on the avatar object.

Motion detection module 1425 detects motion of user 5 . The motion detection module 1425 detects motion of the hand of the user 5 according to the output of the controller 300, for example. The motion detection module 1425 detects body motion of the user 5 according to the output of a motion sensor attached to the body of the user 5, for example. The motion detection module 1425 can also detect motion of the user's 5 facial features.

The collision detection module 1426 detects a collision when each virtual object placed in the virtual space 11 collides with another virtual object. Collision detection module 1426 may, for example, detect when one virtual object touches another virtual object. Collision detection module 1426 can detect when one virtual object is out of touch with another virtual object. Collision detection module 1426 can also detect when one virtual object is in contact with another virtual object. The collision detection module 1426 detects that the operating object touches another virtual object, for example, when the operating object touches another virtual object. Collision detection module 1426 performs predetermined processing based on these detection results.

The virtual object control module 1427 controls behavior of virtual objects other than avatar objects in the virtual space 11 . As an example, virtual object control module 1427 deforms the virtual object. As another example, the virtual object control module 1427 changes the placement position of the virtual object. As another example, virtual object control module 1427 moves virtual objects.

[Viewer's virtual space]
FIG. 15 is a diagram showing virtual space 11A and view image 1517A according to one embodiment. In FIG. 15A, at least avatar objects 6A to 6D, a virtual camera 14A, and a stage object 1532 are placed in a virtual space 11A for providing a virtual experience to a user 5A (first user). User 5A wears HMD 120A on the head. The user 5A holds the right controller 300RA with the right hand (first part) forming part of the right side of user 5A's body, and holds the left controller 300RA with the left hand (second part) forming part of the left side of user 5A's body. It holds the controller 300LA. Avatar object 6A (first avatar) includes virtual right hand 1531RA and virtual left hand 1531LA. The virtual right hand 1531RA is a kind of operation object, and can move in the virtual space 11A according to the movement of the right hand of the user 5A. The virtual left hand 1531LA is a kind of operation object, and can move in the virtual space 11A according to the movement of the left hand of the user 5A.

A virtual space 11A shown in FIG. 15A is constructed by playing live content on computer 200A. In virtual space 11A, avatar object 6B performs a performance as a live performer, and other avatar objects, including avatar object 6A, watch the performance as live viewers. In the virtual space 11A, avatar objects 6A-6D are individually associated with users 5A-5D, respectively.

The avatar object 6B (second avatar) is placed on the stage object 1532 in the virtual space 11A. The stage object 1532 has an appearance that imitates a stage in a real live venue. Avatar objects 6A, 6C, and 6D are all placed in front of stage object 1532 . In the virtual space 11A, the avatar object 6B performs a live performance by moving according to the movement of the user 5B (second user). In the virtual space 11A, the avatar object 6A watches the performance by the avatar object 6B. At this time, the avatar object 6C watches the performance performed by the avatar object 6B in the virtual space 11C provided to the user 5C. Similarly, in virtual space 11D provided to user 5D, avatar object 6D watches the performance performed by avatar object 6B. Therefore, it can be said that the user 5B is the performer and the users 5A, 5C and 5D are the viewers.

In FIG. 15A, the virtual camera 14A is arranged on the head of the avatar object 6A. The virtual camera 14A defines a viewing area 15A according to the position and orientation of the virtual camera 14A. The virtual camera 14A generates a field-of-view image 1517A corresponding to the field-of-view area 15A, and displays it on the HMD 120A as shown in FIG. 15(B). The user 5A visually recognizes a part of the virtual space from the viewpoint of the avatar object 6A by visually recognizing the view image 1517A. Thereby, the user 5A can obtain a virtual experience as if the user 5A himself were the avatar object 6A. View image 1517A includes avatar object 6B performing a performance. Therefore, the user 5A can view the performance by the avatar object 6B from the viewpoint of the avatar object 6A.

A plurality of different avatar objects 6B can also be arranged in the virtual space 11A. In one aspect, different users 5 are associated with the plurality of avatar objects 6B. In another aspect, the same user 5B is associated with multiple avatar objects 6B.

[Performer's virtual space]
FIG. 16 is a diagram showing virtual space 11B and view image 1617B according to one embodiment. In FIG. 16A, at least avatar objects 6A-6D, a virtual camera 14B, and a stage object 1532 are placed in a virtual space 11B for providing a virtual experience to a user 5B. A user 5B wears an HMD 120B (first head-mounted device) on the head. The user 5B holds the right controller 300RB with the right hand (first part) forming part of the right side of the body of the user 5B, and holds the right controller 300RB with the left hand (second part) forming part of the left side of the body of the user 5B. It holds the controller 300LB.

HMD 120B includes sensor 190 that functions as a motion sensor. Right controller 300RB and left controller 300LB include motion sensors 420 . User 5B also wears motion sensors 1641-1643. A motion sensor 1641 is attached to the waist of the user 5B by a belt 1644. FIG. The motion sensor 1642 is attached to the instep of the right foot of the user 5B. The motion sensor 1643 is attached to the instep of the user 5B's left foot. Motion sensors 1641-1643 are wired or wirelessly connected to computer 200B.

In one aspect, a motion sensor worn by user 5B detects the arrival time and angle of a signal (eg, an infrared laser) emitted from a base station (not shown). A processor 210B (hereinafter simply processor 210B) of computer 200B detects the position of the motion sensor with respect to the base station based on the detection result of the motion sensor. Processor 210B may also normalize the positions of the motion sensors relative to the base station relative to a predetermined point (eg, the position of head-mounted sensor 190).

Avatar object 6B includes virtual right hand 1531RB and virtual left hand 1531LB. The virtual right hand 1531RB is a kind of operation object, and can move in the virtual space 11B according to the movement of the right hand of the user 5B. The virtual left hand 1531LA is a kind of operation object, and can move in the virtual space 11B according to the movement of the left hand of the user 5B.

A virtual space 11B shown in FIG. 16A is constructed by playing live content on a computer 200B. The virtual spaces 11A and 11B are synchronized with each other under the control of the server 600. FIG. The user 5B moves his/her own body to cause the avatar object 6B to perform the performance. The computer 200B detects the movement of the user 5B based on the outputs of various motion sensors attached to the user 5B. In the virtual space 11B, the avatar object 6B performs a performance reflecting the movement of the user 5B in the real space according to the specified movement of the user 5B. In virtual space 11B, avatar objects 6A, 6C, and 6D watch a performance by avatar object 6B. When the avatar object 6B performs a performance according to the movement of the user 5B in the virtual space 11B, the avatar object 6B also performs the same performance in the virtual spaces 11A, 11C, and 11D in synchronization therewith. In this way, user 5B has a role as a distributor who distributes a live performance by avatar object 6B to users 5A, 5C, and 5D, respectively.

In FIG. 16A, the virtual camera 14B is placed on the head of the avatar object 6B. The virtual camera 14B defines a viewing area 15B according to the position and orientation of the virtual camera 14B. The virtual camera 14B generates a field-of-view image 1617B corresponding to the field-of-view area 15B, and displays it on the HMD 120B as shown in FIG. 16(B). The user 5B visually recognizes a part of the virtual space from the viewpoint of the avatar object 6B by visually recognizing the view image 1617B. Thereby, the user 5B can obtain a virtual experience as if the user 5B himself were the avatar object 6B. View image 1617B includes avatar objects 6A, 6C, and 6D viewing the performance. Therefore, the user 5B can grasp the state of the avatar objects 6A, 6C, and 6D watching the performance by the avatar object 6B from the viewpoint of the avatar object 6B.

[Live streaming flow]
FIG. 17 is a sequence diagram showing an example of processing executed in the HMD system 100. As shown in FIG. A series of processes for distributing the live performance of the avatar object 6B performed in the virtual space 11B from the computer 200B to the computer 200A will be described below. Avatar object 6B is also distributed live to computers 200C and 200D based on a similar series of processing.

In step S1701, processor 210B detects the positions of user 5B's head, waist, hands, and feet from the motion sensors worn by user 5B. Hereinafter, the positions of the parts of the user 5B detected by each motion sensor are also referred to as "positional information". In step S1702, the processor 210B calculates the rotational directions of the joints of the user 5B based on the current position information of the user 5B and the pre-acquired dimension data of the user 5B. The dimension data is data representing the body dimension of the user 5B. Dimensional data and the direction of rotation will be described later. Detecting the current position information, calculating the direction of rotation, and detecting the movement of the user 5B are synonymous.

At step S1703, the processor 210B moves the avatar object 6B placed in the virtual space 11B based on the current position information and the rotation direction. Processor 210B moves the right upper arm of avatar object 6B, for example, based on the rotation direction of the right shoulder. The processor 210B further moves the position of the avatar object 6B in the virtual space 11B based on the current position information (for example, current waist position information). Thereby, the processor 210B reflects the movement of the user 5B in the real space on the avatar object 6B in the virtual space. In other words, the processor 210B causes the avatar object 6B to perform a performance according to the movements of the user 5B.

The processing for reflecting the movement of the user 5B on the avatar object 6B is not limited to the processing according to the above-described positional information and rotation direction. Processor 210B may, for example, move avatar object 6B in response to movement of user 5B without calculating the direction of rotation. For example, the processor 210B may control the position of each part object of the avatar object 6B corresponding to each part of the user 5B so as to correspond to the position of each part constituting the body of the user 5B.

In step S<b>1704 , processor 210</b>B generates motion information representing the motion of avatar object 6</b>B when performing the performance, and transmits avatar information of avatar object 6</b>B including this motion information to server 600 .

In step S1710, processor 210A of computer 200A (hereinafter simply processor 210A) transmits avatar information of avatar object 6A to the server. In step S1720, server 600 executes synchronization processing for synchronizing virtual spaces 11A and 11B. Specifically, server 600 transmits the avatar information of avatar object 6B received from computer 200B to computer 200A. Server 600 further transmits the avatar information of avatar object 6A received from computer 200A to computer 200B.

At step S<b>1705 , processor 210</b>B receives the avatar information of avatar object 6</b>A transmitted from server 600 . At step S1706, processor 210B controls the avatar object in virtual space 11B based on the received avatar information. Thereby, the behavior of the avatar object 6A in the virtual space 11A is reflected in the avatar object 6A in the virtual space 11B. In other words, the behavior of avatar object 6A is synchronized in virtual spaces 11A and 11B. For example, when the avatar object 6A moves in the virtual space 11A, the avatar object 6A also moves in the virtual space 11B.

At step S1711, the processor 210A receives the avatar information of the avatar object 6B transmitted from the server 600. FIG. At step S1712, the processor 210A moves the avatar object 6B based on the motion information included in the received avatar information. Thereby, the processor 210B reflects the movement of the user 5B in the physical space on the avatar object 6B arranged in the virtual space 11A. In other words, the processor 210A causes the avatar object 6B to perform a performance according to the movement of the user 5B in the virtual space 11A. Thereby, the behavior of the avatar object 6B in the virtual space 11B is reflected in the avatar object 6B in the virtual space 11A. In other words, the behavior of avatar object 6B is synchronized in virtual spaces 11A and 11B. For example, when the avatar object 6B performs the first performance in the virtual space 11B, the avatar object 6B similarly performs the first performance in the virtual space 11A. In this way, the live performance of the avatar object 6B in the virtual space 11B is delivered to the virtual space 11A.

Although not shown, the processor 210B records the voice uttered by the user 5B using the microphone 170B. The processor 210B generates voice data representing the voice of the user 5B and transmits it to the server 600. FIG. The server 600 transmits the received voice data of the user 5B to the computer 200A by synchronous processing. The processor 210A outputs the voice represented by the received voice data of the user 5B to the speaker 180A. As a result of these series of processes, the user 5A can listen in real time to the voice uttered by the user 5B during the live performance.

[Acquisition of dimension data]
FIG. 18 is a diagram for explaining a method of acquiring dimension data. FIG. 18A shows a state in which the user 5B faces the front, spreads both hands horizontally, and stands up. Hereinafter, the state shown in FIG. 18A is also referred to as the first posture. FIG. 18B shows a state in which the user 5B faces the front, puts both hands on the sides of the thighs, and stands up. Hereinafter, the state shown in FIG. 18B is also referred to as the second posture.

In one aspect, processor 210B prompts user 5B to take a first pose and a second pose. As an example, processor 210B displays the characters in the first and second poses on monitor 130B and displays a message to the effect that they will take similar poses. As another example, the processor 210B may output a voice indicating that the first posture and the second posture are taken from the speaker 180B.

The processor 210B acquires position information of the head, waist, hands, and feet of the user 5B based on the outputs of the motion sensors worn by the user 5B in each of the two postures (the first posture and the second posture). . These position information can be obtained as positions in the real coordinate system (x, y, z) as shown in FIG.

The processor 210B calculates the dimension data of the user 5B from the position information corresponding to the two postures. In one embodiment, the processor 210B calculates the height, shoulder width, arm length, leg length, and head-to-shoulder height of the user 5B as dimension data, as shown in FIG. Processor 210B may calculate the distance between both hands in the second posture as the shoulder width. The processor 210B may calculate half the distance between the hands in the first posture minus the shoulder width as the arm length. The processor 210B may calculate the height as the distance from the height of the feet to the height of the head. Processor 210B may calculate the distance from the height of the legs to the height of the hips as the length of the legs. The processor 210B may calculate the height from the hand height to the head in the first posture as the height from the head to the shoulder.

FIG. 21 is a flowchart showing processing for acquiring dimension data. In step S2110, processor 210B arranges virtual camera 14B in virtual space 11B. Processor 210B further outputs field-of-view image 17B corresponding to the imaging range of virtual camera 14B to monitor 130B.

At step S2120, the processor 210B instructs the user 5B to take the first posture. For example, the processor 210B implements the process of step S2120 by arranging the object on which the instruction is written in the virtual space 11B. At step S2130, the processor 210B obtains position information corresponding to the first posture.

At step S2140, processor 210B instructs user 5B to take the second posture. At step S2150, the processor 210B obtains position information corresponding to the second posture.

In step S2160, processor 210B calculates dimension data of user 5B from the position information corresponding to the first posture and the position information corresponding to the second posture. Processor 210B stores the dimension data in storage 230B.

As described above, the user 5B can easily input his/her own dimensions to the computer 200B simply by taking two postures. In another aspect, user 5B may input his or her dimensions into computer 200B using an input device such as a keyboard.

[Direction of joint rotation]
In one embodiment, the processor 210B estimates the rotational direction of the joints of the user 5B based on the outputs (positional information) of the six motion sensors attached to the user 5B and the dimensional data. As an example, the processor 210B estimates the position of the shoulder based on the head position information, the shoulder width, and the height from the head to the shoulder. Processor 210B estimates the elbow position from the shoulder position and hand position information. This estimation can be performed by known applications using Inverse Kinematics.

In one embodiment, the processor 210B acquires the joint tilts (rotational directions) of the neck (head), waist, both wrists, and both ankles of the user 5B from six motion sensors. In addition, the processor 210B estimates the rotational directions of the shoulder, elbow, hip, and knee joints based on inverse kinematics. As shown in FIG. 22, the processor 210B obtains or estimates the rotation direction of each joint in the uvw field coordinate system.

Note that when the rotation direction is calculated based on the position information and the dimension data, the processor 210B calculates the rotation direction when the user 5B is not facing the front (that is, when the head and waist are facing in different directions). The position of the shoulder, etc. cannot be estimated accurately. Therefore, in another embodiment, the computer 200B may further consider the inclination of the part of the user 5B detected by the motion sensor to estimate the rotational direction of the joint. For example, the computer 200B estimates the shoulder positions based on the head position information, the tilt of the head, the tilt of the waist, the width of the shoulders, and the height from the head to the shoulders. According to this configuration, the computer 200B can improve the accuracy of the rotational direction of the joint.

[Live progress processing flow]
FIG. 23 is a sequence chart representing part of the processing performed in HMD set 110B according to one embodiment. FIG. 24 is a diagram illustrating virtual space 2411B and view image 2417B in one embodiment. In this embodiment, at least the HMD set 110B executes a series of processes for making the avatar object 6B go live.

At step S2301, the processor 210B defines a virtual space 2411B as shown in FIG. 24(A). This process corresponds to the process of step S1110 in FIG. Specifically, processor 210B defines virtual space 2411B represented by the virtual space data by specifying the virtual space data. In FIG. 24A, virtual space 2411B includes avatar objects 6A-6D. The virtual space 2411B is a virtual space in which the avatar object 6A and the like watch the live performance of the avatar object 6B. In other words, the virtual space 2411B is the virtual space 2411B in which the performance is performed by the avatar object 6B.

In step S2302, the processor 210B, as the virtual object generation module 1421, generates the stage object 1532 and places it in the virtual space 2411B. The stage object 1532 is a kind of virtual object on which the avatar object 6B performs performance. At step 2303, the processor 210B, as the virtual object generation module 1421, generates an avatar object 6B (first avatar) associated with the user 5B (first user) and places it in the virtual space 2411B. In FIG. 24A, avatar object 6B is placed on stage object 1532 . Although not shown, the processor 210B generates avatar information of the avatar object 6B and transmits it to the server 600 at arbitrary timing. In step S2304, the processor 210B creates the virtual camera 14B and places it in the virtual space 2411B.

At step S2305, the processor 210B receives from the server 600 the avatar information of each of the avatar objects 6A, 6C and 6D. At step S2306, the processor 210B, as the virtual object generation module 1421, generates avatar objects 6A (second avatars), 6C associated with the users 5A (second user), 5C, and 5D, respectively, based on the received avatar information. , and 6D are placed in the virtual space 2411B. Processor 210B also receives avatar information of other avatar objects 6 and also places other avatar objects 6 in virtual space 2411B. A large number of other avatar objects 6 are arranged in the virtual space 2411B, but illustration thereof is omitted in FIG. 24 and the like for convenience of explanation. Henceforth, other avatar objects 6 will not be referred to unless particularly necessary.

At step 2307, the processor 210B, as the virtual object generation module 1421, generates a specular object 2441 (first object) set to transparency, and places it within the field of view 15B (first field of view) from the avatar object 6B. Specular object 2441 is visualized within viewing area 15B in virtual space 2411B. In FIG. 24A, the position where the specular object 2441 is arranged is the position facing the avatar object 6B. When viewed from the avatar object 6B, the specular object 2441 is superimposed on the avatar object 6A. In other words, specular object 2441 is placed between avatar object 6A and avatar object 6B in virtual space 2411B.

The specular object 2441 has a display surface capable of displaying various types of information such as text and images (still images and moving images). The processor 210B arranges the specular object 2441 in the virtual space 2411B with the display surface of the specular object 2441 facing the avatar object 6B. The transparency of the specular object 2441 can take any value that indicates the state of the specular object 2441 in which the user 5B can see behind the specular object 2441 . The transmittance of the specular object 2441 when the specular object 2441 is completely transparent is 100 (maximum value), and the transmittance of the specular object 2441 when the specular object 2441 is completely opaque is 0 (minimum value). Then, the transmittance of the specular object 2441 takes any value between 0 and 100 or less. Processor 210B can, for example, maintain the transparency value of specular object 2441 at the same value during a live performance.

In step S<b>2308 , the processor 210</b>B, as the virtual object generation module 1421 , generates a virtual camera 2442 (virtual viewpoint) and arranges it in association with the specular object 2441 . The virtual camera 2442 is a virtual camera that generates an avatar image 2444 showing at least part of the avatar object 6B by photographing at least part of the avatar object 6B. In FIG. 24A, the virtual camera 2442 is arranged on the specular object 2441 with the photographing direction of the virtual camera 2442 facing the avatar object 6B. The virtual camera 2442 may be placed at a different position from the specular object 2441 while being associated with the specular object 2441 .

Processor 210B defines a field of view 2443 (third field of view) from virtual camera 2442 based on the imaging direction of virtual camera 2442 . A field of view area 2443 corresponds to an imaging range of the virtual camera 2442 . Processor 210B controls viewing area 2443 such that at least a portion of avatar object 6B is contained within viewing area 2443 . In FIG. 24(A), the entire avatar object 6B is contained within the view area 2443. In FIG. A virtual camera 2442 is a virtual camera that is not viewed by user B. Processor 210B sets virtual camera 14B so that virtual camera 14B is not photographed by virtual camera 2442 . Processor 210B also sets virtual camera 2442 such that virtual camera 2442 is not photographed by virtual camera 14B.

In step S2309, the processor 210B, as the virtual camera control module 1422, uses the virtual camera 2442 to photograph the avatar object 6B, thereby generating an avatar image 2444 representing the current front of the avatar object 6B in real time. Avatar image 2444 is a view image (second view image) corresponding to view area 2443 from virtual camera 2442 . In the virtual space 2411B, the virtual camera 14B is placed in the field of view 2443 of the virtual camera 2442. However, since the virtual camera 2442 is set to be unable to take pictures, the avatar image 2444 is the image of the virtual camera 14B. is not included. In other words, the processor 210B excludes the virtual camera 14B from the rendering target when rendering the field of view area 2443 of the virtual camera 2442 in the virtual space 2411B.

Processor 210B sets a predetermined transparency to generated avatar image 2444 . The transparency of the avatar image 2444 is, for example, the same as the transparency of the specular object 2441 . The transparency of the avatar image 2444 may be different from the transparency of the specular object 2441 . In step S2310, the processor 210B, as the virtual object control module 1427, displays the avatar image 2444 set to be transparent on the specular object 2441 in real time. The processor 210B, for example, displays the avatar image 2444 on the specular object 2441 in a horizontally reversed state. As a result, the mirror image of the avatar object 6B is displayed on the specular object 2441 in real time. As a result, the specular object 2441 functions as a virtual mirror reflecting the current front of the avatar object 6B.

In step S2311, the processor 210B, as the virtual camera control module 1422, controls the field of view 15B from the avatar object 6B according to the movement of the head of the user 5B associated with the HMD 120B. Specifically, processor 210B determines the position and tilt of virtual camera 14B in virtual space 2411B according to the movement of HMD 120B, and controls field of view area 15B according to the determined position and tilt of virtual camera 14B. This process corresponds to part of the process of step S1140 in FIG. Since the virtual camera 14B is arranged at the same position as the avatar object 6B, the position of the virtual camera 14B is synonymous with the position of the avatar object 6B. Furthermore, the field of view from the virtual camera 14B is synonymous with the field of view from the avatar object 6B.

At step S2312, processor 210B displays view image 2417B on monitor 130B. Specifically, the processor 210B defines a view image 2417B corresponding to the view area 15B based on the movement of the HMD 120B (that is, the position and tilt of the virtual camera 14B) and the virtual space data defining the virtual space 2411B. do. Defining the field of view image 2417 is synonymous with generating the field of view image 2417B. Processor 210B further causes HMD 120B to display field-of-view image 2417B by outputting field-of-view image 2417B to monitor 130B of HMD 120B. This processing corresponds to the processing of steps S1180 and S1190 in FIG.

The processor 210B, for example, displays a view image 2417B corresponding to the virtual space 2411B shown in FIG. 24(A) on the monitor 130B as shown in FIG. 24(B). User 5B recognizes that avatar objects 6C and 6D are watching the performance of avatar object 6B by visually recognizing view image 2417B. Further, the user 5B can confirm the current appearance of the avatar object 6B in real time by viewing the avatar image 2444 displayed on the specular object 2441 . Thereby, the user 5B can easily grasp whether or not the movement of the user 5B is correctly reflected on the avatar object 6B during the live performance. The user 5B can also visually recognize the state of the avatar object 6A placed behind the specular object 2441 through the specular object 2441 and the avatar image 2444 set to be transparent. Therefore, the user 5B can recognize that the avatar object 6A is also watching the performance of the avatar object 6B. Thus, the user 5B can accurately grasp the current appearance of the avatar object 6B while recognizing the entire field of view 15B in the virtual space 2411B.

(Viewer's virtual space 2411A)
FIG. 25 is a diagram showing virtual space 2411A and view image 2517A according to an embodiment. The processor 210A defines a virtual space 2411A for providing a virtual experience to the user 5A, as shown in FIG. 25(A). Virtual space 2411A is a virtual space that is basically synchronized with virtual space 2411B. The virtual space 2411A is also a virtual space in which the performance by the avatar object 6B is performed, like the virtual space 2411B. Processor 210A places virtual camera 14A, avatar object 6A, and stage object 1532 in virtual space 2411A, respectively. Processor 210A receives avatar information for each of avatar objects 6B-6D from server 600, and arranges avatar objects 6B-6D in virtual space 2411B based on this avatar information. Each position where the avatar objects 6A to 6D are arranged in the virtual space 2411A is the same as each position in the virtual space 2411B.

A portion of virtual space 2411B is not synchronized with virtual space 2411A. Specifically, processor 210A does not place specular object 2441 and virtual camera 2442 in virtual space 2411A. Therefore, unlike the virtual space 2411B, the virtual space 2411A does not include the specular object 2441, the virtual camera 2442, and the avatar image 2444 therein. Thus, the specular object 2441 is an object that cannot be visualized within the field of view 15A (second field of view) from the avatar object 6A in the virtual space 2411A.

The processor 210A, for example, displays a view image 2517A corresponding to the virtual space 2411A shown in FIG. 25(A) on the monitor 130A as shown in FIG. 25(B). The user 5A visually recognizes the performance of the avatar object 6B by visually recognizing the view image 2517A. Since specular object 2441 is not placed in virtual space 2411A, view image 2517A does not include specular object 2441 and avatar image 2444. FIG. Therefore, the user 5B does not visually recognize the specular object 2441 and the avatar image 2444 even if the user 5B visually recognizes the view image 2517A. Thus, the specular object 2441 is a virtual object that is visible to the user 5B but invisible to the user 5A. Also, the avatar image 2444 is information that can be visually recognized by the user 5B but cannot be visually recognized by the user 5A.

Since the user 5A does not visually recognize the specular object 2441 and the avatar image 2444 during the performance of the avatar object 6B, the user 5A does not feel uncomfortable with the performance of the avatar object 6B. Thereby, the user 5A can concentrate more on the performance of the avatar object 6A and enjoy the performance more. In particular, the user 5A does not know that the user 5B is checking the avatar image 2444 during the live performance, and therefore does not lose interest during the live performance.

FIG. 26 is a diagram showing an example of the posture of the user 5B according to an embodiment. FIG. 27 is a diagram illustrating virtual space 2411B and view image 2717B according to one embodiment. After starting the live performance, the user 5B moves his or her body so as to take the posture shown in FIG. 26, for example. The posture shown in FIG. 26 is a posture corresponding to the first performance. The processor 210B causes the avatar object 6B to perform the first performance as shown in FIG. 27(A) based on the body motion of the user 5B for taking the posture shown in FIG. The processor 210B captures the avatar object 6B using the virtual camera 2442 to generate in real time an avatar image 2744 representing the current front of the avatar object 6B performing the first performance. The processor 210B sets a predetermined degree of transparency to the generated avatar image 2744, and displays the avatar image 2744 set to be transparent on the specular object 2441 in a horizontally reversed state in real time.

The processor 210B, for example, displays a view image 2717B corresponding to the virtual space 2411B shown in FIG. 27(A) on the monitor 130B as shown in FIG. 27(B). View image 2717B includes specular object 2441 and avatar image 2744 . User 5B can confirm that avatar object 6B is correctly executing the first performance linked to the movement of user 5B by visually recognizing specular object 2441 and avatar image 2744 included in view image 2717B. can be done. The user 5B can also visually recognize the appearance of the avatar object 6A placed behind the specular object 2441 through the specular object 2441 and the avatar image 2744 . In this way, the avatar image showing the avatar object 6B is always set to be transparent regardless of how the user 5B moves. Therefore, the user 5B can always visually recognize the state behind the specular object 2441 in the virtual space during the live performance.

[Main advantages of this embodiment]
As described above, the processor 210B displays the avatar image 2744 set to be transparent on the specular object 2441 set to be transparent. Thereby, the user 5B can easily grasp the current behavior of the avatar object 6B by visually recognizing the avatar image 2744 . Further, the user 5B can easily grasp the situation behind the specular object 2441 in the virtual space 2411B through the specular object 2441 and the avatar image 2744 set to be transparent. Thus, the processor 210B can provide the user 5B with useful information without reducing the visibility of the user 5B in the virtual space 2411B.

(Example of transparency control)
FIG. 28 is a diagram illustrating virtual space 2411B and view image 2817B according to one embodiment. In FIG. 28A, a specular object 2441 is arranged between the avatar object 6B and the avatar object 6A. Processor 210B generates in real time an avatar image 2844 representing the current front of avatar object 6B by photographing avatar object 6B using virtual camera 2442 . Processor 210B further controls the transparency of specular object 2441 and avatar image 2844 based on the positional relationship between avatar object 6B and avatar object 6A. In FIG. 28A, the distance between the avatar object 6B and the avatar object 6A is less than or equal to the first threshold. Processor 210B increases the transparency of specular object 2441 and avatar image 2844 based on the fact that the distance between avatar object 6B and avatar object 6A is equal to or less than the first threshold. Specifically, processor 210B makes the transparency of specular object 2441 and avatar image 2844 higher than the transparency when the distance between avatar object 6B and avatar object 6A exceeds the first threshold.

The processor 210B, for example, displays a view image 2817B corresponding to the virtual space 2411B shown in FIG. 28(A) on the monitor 130B as shown in FIG. 28(B). When viewing the view image 2817B, the user 5B can more clearly view the figure of the avatar object 6A placed near the avatar object 6B through the specular object 2441 and the avatar image 2844. FIG. Therefore, the user 5B can grasp the appearance of the avatar object 6A more accurately.

FIG. 29 is a diagram illustrating virtual space 2411B and view image 2917B according to one embodiment. In FIG. 29A, a specular object 2441 is arranged between the avatar object 6B and the avatar object 6A. Processor 210B generates avatar image 2944 representing the current front of avatar object 6B in real time by photographing avatar object 6B using virtual camera 2442 . Processor 210B adjusts the transparency of specular object 2441 and avatar image 2944 based on the fact that the positional relationship between avatar object 6B and avatar object 6A shown in FIG. 28(A). Specifically, the position of the avatar object 6A in FIG. 29A is farther from the position of the avatar object 6B than the position of the avatar object 6A in FIG. 28A. As a result, the distance between the avatar object 6B and the avatar object 6A exceeds the first threshold. Processor 210B reduces the transparency of specular object 2441 and avatar image 2844 based on the fact that the distance between avatar object 6B and avatar object 6A exceeds the first threshold. Specifically, processor 210B makes the transparency of specular object 2441 and avatar image 2944 lower than the transparency when the distance between avatar object 6B and avatar object 6A is equal to or less than the first threshold.

The processor 210B, for example, displays a view image 2917B corresponding to the virtual space 2411B shown in FIG. 29(B) on the monitor 130B as shown in FIG. 29(B). When the user 5B visually recognizes the view image 2917B, the user 5B can visually recognize the avatar image 2944 displayed on the specular object 2441 more clearly than in the case of FIG. 28(B). Therefore, the user 5B can more reliably grasp whether the avatar object 6B is acting as intended. In FIG. 29(B), it is more difficult for the user 5B to visually recognize the appearance of the avatar object 6A than in FIG. 28(B). However, since the avatar object 6B is not near the avatar object 6A, even if the behavior of the avatar object 6B is difficult for the user 5B to grasp, it does not greatly affect the progress of the live performance.

(Position control of specular object 2441)
FIG. 30 is a diagram illustrating virtual space 2411B and view image 3017B according to one embodiment. In FIG. 30(A), the avatar object 6B is placed on the stage object 1532 at the right end as viewed from the avatar object 6B. The avatar object 6A is arranged at a position near the stage object 1532 in the virtual space 2411B so as to face the avatar object 6B. Avatar object 6D is placed behind avatar object 6A. Avatar object 6C is placed at a position distant from avatar objects 6A and 6D. In FIG. 30A, a specular object 2441 is arranged between the avatar object 6B and the avatar object 6A. The processor 210B captures the avatar object 6B using the virtual camera 2442 to generate an avatar image 2944 representing the current front of the avatar object 6B in real time and display it on the specular object 2441 .

The processor 210B, for example, displays a view image 3017B corresponding to the virtual space 2411B shown in FIG. 30(A) on the monitor 130B as shown in FIG. 30(B). User 5B can easily confirm the current appearance of avatar object 6B by viewing avatar image 3044 included in view image 2817B. The user 5B can also easily grasp the appearance of the avatar object 6A placed behind the specular object 2441 through the specular object 2441 and the avatar image 3044 .

FIG. 31 is a diagram illustrating virtual space 2411B and view image 3117B according to one embodiment. The user 5B performs an operation to move the avatar object 6B to the left end on the stage object 1532 when the avatar object 6B is arranged as shown in FIG. This operation is, for example, the user 5B pressing any button of the right controller 300RB. Processor 210B detects an operation by user 5B to move avatar object 6B. When the processor 210B detects the operation of the user 5B, the processor 210B moves the avatar object 6B to the leftmost position on the stage object 1532 as shown in FIG. 30(B). The processor 210B also moves the virtual camera 14B in conjunction with the avatar object 6B. As a result, the position of the virtual camera 14B also changes from the right end to the left end on the stage object 1532 .

Processor 210B moves viewing area 15B from the position shown in FIG. 30(A) to the position shown in FIG. 31(A) in conjunction with the movement of avatar object 6B and virtual camera 14B. Specifically, when avatar object 6B and virtual camera 14B move as shown in FIG. In response to , the visual field region 15B, which is the visual field from the virtual camera 14B in the virtual space 2411B, is controlled as shown in FIG. 31(A).

Processor 210B moves specular object 2441 so that specular object 2441 and virtual camera 2442 are positioned at predetermined positions within visual field 15B as shown in FIG. and virtual camera 2442 are moved. In FIG. 31A, the predetermined position is a position facing the avatar object 6B. As shown in FIGS. 30A and 31A, the relative positions of specular object 2441 and virtual camera 2442 within viewing area 15B do not change before and after viewing area 15B moves.

Processor 210B generates avatar image 3144 representing the front of avatar object 6B by photographing avatar object 6B using virtual camera 2442 after movement. The processor 210B further displays the avatar image 3144, which is set to be transparent, on the specular object 2441 after the movement, in a horizontally reversed state.

The processor 210B, for example, displays a view image 3117B corresponding to the virtual space 2411B shown in FIG. 31(B) on the monitor 130B as shown in FIG. 31(B). User 5B visually recognizes other locations in virtual space 2411B by visually recognizing view image 3117B. The user 5B further recognizes that the specular object 2441 has moved within the virtual space 2411B following the movement of the avatar object 6B. The user 5B further confirms the current appearance of the avatar object 6B by viewing the avatar image 3144 displayed on the specular object 2441 .

FIG. 32 is a diagram illustrating virtual space 2411B and view image 3217B according to one embodiment. In the example shown in FIG. 32, the user 5B faces directly to the left of himself after the avatar object 6B is placed as shown in FIG. 30(A). As a result, the whole body including the head of the user 5B rotates to the left by 90°. Processor 210B rotates avatar object 6B and virtual camera 14B by 90 degrees to the left of avatar object 6B in virtual space 2411B as shown in FIG. 32(A) in conjunction with such movement of user 5B. . The processor 210B moves the viewing area 15B from the position shown in FIG. 30(A) to the position shown in FIG. 32 in conjunction with the movement of the head of the user 5B. Specifically, when avatar object 6B and virtual camera 14B move as shown in FIG. , the visual field region 15B, which is the visual field from the virtual camera 14B in the virtual space 2411B, is controlled as shown in FIG.

Processor 210B moves specular object 2441 so that specular object 2441 and virtual camera 2442 are positioned at predetermined positions within visual field 15B as shown in FIG. and virtual camera 2442 are moved. In FIG. 32A, the predetermined position is a position facing the avatar object 6B. As shown in FIGS. 30A and 32A, the relative positions of specular object 2441 and virtual camera 2442 within viewing area 15B do not change before and after viewing area 15B moves.

Processor 210B generates avatar image 3244 representing the front of avatar object 6B by photographing avatar object 6B using virtual camera 2442 after movement. The processor 210B further displays the avatar image 3244, which is set to be transparent, on the specular object 2441 after movement in a horizontally reversed state.

The processor 210B, for example, displays a view image 3217B corresponding to the virtual space 2411B shown in FIG. 32(B) on the monitor 130B as shown in FIG. 32(B). The user 5B visually recognizes another location in the virtual space 2411B by visually recognizing the view image 3217B. The user 5B further recognizes that the specular object 2441 has moved within the virtual space 2411B following the change in orientation of the avatar object 6B. The user 5B further confirms the current appearance of the avatar object 6B by viewing the avatar image 3144 displayed on the specular object 2441 .

As shown in FIGS. 30 to 32, the specular object 2441 always stays at a predetermined position within the viewing area 15B regardless of where the user 5B views the virtual space 2411B. Therefore, the user 5B can easily visually recognize the appearance of the avatar object 6B displayed on the specular object 2441 at any time.

(Modification)
The processor 210B can display on the specular object 2441 an image showing at least a portion of the avatar object 6B and set to transparency. Displaying the avatar image 2844 on the specular object 2441 is an example of displaying on the specular object 2441 an image that shows at least part of the avatar object 6B and is set to be transparent. The processor 210B may also extract an image showing, for example, the upper body, which is a part of the avatar object 6B, from the image captured by the virtual camera 2442, and display the image on the specular object 2441.

Processor 210B does not necessarily need to display avatar image 2844 on specular object 2441 in real time. For example, the processor 210B can display the avatar image 2844 on the specular object 2441 after a certain period of time has elapsed after the generation of the avatar image 2844 . Also in this case, the user 5B can accurately grasp the behavior of the avatar object 6B through the avatar image 2844 displayed on the specular object 2441 .

The processor 210B can also display the image showing at least part of the avatar object 6B as it is on the specular object 2441 without horizontally reversing it. Also in this case, the user 5B can easily confirm the appearance of the avatar object 6B by viewing the image displayed on the specular object 2441 .

Processor 210B can display any first information on specular object 2441 . The first information may be information at least partially set to be transparent. The avatar image 2844 described above is an example of the first information set to be transparent and displayed on the specular object 2441 . The first information may be information that is not set to be transparent, such as a character string with a transparent background and an opaque character.

The first information can be any information (text, image) other than the avatar image 2844 regarding the live progress of the avatar object 6B. The first information may be, for example, information instructing action details for the avatar object 6B. Such first information includes, for example, information instructing the user 5B on a specific action that the avatar object 6B should take at a certain point in time during the live performance. By referring to the first information displayed on the specular object 2441 as a cheat sheet during the live performance, the user 5B can easily grasp what action should be performed by the avatar object 6B during the live performance. can be done. Therefore, the user 5B can smoothly progress the live performance of the avatar object 6B. Further, when viewing the first information, the user 5B can visually recognize the state behind the specular object 2441 in the virtual space 2411B through the specular object 2441. Therefore, the visibility of the user 5B in the virtual space 2411B can be improved. can be enhanced.

Processor 210B may also determine whether to visualize specular object 2441 within viewing area 15B based on an instruction by user 5B. For example, when the processor 210B detects the first instruction by the user 5B, it sets the visualization permission to the specular object 2441, and when it detects the second instruction by the user 5B, it sets the visualization prohibited to the specular object 2441. do. The processor 210B displays the visual field image including the specular object 2441 on the HMD 120 when the specular object 2441 is set to be visualized, and the specular object 2441 is displayed when the specular object 2441 is set not to be visualized. is displayed on the HMD 120. Therefore, the user 5B can determine whether or not to visually recognize the first information during the live performance, as necessary.

[Additional notes]
The contents of one aspect of the present invention are listed below.

(Item 1) Explained the program. According to one aspect of the present disclosure, the program is executed by a computer (200B) having a processor (210B). The program causes the processor to enter a virtual space ( 2411B), and controlling the first field of view (viewing area 15B) from the first avatar according to the movement of the head of the first user associated with the first head-mounted device (HMD 120B). a step (S2311), and the first object (specular object 2441) set to be transparent, which is visualized within the first field of view but not within the second field of view (viewing area 15A) from the second avatar, The step of placing in the first field of view (S2307), the step of displaying the first information (avatar image 2444) on the first object (S2310), and the first field of view image (field of view image 2417B) corresponding to the first field of view. is displayed on the first head mounted device (S2312).

(Item 2) In (Item 1), the step of controlling the first field of view includes moving the first field of view in conjunction with movement of the head of the first user, and the program instructs the processor to is further executed to move the first object such that the first object is positioned at a predetermined position within the first field of view in conjunction with the movement of .

(Item 3) In (Item 1) or (Item 2), the placing step includes placing a virtual viewpoint (virtual camera 2442) in association with the first object, and the program instructs the processor to A step of controlling the third field of view (field of view area 2443) from the virtual viewpoint so as to include at least a part thereof is executed, and the first information is a second field of view image corresponding to the third field of view set to be transparent. .

(Item 4) In (Item 3), the virtual viewpoint is arranged on the first object, and the predetermined position is a position facing the first avatar.

(Item 5) In (item 1) or (item 2), the first information is information indicating action details for the first avatar.

(Item 6) In any one of (Item 1) to (Item 5), the program causes the processor to determine whether or not to visualize the first object within the first field of view based on an instruction by the first user. perform the steps to

(Item 7) In (Item 3) or (Item 4), in the step of displaying the first information, the second field-of-view image is displayed on the first object in real time.

(Item 8) In any one of (Item 1) to (Item 7), in the placing step, the first object is placed between the first avatar and the second avatar.

(Item 9) In (Item 8), the program causes the processor to execute a step of controlling transparency of the first object based on the positional relationship between the first avatar and the second avatar.

(Item 10) In (Item 9), in the step of controlling the degree of transparency, if the distance between the first avatar and the second avatar is equal to or less than the first threshold value, the degree of transparency is increased and the distance between the first avatar and the second avatar is If the distance to the avatar exceeds the first threshold, the transparency is made lower.

(Item 11) The information processing apparatus has been described. According to one aspect of the present disclosure, the information processing device (computer 200B) includes a storage unit (storage 230B) that stores programs to be executed by the information processing device, and controls operations of the information processing device by executing the programs. and a control unit (processor 210B). The control unit controls a virtual space (2411B) including a first avatar (avatar object 6B) associated with a first user (user 5B) and a second avatar (avatar object 6A) associated with a second user (user 5). and controls the first field of view (viewing area 15B) from the first avatar according to the movement of the head of the first user associated with the first head-mounted device (HMD 120B), and within the first field of view A first object (mirror object 2441) that is visualized and is not visualized within the second field of view (viewing area 15A) from the second avatar and is set to be transparent is arranged within the first field of view, and the first object is , the step of displaying the first information (avatar image 2444) (S2310), and displaying the first view image (view image 2417B) corresponding to the first view on the first head mounted device.

(Item 12) I explained how to run the program. According to one aspect of the present disclosure, the method is performed by a computer with a processor. A method comprises: a processor creating a virtual space ( 2411B), and controlling the first field of view (viewing area 15B) from the first avatar according to the movement of the head of the first user associated with the first head-mounted device (HMD 120B). a step (S2311), and the first object (specular object 2441) set to be transparent, which is visualized within the first field of view but not within the second field of view (viewing area 15A) from the second avatar, The step of placing in the first field of view (S2307), the step of displaying the first information (avatar image 2444) on the first object (S2310), and the first field of view image (field of view image 2417B) corresponding to the first field of view. on the first head mounted device (S2312).

In the above embodiments, the virtual space (VR space) in which the user is immersed by the HMD has been exemplified and explained, but a transmissive HMD may be employed as the HMD. In this case, by outputting a visual field image obtained by synthesizing a part of an image constituting a virtual space with a real space visually recognized by a user through a transmissive HMD, an augmented reality (AR) space or mixed reality ( A virtual experience in MR (Mixed Reality) space may be provided to the user. In this case, an action may be generated on the target object in the virtual space based on the movement of the user's hand instead of the operation object. Specifically, the processor may identify the coordinate information of the position of the user's hand in the real space and define the position of the target object in the virtual space in relation to the coordinate information in the real space. As a result, the processor can grasp the positional relationship between the user's hand in the real space and the target object in the virtual space, and execute processing corresponding to the above-described collision control or the like between the user's hand and the target object. . As a result, it is possible to give an effect to the target object based on the movement of the user's hand.

2 network, 5, 5A, 5B, 5C, 5D user, 6, 6A, 6B, 6C, 6D avatar object, 11, 11A, 11B, 11C, 11D virtual space, 12 center, 13 panoramic image, 14, 14A, 14B virtual camera, 15, 15A, 15B, 15C field of view, 16 reference line of sight, 17, 17A, 17B field of view image, 18, 19 field, 100 HMD system, 110, 110A, 110B, 110C, 110D HMD set, 120, 120A, 120B, 120C, HMD, 130, 130A, 130B, 130C monitor, 140 gaze sensor, 150 first camera, 160 second camera, 170, 170A, 170B microphone, 180, 180A, 180B speaker, 190 sensor, 200, 200A, 200B computer, 210, 210A, 210B, 210C, 210D, 610 processor, 220, 620 memory, 230, 230A, 230B, 630 storage, 240, 640 input/output interface, 250, 650 communication interface, 260, 660 bus, 300, 300B controller, 300R right controller, 300L left controller, 310 grip, 320 frame, 330 top surface, 340, 340, 350, 370, 380 buttons, 360 infrared LED, 390 analog stick, 410 HMD sensor, 420, 420A motion sensor, 430, 430A display, 510 control module, 520 rendering module, 530 memory module, 540 communication control module, 600 server, 700 external device, 1421 virtual object generation module, 1422 virtual camera control module, 1423 operation object control module, 1424 avatar object Control Module, 1425 Motion Detection Module, 1426 Collision Detection Module, 1427 Virtual Object Control Module, 1517A, 1617B, 2417, 2417B, 2517A, 2717B, 2817B, 2917B, 3017B, 3117B Vision Image, 1531LA, 1531LB Virtual Left Hand, 1531RA, 1531RB virtual right hand, 1532 Stage object, 1641, 1642, 1643 Motion sensor, 1644 Belt, 2411A, 2411B Virtual space, 2441 Specular body object, 2442 Virtual camera, 2443 Viewing area, 2444, 2744, 2844, 2944, 3044, 3144, 3244 Avatar image

Claims (11)

A program executed by a computer having a processor,
The program causes the processor to:
defining a virtual space including a first avatar associated with a first user and a second avatar associated with a second user;
controlling the first avatar according to the first user's operation ;
generating an image of a first field of view from the first avatar in the virtual space;
arranging in the first field of view a first object set to be transparent, which is visualized in the first field of view but not in the second field of view from the second avatar;
displaying an image of the first avatar with transparency set on the first object;
and displaying a first view image corresponding to the first view on a display device associated with the first user . The program causes the processor to:
2. The program according to claim 1, further causing a step of moving said first object such that said first object is positioned at a predetermined position within said first field of view in conjunction with the movement of said first field of view. the placing step includes placing a virtual viewpoint in association with the first object;
The program causes the processor to:
causing the step of controlling a third field of view from the virtual viewpoint to include at least a portion of the first avatar;
3. The program according to claim 1, wherein said image of said first avatar is a second field of view image corresponding to said third field of view, which is set to be transparent . 3. The program according to claim 1 or 2, wherein said image of said first avatar is information indicating action details for said first avatar. 5. The program according to any one of claims 1 to 4 , wherein the program causes the processor to determine whether to visualize the first object within the first field of view based on an instruction by the first user. or the program according to item 1. 4. The program according to claim 3 , wherein displaying the image of the first avatar displays the second field-of-view image on the first object in real time. 7. The program according to any one of claims 1 to 6 , wherein in said arranging step, said first object is arranged between said first avatar and said second avatar. The program causes the processor to:
8. The program according to claim 7 , causing execution of a step of controlling transparency of said first object based on a positional relationship between said first avatar and said second avatar. In the step of controlling the transparency of the first object, if the distance between the first avatar and the second avatar is equal to or less than a first threshold, the transparency is increased and the first avatar and the second avatar are separated from each other. 9. The program according to claim 8 , wherein if the distance to 2 avatars exceeds the first threshold, the transparency is made lower. An information processing device,
The information processing device is
a storage unit that stores a program to be executed by the information processing device;
A control unit that controls the operation of the information processing device by executing the program,
The control unit
defining a virtual space including a first avatar associated with a first user and a second avatar associated with a second user;
controlling the first avatar according to the operation of the first user;
generating an image of a first field of view from the first avatar in the virtual space;
arranging in the first field of view a first object set to be transparent, which is visualized in the first field of view and not visualized in the second field of view from the second avatar;
displaying first information including an image of the first avatar with transparency set on the first object;
An information processing apparatus that displays a first view image corresponding to the first view on a display device associated with the first user . A method for a computer having a processor to execute a program comprising:
The method comprises: the processor;
defining a virtual space including a first avatar associated with a first user and a second avatar associated with a second user;
controlling the first avatar according to the first user's operation ;
generating an image of a first field of view from the first avatar in the virtual space;
arranging in the first field of view a first object set to be transparent, which is visualized in the first field of view but not in the second field of view from the second avatar;
displaying, on the first object, first information including an image of the first avatar with transparency set ;
displaying a first view image corresponding to said first view on a display device associated with said first user .

Images