JP7589268B2 - program - Google Patents

Description

The present invention relates to a program.

Technology has been known for some time that captures a user's movements and causes a character in a VR space to move. Using this technology, a character in a VR space performing (e.g., singing, playing an instrument, dancing, etc.) is distributed as video content.

One way to enhance the appeal of such video content is to combine footage of the character taken from various angles (e.g., the front, the side, a close-up of the upper body, etc.). Patent Document 1 discloses a technology for switching the position of a virtual camera placed in a virtual space.

JP 2021-099825 A

However, if you try to change the position of the virtual camera while controlling a character, not only does the process become cumbersome, but the continuity of the character's movements is lost, reducing the appeal of the video content.

The present invention was made in consideration of the above circumstances, and its purpose is to provide a program that allows users to obtain attractive video content materials with simple operations.

In order to solve the above-mentioned problem, the present invention provides a program that causes a computer equipped with an operation device to function as a camera setting means for setting a plurality of virtual cameras that capture an image of a target object placed in a virtual space from different set viewpoints, and as an image generation means for generating a plurality of set viewpoint images captured by each of the plurality of virtual cameras, the camera setting means setting imaging conditions for each of the plurality of virtual cameras including at least one of a distance from the target object, an angle relative to the target object, and an imaging portion of the target object in accordance with a user's operation via the operation device, and the computer functions as a camera movement means for moving each of the plurality of virtual cameras in accordance with a movement of the target object in accordance with the user's operation via the operation device so as to satisfy the set imaging conditions , the imaging conditions including whether or not to include in the set viewpoint image for each of the plurality of virtual cameras a peripheral object that is different from the target object and that overlaps with the target object as viewed from the virtual camera, and the image generation means switching whether or not to include the peripheral object in the set viewpoint image for each of the plurality of virtual cameras in accordance with the imaging conditions .

According to the present invention, attractive video content materials can be obtained with simple operations.

FIG. 1 is a diagram showing an overview of a system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of a server. FIG. 2 is a hardware configuration diagram of an HMD set which is an example of a user terminal. FIG. 11 is a hardware configuration diagram of a tablet terminal, which is another example of a user terminal. FIG. 1 is a diagram conceptually illustrating one aspect of a virtual space. 1 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from an X direction. 13 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from a Y direction. FIG. 2 is a functional block diagram of a server and a user terminal. 13 is a screen example of an imaging condition setting screen. 13 is a flowchart showing a process of a user terminal. FIG. 11 is a conceptual diagram of a virtual space corresponding to steps S11 and S12. 13 is a flowchart of a self-viewpoint video generation and display process. 13A and 13B are diagrams illustrating an example of a self-viewpoint image when display/non-display of a set viewpoint camera is switched. 13 is a flowchart of a setting viewpoint video generation and storage process. 13 is a diagram showing an example of a set viewpoint image when switching between displaying and hiding other people's avatars using the set viewpoint camera. FIG. 13A and 13B are diagrams illustrating an example of a set viewpoint image when a threshold distance is changed in the set viewpoint camera. 13 is a flowchart of a process for editing a set viewpoint video. 13 is an example of an editing screen and a viewed image.

Below, the system 1 according to the embodiment will be described with reference to the drawings. Note that the embodiment of the present invention described below is an example of how the present invention can be realized, and does not limit the scope of the present invention to the scope of the described embodiment. Therefore, the present invention can be implemented by making various modifications to the embodiment.

[Overview of System 1]
FIG. 1 is a diagram showing an overview of a system 1 according to this embodiment. As shown in FIG. 1, the system 1 mainly includes a server 10 and user terminals 20A, 20B, and 20C (hereinafter, these may be collectively referred to as "user terminals 20"). Although three user terminals 20 are illustrated in FIG. 1, examples of the user terminals 20 included in the system 1 are not limited to these. The server 10 and the user terminals 20 are connected to each other so as to be able to communicate with each other via a communication network 2. Specific examples of the communication network 2 are not particularly limited, but may be, for example, the Internet, a mobile communication system (e.g., 4G, 5G, etc.), a wireless network such as Wi-Fi (registered trademark), or a combination thereof.

The system 1 according to this embodiment is a system that allows each user of a plurality of user terminals 20A to 20C to experience a common virtual space provided by a server 10. More specifically, the user terminal 20 places a user character (hereinafter, sometimes referred to as an "avatar") associated with the user in the virtual space, and causes the user terminal 20 to display an image of the virtual space viewed from a preset viewpoint.

The user terminal 20 also operates the avatar in the virtual space in conjunction with the user's operation of the operating device. The avatar's movements include, for example, moving within the virtual space, moving various parts of the body, changing posture, changing facial expressions, speaking, and moving objects placed in the virtual space.

The user terminal 20 also transmits avatar data indicating the avatar's movements and status changes to the server 10 via the communication network 2. The server 10 transmits the avatar data received from the user terminal 20 to other user terminals 20 via the communication network 2. The user terminal 20 then updates the movements and status of the corresponding avatar in the virtual space based on the avatar data received from the server 10.

The user terminal 20 also captures an image of an avatar in the virtual space from a preset viewpoint to generate a set viewpoint video. The user terminal 20 also edits the generated set viewpoint video to generate a viewing video, and uploads the generated viewing video to the server 10. The user terminal 20 also downloads the viewing video from the server 10 and allows the user to view it.

Hereinafter, the user terminal 20 for operating the avatar may be referred to as the "play terminal", the user terminal 20 for editing the set viewpoint video may be referred to as the "editing terminal", and the user terminal 20 for viewing the viewing video may be referred to as the "viewing terminal". The play terminal, editing terminal, and viewing terminal may be the same terminal or different terminals. For example, an HMD set is suitable as the play terminal, a laptop computer or desktop computer is suitable as the editing terminal, and a tablet terminal or smartphone is suitable as the viewing terminal. However, the specific form of each terminal is not limited to the above examples.

[Configuration of Server 10]
Fig. 2 is a hardware configuration diagram of the server 10. The server 10 is realized by, for example, a general-purpose computer such as a workstation or a personal computer. As shown in Fig. 2, the server 10 mainly includes a processor 11, a memory 12, a storage 13, an input/output interface 14, and a communication interface 15. Each component of the server 10 is connected to a communication bus 19.

The processor 11 performs the processes described below by executing a series of instructions contained in the server program 13P stored in the memory 12 or the storage 13. The processor 11 is realized, for example, as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array), or other device.

The memory 12 temporarily holds the server program 13P and data. The server program 13P is loaded, for example, from the storage 13. The data includes data input to the server 10 and data generated by the processor 11. For example, the memory 12 is realized as a RAM (Random Access Memory) or other volatile memory.

The storage 13 permanently holds the server program 13P and data. The storage 13 is realized, for example, as a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The storage 13 may also be realized as a removable storage device such as a memory card. As yet another example, instead of being built into the server 10, the storage 13 may be connected to the server 10 as an external storage device. With this configuration, for example, in a situation where multiple user terminals 20 are used, such as an amusement facility, it becomes possible to collectively update the server program 13P and data.

The input/output interface 14 is an interface for connecting external devices such as a monitor, an input device (e.g., a keyboard, a pointing device), an external storage device, a speaker, a camera, a microphone, and a sensor to the server 10. The processor 11 communicates with the external devices through the input/output interface 14. The input/output interface 14 is realized using, for example, a Universal Serial Bus (USB), a Digital Visual Interface (DVI), a High-Definition Multimedia Interface (HDMI, registered trademark), or other terminals.

The communication interface 15 communicates with other devices (e.g., user terminal 20) connected to the communication network 2. The communication interface 15 is realized, for example, as a wired communication interface such as a LAN (Local Area Network), or a wireless communication interface such as Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), or NFC (Near Field Communication).

[Configuration of user terminal 20]
Fig. 3 is a hardware configuration diagram of an HMD (Head Mounted Display) set, which is an example of the user terminal 20. Fig. 4 is a hardware configuration diagram of a tablet terminal, which is another example of the user terminal 20. The user terminal 20 is realized as, for example, the HMD set shown in Fig. 3, the tablet terminal shown in Fig. 4, a smartphone, a feature phone, a laptop computer, a desktop computer, or the like.

As shown in FIG. 3, the user terminal 20 realized as an HMD set includes a computer 26 having a processor 21, memory 22, storage 23, input/output interface 24, and communication interface 25. Each component of the computer 26 is connected to a communication bus 29. The basic configuration of the processor 21, memory 22, storage 23, input/output interface 24, communication interface 25, and communication bus 29 is the same as that of the processor 11, memory 12, storage 13, input/output interface 14, communication interface 15, and communication bus 19 shown in FIG. 2. In addition, the storage 23 holds a terminal program 23P.

The user terminal 20 realized as an HMD set also includes an HMD 30, a motion sensor 41, and an operation device 42 as external devices of the computer 26. The HMD 30, the motion sensor 41, and the operation device 42 are connected to the processor 21 via the input/output interface 24.

The HMD 30 is worn on the user's head to provide the user with a virtual space. More specifically, the HMD 30 may include both a so-called head-mounted display equipped with a monitor and a head-mounted device to which a smartphone or other terminal equipped with a monitor can be attached. The HMD 30 mainly includes a monitor 31 (display device), a gaze sensor 32, cameras 33 and 34, a microphone 35, and a speaker 36.

As an example, the monitor 31 is realized as a non-transmissive display device. The monitor 31 is disposed on the main body of the HMD 30 so as to be located, for example, in front of both eyes of the user. The non-transmissive monitor 31 is realized, for example, as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor.

As another example, the monitor 31 is realized as a transmissive display device. In this case, the HMD 30 is not a closed type that covers the user's eyes, but an open type such as glasses. The monitor 31 may include a configuration that simultaneously displays a part of an image that constitutes a virtual space and the real space. As one example, the transmissive monitor 31 may display an image of the real space captured by a camera mounted on the HMD 30. As another example, the transmissive monitor 31 may be configured to have an adjustable transmittance. The transmissive monitor 31 may also set the transmittance of a part of the display area high so that the real space can be directly viewed.

The monitor 31 may also have the following configuration to allow the user to view a three-dimensional image. As one example, the monitor 31 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. As another example, the monitor 31 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the monitor 31 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

The gaze sensor 32 detects the direction in which the user's right and left eyes are looking. In other words, the gaze sensor 32 detects the user's gaze. The gaze sensor 32 is realized, for example, by a sensor with an eye tracking function. It is preferable that the gaze sensor 32 includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 32 irradiates the user's right and left eyes with infrared light, and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. The gaze sensor 32 then identifies the user's gaze based on each detected rotation angle.

Camera 33 captures the upper part of the face of the user wearing HMD 30 (more specifically, the user's eyes, eyebrows, etc.). Camera 34 captures the lower part of the face of the user wearing HMD 30 (more specifically, the user's nose, mouth, etc.). For example, camera 33 is attached to the side of the housing of HMD 30 that faces the user, and camera 34 is attached to the side opposite the side facing the user. Note that HMD 30 may be provided with one camera that captures the entire face of the user instead of the two cameras 33 and 34.

The microphone 35 converts the user's speech into an audio signal (electrical signal) and outputs it to the computer 26. The speaker 36 converts the audio signal output from the computer 26 into sound and outputs it to the user. Note that the HMD 30 may include earphones instead of the speaker 36.

The motion sensor 41 has a position tracking function for detecting the motion of the HMD 30. As one example, the motion sensor 41 may read multiple infrared rays emitted by the HMD 30 to detect the position and inclination of the HMD 30 in real space. As another example, the motion sensor 41 may be realized by a camera. In this case, the motion sensor 41 analyzes the image information of the HMD 30 output from the camera to detect the position and inclination of the HMD 30. As yet another example, the motion sensor 41 may be realized by an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor.

The operation device 42 is connected to the computer 26 by wire or wirelessly. The operation device 42 accepts input (operation) of commands to the computer 26 by the user. As one example, the operation device 42 may be a so-called controller that is held and operated by the user. As another example, the operation device 42 may be configured to be wearable on the user's body or part of his/her clothing, and may detect the user's movements by a motion sensor. However, specific examples of the operation device 42 are not limited to these, and may be a keyboard, a pointing device, a touch panel, etc.

As shown in FIG. 4, the user terminal 20 realized as a tablet terminal mainly comprises a processor 21, a memory 22, a storage 23, a communication interface 25, a monitor 31, cameras 33 and 34, a microphone 35, a speaker 36, a motion sensor 41, and an operation device 42. Each component of the tablet terminal is connected to a communication bus 29. The basic configuration of the processor 21, memory 22, storage 23, communication interface 25, monitor 31, cameras 33 and 34, microphone 35, speaker 36, motion sensor 41, and operation device 42 is the same as that of an HMD set, so the configuration specific to the tablet terminal will be described below.

The monitor 31 is provided on the surface of a flat housing. The camera 33 is a so-called in-camera that is attached to the surface of the flat housing and captures an image of the face of a user viewing the monitor 31. The camera 34 is a so-called out-camera that is attached to the back surface of the flat housing (the surface opposite the monitor 31) and captures an image of the surroundings. The motion sensor 41 detects the motion of the housing (e.g., rotation around three mutually orthogonal axes). An example of an operation device 42 suitable for a tablet terminal is a touch panel that is superimposed on the monitor 31 and accepts various touch operations by the user (e.g., tapping, sliding, flicking, pinching in, pinching out, etc.).

[Outline of Virtual Space 90]
Fig. 5 is a diagram conceptually illustrating one aspect of the virtual space 90. Fig. 6 is a diagram illustrating a YZ cross section of the field of view 94 in the virtual space 90 as viewed from the X direction. Fig. 7 is a diagram illustrating an XZ cross section of the field of view 94 in the virtual space 90 as viewed from the Y direction.

As shown in FIG. 5, the virtual space 90 has a spherical structure that covers the entire 360-degree area from the center C. In FIG. 5, the celestial sphere in the upper half of the virtual space 90 is illustrated to avoid complicating the explanation. Meshes are defined in the virtual space 90. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system, which is a global coordinate system defined in the virtual space 90. Each partial image that constitutes a panoramic image 91 (still image, video, etc.) that can be deployed in the virtual space 90 is associated with a corresponding mesh in the virtual space 90.

For example, in virtual space 90, an XYZ coordinate system is defined with center C as the origin. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the Z-axis (front-to-back direction) of the XYZ coordinate system is parallel to the z-axis of the real coordinate system.

A virtual camera 92 associated with the user terminal 20 is placed in the virtual space 90. The position of the virtual camera 92 in the virtual space 90 corresponds to the user's viewpoint in the virtual space 90. The orientation of the virtual camera 92 corresponds to the user's line of sight (reference line of sight 93) in the virtual space 90. The processor 21 then determines a viewing area 94 (the angle of view of the virtual camera 92) in the virtual space 90 based on the position and orientation of the virtual camera 92.

As shown in FIG. 6, the field of view 94 includes an area 95 in the YZ cross section. Area 95 is a range of polar angle α centered on the reference line of sight 93 in a vertical cross section (YZ cross section) including the reference line of sight 93 in the virtual space 90. As shown in FIG. 7, the field of view 94 includes an area 96 in the XZ cross section. Area 96 is a range of azimuth angle β centered on the reference line of sight 93 in a horizontal cross section (XZ cross section) including the reference line of sight 93 in the virtual space 90.

The processor 21 generates (extracts) a partial image included in the field of view 94 of the panoramic image 91 deployed in the virtual space 90 as a virtual space image 97 captured by the virtual camera 92. The processor 21 then displays the generated virtual space image 97 on the monitor 31. That is, the field of view 94 corresponds to the user's field of view in the virtual space 90. Furthermore, the field of view 94 moves in response to changes in the position and orientation of the virtual camera 92 in the virtual space 90, and the virtual space image 97 displayed on the monitor 31 is updated. That is, the user's field of view moves.

For example, the processor 21 moves the virtual camera 92 in the virtual space 90 in conjunction with the user's operation received by the operation device 42. The processor 21 also changes the orientation of the virtual camera 92 (i.e., the reference line of sight 93) in conjunction with the movement of the user terminal 20 detected by the motion sensor 41 (e.g., rotation around three mutually orthogonal axes). Furthermore, the processor 21 displays on the monitor 31 a virtual space image 97 captured by the virtual camera 92 after the position and orientation have been changed.

[Functional block diagram of server 10 and user terminal 20]
Fig. 8 is a functional block diagram of the server 10 and the user terminal 20. As shown in Fig. 8, the server program 13P loaded into the memory 12 causes the server 10 to function as an avatar synchronization means 110, an editing assistance means 120, and a viewing assistance means 130. In addition, the terminal program 23P loaded into the memory 22 causes the user terminal 20 (computer 26) to function as a virtual space definition means 210, a placement means 220, a camera setting means 230, a camera movement means 240, an image generation means 250, an image display means 260, and an editing means 270.

The avatar synchronization means 110 transmits (relays) avatar data received from one of the user terminals 20 via the communication network 2 to the other user terminals 20. This causes all user terminals 20A to 20C to hold the same avatar data. As a result, the same avatar is placed in the same position in the virtual space 90 defined by each of the user terminals 20A to 20C. In other words, the avatars are synchronized in all user terminals 20A to 20C.

The editing assistance means 120, for example, assists in editing the set viewpoint video generated at the play terminal (user terminal 20A) at the editing terminal (user terminal 20B). More specifically, the editing assistance means 120 stores the set viewpoint video data received from the play terminal through the communication network 2 in the storage 13. Also, in response to a request from the editing terminal, the editing assistance means 120 transmits the set viewpoint video data stored in the storage 13 to the editing terminal through the communication network 2. Furthermore, the editing assistance means 120 stores the viewing video data received from the editing terminal through the communication network 2 in the storage 13. In response to a request from the viewing terminal (user terminal 20C), the viewing video data stored in the storage 13 transmits the viewing video data to the viewing terminal through the communication network 2.

In addition, some or all of the functions of the editing assistance means 120 and the viewing assistance means 130 may be realized on a computer independent of the server 10. As one example, the editing assistance means 120 can be realized by a storage server that provides storage space on the cloud. As another example, the viewing assistance means 130 can be realized by a server that provides a video viewing service (streaming service).

The virtual space definition means 210 defines a virtual space 90 corresponding to the real space. More specifically, the virtual space definition means 210 expands virtual space data indicating the virtual space 90 in the memory 22. The virtual space data indicates, for example, a panoramic image 91 and the shapes and positions of objects (e.g., buildings, plants) to be placed in the virtual space 90. Note that the virtual space data may be downloaded in advance from the server 10 and stored in the storage 23, or may be downloaded from the server 10 when the virtual space 90 is defined. The virtual space definition means 210 may also define a virtual space 90 selected from among a plurality of virtual spaces 90 via the operation device 42. The specific process of defining the virtual space 90 is already known, and therefore a detailed description thereof will be omitted.

The placement means 220 places characters in the virtual space 90 defined by the virtual space definition means 210. Here, a case where the terminal program 23P is executed on the user terminal 20A will be described. The characters that the placement means 220 places in the virtual space 90 include the avatar of the user of the user terminal 20A (hereinafter referred to as the "user's avatar") and the avatars of the users of the other user terminals 20B and 20C (hereinafter referred to as the "other users' avatars").

First, the placement means 220 places the principal avatar in the virtual space 90 based on the avatar data pre-stored in the storage 23. Then, the placement means 220 transmits the avatar data of the principal avatar to the server 10 via the communication network 2. Furthermore, when the placement means 220 receives a user's operation via the operation device 42 instructing the principal avatar to move, it moves the principal avatar in the virtual space 90 in accordance with the operation. Then, the placement means 220 updates the avatar data to indicate the state of the principal avatar after the movement, and transmits the updated avatar data to the server 10 via the communication network 2.

The placement means 220 also receives avatar data of the other person's avatar from the server 10 via the communication network 2. The placement means 220 then places the other person's avatar in the virtual space 90 based on the avatar data received from the server 10. Furthermore, the placement means 220 rearranges the other person's avatar every time the avatar data is updated, thereby causing the other person's avatar to operate within the virtual space 90.

The camera setting means 230 sets the imaging conditions of the virtual camera 92 (more specifically, the set viewpoint cameras 92B to 92D) in accordance with the user's operation via the operation device 42. FIG. 9 is an example of an imaging condition setting screen. As shown in FIG. 9, the imaging condition setting screen accepts settings of the set distance, set angle, imaging part, display/non-display of the set viewpoint camera, display/non-display of other people's avatars, and threshold distance as imaging conditions. However, specific examples of imaging conditions are not limited to these. Furthermore, the interface for setting the imaging conditions is not limited to the example in FIG. 9. The camera setting means 230 then positions the virtual camera 92 in the virtual space 90 so as to satisfy the imaging conditions specified by the user via the operation device 42.

As shown in FIG. 11, the virtual camera 92 according to this embodiment includes a self-viewpoint camera 92A and multiple set viewpoint cameras 92B, 92C, and 92D. Hereinafter, the image captured by the self-viewpoint camera 92A will be referred to as the "self-viewpoint image," and the images captured by the set viewpoint cameras 92B to 92D will be referred to as the "set viewpoint image." The self-viewpoint image and the set viewpoint image are constructed by arranging multiple virtual space images 97 in chronological order.

The self-viewpoint camera 92A is a virtual camera that captures the virtual space 90 from the self-viewpoint of the user's avatar toward the reference line of sight 93. In other words, the imaging range of the self-viewpoint camera 92A matches the field of view of the user's avatar. However, the self-viewpoint camera 92A may capture the virtual space 90 from behind the user's avatar toward the reference line of sight 93. The set viewpoint cameras 92B, 92C, and 92D are virtual cameras that capture the user's avatar from a set viewpoint that is set in advance at a position different from the user's avatar. Note that the number of set viewpoint cameras 92B, 92C, and 92D is not limited to three, and may be two or more. The imaging conditions of the set viewpoint camera 92B are explained below.

"Set distance" indicates the distance between the user's avatar and the set viewpoint camera 92B. "Set angle" indicates the angle between the user's avatar's reference line of sight 93 and the line connecting the user's avatar and the set viewpoint camera 92B. In other words, the set angle is 0° when capturing an image of the front of the user's avatar, and the set angle is 90° when capturing an image of the right side of the user's avatar. "Image-capturing part" indicates the part of the user's avatar that is captured by the set viewpoint camera 92B (e.g., whole body, upper body, face, etc.). "Show/hide set viewpoint camera" indicates whether or not to include the set viewpoint camera 92B in the user's viewpoint image.

"Show/hide other person's avatar" indicates whether or not other person's avatars located between the user's own avatar and the set viewpoint camera 92B are included in the set viewpoint image. "Threshold distance" indicates the distance from the set viewpoint camera 92B at which other person's avatars are hidden when hiding of other person's avatars is selected. As an example, when the threshold distance is set to 5m, other person's avatars located between the user's own avatar and the set viewpoint camera 92B that are less than 5m away from the set viewpoint camera 92B are excluded from the set viewpoint image, and other person's avatars that are 5m or more away from the set viewpoint camera 92B are included in the set viewpoint image. As another example, when the set distance is set to the threshold distance, all other person's avatars located between the user's own avatar and the set viewpoint camera 92B are excluded from the set viewpoint image.

The camera moving means 240 moves the virtual camera 92 within the virtual space 90 based on the movement of the user terminal 20 detected by the motion sensor 41 or the avatar's operation instruction via the operation device 42. Moving the user terminal 20 and instructing the avatar's operation via the operation device 42 are examples of movement operations. The camera moving means 240 then moves the virtual camera 92 to follow the user's avatar moved by the placement means 220 and to satisfy the imaging conditions set by the camera setting means 230.

The image generating means 250 generates an image captured within the virtual space 90 by the virtual camera 92. More specifically, the image generating means 250 extracts an image corresponding to the field of view 94 from the panoramic image 91 as a virtual space image 97. The image generating means 250 also generates an image by arranging the virtual space images 97 extracted at a predetermined time interval (e.g., 1/60 second intervals) in chronological order.

The image generating means 250 generates a self-viewpoint image captured by the self-viewpoint camera 92A of the virtual space 90, and a plurality of set viewpoint images captured by the set viewpoint cameras 92B to 92D of the virtual space 90. The image generating means 250 then stores self-viewpoint image data representing the self-viewpoint image in the memory 22. The image generating means 250 also stores set viewpoint image data representing the set viewpoint image in the storage 23.

The video display means 260 displays the self-viewpoint video on the monitor 31. More specifically, the video display means 260 expands the self-viewpoint video data generated by the video generation means 250 into the graphic memory of the monitor 31. Then, every time the character in the virtual space 90 is updated, the video generation means 250 generates a new virtual space image 97, and the video display means 260 displays the newly generated virtual space image 97 on the monitor 31. This allows the user of the user terminal 20 to view the self-viewpoint video showing the field of view of the character operating in the virtual space 90 through the monitor 31.

The editing means 270 edits the multiple set viewpoint videos generated by the video generating means 250 in accordance with user operations via the operating device 42 to generate a viewing video. Editing refers to, for example, arranging still images extracted from each of the multiple set viewpoint videos in chronological order. Note that, although the editing means 270 and the other means 210 to 270 are implemented in different user terminals 20A and 20B in the example of FIG. 8, they may be implemented in the same user terminal 20. Also, the editing means 270 can be replaced by a well-known video editing program.

The operation of system 1 when generating a set viewpoint video will be described with reference to Figures 10 to 16. In this embodiment, it is assumed that a user of a user terminal 20 watches a performance (e.g., singing, playing an instrument, dancing, etc.) given by two avatars 50, 51 in a virtual space 90 through avatars 52, 53. Below, the processing of user terminal 20A associated with avatar 50 will be mainly described. The avatar 50 of user terminal 20A will be referred to as "user avatar 50", and the avatars 51, 52, 53 of other user terminals 20 will be referred to as "other avatars 51, 52, 53".

In this embodiment, the performance of the person's avatar 50 is captured by three set viewpoint cameras 92B to 92D to generate three set viewpoint videos. In this embodiment, the imaging conditions of the set viewpoint camera 92B are set distance = 15 m, set angle = 0°, imaging part = whole body, set viewpoint camera = displayed, other avatar = hidden, threshold distance = 5 m. The imaging conditions of the set viewpoint camera 92C are set distance = 10 m, set angle = 0°, imaging part = upper body, set viewpoint camera = hidden, other avatar = hidden, threshold distance = 10 m. The imaging conditions of the set viewpoint camera 92D are set distance = 10 m, set angle = 90°, imaging part = whole body, set viewpoint camera = hidden, other avatar = hidden, threshold distance = 3 m.

Figure 10 is a flowchart showing the processing of the user terminal 20A. Figure 11 is a conceptual diagram of the virtual space 90 corresponding to steps S11 and S12. Figure 12 is a flowchart of the self-viewpoint image generation and display processing. Figure 13 is a diagram showing an example of the self-viewpoint image when the display/non-display of the set viewpoint camera 92B is switched. Figure 14 is a flowchart of the set viewpoint image generation and storage processing. Figure 15 is a diagram showing an example of the set viewpoint image when the display/non-display of another person's avatar is switched in the set viewpoint camera 92C. Figure 16 is a diagram showing an example of the set viewpoint image when the threshold distance is changed in the set viewpoint camera 92D.

As shown in FIG. 10, the user terminal 20A (virtual space definition means 210) defines the virtual space 90 by expanding the virtual space data stored in the storage 23 in the memory 12 (S11). Next, the user terminal 20A (placement means 220 and camera setting means 230) places the user's avatar 50 and the virtual camera 92 in the virtual space 90 defined in step S11 as shown in FIG. 11, and transmits the avatar data to the server 10 via the communication network 2 (S12). The initial position of the user's avatar 50 is, for example, the center C of the virtual space 90. The initial position of the user's viewpoint camera 92A is the head position of the user's avatar 50. The initial positions of the set viewpoint cameras 92B to 92D are positions indicated by the imaging conditions set on the imaging condition setting screen.

Next, when the user terminal 20A (placement means 220) receives avatar data from the server 10 via the communication network 2 (S13: Yes), it places other people's avatars 51, 52, and 53 in the virtual space 90 according to the received avatar data (S14), as shown in FIG. 11 for example. Note that the other people's avatars 51, 52, and 53 correspond to avatar data transmitted from a user terminal 20 other than the user terminal 20A.

Next, the user terminal 20A (image generating means 250 and image display means 260) executes the self-viewpoint image generating and displaying process shown in FIG. 12 in order to reflect the other person's avatars 51, 52, and 53 in the virtual space image 97 that constitutes part of the self-viewpoint image (S17). As a result, the virtual space image 97 that constitutes part of the self-viewpoint image is generated, and the user of the user terminal 20A views it through the monitor 31.

Furthermore, the user terminal 20A (image generating means 250) executes the setting viewpoint image generating and saving process shown in FIG. 14 for each of the setting viewpoint cameras 92B to 92D in order to reflect the other person avatars 51, 52, and 53 in the virtual space image 97 that constitutes part of the setting viewpoint image (S18). As a result, the virtual space image 97 that constitutes part of the setting viewpoint image is generated and stored in the storage 23 as part of the setting viewpoint image data.

When the user terminal 20A (positioning means 220 and camera moving means 240) receives an operation instructing the operation of the principal avatar 50 through the operation device 42 (S15: Yes), the user terminal 20A (placement means 220 and camera moving means 240) moves the principal avatar 50 in accordance with the user's operation on the operation device 42, moves the virtual camera 92 following the principal avatar 50, and transmits avatar data showing the principal avatar 50 after the operation to the server 10 through the communication network 2 (S16). The user terminal 20A (camera moving means 240) also moves the virtual camera 92 following the movement of the user terminal 20A detected by the motion sensor 41.

Next, the user terminal 20A (image generating means 250 and image display means 260) executes a self-viewpoint image generation and display process (S17) to generate a virtual space image 97 corresponding to the angle of view after the movement of the self-viewpoint camera 92A. The self-viewpoint image displayed on the monitor 31 is updated. Furthermore, the user terminal 20A (image generating means 250) executes a set viewpoint image generation and storage process (S18) to generate a virtual space image 97 corresponding to the angle of view after the movement of each of the set viewpoint cameras 92B to 92D. As a result, a new virtual space image 97 is added to the set viewpoint image.

The user terminal 20A repeatedly executes the processes of steps S13 to S18 until an operation (e.g., logging out) instructing the user to terminate the terminal program 23P is received through the operation device 42 (S19: No). The user terminal 20A irregularly receives avatar data and irregularly receives user operations. That is, the processes of steps S14 and S16 are executed at any timing and frequency. Then, when the user terminal 20A receives an operation instructing the user to terminate the terminal program 23P through the operation device 42 (S19: Yes), the user terminal 20A terminates the process of FIG. 10.

As shown in FIG. 12, the user terminal 20A (image generating means 250) identifies the field of view 94 of the self-viewpoint camera 92A (S21). Next, the user terminal 20A (image generating means 250) determines whether to display or hide the set viewpoint cameras 92B and 92C included in the field of view 94 based on the imaging conditions (S22). The user terminal 20A (image generating means 250) then includes the set viewpoint camera 92B, for which "set viewpoint camera = display" is set, in the virtual space image 97 (S22: Yes → S23), and excludes the set viewpoint camera 92C, for which "set viewpoint camera = hide" is set, from the virtual space image 97 (S22: No → S24).

Next, the user terminal 20A (image generating means 250) generates a virtual space image 97 including the other person avatars 52, 53 and the set viewpoint camera 92B out of the other person avatars 52, 53 and the set viewpoint cameras 92B, 92C included in the field of view 94 in FIG. 11 as part of the self-viewpoint image (S25). Next, the user terminal 20A (image display means 260) displays the virtual space image 97 generated in step S25 on the monitor 31 (S26).

As a result, as shown in FIG. 13(A), the user of the user terminal 20A can have his/her avatar 50 perform while being conscious of the position of the set viewpoint camera 92B. On the other hand, if the set viewpoint camera 92B is in the way, by setting the imaging condition to "set viewpoint camera = hidden", a virtual space image 97 that does not include the set viewpoint camera 92B is generated, as shown in FIG. 13(B).

Furthermore, as shown in FIG. 14, the user terminal 20A (image generating means 250) identifies the field of view 94 of each of the set viewpoint cameras 92B to 92D (S31). Next, the user terminal 20A (image generating means 250) determines whether to display or hide the other person's avatars 51 to 53 included in the field of view 94 based on the imaging conditions (S32). Next, when the user terminal 20A (image generating means 250) determines that the imaging conditions of the set viewpoint cameras 92B to 92D are set to "other person's avatar = not displayed" (S32: No), it determines whether the other person's avatars 51 to 53 included in the field of view 94 are located between the user's avatar 50 and each of the set viewpoint cameras 92B to 92D (S33). Next, if the user terminal 20A (image generating means 250) determines that the other person's avatars 51-53 in the field of view 94 are located between the user's avatar 50 and each of the set viewpoint cameras 92B-92D (S33: Yes), it compares the distance from the set viewpoint cameras 92B-92D to the other person's avatars 51-53 with a threshold distance (S34).

If the imaging conditions of the set viewpoint camera 92B are set to "other avatars = display" (S32: Yes), the user terminal 20A (image generating means 250) skips the determination process of steps S33 to S34, and includes all other avatars 51, 52 included in the field of view 94 of the set viewpoint camera 92B in the virtual space image 97 (S35). As a result, as shown in FIG. 15(A), other avatars 52 who are near the set viewpoint camera 92B as audience members are reflected large in the virtual space image 97, obscuring most of the user's avatar 50.

Therefore, as described above, it is desirable to set the imaging conditions of the set viewpoint camera 92B to "Other avatars = hidden, threshold distance = 5m". As a result, the user terminal 20A (image generating means 250) includes in the virtual space image 97, among the other avatars 51 and 52 included in the field of view 94 of the set viewpoint camera 92B, the other avatars 51 that do not overlap with the user's avatar 50 (S33: No → S35), and excludes from the virtual space image 97 the other avatars 52 that overlap with the user's avatar 50 and are located less than the threshold distance from the set viewpoint camera 92B (S33: Yes & S34: No → S36). As a result, as shown in FIG. 15(B), the user's avatar 50 can be appropriately imaged without being disturbed by the other avatars 52 that are near the set viewpoint camera 92B as an audience member.

In addition, by setting the imaging conditions of the set viewpoint camera 92D to "Other avatars = hidden, threshold distance = 3 m", the user terminal 20A (image generating means 250) includes other avatars 51 that overlap the user's avatar 50 and whose distance from the set viewpoint camera 92B is equal to or greater than the threshold distance in the virtual space image 97 (S33: Yes & S34: Yes → S35). This allows other avatars 51 performing together near the user's avatar 50 to be included in the set viewpoint image together with the user's avatar 50, as shown in FIG. 16 (A).

On the other hand, by setting the imaging conditions of the set viewpoint camera 92D to "set distance = 10 m, other avatars = hidden, threshold distance = 10 m," the user terminal 20A (image generating means 250) removes all other avatars 51 that overlap the user's avatar 50 from the virtual space image 97 (S33: Yes & S34: No → S36). As a result, as shown in FIG. 16 (B), all other avatars 51 that overlap the user's avatar 50 can be hidden regardless of the distance from the set viewpoint camera 92D.

Next, the user terminal 20A (image generating means 250) generates, for example, a virtual space image 97 as shown in Figures 15-16 as part of the set viewpoint video (S37). Next, the user terminal 20A (image generating means 250) saves the virtual space image 97 generated in step S37 in the storage 23 (i.e., adds it to the set viewpoint video data) (S38). Then, by repeatedly executing the set viewpoint video generation & saving process, the set viewpoint video is accumulated in the storage 23.

Next, the operation of the system 1 when editing the set viewpoint video will be described with reference to Figs. 17 and 18. Fig. 17 is a flowchart of the process of editing the set viewpoint video. Fig. 18 shows an example of the editing screen and the video to be viewed.

First, the user terminal 20A (image generating means 250), which is a play terminal, transmits the generated multiple set viewpoint images to the server 10 via the communication network 2 (S41). Next, the server 10 (editing assistance means 120) stores the multiple set viewpoint images received from the user terminal 20A via the communication network 2 in the storage 13 (S42).

Next, when starting to edit the set viewpoint video, the user terminal 20B (editing means 270), which is an editing terminal, transmits an editing request to the server 10 via the communication network 2 (S43). The server 10 (editing assistance means 120) transmits the multiple set viewpoint videos stored in the storage 13 to the user terminal 20B via the communication network 2 as a reply to the editing request received from the user terminal 20B via the communication network 2 (S44).

Next, the user terminal 20B (editing means 270) edits the multiple set viewpoint videos received from the server 10 via the communication network 2 in accordance with the user's operation via the operation device 42 to generate a viewing video (S45). The user terminal 20B (editing means 270) displays, for example, the editing screen shown in FIG. 18(A) on the monitor 31, and accepts the user's operation to edit the set viewpoint video via the operation device 42.

The editing screen shown in FIG. 18(A) displays multiple still images constituting each of multiple set viewpoint videos arranged in chronological order. Next, the user terminal 20B (editing means 270) accepts a user operation via the operation device 42 to select one of the multiple still images for the same time for each of times t1 to t4. Then, as shown in FIG. 18(B), the user terminal 20B (editing means 270) extracts the still image selected via the operation device 42 (the still image surrounded by a thick frame in FIG. 18(A)), and arranges the extracted multiple still images in chronological order to generate a viewing video.

Next, user terminal 20B (editing means 270) transmits the viewing video generated in step S45 to server 10 via communication network 2 (S46). Next, server 10 (editing assistance means 120) stores the viewing video received from user terminal 20A via communication network 2 in storage 13 (S47). Furthermore, in response to a request from user terminal 20C, which is a viewing terminal, server 10 (viewing assistance means 130) transmits the viewing video stored in storage 13 to the viewing terminal via communication network 2. Then, user terminal 20C displays the viewing video received from server 10 via communication network 2 on monitor 31.

As an example, the process shown in FIG. 17 may be started after the process shown in FIG. 10 is completed. That is, the process shown in FIG. 17 may be executed asynchronously with the process shown in FIG. 10 after the set viewpoint video is completed. As another example, the process shown in FIG. 10 and the process shown in FIG. 17 may be executed in parallel. That is, the user terminal 20A (video generating means 250) may execute the process of step S41 every time it generates a virtual space image 97 that constitutes part of the set viewpoint video. Also, the user terminal 20B (editing means 270) may execute the processes of steps S43 to S46 every time the server 10 receives the virtual space image 97 from the user terminal 20A. In this way, the viewing terminal is generated simultaneously with (strictly speaking, with a slight delay from) the performance of the user's avatar 50.

The user terminal 20B (editing means 270) may also obtain multiple sets of set viewpoint images from the server 10 and generate viewing images by arranging virtual space images 97 extracted from these in chronological order. For example, the user terminal 20B (editing means 270) may generate viewing images using multiple set viewpoint images in which avatar 50 is the subject avatar and multiple set viewpoint images in which avatar 51 is the subject avatar.

[Effects of this embodiment]
According to the above embodiment, a plurality of set viewpoint images that serve as material for the viewing video are generated by having the person's avatar 50 perform with a plurality of set viewpoint cameras 92B-92D set in advance. In other words, by setting the set viewpoint cameras 92B-92D and performing the person's avatar 50 at different times, the continuity of the movements of the avatar 50 is not lost. As a result, attractive material for the viewing video can be obtained with simple operations.

Furthermore, according to the above embodiment, a desired set viewpoint image can be obtained by setting the imaging conditions of the set viewpoint cameras 92B to 92D in advance. Note that specific examples of imaging conditions are not limited to the example in FIG. 9. As another example, the angle between the horizon and the straight line connecting the user's avatar 50 and the set viewpoint camera 92B (i.e., the elevation angle or depression angle) may be added to the imaging conditions.

In addition, according to the above embodiment, by switching whether or not to include the set viewpoint cameras 92B-92D in the self-viewpoint image, an appropriate self-viewpoint image can be displayed on the monitor 31 according to the preferences of the user operating the personal avatar 50 (i.e., whether or not the user is aware of the set viewpoint cameras 92B-92D).

Furthermore, according to the above embodiment, by switching between displaying and hiding the other person's avatars 51-53 located between the user's avatar 50 and the set viewpoint cameras 92B-92D, it is possible to obtain more attractive video material for viewing. For example, the other person's avatar 51 that enhances the appeal of the user's avatar 50's performance may be displayed, and the other person's avatar 52 that interferes with the user's avatar 50's performance may be hidden.

For example, the other person's avatar 51 near the user's avatar 50 is likely to improve the attractiveness of the performance, while the other person's avatars 52, 53 near the set viewpoint cameras 92B-92D are likely to hinder the performance. Therefore, according to the above embodiment, by switching between displaying and hiding the other person's avatars 51-53 depending on the distance from the set viewpoint cameras 92B-92D, it is possible to obtain attractive viewing video material with simple settings. However, the parameters for switching between displaying and hiding the other person's avatars 51-53 are not limited to the above example. As another example, the user terminal 20A (video generating means 250) may include the other person's avatar located in a display area preset in the virtual space 90 in the set viewpoint video, and may exclude the other person's avatar located in a non-display area preset in the virtual space 90 from the set viewpoint video.

Furthermore, according to the above embodiment, by generating viewing images using multiple sets of set viewpoint images, it is possible to generate even more attractive viewing images, for example, when multiple avatars 50, 51 are performing as a group.

In this embodiment, the user's avatar 50 is an example of a "target object," and the other user's avatars 51 to 53 are examples of "peripheral objects." However, the objects (target objects, peripheral objects) are not limited to human-shaped "characters," as long as they are placed in the virtual space 90 and can move or change shape. As other examples, the objects may be automobiles, ships, aircraft, buildings under construction, animals, etc. Furthermore, what is captured by the set viewpoint cameras 92B to 92D is not limited to the performance of the user's avatar 50 (target object), but may be any movement within the virtual space 90.

The program according to the present invention is not limited to a single program, but may be a collection of multiple programs. The program according to the present invention is not limited to being executed by a single device, but may be executed by multiple devices in a shared manner. Furthermore, the division of roles between the server 10 and the user terminal 20 is not limited to the above example. That is, part of the processing of the server 10 may be executed by the user terminal 20, and part of the processing of the user terminal 20 may be executed by the server 10.

Furthermore, some or all of the means realized by the program can be realized by hardware such as an integrated circuit. Furthermore, the program may be provided in a form recorded on a non-transitory recording medium that can be read by a computer. Examples of recording media include hard disks, SD cards, DVDs, and servers on the Internet.

1...system, 2...communication network, 3...detection device, 10...server, 11, 21...processor, 12, 22...memory, 13, 23...storage, 13P...server program, 14, 24...input/output interface, 15, 25...communication interface, 19, 29...communication bus, 20...user terminal, 23P...terminal program, 26...computer, 30...HMD, 31...monitor, 32...gaze sensor, 33, 34...camera, 35...microphone, 36...speaker, 41...motion sensor, 42...operation device, 50...user avatar , 51, 52, 53...Other avatars, 90...Virtual space, 91...Panoramic image, 92...Virtual camera, 92A...Self-viewpoint camera, 92B, 92B, 92C, 92D...Set viewpoint camera, 93...Reference line of sight, 94...Field of view area, 95, 96...Area, 97...Virtual space image, 110...Avatar synchronization means, 120...Editing assistance means, 130...Viewing assistance means, 210...Virtual space definition means, 220...Placement means, 230...Camera setting means, 240...Camera movement means, 250...Image generation means, 260...Image display means, 270...Editing means

Claims (5)

A computer having an operating device,
a camera setting means for setting a plurality of virtual cameras for capturing images of a target object placed in a virtual space from different set viewpoints;
functioning as an image generating means for generating a plurality of set viewpoint images captured by the plurality of virtual cameras,
the camera setting means sets, for each of the plurality of virtual cameras, imaging conditions including at least one of a distance from the target object, an angle with respect to the target object, and an imaging portion of the target object in accordance with a user's operation via the operation device;
causing the computer to function as a camera moving unit that moves each of the plurality of virtual cameras in accordance with a movement of the target object in accordance with a user's operation through the operation device so as to satisfy the set imaging condition ;
the imaging condition includes, for each of the plurality of virtual cameras, whether or not to include, in the set viewpoint image, a peripheral object that overlaps with the target object as viewed from the virtual camera, among peripheral objects different from the target object;
The image generating means switches whether or not to include the peripheral object in the set viewpoint image of each of the multiple virtual cameras in accordance with the imaging conditions . The program according to claim 1 ,
The image generating means includes:
Among the peripheral objects that overlap the target object as viewed from the virtual camera, only the peripheral objects whose distance from the virtual camera is equal to or greater than a threshold distance are included in the set viewpoint image ;
The program includes, in the set viewpoint image, the peripheral objects that do not overlap the target object as viewed from the virtual camera, regardless of their distance from the virtual camera . The program according to claim 1,
The target object is a user character operated through a user terminal,
The image generating means further generates a self-viewpoint image capturing an image of the virtual space from a self-viewpoint of the user character,
the imaging conditions include whether or not to include each of the plurality of virtual cameras in the self-viewpoint image;
A program that causes the computer to function as an image display means that displays the self-viewpoint image, in which the inclusion or non-inclusion of each of the multiple virtual cameras is switched according to the imaging conditions, on the user terminal that controls the user character. The program according to claim 1,
A program that causes the computer to function as an editing unit that edits a plurality of the set viewpoint videos to generate a viewing video. The program according to claim 4 ,
The editing means generates the viewing image using a plurality of sets of the set viewpoint images each capturing a plurality of the target objects.

Images