CN116114252A - Information processing system, information processing method, information processing program - Google Patents

Abstract

The present invention provides a technique for distributing suitable content in a virtual space, and discloses an information processing system (1) which has: a first detection unit (211A) that detects a state of a first user who distributes content of an image that includes a first virtual space (H20); a first rendering processing unit (200A) that renders, by a first terminal device (20A) associated with a first user, an image of a first virtual space that can be viewed by the first user; a first storage unit (240A) for storing a first display medium corresponding to a first user, the image of the first virtual space having a plurality of layers, the first rendering unit comprising: a first drawing unit (201A) for drawing a first image area of a layer related to a first display medium among a plurality of layers; and a second drawing unit (202A) for drawing a second image area of a layer related to the first user interface in the plurality of layers, wherein the first drawing unit changes the state of the first display medium in the first image area based on the state of the first user detected by the first detection unit.

Description

Information processing system, information processing method, information processing program

technical field

The present invention relates to an information processing system, an information processing method, and an information processing program.

Background technique

Conventionally, "Mirrativ" is known as a service for distributing music and video to a plurality of users (for example, refer to Non-Patent Document 1).

prior art literature

non-patent literature

Non-Patent Document 1: "Mirrativ", [online], Milaive Co., Ltd., [retrieved on December 18, 2019], Internet (URL: https://www.mirrativ.com/).

Contents of the invention

The problem to be solved by the invention

Conventional technologies such as those described above are services that can be viewed in a real space, and it is difficult to develop in a virtual space.

Therefore, an object of the present invention is to provide a content distribution technology suitable in a virtual space.

solutions to problems

According to one aspect of the present invention, an information processing system is disclosed, which includes: a first detection unit that detects a state of a first user who distributes content including an image of a first virtual space; a first rendering process A part, which uses the first terminal device related to the first user or the second terminal device related to the second user who watches the content, to describe the first user or the second user to view the content An image of a virtual space; a first storage unit that stores a first display medium corresponding to the first user, the image of the first virtual space has multiple levels, and the first rendering processing unit has: a first a rendering unit for rendering a first image area of a layer related to the first display medium among the plurality of layers; a second rendering unit for rendering a first image area of a layer related to the first user interface among the plurality of layers In the second image area, the first drawing unit changes the state of the first display medium in the first image area based on the state of the first user detected by the first detection unit.

Invention effect

According to the present invention, it is possible to obtain a content distribution technique suitable in a virtual space.

Description of drawings

FIG. 1 is a block diagram of a virtual reality creation system according to this embodiment.

FIG. 2 is an explanatory diagram of images viewed by left and right eyes, respectively.

FIG. 3 is an explanatory diagram of the structure of an image in a virtual space according to the present embodiment.

FIG. 4 is an explanatory diagram of a virtual space corresponding to a home screen (lobby screen).

FIG. 5 is an explanatory diagram of a hierarchical structure of homepage images.

FIG. 5A is an enlarged view of the field of view (when viewing the front) from the viewpoint of the user in FIG. 5 .

FIG. 6 is a diagram showing an example of a homepage image viewed from a wearable device.

FIG. 7A is an explanatory diagram of the manner in which the operation area is switched back and forth.

FIG. 7B is an explanatory diagram of the manner in which the operation area is switched back and forth.

Fig. 8A is a schematic side view of a virtual space corresponding to a distributed image.

FIG. 8B is a schematic top view of the virtual space corresponding to the distribution image when viewed from above.

FIG. 9 is an explanatory diagram of a hierarchical structure of distribution images.

FIG. 10 is an explanatory diagram showing the relationship between the mirror and each layer for distributing images in plan view.

FIG. 11 is an explanatory diagram of an interface display layer related to a distributed image.

FIG. 12 is a diagram showing an example of a distribution image viewed by a distribution user through a wearable device.

FIG. 13 is a diagram showing an example of a distributed image viewed by a viewing user through a wearable device.

FIG. 14 is a diagram showing an example of a distribution image for a smartphone.

Fig. 15 is a schematic block diagram showing functions of a terminal device on the content distribution side.

FIG. 16 is an explanatory diagram of an example of drawing information of an avatar.

Fig. 17 is a schematic block diagram showing functions of a terminal device on the content viewing side.

Fig. 18 is a schematic block diagram showing the functions of the server device.

FIG. 19 is a flowchart showing an example of operations up to the start of distribution of specific content.

FIG. 20 is a flowchart showing an example of operations until viewing of specific content is started.

FIG. 21 is a flowchart showing an example of operations when the specific content is being distributed from the distribution user (that is, when the viewing user is viewing the specific content).

Detailed ways

Embodiments will be described in detail below with reference to the drawings.

The outline of a virtual reality generation system 1 according to an embodiment of the present invention will be described with reference to FIG. 1 . FIG. 1 is a block diagram of a virtual reality creation system 1 according to this embodiment.

The virtual reality generation system 1 has a server device 10 (an example of an external processing device) and one or more terminal devices 20 . In FIG. 1 , three terminal devices 20 are illustrated for simplicity, but the number of terminal devices 20 may be two or more.

The server device 10 is, for example, an information processing system such as a server managed by an operator who provides one or more virtual reality. The terminal device 20 is a device used by a user, such as a mobile phone, a smart phone, a tablet computer terminal, a PC (Personal Computer, personal computer), a wearable device (head-mounted display, glasses-type device, etc.) or a game device. Typically, a plurality of terminal devices 20 can be connected to the server device 10 via the network 3 in different ways for each user.

The terminal device 20 can execute the virtual reality application related to this embodiment. The virtual reality application may be received by the terminal device 20 from the server device 10 or a prescribed application distribution server through the network 3, or may be pre-stored in a storage device possessed by the terminal device 20 or a memory card readable by the terminal device 20, etc. storage medium. The server device 10 and the terminal device 20 are communicably connected via the network 3 . For example, the server device 10 and the terminal device 20 cooperate to execute various processes related to virtual reality.

The terminal devices 20 include a terminal device 20A on the content distribution side (an example of a first terminal device) and a terminal device 20B on a content viewing side (an example of a second terminal device). In addition, in the following description, the terminal device 20A on the content distribution side and the terminal device 20B on the content viewing side are described as separate terminal devices, but there may also be a case where the terminal device 20A on the content distribution side becomes the terminal device on the content viewing side The case of 20B, or the opposite case. In addition, hereinafter, when the terminal device 20A and the terminal device 20B are not particularly distinguished, they may be simply referred to as "terminal device 20".

The terminal devices 20 are communicably connected to each other via the server device 10 . In addition, hereinafter, "one terminal device 20 transmits information to other terminal devices 20" means "one terminal device 20 transmits information to other terminal devices 20 through the server device 10". Likewise, "one terminal device 20 receives information from other terminal devices 20" means "one terminal device 20 receives information from other terminal devices 20 through the server device 10". However, in a modified example, each terminal device 20 may be connected so as to be communicable without going through the server device 10 .

In addition, the network 3 may include a wireless communication network, the Internet, a VPN (Virtual Private Network, virtual private network), a WAN (Wide Area Network, wide area network), a wired network, or any combination thereof.

In the example shown in FIG. 1 , the virtual reality generation system 1 includes studio units 30A, 30B. Like the terminal device 20A on the content distribution side, the studio units 30A and 30B are devices on the content distribution side. The studio units 30A and 30B can be arranged in studios, rooms, venues, etc. for content creation.

Each studio unit 30 can have the same function as the terminal device 20A and/or the server device 10 on the content distribution side. Hereinafter, for simplicity of description, the case where the terminal device 20A on the content distribution side distributes various contents to the terminal devices 20B on the content viewing side through the server device 10 will be mainly described. However, instead of or in addition to it, the studio units 30A, 30B facing the distribution user may have the same function as the terminal device 20A on the content distribution side, so that the server device 10 may send the content to the terminal device on the content viewing side. 20B distributes various contents. In addition, in a modified example, the virtual reality generation system 1 may not have the studio units 30A, 30B.

Hereinafter, although the virtual reality generation system 1 realizes an example of an information processing system, each element of a specific terminal device 20 (refer to the terminal communication unit 21 to the terminal control unit 25 in FIG. 1 ) can realize an example of an information processing system, An example of an information processing system can also be realized by a plurality of terminal devices 20 working together. In addition, the server device 10 may implement an example of an information processing system alone, or the server device 10 and more than one terminal device 20 may cooperate to implement an example of an information processing system.

Here, an outline of virtual reality according to this embodiment will be described. The virtual reality related to this embodiment is a virtual reality corresponding to any reality, such as entertainment activities such as education, travel, role-playing, simulation, games, and concerts. (Avatar) such a virtual reality medium. For example, virtual reality according to this embodiment is realized by a three-dimensional virtual space, various virtual reality media appearing in the virtual space, and various contents provided in the virtual space.

Virtual reality media is electronic data used in virtual reality, including arbitrary media such as cards, props, points, currency in services (or currency in virtual reality), tickets, characters, avatars, and parameters. In addition, the virtual reality medium may also be virtual reality-related information such as level information, status information, parameter information (physical strength, attack power, etc.) or ability information (skills, abilities, spells, occupations, etc.). In addition, virtual reality media is electronic data that users can acquire, own, use, manage, exchange, synthesize, enhance, sell, discard, or give away in virtual reality, but the use of virtual reality media is not limited to way of recording.

In this embodiment, the users include a viewing user (an example of a second user) who watches various contents in the virtual space, and a user in the virtual space through the distributor avatar M2 (an example of the first display medium) described later. A distribution user (an example of a first user) who distributes specific content to be described later.

In addition, a distribution user can view specific content from other distribution users as a viewing user, and conversely, a viewing user can also distribute specific content as a distribution user. However, below, in order to prevent the explanation from becoming complicated, the viewing user is referred to as the current viewing user, and the distribution user is referred to as the current distribution user. In addition, in the following, when a distribution user and a viewing user are not particularly distinguished, they may be simply referred to as "users". In addition, when the viewer avatar M1 (an example of the second display medium) and the distributor avatar M2 related to the viewing user are not particularly distinguished, they are sometimes simply referred to as "avatar".

In addition, the avatar is typically in the form of a character with a frontal orientation, and may have the form of a human, animal, or the like. The avatar can have various appearances (appearances when drawn) by corresponding to various avatar props.

Basically, the viewing user and the distribution user wear a wearable device on, for example, a part of the head or face, and view the virtual space through the wearable device. In addition, the wearable device may also be a head-mounted display or a glasses-type device. The glasses-type device may be so-called AR (Augmented Reality, augmented reality) glasses or MR (Mixed Reality, mixed reality) glasses. In any case, the wearable device can be set separately from the terminal device 20 , and can also realize a part or all of the functions of the terminal device 20 . In this embodiment, as an example, the terminal device 20 is a terminal device realized by a head-mounted display.

Hereinafter, among various contents distributed by the server device 10 , specific contents based on the distribution user will be mainly described. In addition, in the following description, content suitable for viewing on a head-mounted display will be described.

The specific content based on the distribution user refers to the content in which the distribution user's avatar M2, whose orientation, position, and movement changes according to the distribution user's orientation, position, and movement, appears in the virtual space. In addition, the orientation, position, and action of the distribution user include not only the orientation, position, and action of a part or all of the user's face, hands, etc., but also the orientation, position, and action of the user's line of sight, or similar concepts. . The specific content based on distributing users is typically video content.

The specific content based on the distributing user is typically content that provides entertainment in any manner through the distributor avatar M2. For example, specific content based on distributing users may be related to various performances such as dancing, music, magic, etc., chatting, parties, conventions, conferences, or similar activities.

Additionally, specific content based on distribution users may be educational. For example, the distribution user-based specific content may include guidance, advice, etc. from the distribution user via the distributor avatar M2. For example, as content provided by virtual reality related to dance lessons, guidance, advice, etc. from a dance teacher may be included. In this case, the dance teacher becomes the distribution user, the students become the viewing users, and the students can receive individual guidance from the teacher in virtual reality.

In addition, the specific content based on the distribution user may include a cooperative cooperation (hereinafter referred to as "cooperation") of two or more distribution users. In this way, distribution using various methods can be realized, and communication among distribution users can be promoted.

In addition, the server device 10 is also capable of distributing content other than specific content based on the distribution user. The type and quantity of content provided by the server device 10 (content provided by virtual reality) are arbitrary, but in this embodiment, as an example, the content provided by the server device 10 may include digital content such as various videos. The image can be a real-time image or a non-real-time image. In addition, the video may be a video based on an actual image or a video based on CG (Computer Graphics, computer graphics). The image may be an image for informational purposes. In this case, the video may be an information provision service about a specific category (information provision service related to travel, living, food, fashion, health, beauty, etc.), a broadcast service based on a specific user (such as Youtube (registered trademark )) etc. images.

There are various methods of providing content in virtual reality, and methods other than those provided using the display function of the head-mounted display may be used. For example, when the content is video, the content can be provided by drawing the video on a display screen of a display device (virtual reality medium) in a virtual space. In addition, the display device in the virtual space can be in any form, and can be a screen installed in the virtual space, a large display screen installed in the virtual space, a display screen of a mobile terminal in the virtual space, and the like.

In addition, the content in virtual reality may be content that can be viewed by means other than the head-mounted display. For example, content in virtual reality can be viewed directly (without going through a head-mounted display) via a smartphone, tablet, etc.

(Structure of server device)

The configuration of the server device 10 will be specifically described. The server device 10 is constituted by a server computer. The server device 10 can be realized through the cooperative work of multiple server computers. For example, the server device 10 can be realized by a server computer providing various contents, a server computer implementing various authentication servers, and the like working in cooperation. Furthermore, the server device 10 may include a Web server. In this case, part of the functions of the terminal device 20 described later can be realized by the browser processing the HTML file received from the Web server and various programs (Javascript) attached thereto.

As shown in FIG. 1 , the server device 10 has a server communication unit 11 , a server storage unit 12 , and a server control unit 13 .

The server communication unit 11 communicates with external devices wirelessly or by wire, and includes an interface for transmitting and receiving information. The server communication unit 11 may include, for example, a wireless LAN (Local Area Network, local area network) communication module or a wired LAN communication module. The server communication unit 11 can exchange information with the terminal device 20 via the network 3 .

The server storage unit 12 is, for example, a storage device, and stores various information and programs necessary for various processes related to virtual reality.

The server control unit 13 may include a dedicated microprocessor or a CPU (Central Processing Unit, central processing unit) or a GPU (Graphics Processing Unit, graphics processing unit) etc. which realize specific functions by reading a specific program. For example, the server control unit 13 cooperates with the terminal device 20 to execute the virtual reality application according to the user's operation on the display unit 23 of the terminal device 20 .

(Structure of terminal equipment)

The configuration of the terminal device 20 will be described. As shown in FIG. 1 , the terminal device 20 has a terminal communication unit 21 , a terminal storage unit 22 , a display unit 23 , an input unit 24 , and a terminal control unit 25 .

The terminal communication unit 21 communicates with external devices wirelessly or by wire, and includes an interface for transmitting and receiving information. The terminal communication unit 21 may include, for example, an LTE (Long Term Evolution, long-term evolution technology) (registered trademark), LTE-A (LTE-Advanced, long-term evolution technology upgrade version), the fifth generation mobile communication system, UMB (Ultra Mobile Broadband, Wireless communication modules, wireless LAN communication modules, or wired LAN communication modules corresponding to mobile communication standards such as Ultra Mobile Broadband). The terminal communication unit 21 can send and receive information with the server device 10 via the network 3 .

The terminal storage section 22 includes, for example, a primary storage device and a secondary storage device. For example, the terminal storage unit 22 may include a semiconductor memory, a magnetic memory, an optical memory, or the like. The terminal storage unit 22 stores various information and programs received from the server device 10 and used in virtual reality processing. Information and programs used in virtual reality processing can be obtained from external devices through the terminal communication unit 21 . For example, a virtual reality application can be obtained from a prescribed application distribution server. Hereinafter, the application program is also only referred to as an application.

In addition, the terminal storage unit 22 stores data for drawing a virtual space, for example, images of indoor spaces such as buildings, outdoor spaces, and the like. In addition, multiple types of data for describing the virtual space may be prepared for each virtual space and used flexibly.

In addition, the terminal storage unit 22 stores various images (texture images) for projecting (texture mapping) various items arranged in a three-dimensional virtual space.

For example, the terminal storage unit 22 stores drawing information of the viewer avatar M1 as a virtual reality medium corresponding to each user. In the virtual space, the viewer avatar M1 is drawn based on the drawing information of the viewer avatar M1.

Furthermore, the terminal storage unit 22 stores drawing information of the distributor avatar M2 as a virtual reality medium corresponding to each distribution user. In the virtual space, the distributor avatar M2 is drawn based on the drawing information of the distributor avatar M2.

In addition, the terminal storage unit 22 stores, for example, various gift items, buildings, walls, or NPCs (Non Player Characters, non-player characters), which are related to various items that are different from the viewer avatar M1 and the distributor avatar M2. description information. In the virtual space, various items are drawn based on such drawing information. In addition, a gift item is an item corresponding to a present (premium) given by one user to another user, and is a part of an item. Gift items can be items worn by the avatar (clothing, decorations), items that decorate the distribution image (fireworks, flowers, etc.), backgrounds (wallpapers) or similar items, tickets that can reverse the gashapon machine (drawing lots) or similar items.

The display unit 23 includes, for example, a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 23 can display various images. The display unit 23 is constituted by, for example, a touch panel, and functions as an interface for detecting various user operations. In addition, the display unit 23 may be built in a head-mounted display as described above.

The input unit 24 may include physical keys, and may further include any input interface typified by a pointing device such as a mouse. In addition, the input unit 24 can also accept non-contact user input such as voice input, gesture input, and line-of-sight input. In addition, sensors for detecting various states of the user (image sensor, acceleration sensor, distance sensor, etc.), dedicated motion capture that integrates sensor technology and cameras, controllers such as handles, etc.) for detecting various states of the user can be used in gesture input. In addition, a camera for line-of-sight detection may be placed in the head-mounted display. In addition, as mentioned above, the various states of the user are, for example, the user's orientation, position, action, or similar states. In this case, the user's orientation, position, and action refer to not only the user's hand, body orientation, The position and action also include the orientation, position, action or similar concepts of the distributed user's line of sight.

The terminal control unit 25 includes one or more processors. The terminal control unit 25 controls the overall operation of the terminal device 20 .

The terminal control unit 25 transmits and receives information through the terminal communication unit 21 . For example, the terminal control section 25 receives various information and programs used in various processes related to virtual reality from at least one of the server device 10 and other external servers. The terminal control unit 25 stores the received information and programs in the terminal storage unit 22 . For example, a browser (Internet browser) for connecting to a Web server may be stored in the terminal storage unit 22 .

The terminal control unit 25 activates the virtual reality application according to the user's operation. The terminal control unit 25 cooperates with the server device 10 to execute various processes related to virtual reality. For example, the terminal control unit 25 displays an image of the virtual space on the display unit 23 . For example, a GUI (Graphic User Interface) for detecting user operations may be displayed on the screen. The terminal control unit 25 can detect user operations through the input unit 24 . For example, the terminal control unit 25 can detect various operations (operations corresponding to tap operations, long-press operations, flick operations, and swipe operations) by gestures of the user. The terminal control unit 25 transmits the operation information to the server device 10 .

The terminal control unit 25 draws the distributor avatar M2 and the like together with the virtual space (image), and displays it on the display unit 23 . In this case, for example, as shown in FIG. 2 , a stereoscopic image can be generated by generating images G200 and G201 seen by the left and right eyes, respectively. FIG. 2 schematically shows images G200 , G201 seen by the left and right eyes, respectively. In addition, unless otherwise specified below, the image of the virtual space means the whole image expressed by the images G200 and G201. In addition, the terminal control unit 25 realizes various actions of the distributor avatar M2 in the virtual space according to various operations of the distributor user, for example. The specific drawing processing by the terminal control unit 25 will be described later.

FIG. 3 is an explanatory diagram of the structure of an image in a virtual space according to the present embodiment.

In the present embodiment, the image of the virtual space has a plurality of layers 300 as conceptually shown in FIG. 3 . In this case, the terminal control unit 25 draws the image area of each layer and superimposes (overlays) the image area of each layer to generate an image of the virtual space. In addition, in the example of FIG. 3, although the hierarchy 300 is formed of four layers 301 to 304, the number of layers is arbitrary.

Multiple layers may include a distribution avatar display layer J0 for displaying the distributor avatar M2, a user avatar display layer J1 for displaying the viewer avatar M1, an interface display layer J2 for displaying a user interface, a background layer J3, and displaying various auxiliary images. The supplementary information of the information display layer J5, or any combination of two or more of similar layers.

Hereinafter, the front-back relationship (or rear) of multiple layers is based on the viewpoint of the user viewing the image in the virtual space, and the distance between the layers is based on the distance along the line of sight of the user viewing the image in the virtual space. benchmark.

When the background layer J3 is included in a plurality of layers, the background layer J3 is the rearmost layer, but the front-rear relationship of other layers can be appropriately set. For example, when a plurality of layers related to images in a certain virtual space include the distribution avatar display layer J0, the interface display layer J2, the background layer J3, and the auxiliary information display layer J5, the distribution avatar display layer is included sequentially from the front side. J0, an interface display layer J2, an auxiliary information display layer J5, and a background layer J3.

The interface display layer J2 is a layer related to the above-mentioned input unit 24, and can accept non-contact user input. The interface display layer J2 is preferably arranged within a distance within the virtual space where the user's hand can reach (that is, within an operable distance). Accordingly, the user can perform various inputs without moving in the virtual space while wearing the head-mounted display, and can perform various inputs through the interface display layer J2 without causing dizziness.

The auxiliary information display layer J5 is preferably arranged behind the interface display layer J2. In this case, the auxiliary information display layer J5 is arranged at a distance that cannot be reached by the user's hands (unreachable without moving) in the virtual space. Thereby, the visibility of the interface display layer J2 can be improved compared with the case where the auxiliary information display layer J5 is arrange|positioned at the front side of the interface display layer J2.

The auxiliary information displayed on the auxiliary information display layer J5 is arbitrary, for example, may include input information from the viewing user of the specific content (that is, the viewing user who is watching the specific content), guidance/notification to the distribution user and/or other users Information, and at least any one of item information including gift items presented to distribution users.

In addition, the auxiliary information display layer J5 may be composed of a plurality of layers such that different layers are formed according to attributes of auxiliary information. For example, the auxiliary information display layer J5 depicting gift items may be set separately from other auxiliary information display layers J5. In this case, the supplementary information display layer J5 in which the gift item is drawn may be arranged behind the distribution avatar display layer J0 than the other supplementary information display layers J5. Alternatively, the supplementary information display layer J5 depicting gift items may be arranged on the front side of the distribution avatar display layer J0 differently from the other supplementary information display layers J5.

The background layer J3 may have a function of defining the outer boundary (shell) of the virtual space. In addition, the virtual space may be an infinite space substantially not having the background layer J3, or a cylindrical or celestial spherical space.

(Example of virtual reality)

The server control unit 13 cooperates with the terminal device 20 to display the image of the virtual space on the display unit 23 and update the image of the virtual space according to the operation of the virtual reality and the operation of the user.

In addition, although the rendering processing described below is realized by the terminal device 20 , part or all of the rendering processing described below may be realized by the server control unit 13 in other embodiments. For example, in the following description, at least a part of the image of the virtual space displayed on the terminal device 20 may be a page display (web display) displayed on the terminal device 20 based on the data generated by the server device 10, and the At least a part is set as a native display displayed by a native application (native APP) installed in the terminal device 20 .

Here, some examples of images of the virtual space generated by the virtual reality generation system 1 will be described with reference to FIGS. 4 to 14 .

FIG. 4 is an explanatory diagram of a virtual space corresponding to a home screen (lobby screen). FIG. 5 is an explanatory diagram of a hierarchical structure of an image on the home screen (hereinafter referred to as "home image H1"). FIG. 5A is an enlarged view of the field of view (when viewing the front) R500 from the viewpoint of the user in FIG. 4 . FIG. 6 is a diagram showing an example of a homepage image H1 viewed from a head-mounted display. 7A and 7B are explanatory views of the manner in which the operation area is switched back and forth.

The homepage image H1 shows a virtual space (hereinafter referred to as "homepage space") which is an entrance when moving to various virtual spaces. In Fig. 4, users located in the homepage space are schematically shown. As will be described later, an operation area G300 for selecting various contents is arranged in the home space. Users can perform various activities from the homepage space, such as viewing specific content based on desired distribution users, distributing specific content as a distribution user themselves, and distributing specific content in cooperation with other users (hereinafter also referred to as "cooperative distribution") )wait.

In this embodiment, as an example, the homepage image H1 includes an interface display layer J2 (an example of a layer related to the second user interface), an auxiliary information display layer J5 , and a background layer J3 sequentially from the front side. In addition, in the modified example, other layers as described above may be included, and some layers (for example, the auxiliary information display layer J5 and the background layer J3 ) may be omitted.

In FIG. 5, an example of the layout of the interface display layer J2, the auxiliary information display layer J5, and the background layer J3, and the position of the user (the position of the viewer avatar M1 or the distributor avatar M2) are shown in plan view. Specifies the reference axis I. In this specification, unless otherwise specified, a plan view refers to a plan view based on the vertical direction of the virtual space.

In this embodiment, as shown in FIG. 5 , the interface display layer J2, the auxiliary information display layer J5, and the background layer J3 form cylinder surfaces with different radii r1, r2, and r3 around a predetermined reference axis I when viewed from above. configuration, each radius r1, r2, r3 corresponds to the distance between each layer and the user. In addition, the radius r3 of the background layer J3 may be infinite (infinity). In addition, as mentioned above, the radius r1 of the interface display layer J2 can be set within the reachable range of the user's hand. In addition, the radius r1 can be adjusted for each user. Thus, the interface display layer J2 can be formed with a sense of distance according to the length and preference of the user's hand.

The interface display layer J2 is preferably arranged based on the position of the user in the homepage space. In other words, when the user enters the homepage space, the user is positioned in a prescribed positional relationship relative to the interface display layer J2. In this embodiment, the interface display layer J2 is based on the position of the user in the homepage space, and is arranged within the reach of the user at that time. Accordingly, the user can perform various inputs without moving in the virtual space while wearing the head-mounted display, and can perform various inputs through the interface display layer J2 without causing dizziness.

The interface display layer J2 preferably includes a plurality of planar operation areas G300 functioning as the input unit 24 . In this case, the plurality of planar operation regions G300 can function as selection buttons that can be selected by the user. In addition, the size and shape of the planar operation region G300 are arbitrary, and may vary according to the attributes of the operation region G300. Thereby, the user can easily recognize the operation area G300 for each attribute.

In this case, various processes (an example of various second processes) realized by operating the operation area G300 are arbitrary, and may include, for example, changing the user interface (e.g., a plurality of planar operation areas G300). Processing of placement, processing of moving a user to an arbitrary location or a specific location in virtual space (for example, processing of moving a distribution user to a location for distribution), processing of starting distribution of various contents, processing of starting viewing of various contents, Viewing end processing of various contents, voice input processing, processing of presenting presents to distribution users, processing of lottery of avatar items, processing of selection/exchange of avatar items, processing of character input, and transfer to a place where these processes can be performed The processing of any one of the states, or any combination of these processing. Thus, the user can realize various operations through the interface display layer J2.

A plurality of planar operation areas G300 corresponding to content viewing start processing may be drawn in a manner corresponding to content such as alternative content. For example, thumbnail images and real-time images of alternative content (for example, specific content based on distribution users) may be drawn in each operation area G300 (refer to FIG. 4 ). In addition, the distributor avatars M2 of a plurality of distribution users performing the cooperative distribution may be drawn on the thumbnail image of the cooperative distribution. Accordingly, the user can easily grasp what content can be viewed, and can easily select a desired operation area G300 (for example, an operation area G300 related to desired specific content).

In addition, it is preferable that the plurality of planar operation regions G300 are arranged to form a plurality of columns as shown in FIG. 4 . In this way, even when the number of planar operation areas G300 increases due to an increase in the number of distribution users (and the increase in the number of specific content accompanying this distribution), for example, it is possible to efficiently arrange them within the range that users can view. Many operating areas.

In this case, at least some of the plurality of planar operation regions G300 are arranged in a plurality of rows along the first curved surface 501 (see FIGS. 4 and 5A ) around the predetermined reference axis I. The predetermined reference axis I is an axis that passes through the user's position and extends in the vertical direction of the home page space. At this time, when viewing the homepage space from above, the first curved surface 501 may be in the form of a circle that gives the interface display layer J2 a shape, and the center of curvature may be located on the predetermined reference axis I. However, in a modified example, the first curved surface 501 may be an elliptical shape or a similar shape when viewed from above the homepage space. Accordingly, the user can view a plurality of planar operation regions G300 at the same distance from the user, thereby improving operability. In addition, arranging the planar operation region G300 along the first curved surface 501 means that the in-plane direction related to the planar shape of the operation region G300 may be parallel to the tangential direction of the first curved surface 501 in a plan view of the homepage space. . Or, in the case where the radius of curvature of the first curved surface 501 is sufficiently small, the operation region G300 may be projected onto the first curved surface 501 .

In addition, it is preferable that a plurality of planar operation areas G300 are arranged collectively for each category. For example, in a frontal area along the first curved surface 501 when viewed from the user's point of view, an operation area G300 related to the user's "recommended" category can be configured, and different categories (such as , "Cooperation", "Game", "Newcomer", "Follow", etc.). The arrangement of each operation area G300 can be customized for each user in the same manner as the arrangement of icons of various applications on the screen of a smartphone. In FIG. 4 , a region R500 (refer to FIG. 5 ) schematically shows a frontal region viewed from the user's point of view. The home page image H1 is generated based on the frontal area viewed from the user's point of view. Also, if the direction of the user's line of sight or the orientation of the face changes, the position of the frontal region R500 when viewed from the user's point of view may change accordingly. Accordingly, it is possible to improve the consistency between the change of the field of view in the real space and the change of the field of view in the virtual space.

In addition, it is preferable that the plurality of planar operation regions G300 are arranged in a plurality of layers in front and back. For example, the plurality of planar operation regions G300 may include a first group arranged in a plurality of rows along a first curved surface 501 around a predetermined reference axis I, and a group arranged in rows along a second curved surface 502 around a predetermined reference axis I. The second set of multiple column configurations. In this case, as shown in FIG. 4 , the second curved surface 502 may be shifted rearward with respect to the first curved surface 501 . In this case, when viewed from the user's point of view, the second group of a plurality of operation regions G300 arranged along the second curved surface 502 (hereinafter also referred to as "operation region G300-2" to distinguish it from the first group) An overlapping relationship is formed behind the first group of a plurality of operation regions G300 arranged along the first curved surface 501 (hereinafter also referred to as “operation regions G300-1” to distinguish them from the second group). At this time, a part of the operation area G300-2 of the second group can be viewed from behind the operation area G300-1 of the first group when viewed from the user's point of view. In this case, the user can be informed that there is an operation area on the rear side, and the number of operation areas that can be seen by the user can be effectively increased. In addition, it is also possible to set a third curved surface that is shifted to the rear side with respect to the second curved surface, and further arrange a planar operation region G300. In this manner, any number of two or more planar operation regions G300 may be arranged to overlap each other when viewed from the user's point of view.

In addition, in the case of a plurality of planar operation regions G300 overlapping one another in this way, most of the operation regions G300 can be arranged so that the user can operate, and the processing load related to drawing can be effectively reduced. For example, by performing complete drawing of thumbnail images and real-time images only on the operation area G300-1, and performing incomplete drawing (such as processing such as changing the texture) on the operation area G300-2, it is possible to reduce the overall cost. Processing load associated with rendering. In addition, from the same point of view, thumbnail images and real-time images are completely drawn only in the operation region G300-1 in the region R500 which is the frontal region R500 viewed from the user's point of view in the operation region G300-1, and Incomplete rendering (for example, processing such as changing the texture) is performed on the operation region G300-1 other than the front region R500, thereby reducing the overall processing load related to rendering. For example, by pre-caching or pre-loading data with a high possibility of being the operation area G300-1 in the front area R500 in the user's terminal device 20, delays can be reduced, and the number of requests from the network 3 can be effectively reduced. The number of requests, the amount of requests burdened by the network 3 accompanying this, and the computing resources used to respond to the requests. In this case, the data with a high possibility of being the operation region G300-1 in the front region R500 may be predicted based on the tendency of each user, or may be determined by machine learning based on artificial intelligence.

In FIG. 5A , the relationship between the first curved surface 501 and the second curved surface 502 is schematically shown in plan view. The offset distance between the first curved surface 501 and the second curved surface 502 can be intentionally made smaller than the distance between layers of different attributes (for example, the distance between the interface display layer J2 and the auxiliary information display layer J5 ). Thereby, the user can easily understand that the same interface display layer J2 exists, and also easily and intuitively understand that there are a plurality of planar operation regions G300 on the rear side.

When the interface display layer J2 of such a plurality of planar operation regions G300 is arranged, the user can change the arrangement of the user interface (for example, the plurality of planar operation regions G300 ) through a specific operation. Thereby, the user interface (for example, a plurality of planar operation regions G300 ) can be arranged according to the user's preference and preference.

For example, some of the plurality of planar operation regions G300 may function as buttons for changing the arrangement of the plurality of planar operation regions G300. Alternatively, the user may view and move the plurality of planar operation regions G300 left and right from the user's point of view by inputting a gesture of moving the hand left and right. FIG. 6 shows a state in which the operation region G300 on the left (indicated as “G300L” in FIG. 6 ) is moved to the front side viewed from the user's point of view by the user's operation. In this way, the plurality of planar operation regions G300 can also move left and right along the first curved surface 501 . Thereby, among the plurality of planar operation regions G300 , it is possible to easily change the operation region G300 located on the front side viewed from the user's point of view. In this case, the plurality of planar operation regions G300 may be moved such that the planar operation regions G300 located in the front region vary according to the category. In this way, since the aggregation of different categories can be maintained, it is possible to achieve both ease of viewing and operability.

Alternatively, a part or all of the plurality of planar operation regions G300 may move left and right along the first curved surface 501 (and/or the second curved surface 502 ) continuously, periodically or irregularly, or may move in a rotational manner. Such movement can be appropriately changed based on the user's settings, and can also be realized based on the fulfillment event of a predetermined movement condition.

In addition, in the case of a plurality of planar operation regions G300 overlapping front and rear as described above, the user can switch front and rear by performing a predetermined input. For example, as described above, when viewed from the user's point of view, the second group of multiple operation regions G300-2 arranged along the second curved surface 502 is in contrast to the first group of multiple operation regions G300 arranged along the first curved surface 501. In the case of overlapping behind -1, as schematically shown in FIG. 7A , the user can switch the first group and the second group by a predetermined input of moving the hand in a predetermined manner. This enables intuitive operation and improves operability. In this case, the second group of operation regions G300-2 arranged along the second curved surface 502 is arranged as the first group along the first curved surface 501, and the first group of operation regions G300-2 arranged along the first curved surface 501 The region G300-1 is arranged along the second curved surface 502 as the second group. In addition, such front-rear switching operation may be applied only to the operation area G300 located on the front side viewed from the user's point of view among the plurality of planar operation areas G300. Thereby, compared with the case where the whole of the plurality of planar operation regions G300 is replaced, the processing load can be reduced.

In addition, the forward and backward switching method can also be as shown schematically by arrows in FIG. 7A , where the entirety of the first group of multiple operating areas G300-1 arranged along the first curved surface 501 is directly rotated backward. As schematically shown by arrows in 7B, the upper side of the plurality of operating regions G300-1 of the first group arranged along the first curved surface 501 rotates backward from the upper side, and the lower side rotates backward from the lower side. . In addition, conversely, as schematically shown by arrows in FIG. 7B , the exchange of the first group and the second group can be realized in the following manner, the multiple operations of the second group arranged along the second curved surface 502 The upper side in the region G300-2 rotates forward from the upper side, and the lower side rotates forward from the lower side. In addition, although two groups have been described here, the same applies to the case of three or more groups (that is, a plurality of planar operation regions G300 overlapping three or more layers one after the other).

In addition, in this embodiment, although the home page image H1 does not include the user avatar display layer J1, it is not limited thereto. For example, the homepage image H1 may include the user avatar display layer J1 on the front side (closer to the user's point of view) than the interface display layer J2 . In this case, in the user avatar display layer J1 of the homepage image H1, a hand may be drawn while the user stretches his hand forward or the like. Accordingly, the user can operate the plurality of planar operation regions G300 while watching the movement of the hand, thereby improving operability.

8A and 8B are explanatory diagrams of a virtual space corresponding to a distribution image, FIG. 8A is a schematic side view, and FIG. 8B is a schematic top view when viewed from above. In FIG. 8A and FIG. 8B , the distribution user is schematically indicated by a mark 808 , and the position of the corresponding distributor avatar M2 is schematically shown by a dotted line box. FIG. 9 is an explanatory diagram of a hierarchical structure of an image (hereinafter, referred to as "distribution image H2") based on a distribution user's specific content. FIG. 10 is an explanatory diagram showing the relationship between the mirror and each layer of the distribution image H2 in plan view. FIG. 11 is an explanatory diagram of the interface display layer J2 related to the distribution image H20 for distribution users. FIG. 12 is a diagram showing an example of a distribution image H2 viewed by a distribution user through a head-mounted display (an example of a first wearable device). FIG. 13 is a diagram showing an example of a distribution image H2 viewed by a viewing user through a head-mounted display (an example of a second wearable device). FIG. 14 is a diagram showing an example of a distribution image H2 for a smartphone. In addition, the distribution image H2 shown in FIG. 14 is an image when viewed without a head-mounted display, and can also be used as an image for similar terminals other than a smartphone.

The distribution image H2 is a virtual space for creating specific content based on the distribution user (hereinafter referred to as "content creation space") or the like, and corresponds to an image related to the virtual space where the distributor avatar M2 is arranged. In addition, the content production space may correspond to a space such as a studio in the real world. The distribution user can perform specific content production activities in the content production space.

In this embodiment, the distribution image H2 includes: a distribution image H20 for distribution users (see FIG. 12 ); a distribution image H21 (refer to FIG. The distribution image H22 (hereinafter, also referred to as "distribution image H22 for smartphone") that can be viewed without the head-mounted display is viewed as the distribution image for the user (see FIG. 14 ). Unless otherwise specified below, the distribution image H2 corresponds to these arbitrary distribution images.

In this embodiment, as an example, the distribution image H2 sequentially includes the distribution avatar display layer J0, the interface display layer J2 (an example of a layer related to the first user interface), the auxiliary information display layer J5, the user avatar Image display layer J1, and background layer J3. In addition, in the modified example, other layers as described above may be included, and some layers (for example, the auxiliary information display layer J5 and the background layer J3 ) may be omitted. In addition, the interface display layer J2 and the auxiliary information display layer J5 can be properly combined.

In FIG. 9 , an arrangement example of the distribution avatar display layer J0, the interface display layer J2, the auxiliary information display layer J5, the user avatar display layer J1, and the background layer J3, as well as the positions passing through the distribution users are shown in a top view. (The position of the distributor avatar M2) defines the reference axis I.

In this embodiment, as shown in FIG. 9, the distribution avatar display layer J0, the interface display layer J2, the auxiliary information display layer J5, the user avatar display layer J1, and the background layer J3 are arranged in the following manner, that is, in a plan view Vertical surfaces with different distances d1 to d5 are formed along the line of sight of the user passing through the specified reference axis I. In this case, the distances d1 to d5 respectively correspond to the distances between the respective floors and the user. In addition, the distance d5 related to the background layer J3 may be infinite, or may be a distance corresponding to the plane farthest from the viewpoint of the virtual camera (far clipping plane). In this case, the interface display layer J2 may be set on the surface closest to the far cropping plane (near cropping plane) of the virtual camera. In addition, in this case, the point of view of the distribution user and the virtual camera may be set at the same positions as the face and head item of the distributor avatar M2, or may be set in a direction facing forward from the viewpoint of the distributor avatar M2. The optical axis of the virtual camera. In this case, from the point of view of the distribution user, the face of the distributor avatar M2 and the head of the distributor avatar M2 are not drawn as objects; on the other hand, the wrist and hands of the distributor avatar M2 (forward The movement of the hand and wrist when reaching out to operate) can also be used as the object of drawing. In addition, in this case, the distributor avatar M2 can be drawn as the distribution avatar display layer J0 from the perspective of other users (for example, viewing users). In addition, as mentioned above, the distance d2 of the interface display layer J2 can be set within the reach of the user's hand. In addition, the distance d2 can also be adjusted per user. In addition, distance d2 may be the same as radius r1 mentioned above (refer FIG. 5). In this way, the interface display layer J2 can be formed with a sense of distance corresponding to the length of the user's hand and preference. Also, when the distance d2 is the same as the radius r1 (see FIG. 5 ), the same (common) operability can be realized between the distribution image H20 for distribution users and the home page image H1, and convenience can be improved.

The content creation space of the virtual space where the distributor avatar M2 is active is any space. For example, in the example shown in FIG. 8A and FIG. Wardrobe 801, desk 802. The distribution user can select a desired avatar item from the avatar items in the closet 801 and prepare (standby) the distributor avatar M2. In addition, avatar props refer to props corresponding to and depicted on avatars such as the avatar M2 of the distributor, and may include, for example, hairstyles, clothes, wearables, skin colors, and the like.

In the present embodiment, a mirror 805 is disposed in the distribution room 800 as a preferable example in the content creation space. Like mirrors in the real world, the mirror 805 has a property of reflecting light (visible light). The image reflected in the mirror 805 corresponds to the distribution image H20. Therefore, by placing the distributor avatar M2 corresponding to the user in front of the mirror 805, the distributor can confirm the distributor corresponding to the various states (orientation, position, and action) of the user while looking at the front (mirror 805). Various states of the avatar M2 (and the accompanying states of the distribution image H20). Thus, in the state where the distribution user is facing the mirror 805 (the state facing the front) and the line of sight is directed to the mirror 805, he can confirm various states (orientation, position, and action) of himself and the distributor's virtual reality associated with it. The state of the avatar M2 (and the state of the distribution image H20 accompanying this). In this way, since the distribution user can properly correct his own motion in real time while confirming his own motion while facing the mirror 805 directly, he can easily generate the desired distribution image H20 according to his imagination.

FIG. 10 shows the positional relationship of each layer of the mirror 805 in the distribution room 800, the distribution user, and the distribution image H2 in plan view. In FIG. 10 , the region R502 corresponds to the region R501 shown in FIG. 9 , and the region R502 corresponding to the field of view through the mirror 805 when looking at the front from the point of view of the distribution user is the same as when a virtual camera is placed on the mirror 805. The field of view of the virtual camera corresponds. In this case, the virtual camera in the mirror 805 can realize the same function as the mirror 805 by displaying the captured image on the mirror 805 .

The distributor avatar is drawn on the distribution avatar display layer J0. In this case, on the delivery avatar display layer J0 , the distributor avatar M2 is drawn in a state corresponding to various states (orientation, position, and movement) of the delivery user standing in front of the mirror 805 . In addition, as described above, various states (orientation, position, and motion) of the distribution user can be obtained through the input unit 24 .

The interface display layer J2 is preferably configured based on the distribution user's position standing in front of the mirror 805 . In other words, when the distribution user stands in front of the mirror 805, the distribution user is positioned in a prescribed positional relationship relative to the interface display layer J2. In this embodiment, the interface display layer J2 is placed within the reach of the hand of the user at the position where the delivery user stands in front of the mirror 805 as a reference. In addition, the predetermined positional relationship may be appropriately changed by the user. Thereby, the interface display layer J2 corresponding to the user's preference and physique can be realized.

The interface display layer J2 preferably includes one or more operation areas that function as the input unit 24 . Various operation areas can function as selection buttons that the distribution user can select. For example, the distribution user can select (operate) a desired selection button through non-contact user input.

In this case, various processes (an example of various first processes) realized by operating the operation area may be arbitrary processes, and may include, for example, process for starting distribution of specific content and processing for ending distribution of specific content. , voice input processing, distribution user receiving gift processing, item lottery processing (such as avatar-related items), selection/exchange processing of various items, text input processing, consent processing to other users, and parameters for distributing content At least any one of processing for changing values (for example, various parameters of the distribution image H2 described later) and processing for transitioning to a state where these processings can be executed.

FIG. 11 shows an example of the operation area G100 in the form of various icons in the distribution image H20 for the distribution user. In the example shown in FIG. 11 , it includes: various operation areas G100, an operation area G101 for chatting (comments), an operation area G102 for obtaining screenshots, an operation area G103 for playing games, an operation area G103 for An operation area G104 for sharing other video content (including sharing of the way users watch and distribute users watch the same video, etc.), an operation area G105 for twisting the gashapon machine (drawing lots), etc. In addition, the lottery object in the gashapon machine may be arbitrary, for example, it may be an avatar item.

The operation area in the interface display layer J2 can be displayed all the time, or only in specific cases. Thereby, while operations can be performed through the operation area in the interface display layer J2, the display area of the distributor avatar M2 can be enlarged to display the entire distributor avatar M2.

In addition, the specific operation area in the interface display layer J2 can be displayed in the same manner as the operation area G300 related to the above-mentioned homepage image H1. In this case, referring to FIG. 7A and FIG. 7B , various operation areas in the interface display layer J2 can be switched back and forth in the above-mentioned manner.

In addition, in a preferred embodiment, the operation area in the interface display layer J2 includes an operation area G120 (hereinafter also referred to as "smartphone small window area G120") for adjusting various parameters of the distribution image H2. Part or all of the distribution image H21 for viewing users and/or part or all of the distribution image H22 for smartphones may be displayed in the smartphone small window area G120.

The various parameters of the distribution image H2 that can be adjusted through the small window area G120 of the smart phone are, for example, parameters related to the virtual camera, specifically, may include: brightness, distance from the virtual camera (that is, the distance between the mirror 805 and the distribution user) distance), the optical axis direction of the virtual camera, the horizontal position of the virtual camera (the position relative to the distribution user in the horizontal direction intersecting the optical axis direction), zoom (magnification), and the like. In this case, for example, distribute the user to operate the smart phone small window area G120 as a smart phone, such as by zooming in (pinch out) or zooming out (pinch in) so that the smart phone small window area G120 can be changed. The image in the small window area G120 of the smart phone can also be moved in parallel by sliding. Alternatively, the distribution user can change the position of the virtual camera (relative position to itself) by performing an operation to move the position of the smartphone small window area G120. In addition, the distribution user can also control the state of the virtual camera through the small window area G120 of the smartphone, so that it is possible to implement either the following mode in which the position of the virtual camera automatically follows the movement of the distribution user, or the fixed mode in which the position of the virtual camera is fixed. switch between. In this case, mobile distribution can be performed using the follow-up mode. In addition, the distribution user can also control the state of the virtual camera through the small window area G120 of the smart phone, so as to form a state that does not want the viewing user to watch such as the state of AFK (Away From Keyboard, temporarily leaving) and the mute state. Thus, it is possible to prevent an increase in the processing load due to drawing unnecessary objects, and to improve the convenience of the distribution user.

However, when the user is wearing a head-mounted display, etc., there is a difference from the operation in the real space, and the operability when adjusting various parameters of the distribution image H2 is likely to deteriorate. However, by setting the smartphone small window area G120, Operability becomes good.

In addition, when the distribution image H22 for the smartphone is displayed on the smartphone small window area G120, the distribution user can confirm the distribution image H20 approximately identical to the distribution image H21 when viewing on the head-mounted display, and the distribution user can view the distribution image H22 with the smartphone. Specific content is created while performing the distribution image H22 at the time of viewing. In addition, instead of the smartphone small window area G120 or in addition to it, a small window section for other viewing devices such as a tablet computer small window section may be set. The distribution user can also select a desired widget and display it on the distribution image H20.

In the example shown in FIG. 12 , the smartphone widget area G120 is displayed adjacent to the mirror 805 on which the distributor avatar M2 is reflected. The smartphone small window area G120 can display the distribution image H22 (see FIG. 14 ) when displayed on the smartphone. In this case, the distribution user can perform distribution while checking in real time the state of the distribution image H22 on the viewing user's terminal device 20 viewed with a smartphone.

In addition, in a modified example, the distribution image H20 may include two concepts of an image being distributed and a preparation image before distribution. In the case of a preparation image before distribution, an operation region G106 (refer to FIG. 11 ) for opening the closet 801 (refer to FIG. 8A ) to change clothes, an operation region G107 ( Refer to Figure 11) and so on.

In the auxiliary information display layer J5, text information such as presents, props, and help messages from viewing users that distribute specific content related to the image H2 can be depicted in real time. Thereby, the distribution user can distribute the specific content while enjoying the reaction of the viewing user, communication with the viewing user, and the like. In addition, the distribution name, title, and the like of specific content may be drawn on the auxiliary information display layer J5. For example, in the example shown in FIG. 12 , a distribution name G11 such as "casual chat distribution", a heart-shaped gift item G12, and various comments G13 such as "meeting for the first time..." are drawn.

In the user avatar display layer J1, the viewer avatar M1 of the viewer viewing the specific content related to the distribution image H2 can be drawn. Accordingly, the distribution user can perform distribution while grasping the number and status of viewing users. In addition, the viewer avatar M1 drawn by the user avatar display layer J1 may be drawn in a state corresponding to the state (orientation, position, and action) of the corresponding viewing user. Accordingly, the distribution user can perform distribution while grasping the state of the viewing user.

In this case, the distributor avatar M2 drawn on the delivery avatar display layer J0 may be drawn translucently so that the distribution user can easily grasp the state of the user avatar display layer J1. In this case, the distribution user can easily confirm the state of the user avatar display layer J1 behind the distributor avatar M2. In addition, the distributor avatar M2 can also be switched between a translucent state and an opaque state through the input of the distribution user. In this case, the distribution user can easily selectively confirm the state of the distributor avatar M2 and the state of the user avatar display layer J1 behind the distributor avatar M2.

The wall of the delivery room 800 and the like may be drawn in the background layer J3. Alternatively, the scenery through the wall of the delivery room 800 may be drawn on the background layer J3. In addition, the viewer avatar M1 of the viewer who distributes the specific content related to the image H2 can enter a virtual space (hereinafter referred to as "viewing space") different from the content creation space. In this case, a viewing space-related background can be drawn on the background layer J3. In this case, the distribution image H21 for viewing users (see FIG. 13 ), which is an image of the viewing space, may have a different background layer J3 from the distribution image H20 for distribution users (see FIG. 12 ). In this way, there may be inconsistency between the viewing space and the virtual space (content creation space) represented by the distribution image H20 for distribution users. In this way, the distribution image H21 for viewing users can be optimized from a viewpoint different from that of the distribution image H20 for distribution users.

In addition, in a modified example, as schematically shown in FIG. 8B , the distribution chamber 800 may have a wall 810 of a half mirror (magic mirror) on the front side. In this case, the wall body 810 has the same reflective function as the mirror 805 that can see the room from the outside but cannot see the outside from the inside. In this case, the distribution user can perform distribution using the wall body 810 in the same manner as the mirror 805 described above. On the other hand, the viewing user can view the distribution user (distributor avatar M2 ) in the distribution room 800 through the wall 810 . In addition, in this case, the background layer J3 of the delivery image H2 may include a drawing of a specially made stage or the like instead of the interior of the delivery room 800 .

Next, a configuration example of the aforementioned virtual reality generation system 1 will be sequentially described with reference to the drawings following FIG. 15 .

FIG. 15 is a schematic block diagram showing the functions of the terminal device 20A on the content distribution side. FIG. 16 is an explanatory diagram of an example of drawing information of an avatar. 15 and 16, unless otherwise specified, a distribution user refers to one distribution user, and a distribution image H2 (H20) refers to a distribution image H2 that forms specific content based on the one distribution user.

As shown in FIG. 15 , the terminal device 20A includes a first rendering processing unit 200A, a second rendering processing unit 200B, a first information generation unit 210A, a first communication unit 220A, a first display unit 230A, a first storage unit 240A, a second A user interface part 250A.

The respective functions of the first rendering processing unit 200A, the second rendering processing unit 200B, the first information generating unit 210A, the first communication unit 220A, the first display unit 230A, and the first user interface unit 250A are implemented through the terminal shown in FIG. The terminal control unit 25 of the device 20A cooperates with the terminal communication unit 21 , the terminal storage unit 22 , the display unit 23 , and the input unit 24 to execute the virtual reality application, thereby achieving realization. Furthermore, the first storage unit 240A can be realized by the terminal storage unit 22 of the terminal device 20A shown in FIG. 1 .

The first drawing processing unit 200A draws the above-mentioned distribution image H20 (see FIG. 12 ) for distribution users (an example of an image in the first virtual space). The first rendering processing unit 200A generates, for example, a distribution image H20 for distribution users (see FIG. 12 ) at a predetermined frame rate.

The first rendering processing unit 200A includes a first rendering unit 201A, a second rendering unit 202A, and a third rendering unit 203A.

The first drawing unit 201A draws the image area (an example of the first image area) of the distribution avatar display layer J0 in the distribution image H20 (see FIG. 12 ) for the distribution user. The distribution avatar display layer J0 in the distribution image H20 is as described above. The first drawing unit 201A may draw the distributor avatar M2 in the image area of the distribution avatar display layer J0 based on, for example, the drawing information of the avatar stored in the first storage unit 240A (see table 700 in FIG. 16 ). In FIG. 16 , an example of the drawing information of the avatar is schematically shown by a table 700 .

In this case, in the drawing information of the avatar, a face part ID, a hairstyle part ID, a clothing part ID, and the like are associated with each avatar ID. Appearance-related component information such as face component ID, hairstyle component ID, and clothing component ID are parameters that characterize the avatar and can be selected by each user. For example, various types of appearance-related information such as face part ID, hairstyle part ID, and clothing part ID related to the avatar are prepared. In addition, for face part IDs, part IDs are prepared for each type of face shape, eyes, mouth, nose, etc., and information related to face part IDs can be managed by combining the IDs of the parts constituting the face. In this case, not only the server device 10 but also the terminal device 20 side can draw each avatar based on each ID related to the appearance associated with each avatar ID.

In addition, when, for example, the distributor avatar M2 opens the wardrobe 801 (see FIG. 8A ) to change clothes (that is, changes IDs related to hairstyles and clothing), the first drawing unit 201A updates the distributor avatar accordingly. The appearance of the M2.

Also, the first drawing unit 201A changes the status of the distributor avatar M2 based on the detection result (the status detection result of the distribution user) of the first detection unit 211A (described later) of the first information generation unit 210A. In this embodiment, as described above, the first drawing unit 201A draws the distributor avatar M2 in such a manner that the state of the delivery user facing the mirror 805 (see FIG. 8A ) is replaced with the distributor avatar M2.

For example, when the distributor avatar M2 is in the form of a character with a frontal orientation, the first drawing unit 201A may link the orientation of the distributor avatar M2 with the orientation of the distribution user in the following manner, that is, when the distribution user faces The distributor virtual image M2 faces left when it is right, and the distributor virtual image M2 faces downward when the distribution user faces downward. In addition, in this case, the orientation may be only the orientation of the face, only the orientation of the body, or a combination thereof. Thereby, the alignment (linkage) of the directions of the distributor avatar M2 and the distribution user becomes high, and it is possible to diversify the expression by the direction of the distributor avatar M2.

In addition, when the distributor avatar M2 is in the form of a character with a gaze direction, the first drawing unit 201A may link the direction of the gaze of the distributor avatar M2 with the direction of the gaze of the distribution user as follows: When the distribution user's gaze is directed to the right, the distributor's avatar M2's gaze is directed to the left, and when the distribution user's gaze is directed downward, the distributor's avatar M2's gaze is directed downward. In addition, it is also possible to link various eye movements such as blinking. In addition, it is also possible to link the movements of the nose and mouth. As a result, the consistency (linkage) of each component between the distributor avatar M2 and the delivery user becomes high, and the expressions produced by the distributor avatar M2 can be varied.

In addition, when the distributor avatar M2 is in the form of a character with hands, the first drawing unit 201A may link the movement of the hand of the distributor avatar M2 with the movement of the hand of the distribution user in such a manner that when the distribution When the delivery user raises his right hand, the distributor avatar M2 raises his left hand, and when the delivery user raises his hands, the distributor avatar M2 raises his hands. In addition, it is also possible to link the movements of various parts of the hand such as fingers. In addition, other members such as legs may also be linked in the same manner. As a result, the coherence (linkage) between the distributor avatar M2 and the delivery user becomes high, and it is possible to diversify expressions such as actions of parts of the distributor avatar M2.

Also, when the position (drawing position) of the distributor avatar M2 in the distribution image H20 is variable (that is, when the distributor avatar M2 is a character whose position changes), the first drawing unit 201A may perform the following: The position of the distributor avatar M2 in the distribution image H20 is changed according to the position of the distribution user (relative to the position of the mirror 805), that is, when the distribution user moves to the right, the distributor avatar M2 moves to the left, and when the distribution user moves away The distributor virtual image M2 is far away from the mirror 805 . Thereby, the coherence (linkage) of the movement (position) between the distributor avatar M2 and the delivery user becomes high, and the expression due to the position change of the distributor avatar M2 can be diversified.

The second drawing unit 202A draws the image area (an example of the second image area) of the interface display layer J2 in the distribution image H20 (see FIG. 12 ) for the distribution user. The interface display layer J2 in the distribution image H20 is as described above.

The second drawing unit 202A draws the aforementioned various operation areas G100 (see FIG. 11 ) and the smartphone widget area G120 (see FIG. 12 ). In this case, the second rendering unit 202A may always render the various operation regions G100 and the smartphone widget region G120, or may appropriately omit rendering of a part or all of them. For example, the second rendering unit 202A may output cooperation approval/disapproval operation areas G117 and G118 (see FIG. 11 ) in response to a cooperation approval/disapproval instruction from the viewing user information acquisition unit 222A described later. Thereby, the variation (variation) operable by the interface display layer J2 can be increased, and the simplification (easiness of viewing) of the display of the interface display layer J2 can be realized.

The third drawing unit 203A draws the image areas (an example of the third image area) of the remaining layers in the distribution image H20 (see FIG. 12 ) for distribution users. That is, the third drawing unit 203A draws image areas of layers other than the distribution avatar display layer J0 and the interface display layer J2 in the distribution image H20 . For example, the third rendering unit 203A may render each image area of the background layer J3 and the auxiliary information display layer J5 according to the hierarchical structure of the distribution image H20.

In this embodiment, as described above, as an example, the distribution image H20 sequentially includes the distribution avatar display layer J0, the interface display layer J2, the auxiliary information display layer J5, the user avatar display layer J1, and the background layer. J3.

Therefore, in this embodiment, the third rendering unit 203A renders each image area of the auxiliary information display layer J5 , the user avatar display layer J1 , and the background layer J3 .

The third rendering unit 203A may respond to a gift rendering instruction from the viewing user information acquisition unit 222A described later, and render the image area of the auxiliary information display layer J5 based on, for example, the rendering information of gift items stored in the first storage unit 240A. gift items in . In addition, drawing information related to gift items may be stored in the first storage unit 240A for each gift ID. The gift depiction indication may include a gift ID and coordinate information for identifying a gift item to be depicted. The coordinate information is information indicating a drawing position in the distribution image H20, and may change in time series. In addition, the coordinate information may also include action information indicating an action of the gift item. The motion information may indicate motions such as descending while rotating, for example. Thereby, the diversification of the expression of a gift item can be realized.

In addition, the third drawing unit 203A may respond to a viewing user drawing instruction from the viewing user information acquisition unit 222A described later, based on, for example, the drawing information of the avatar stored in the first storage unit 240A (see table 700 in FIG. 16 ). ), drawing the viewer avatar M1 (the viewer avatar M1 of the viewing user) in the image area of the user avatar display layer J1. The viewer drawing instruction may include various information for drawing the viewer avatar M1 related to the viewer who is viewing the specific content. For example, various information used to describe the viewer avatar M1 may include information of the viewer avatar M1 (such as avatar ID, face part ID, hairstyle part ID, clothing part ID, etc.), coordinate information. Also, when the viewer avatar M1 is in motion, the viewer drawing instruction may include information indicating the state of the viewer (or information indicating the state of the viewer avatar M1 based thereon). As a result, the actions of the viewing user can be expressed on the distribution image H20 for the distribution user, and thus the sense of unity between the distribution user and the viewing user can be enhanced.

In addition, the third drawing unit 203A may draw various comments in the image area of the user avatar display layer J1 in response to a comment drawing instruction from the user information acquisition unit 222A, which will be described later. The comment drawing instruction may include comment ID and coordinate information. In addition, drawing information related to fixed-format reviews may be stored in the first storage unit 240A for each review ID. In addition, instead of the comment ID related to the fixed-format comment, text information (for example, a message such as a chat) that can be transmitted from the terminal device 20B on the content viewing side may be used. Thereby, a unique message or the like from each viewing user can be transmitted, and the sense of unity between the distribution user and the viewing user in the distribution image H20 for the distribution user can be enhanced.

In addition, the third drawing unit 203A may draw a background image corresponding to the background ID selected by the distribution user on the background layer J3. The background image can be stored in the first storage unit 240A for each background ID. In addition, the background image can also be customized by the distribution user.

The second drawing processing unit 200B draws the above-mentioned homepage image H1 (see FIG. 6 ) (an example of an image in the second virtual space). The second rendering processing unit 200B generates the homepage image H1 (see FIG. 6 ) at a predetermined frame rate, for example.

The second rendering processing unit 200B includes a fourth rendering unit 204B and a fifth rendering unit 205B.

The fourth drawing unit 204B draws the image area (an example of the fourth image area) of the interface display layer J2 in the home page image H1 (see FIG. 6 ). The interface display layer J2 in the homepage image H1 is as described above. The fourth rendering unit 204B can change the arrangement of the plurality of operation regions G300 and the like in response to an instruction from the first input processing unit 252A described later (see FIG. 6 and FIG. 7A ).

The fifth drawing unit 205B draws the image areas (an example of the fifth image area) of the remaining layers in the homepage image H1 (see FIG. 6 ). That is, the fifth drawing unit 205B draws image areas of layers other than the interface display layer J2 in the home page image H1 . For example, the fifth rendering unit 205B may render each image area of the background layer J3 and the auxiliary information display layer J5 according to the hierarchical structure of the home page image H1. Thereby, various kinds of information that may be required by the user can be efficiently provided without greatly increasing the processing load on drawing.

In the present embodiment, as described above, as an example, the homepage image H1 includes the interface display layer J2, the auxiliary information display layer J5, and the background layer J3 in order from the front side.

Therefore, in this embodiment, the fifth rendering unit 205B renders each image area of the auxiliary information display layer J5 and the background layer J3.

The first information generation unit 210A generates various information on distribution users (hereinafter referred to as "distribution user information"). The delivery user information includes information indicating the status of the delivery user used by the first user interface unit 250A described later (the status of the delivery user detected by the first detection unit 211A). In addition, the distribution user information includes information necessary for drawing a distribution image H21 (see FIG. 13 ) for viewing users. For example, the distribution user information includes information (eg, avatar ID, face part ID, hairstyle part ID, clothing part ID, etc.) for depicting the distributor avatar M2. Furthermore, in the case where the distribution user's specific content includes the distribution user's voice, the distribution user information may also include voice information acquired through the microphone of the input unit 24 or the like. In this way, it is possible to separate various types of information constituting the specific content and perform distributed processing.

In this embodiment, the first information generation unit 210A includes a first detection unit 211A, a first voice information generation unit 212A, and a first text information generation unit 213A.

The first detection section 211A detects the state of the delivery user (the user's orientation, position, motion, or the like). The first detection unit 211A can detect the status of the distribution user through the above-mentioned input unit 24 . The first detection unit 211A can detect the state of the delivery user at a cycle corresponding to the frame rate of the homepage image H1, for example, while the homepage image H1 is being drawn. Furthermore, the first detection unit 211A may detect the state of the distribution user at a cycle corresponding to the frame rate of the distribution image H20 for the distribution user (see FIG. 12 ), for example, while the distribution image H20 for the distribution user is being drawn. Thereby, it is possible to detect (update) the state of the distribution user (and the display of the distributor avatar M2 accompanying this) at a high frequency.

The first detection section 211A can switch the on/off state according to, for example, an operation by a distribution user. For example, when the distribution user operates a distribution start button that can be displayed on the interface display layer J2 (refer to the operation area G107 in FIG. In response to a command from the first input processing unit 252A described later, the first detection unit 211A is turned on. Thereby, it is possible to increase the number of operable variations via the interface display layer J2, and it is possible to realize simplification (easiness of viewing) of the display on the interface display layer J2. By simplifying the display of the interface display layer J2, the visibility of the rear layers can be improved.

The first audio information generation unit 212A generates audio information based on the utterance of the distribution user. For example, the first voice information generation unit 212A generates the voice information of the distribution user through the microphone of the input unit 24 or the like. In addition, the first voice information generation unit 212A may process the voice data obtained through the input unit 24 to generate voice information of the distribution user. In this case, the processing method may be selected by the distributing user, or the voice data may be processed manually by the distributing user.

The first voice information generation section 212A can switch the on/off state according to, for example, an operation by a distribution user. For example, when the distribution user operates a mute button (not shown) that can be displayed on the interface display layer J2, the operation can be detected by the first input detection unit 251A described later, and the input from the first input process described later can be detected. In response to an instruction from the unit 252A, the first voice information generation unit 212A is turned off.

The first text information generating unit 213A generates text information based on utterances and/or character input of the distribution user. For example, the first text information generating unit 213A generates text information when the distribution user and the viewing user have a conversation (chat, etc.).

The first text information generation section 213A can switch the on/off state according to, for example, an operation by a distribution user. For example, when the distribution user operates a button for comments that can be displayed on the interface display layer J2 (refer to the operation area G101 in FIG. In response to an instruction from the first input processing unit 252A described above, the first text information generating unit 213A is turned on. Thereby, it is possible to increase the number of operable variations via the interface display layer J2, and it is possible to realize simplification (easiness of viewing) of the display on the interface display layer J2.

In this case, the first information generation unit 210A may also include in the distribution user information a set of information indicating the state of the distribution user (or information representing the state of the distributor avatar M2 based on this information) in the distribution user information. information) (hereinafter, sometimes simply referred to as "status information of the distributor avatar M2"), information of the distributor avatar M2 (such as avatar ID, face part ID, hairstyle part ID, clothing part ID, etc.), appropriate voice message, text message. In this case, the first information generating unit 210A may also generate the distribution user information by adding a time stamp to dynamically changing information such as status information, voice information, and text information of the distributor avatar M2. Thus, when receiving the distribution user information, the terminal device 20B on the content viewing side can use the distribution image H21 as described above, that is, the distribution image H21 with voice output, as a message from the distribution user based on the received distribution user information. Specific content output.

The first communication unit 220A communicates with the server device 10 and other terminal devices 20 (for example, the terminal device 20B on the content viewing side). In this embodiment, the first communication unit 220A includes a distribution processing unit 221A and a viewing user information acquisition unit 222A.

The distribution processing unit 221A transmits the distribution user information generated by the first information generating unit 210A to the terminal device 20B on the content viewing side. In addition, in a modified example, the distribution processing unit 221A may transmit video data based on the distribution image H20 generated by the first rendering processing unit 200A to the terminal device 20B.

The distribution processing unit 221A can send the distribution user information to the terminal device 20B on the content viewing side in real time according to, for example, the operation of the distribution user, etc. Device 20B sends. In the case of non-real time, the distribution processing unit 221A may send the video data manually edited by the distribution user or the edited distribution user information to the terminal device 20B on the content viewing side.

The viewing user information acquiring unit 222A acquires various viewing user information from the terminal device 20B on the content viewing side. Based on the obtained viewing user information, the viewing user information acquisition unit 222A generates the above-mentioned gift drawing instruction, viewing user drawing instruction, comment drawing instruction, cooperation approval/disapproval instruction, and the like. In addition, the viewing user information includes information necessary for generating these gift drawing instructions, viewing user drawing instructions, comment drawing instructions, cooperation approval/disapproval instructions, and the like.

The first display unit 230A outputs the distribution image H20 (see FIG. 12 ) for distribution users generated by the first rendering processing unit 200A and the homepage image H1 generated by the second rendering processing unit 200B to the display unit 23 of the terminal device 20A. In addition, in the present embodiment, as described above, the display unit 23 is in the form of a head-mounted display as an example.

The above-mentioned avatar drawing information (see table 700 in FIG. 16 ) and the like are stored in the first storage unit 240A.

The first user interface unit 250A detects various inputs from the distribution user through the input unit 24 and executes processing corresponding to the various inputs. The first user interface unit 250A includes a first input detection unit 251A and a first input processing unit 252A.

The first input detection unit 251A detects various inputs of the distribution user through the above-mentioned interface display layer J2. For example, the first input detection unit 251A detects input through the above-mentioned various operation areas G100 (see FIG. 11 ), smartphone widget area G120 (see FIG. 12 ), operation area G300 , and the like.

In this case, the first input detection unit 251A may be based on the state of the distribution user detected by the first detection unit 211A, the state of the interface display layer J2 (for example, the position of various operation areas G100, the smart phone small window area G120, the display state), detecting and distributing the user's input through the interface display layer J2. For example, the first input detection unit 251A may, when detecting an action of clicking one of the various operation areas G100 , detect a selection operation of the one operation area. Thus, the user can perform selection operations with simple actions in the virtual space.

The first input processing unit 252A executes various processes (an example of various second processes) corresponding to various inputs detected by the first input detection unit 251A on the homepage image H1. Various kinds of processing are arbitrary, and as described above, for example, processing of changing the layout of the user interface (for example, a plurality of planar operation areas G300), processing of moving the user to an arbitrary place or a specific place in the virtual space (for example, using The distribution user moves to the location for distribution) and the like. In addition, various processing may be appropriately accompanied by various rendering processing performed by the second rendering processing unit 200B and transition of screens (for example, movement of virtual space).

In addition, the first input processing unit 252A executes various processes (one of various first processes) corresponding to various inputs detected by the first input detection unit 251A in the distribution image H20 (refer to FIG. 12 ) for users. example). Various processes are arbitrary, and as described above, may be, for example, the process of starting the distribution of specific content, the processing of ending the distribution of specific content, voice input processing, processing of receiving a gift by the distribution user, and the like. In addition, various processing may be appropriately accompanied by various rendering processing performed by the first rendering processing unit 200A and transition of screens (for example, movement of virtual space). For example, the process for transitioning to a state where the process of receiving a gift can be performed may be accompanied by the drawing of notification information (notification information indicating that a gift has been given) by the third rendering unit 203A of the first rendering processing unit 200A, the rendering of the third rendering unit 203A, and the third rendering unit 200A. The second drawing unit 202A of the first drawing processing unit 200A draws the receiving button, etc.

Thus, according to the present embodiment, since the distribution image H20 for distribution users has a plurality of layers as described above, it is possible to effectively reduce the processing load by layering the images of the distribution image H20 for distribution users. For example, in the case of distributing specific content in the form of video, the image area of the avatar display layer J0 and the image area of the interface display layer J2 are drawn (updated) at a higher frequency than the image area of the interface display layer J2. The distribution avatar in the display layer J0 and the like can effectively reduce the overall processing load when drawing the distribution image H20 for the distribution user. In addition, by layering the distribution image H20 for the distribution user, it is possible to effectively give the user a sense of distance as "space". In addition, by layering the distribution images H20 for distribution users, it is possible to arrange them at appropriate distances, for example, according to the attributes of the image regions of each layer, and it is possible to balance, for example, the drawing of a user interface with high operability in a manner that does not easily cause dizziness and ease of use. Drawings and the like of the distributor avatar M2 and the like to be viewed.

FIG. 17 is a schematic block diagram showing the functions of the terminal device 20B on the content viewing side.

As shown in FIG. 17 , the terminal device 20B includes a first rendering processing unit 200A′, a second rendering processing unit 200B, a second information generation unit 210B, a second communication unit 220B, a second display unit 230B, a second storage unit 240B, The second user interface part 250B.

The functions of the first rendering processing unit 200A′, the second rendering processing unit 200B, the second information generating unit 210B, the second communication unit 220B, the second display unit 230B, and the second user interface unit 250B are implemented by the This can be achieved by the terminal control unit 25 of the terminal device 20B cooperating with the terminal communication unit 21 , the terminal storage unit 22 , the display unit 23 , and the input unit 24 to execute the virtual reality application. Furthermore, the second storage section 240B can be realized by the terminal storage section 22 of the terminal device 20B shown in FIG. 1 .

The first rendering processing unit 200A' responds to a distribution user rendering instruction from a distribution user information acquisition unit 222B described later, and renders a distribution image H21 for viewing users (see FIG. 13 ). In this case, the distribution user drawing instruction is generated based on the distribution user information (distribution user information generated by the first information generation unit 210A of the terminal device 20A) transmitted from the terminal device 20A on the content distribution side. The first rendering processing unit 200A' can generate, for example, a distribution image H21 for viewing users (see FIG. 13 ) at a frame rate corresponding to the update period of the distribution user information.

The first rendering processing unit 200A' includes a first rendering unit 201B, a second rendering unit 202B, and a third rendering unit 203B.

The first drawing unit 201B draws the image area (an example of the first image area) of the distribution avatar display layer J0 in the distribution image H21 (see FIG. 13 ) for viewing users. The distribution avatar display layer J0 in the distribution image H21 for viewing users is as described above. The first drawing unit 201B may draw the distributor in the image area of the distribution avatar display layer J0 based on the distribution user information and, for example, the drawing information of the avatar stored in the first storage unit 240A (refer to the table 700 in FIG. 16 ). Avatar M2. In this case, since the drawing information of the avatar stored in the first storage unit 240A is used, the distributor avatar M2 can be drawn by processing with a small load.

The operation of the first rendering unit 201B has the same function as that of the first rendering unit 201A of the terminal device 20A on the content delivery side described above. In this case, various information for drawing the distributor avatar M2 may also be included in the distribution user drawing instruction. That is, the distribution user drawing indication may include information of the distributor avatar M2 (such as avatar ID, face part ID, hairstyle part ID, clothing part ID, etc.), information representing the state of the distribution user (or representing the distribution based thereon information about the state of the avatar M2) and the like.

The second drawing unit 202B draws the image area (an example of the second image area) of the interface display layer J2 in the distribution image H21 (see FIG. 13 ) for viewing users. The operation of the second rendering unit 202B has the same function as that of the second rendering unit 202A of the terminal device 20A on the content delivery side described above. However, an operation area different from the interface display layer J2 in the distribution image H20 for distribution users (see FIG. 12 ) may be drawn on the interface display layer J2 in the distribution image H21 for viewing users. For example, as shown in FIG. 13 , the second drawing unit 202B may also draw an operation area G108 for presenting a gift to the distribution user, an operation area G109 for sending a cooperative distribution invitation to the distribution user, and the like. In addition, the cooperative distribution invitation can also be issued from the distribution user side. In this case, the operation area for sending the cooperative distribution invitation to the viewing user may also be drawn as a distribution image H20 for the distribution user (see FIG. 12 ). As a result, cooperative distribution can be promoted, and communication between users can be effectively promoted.

The third drawing unit 203B draws the image areas (an example of the third image area) of the remaining layers in the distribution image H21 (see FIG. 13 ) for viewing users. The operation of the third rendering unit 203B has the same function as that of the third rendering unit 203A of the terminal device 20A on the content delivery side described above. However, the content of the drawing in the auxiliary information display layer J5 may be appropriately different. For example, as shown in FIG. 13 , the viewer avatar M1 (see FIG. 12 ) may not be drawn on the distribution image H21 for the viewing user.

The second drawing processing unit 200B draws the above-mentioned homepage image H1 (see FIG. 6 ) (an example of an image in the second virtual space). The second rendering processing unit 200B generates the homepage image H1 (see FIG. 6 ) at a predetermined frame rate, for example.

The second rendering processing unit 200B includes a fourth rendering unit 204B and a fifth rendering unit 205B. The second rendering processing unit 200B may be substantially the same as the second rendering processing unit 200B of the terminal device 20A. Thus, an efficient configuration using common hardware resources can be realized.

The second information generating unit 210B generates various viewing user information related to the viewing user. The various viewing user information includes information indicating the status of the viewing user used by the second user interface unit 250B described later (the status of the viewing user detected by the second detection unit 211B). In addition, various viewer information includes what is necessary for generating gift drawing instructions, viewing user drawing instructions, comment drawing instructions, cooperation approval/disapproval instructions, etc. information.

In this embodiment, the second information generation unit 210B includes a second detection unit 211B, a second voice information generation unit 212B, and a second text information generation unit 213B.

The second detection unit 211B detects the state of the viewing user (the user's orientation, position, movement, or the like). The second detection unit 211B may also detect the status of the viewing user through the above-mentioned input unit 24 .

The second detection unit 211B may detect the state of the viewing user at a cycle corresponding to, for example, the frame rate of the homepage image H1 while the homepage image H1 is being drawn. Also, the second detection unit 211B may detect the status of the viewing user at a cycle corresponding to, for example, an update cycle of the distribution user information while the viewing user distribution image H21 (see FIG. 13 ) is being drawn.

The second audio information generation unit 212B generates audio information based on the utterance of the viewing user. For example, the second voice information generation unit 212B generates voice information of the viewing user through the microphone of the input unit 24 or the like. In addition, the second voice information generation unit 212B may process the voice data obtained through the input unit 24 to generate voice information of the viewing user. In this case, the processing method may be selected by the viewing user, or the processing of the voice data may be manually performed by the viewing user.

The second voice information generation unit 212B can switch the on/off state according to, for example, an operation of the viewing user. For example, when the viewing user operates a mute button (not shown) that can be displayed on the interface display layer J2 in the distribution image H21 (see FIG. 13 ) for the viewing user, the second input detection unit described later can 251B detects this operation, and in response to a command from the second input processing unit 252B described later, the second voice information generating unit 212B is turned off.

The second text information generating unit 213B generates text information based on utterances and/or character input of the viewing user. For example, the second text information generation unit 213B generates text information when the viewer and the distribution user have a dialogue (chat, etc.).

The second text information generating section 213B can switch the ON/OFF state according to, for example, an operation of the viewing user. For example, when the viewing user operates a comment button (not shown in FIG. 13 ) that can be displayed on the interface display layer J2, the operation is detected by the second input detection unit 251B described later, and the input from the later-described In response to the command from the second input processing unit 252B, the second text information generating unit 213B turns on. Thereby, it is possible to increase the number of operable variations via the interface display layer J2, and it is possible to realize simplification (easiness of viewing) of the display on the interface display layer J2.

In this case, the second information generation unit 210B may also include in the viewing user information a set of information indicating the state of the viewing user (or information representing the state of the viewer avatar M1 based thereon). Information) (hereinafter, sometimes simply referred to as "status information of the viewer's avatar M1"), information of the viewer's avatar M1 (such as avatar ID, face part ID, hairstyle part ID, clothing part ID, etc.), appropriate voice message. In this case, the second information generation unit 210B may generate the viewing user information by adding a time stamp to dynamically changing information such as the status information and voice information of the viewer avatar M1. Thus, when the terminal device 20A on the content distribution side receives the viewing user information, it can draw the viewer avatar M1 in the user avatar display layer J1 of the distribution image H20 as described above based on the received viewing user information. .

The second communication unit 220B communicates with the server device 10 and other terminal devices 20 (for example, the terminal device 20A on the content distribution side). In this embodiment, the second communication unit 220B includes a distribution processing unit 221B and a distribution user information acquisition unit 222B.

The distribution processing unit 221B transmits the viewing user information generated by the second information generating unit 210B to the terminal device 20A on the content distribution side.

The distribution user information obtaining unit 222B obtains the distribution user information from the terminal device 20A on the content distribution side, and provides the above-mentioned distribution user drawing instruction to the first drawing processing unit 200A'. The distribution user information is generated by the distribution processing section 221A of the terminal device 20A on the content distribution side described above.

The second display unit 230B outputs the distribution image H20 and the home page image H1 generated by the first rendering processing unit 200A' and the second rendering processing unit 200B to the display unit 23 of the terminal device 20B. In addition, in the present embodiment, as described above, the display unit 23 is in the form of a head-mounted display as an example.

The above-mentioned avatar drawing information (see table 700 in FIG. 16 ) and the like are stored in the second storage unit 240B.

The second user interface unit 250B detects various inputs of the viewing user through the input unit 24, and executes processing corresponding to the various inputs. The second user interface unit 250B includes a second input detection unit 251B and a second input processing unit 252B.

The second input detection unit 251B detects various inputs made by the viewing user through the above-mentioned interface display layer J2. For example, the second input detection unit 251B detects an input through the interface display layer J2 drawn by the second drawing unit 202B and the fourth drawing unit 204B described above.

In this case, the second input detection unit 251B may be based on the state of the viewing user detected by the second detection unit 211B, the state of the interface display layer J2 (for example, the position of various operation areas G100, the smart phone small window area G120, the display status) to detect the input made by the viewing user through the interface display layer J2. For example, the second input detection unit 251B detects various operations of the viewing user on the above-mentioned plurality of planar operation areas G300 in the homepage image H1, or detects the viewing user's operation in the distribution image H21 (see FIG. 13 ) for the viewing user. Various operations on various operation areas (the operation area for sending invitations for cooperative distribution to distribution users, etc.).

The second input processing unit 252B executes various processes corresponding to various inputs detected by the second input detection unit 251B on the homepage image H1 (an example of various second processes). Various kinds of processing are arbitrary, and as described above, for example, processing of changing the layout of the user interface (for example, a plurality of planar operation areas G300), processing of moving the user to an arbitrary location or a specific location in the virtual space ( For example, the process of moving the viewing user to a place for viewing) and the like. In addition, various kinds of processing may be appropriately accompanied by various kinds of rendering processing performed by the second rendering processing unit 200B and transition of screens (for example, movement of virtual space). For example, the process for transitioning to a state where gift giving can be performed may be accompanied by rendering of a gift selection screen by the fourth rendering unit 204B of the second rendering processing unit 200B.

In addition, the second input processing unit 252B executes various processes corresponding to various inputs detected by the second input detection unit 251B in the distribution image H21 (see FIG. 13 ) for viewing users (an example of various second processes). ). Various kinds of processing are arbitrary, and as described above, for example, end processing of viewing specific content, voice input processing, processing of presenting a present to a distribution user, etc. may be possible. In addition, various processing may be appropriately accompanied by various rendering processing performed by the first rendering processing unit 200A' and transition of screens (for example, movement of virtual space). For example, the process of presenting a gift to the delivery user may be accompanied by rendering of notification information (notification information indicating receipt of a gift) by the third rendering unit 203B of the first rendering processing unit 200A'.

FIG. 18 is a schematic block diagram showing functions of the server device 10 .

The server device 10 mainly includes a communication unit 100 and a storage unit 110 .

The communication unit 100 can exchange various information (such as distribution user information, View user information, etc.) to communicate.

The storage unit 110 can store various information required for distribution and/or viewing of specific content by a distribution user.

Thus, according to this embodiment, since the distribution image H21 for viewing users has a plurality of layers as described above, it is possible to effectively reduce the processing load by layering the distribution image H21 for viewing users. For example, by drawing (updating) the image area in the frontal area when viewed from the user's point of view in the interface display layer J2, etc., with a higher frequency than other image areas, it is possible to effectively reduce the number of users who are viewing the image. The overall processing load when distributing the image H21 is used. In addition, by layering the distribution image H21 for viewing users, it is possible to effectively give the user a sense of distance as "space". In addition, by layering the distribution image H21 for viewing users, for example, it can be arranged at an appropriate distance according to the attribute of the image area of each layer, and it is possible to combine, for example, drawing of a user interface with high operability in a manner that does not cause dizziness easily. Easy-to-view operation area such as G300 and so on.

In addition, in this embodiment, the server device 10 does not have the above-mentioned rendering function of the home page image H1 and the distribution image H2. device 20B to achieve. However, part or all of the rendering function of the home page image H1 and/or the distribution image H2 may also be realized by the server device 10 . The same may be applied to functions other than the rendering function as appropriate.

Next, an example of operations performed in the virtual reality generation system 1 shown in FIG. 1 will be described with reference to the drawings following FIG. 19 .

FIG. 19 is a flowchart showing an example of operations up to the start of distribution of specific content among the operations of the terminal device 20A on the content distribution side performed in the virtual reality generation system 1 shown in FIG. 1 .

First, in step S190, a user who becomes a distribution user wears the terminal device 20A as a head-mounted display, and starts a virtual reality application. Thereby, the user is located in the home space (step S191), and can view the home page image H1. The user operates the operation area of the interface display layer J2 in the home page image H1 to move to the content creation space (step S192). When the distribution user moves to the content creation space, the distribution user avatar M2 is appropriately prepared for changing clothes (step S193), and then stands in front of the mirror 805 to watch the distribution image H20 (preparation image before distribution) (step S194 ). In addition, the distribution image H20 as a preparation image before distribution may include only the distribution avatar display layer J0 and the interface display layer J2, or may include only the distribution avatar display layer J0, the interface display layer J2, and the background layer J3. In addition, in the distribution image H20 that is a preparatory image before distribution, an operation area G107 for starting distribution may be drawn on the interface display layer J2 (see FIG. 11 ). In addition, in another embodiment, distribution may be started simultaneously with moving to the content creation space.

When the distributing user views the distributing image H20 and decides to start distributing, he operates the corresponding operation area G107 (refer to FIG. 11 ) in the interface display layer J2 to start distributing specific content including the distributing image H20 (step S195). In this way, when the distribution of the specific content starts, the interface display layer J2 in the home page image H1 of the terminal device 20B on the content viewing side draws the operation area G300 where the specific content can be selected (see FIGS. 6 and 7A ).

FIG. 20 is a flowchart showing an example of operations up to the start of viewing specific content among the operations of the terminal device 20B on the content viewing side performed in the virtual reality generation system 1 shown in FIG. 1 .

First, in step S200, a user who becomes a viewing user wears the terminal device 20B as a head-mounted display, and starts a virtual reality application. Thereby, the user is located in the home space (step S201), and can view the home page image H1. The user operates the operation area G300 of the interface display layer J2 in the home page image H1 to select desired specific content (step S202). In this way, the viewing user can start viewing specific content (step S203).

FIG. 21 is a diagram showing the specific content based on the distribution user in the operation of the terminal device 20A on the content distribution side, the terminal device 20B on the content viewing side, and the server device 10 performed in the virtual reality generation system 1 shown in FIG. 1 . A flow chart of an example of operations during distribution (that is, when a viewing user is viewing specific content).

In addition, in FIG. 21 , the operation performed by a terminal device 20A on the content distribution side is shown on the left, the operation performed by the server device 10 (one server device 10 here) is shown in the center, and the operation performed by the server device 10 (here, one server device 10 ) is shown on the right side. An action performed by the terminal device 20B on the content viewing side is shown.

In step S210, the distribution user performs a performance in front of the mirror 805 or the like. Thus, the terminal device 20A on the content distribution side generates distribution user information corresponding to performances and the like. The terminal device 20A on the content distribution side transmits such distribution user information to the server device 10 . In addition, as long as the correspondence between the transmitted (already transmitted) information and the time stamp based on the reference time is satisfied, such a condition is clarified in both the terminal device 20A on the content distribution side and the terminal device 20B on the content viewing side , the distribution user information may also be sent to the server device 10 by multiplexing each other in any multiplexing manner. If such a condition is satisfied, then when the terminal device 20B on the content viewing side receives the distribution user information, it can perform appropriate processing according to the time stamp corresponding to the distribution user information. Regarding the multiplexing mode, the distribution user information may be sent through different channels, or part of the distribution user information may be sent through the same channel. Channels may include time slots, frequency bands, and/or spreading codes, among others. In addition, the method of distributing video (specific content) using the reference time can also be realized by the method disclosed in Japanese Patent No. 6803485, which is incorporated herein by reference.

Next, in parallel with the action in step S210, in step S212, the terminal device 20A on the content distribution side continuously sends the distribution user information for drawing the distribution image H21 for the viewing user to the terminal device 20B on the content viewing side, Then, a distribution image H20 (see FIG. 12 ) for the distribution user is output to the terminal device 20A on the content distribution side.

The terminal device 20A on the content distribution side can perform actions in steps S210 and S212 in parallel with actions in steps S214 to S222 described below.

Next, in step S214, the server device 10 transmits (forwards) the distribution user information continuously transmitted from the terminal device 20A on the content distribution side to the terminal device 20B on the content viewing side.

In step S216 , the terminal device 20B on the content viewing side receives the distribution user information from the server device 10 and stores it in the storage unit 140 . In one embodiment, considering that the voice information may have a larger capacity than other information and/or the communication line may fail, the terminal device 20B on the content viewing side can temporarily store the distribution user information received from the server device 10 It is stored (cached) in the terminal storage unit 22 (see FIG. 1 ).

In parallel with receiving and storing such distribution user information, in step S218, the terminal device 20B on the content viewing side uses the distribution user information received and stored from the terminal device 20A on the content distribution side through the server device 10 to generate a user ID for viewing. Distribute image H21 and play specific content.

In parallel with the actions in the above steps S216 and S218, in step S220, the terminal device 20B on the content viewing side generates viewing user information, and sends the viewing user information to the content distribution side terminal device 20A through the server device 10. The viewing user information may be generated only when, for example, the viewing user performs an operation of presenting a gift.

In step S222, the server device 10 transmits (forwards) the viewing user information received from the terminal device 20B on the content viewing side to the terminal device 20A on the content distribution side.

In step S224 , the terminal device 20A on the content distribution side can receive viewing user information through the server device 10 .

In step S226, the terminal device 20A on the content distribution side can basically perform the same operations as in step S210. For example, the terminal device 20A on the content distribution side generates a gift drawing instruction based on the viewing user information received in step S224, and draws a corresponding gift item on the auxiliary information display layer J5 in the distribution image H20.

In this way, the process shown in FIG. 21 can be continuously executed until the distribution of the specific content by the distribution user is completed or until there are no more viewing users of the specific content.

In addition, in the example shown in FIG. 21 , the subject of execution of each process can be changed in various ways as described above. For example, the server device 10 may be used instead of the terminal device 20A to implement the process of generating the distribution image H20 for distribution users in the processing of step S212. In addition, the process of generating the distribution image H21 for viewing users in the process of step S218 can also be realized using the terminal device 20A and the server device 10 . In this case, in step S216, the data of the distribution image H21 for viewing users may be received instead of the distribution user information. In addition, it is also possible to use the server device 10 instead of the terminal device 20A to realize the process of drawing gift items on the auxiliary information display layer J5 in the distribution image H20 based on the viewing user information in the process of step S226.

(Modification)

Although in the above-mentioned embodiment, the form using the wearable device (head-mounted display) as the terminal device 20A, 20B has been described, the virtual reality generation system 1 according to the present invention may also be configured to use a mobile phone, a smartphone , tablet terminal, PC (Personal Computer, personal computer), etc. as the virtual world (virtual space on the Internet) of terminal device 20A (an example of a first terminal device), terminal device 20B (an example of a second terminal device) . Hereinafter, a modified example of the case of such a form will be described, but a settable scene (scene), switching of screens, and a form of a user interface (UI) will be described as a point of view of a case of configuring a virtual world. Regarding matters not covered in the following description, the description of the above-mentioned embodiment is applied within a technically reasonable range. In addition, in the following description, the user interface may be realized by the input unit 24 , the first user interface unit 250A, and/or the second user interface unit 250B in the above-mentioned embodiment.

First, the settable scenes (scenes) and the user interface in each scene in the case of configuring as a virtual world will be described.

For a scenario where each user watches together (hereinafter referred to as "shared viewing"), the user interface is preferably a hierarchical menu with operations such as play, stop, and language setting. That is, even in the case of shared viewing using one large screen, it is possible to independently operate the volume, subtitle language setting, play, stop, fast forward, etc., and it is preferable to set the authority for these operations in addition to direct operations. Stratification. As an example of the user interface, it can be set as follows, for example, [play video > list display > share viewing/private viewing > fix here/operate by all staff], [operate the screen during playback > (if authorized) Play/Stop/Fast Forward/Rewind/End].

For scenes such as giving an expression (emotional expression) to an avatar, when sharing viewing, since each user directs his or her gaze toward the screen, it is preferable not to choose the usual expression and expression, but to show the direction of the target in a simpler way. Quick setup (eg semi-automatic). As an example of the user interface, it can be set as follows, for example, [shared viewing screen > spectator mode > up, down, left, and right of the joystick becomes {smiley face symbol, heart symbol, clapping symbol, semi-automatic (semi-auto)}], [ Semi-automatic > auto-response as others react/cheers in video].

For the presentation mode in which the user acts as a lecturer to perform a presentation, it can be set to allow the lecturer among the users to watch the script or the material to be shared on the screen, the displayed user interface, etc., and not to other users. Displays menus and user interface (but shows avatar). As an example of the user interface, it may be set as follows, for example, [go to the next page/text of the current conversation/next image]. Since the user who becomes the presenter does not turn backward, operations and previews, prepared images, and text are displayed on the screen.

In the seated answer mode, that is, a scene where the user quickly answers only with options for quiz programs or online education, as an example of a user interface, for example, an answer to the options and a raise hand button may be displayed for each user. In order to avoid being too free, when designated as an organizer, it will be fixed in its user interface until certain conditions are met. Combinations of options that form questions and answers and include timers, points, number of coins bet, etc.

For the configuration mode of a scene such as arranging a building, as a hierarchy, it can be in the form of [configuration→store→outdoor→purchase (balance, settlement, loan)→confirm→location→direction→other users can/cannot operate...], but If it is set as a direct arrangement in the three-dimensional space, it can only correspond to the degree of "pick up and put down", so it is difficult to match the orientation. Furthermore, it is also difficult to perform subsequent attribute setting such as "whether other people can see, touch, and move". Therefore, as an example of the user interface, it can be set as follows, for example, [walk freely in the three-dimensional space→open the menu→shop]. No one else can see the menu, store. While displaying these contents, from other people's perspective, it looks like a magnifying glass symbol floating above the head, which can be understood as a state of "store" in the operation state of freezing. In the case where the arrangement is determined and there is an item arrangement that conflicts with others, the "planned construction site" is displayed to others, and if there are other users there, it becomes a waiting process. If it cannot be resolved even after waiting for a certain period of time, you can return to the reconfirmation screen, and when canceling, you can return one to the placement location and orientation.

Next, switching of screens and a user interface in the case of configuring a virtual world will be described.

With regard to the user's point of view, in the case of an operation performed only from one's own point of view, difficulty arises when trying to watch from a bird's-eye view or from a third person's point of view. Therefore, as switching of viewpoints, it is possible to enable viewing in an overhead state when [Overview] is selected on the user interface. As an example of the user interface, it can be set as follows, for example, [menu→drone shooting→confirm and enter the drone operation mode]. At this time, the avatar can be fixed, and the drone can be used to shoot from the sky with the joystick while making movements through facial expressions and hands.

For the layout view when capturing screenshots, a confirmation screen such as "Do you want to share?" is displayed in conjunction with the continuous shooting mode. As an example of the user interface, the following format can be set, for example, [Open the film → Arrange several pictures in the past Shooting and choose left and right / up and down to decide sharing / filter / layout (vertical length, horizontal length, square) / delete and other operations].

In the game view when the game is played in the virtual world space (In-Space-Game), judgments and score sheets according to the rules of the game can be displayed. In addition, information for assisting the game (for example, if the game is bowling, the bowling pins to be targeted, and if the game is fencing, the opponent's punches so far, etc.) may be displayed. On the other hand, depending on the game, the change of weapons according to the character, the display of the opponent's HP, and the contact with our team are information that other users do not want to see, so they can also be set to be hidden.

When the game is a vehicle operation, for example, a racing game, rankings, maps, and the like may be displayed. In the case of a racing game, the steering wheel operation is shift controlled, and if you choose to ride a car, the user interface is limited to a driving UI. When in this mode, it can be set that the handle controller is not used as a substitute for hands but as a substitute for steering wheel and accelerator/brake operations.

In addition to this, it is also possible, for example, to display views related to compatibility with smartphones. This is different from the so-called virtual smartphone display, and is related to the purchase decision of the smartphone. By displaying the user interface (smartphone UI) in such a manner that the three-dimensional background is hidden in the screen, it is easy to make a decision, etc. As a user interface, for example, how to use (swipe, touch, press and hold, drag, etc.) the touchpad and joystick installed on various joystick controllers can be defined according to the user's environment. In addition, including the handle controller in the above-mentioned racing game, the handle controller, touch pad, and joystick mentioned here may be real objects, or virtual controllers drawn in a virtual space.

As mentioned above, although embodiment of this invention was described in detail with reference to drawings, the specific structure is not limited to this embodiment, The design etc. of the range which do not deviate from the summary of this invention are also included in this invention.

Explanation of reference signs

1: Virtual reality generation system

3: Network

10: Server equipment

11: Server Communication Department

12: Server storage department

13: Server Control Department

20, 20A, 20B: terminal equipment

21: Terminal Communication Department

22: Terminal storage department

23: Display part

24: Input part

25: Terminal Control Department

30A, 30B: studio unit

140: Storage Department

200: Department of Communications

200A, 200A': the first rendering processing unit

201A, 201B: First Drawing Department

202A, 202B: the second drawing department

203A, 203B: The third drawing department

200B: Second rendering processing unit

204B: The fourth depiction department

205B: The fifth drawing department

210: Storage Department

210A: First information generation unit

210B: Second information generation unit

211A: The first detection department

211B: Second detection unit

212A: First voice information generation unit

212B: second voice information generation unit

213A: First text information generation unit

213B: Second text information generation unit

220A: First Department of Communications

220B: Second Department of Communications

221A, 221B: distribution processing department

222A: Viewing user information acquisition unit

222B: Distribution user information acquisition department

230A: first display unit

230B: second display unit

240A: the first storage unit

240B: Second storage unit

250A: first user interface unit

250B: second user interface unit

251A: first input detection unit

251B: second input detection unit

252A: first input processing unit

252B: Second input processing unit

Claims (27)

1. An information processing system, comprising:

a first detection unit that detects a state of a first user who distributes content of an image including a first virtual space;

a first drawing processing unit that draws, by a first terminal device related to the first user or a second terminal device related to a second user who views the content, an image of the first virtual space that can be viewed by the first user or the second user; and

a first storage unit that stores a first display medium corresponding to the first user,

the image of the first virtual space has a plurality of levels,

the first drawing processing unit includes:

a first drawing unit that draws a first image area of a layer related to the first display medium among the plurality of layers; and

A second drawing unit that draws a second image area of a layer related to the first user interface among the plurality of layers,

the first drawing unit changes the state of the first display medium in the first image area based on the state of the first user detected by the first detecting unit.

2. The information handling system of claim 1, wherein,

the first display medium is in the form of a character having a front direction,

the first detection unit detects user information of at least any one of a direction, a position, and an action of the first user in real space as a state of the first user,

the first drawing unit draws the first image area so as to realize at least one of the orientation, position, and motion of the character corresponding to the user information.

3. The information processing system according to claim 1 or 2, wherein,

the first drawing unit draws the first display medium semi-transparently.

4. An information processing system according to any one of claims 1 to 3, wherein,

the first display medium functions as an avatar of the first user,

The first drawing unit draws the first display medium disposed in the first virtual space based on an image reflected on a mirror provided in the first virtual space, thereby generating the first image area.

5. The information handling system of claim 4, wherein,

the first drawing processing unit further includes a third drawing unit that draws a third image region of at least one of a layer related to a background in the plurality of layers and a layer located closer to the first user viewpoint than the background and farther from the first user viewpoint than the second image region.

6. The information handling system of claim 5, wherein,

the third drawing unit includes information related to the second user in the third image area.

7. The information handling system of claim 6, further comprising:

a second storage unit that stores a second display medium corresponding to the second user,

the third drawing unit draws the second display medium as the information related to the second user.

8. The information handling system of claim 5, wherein,

The third drawing unit includes at least one of input information from the second user, guidance/notification information for the first user and/or the second user, and prop information including a gift item given to the first user in the third image area.

9. The information processing system according to any one of claims 1 to 8, further comprising:

a first input detection unit that detects an input by the first user through the first user interface, based on the state of the first user and the state of the second image area detected by the first detection unit; and

and a first input processing unit that performs various first processes corresponding to the input detected by the first input detecting unit.

10. The information handling system of claim 9, wherein,

the various first processes include at least any one of a distribution start process of the content, a distribution end process of the content, a voice input process, a process of collecting gifts, a drawing process of props corresponding to the first display medium and being capable of drawing, a selection/exchange process of props corresponding to the first display medium and being capable of drawing, a text input process, an agreement process to other users, a process of changing a value of a parameter of the content, and a process for shifting to a state in which these processes are capable of being executed.

11. The information processing system according to any one of claims 1 to 10, wherein,

the first drawing processing unit is implemented by the first terminal device, an external processing device communicably connected to the first terminal device, the second terminal device communicably connected to the first terminal device, a terminal device communicably related to the second user connected to the first terminal device and different from the second terminal device, or any combination of these devices.

12. The information processing system according to any one of claims 1 to 11, wherein,

the first virtual space is cylindrical or celestial sphere.

13. The information processing system according to any one of claims 1 to 12, further comprising:

a second rendering processing unit that renders, by the first terminal device or the second terminal device, an image of a second virtual space that can be viewed by the first user or the second user,

the image of the second virtual space has a plurality of levels,

the second drawing processing unit includes:

a fourth drawing unit that draws a fourth image area of a layer related to the second user interface among the plurality of layers; and

A fifth drawing unit that draws a fifth image area of a layer located on a rear side of the fourth image area when viewed from a viewpoint for the first user or the second user among the plurality of layers,

the fourth drawing unit draws a plurality of planar operation areas associated with the second user interface so as to be arranged along a spherical surface or a curved surface.

14. The information handling system of claim 13, wherein,

the fourth drawing unit draws at least a part of the plurality of planar operation areas in a plurality of columns along a first curved surface around a predetermined reference axis extending in the up-down direction of the second virtual space, based on the predetermined reference axis, at a position corresponding to the position of the second user in the second virtual space.

15. The information handling system of claim 14, wherein,

the fourth drawing unit draws the other portions of the plurality of planar operation regions in a plurality of rows along a second curved surface that is offset rearward from the first curved surface around the predetermined reference axis when viewed from the viewpoint for the first user or the second user.

16. The information handling system of claim 15, further comprising:

A second detection unit that detects a state of a second user who enters the second virtual space;

a second input detection unit that detects an input of the second user through the second user interface based on the state of the second user and the state of the fourth image area detected by the second detection unit; and

and a second input processing unit that executes various second processes corresponding to the input detected by the second input detecting unit.

17. The information handling system of claim 16, wherein,

the various second processes include at least any one of a process of changing a configuration of the second user interface, a process of moving the first user to an arbitrary place or a specific place of the first virtual space, a distribution start process of the content, a viewing end process of the content, a voice input process, a gift giving process, a drawing process of an avatar prop, a selection/exchange process of an avatar prop, a text input process, and a process for shifting to a state in which any of these processes can be performed.

18. The information handling system of claim 17, wherein,

The specific location includes a location where the first user can select a prop that corresponds to the first display medium and can be depicted.

19. The information handling system of claim 17 or 18, wherein,

the operation region along the second curved surface is drawn so as to overlap with a rear side when viewed from the viewpoint for the second user with respect to the operation region drawn along the first curved surface,

the second input processing unit causes the fourth drawing unit to change the arrangement of the second user interface so that the operation region drawn along the first curved surface is along the second curved surface and the operation region drawn along the second curved surface is along the first curved surface based on the predetermined input detected by the second input detecting unit.

20. The information handling system of any of claims 16 to 19, wherein,

the various second treatments correspond to the plurality of planar operating areas.

21. The information handling system of any of claims 13 to 20, wherein,

the fifth image area includes an image area of a layer related to a background in the plurality of layers, and an image area including information related to at least a part of the plurality of planar operation areas and being an image area of a layer of the plurality of layers located closer to a viewpoint for the first user or the second user than the background.

22. The information handling system of any of claims 13 to 21, wherein,

the second drawing processing unit is implemented by the first terminal device, an external processing device communicably connected to the first terminal device, the second terminal device communicably connected to the first terminal device, a terminal device communicably related to the second user connected to the first terminal device and different from the second terminal device, or any combination of these devices.

23. The information handling system of any of claims 13 to 22, wherein,

the second virtual space is cylindrical or celestial sphere.

24. An information processing method executed by a computer, comprising:

a first detection step of detecting a state of a first user who distributes content of an image containing a first virtual space;

a first rendering processing step of rendering an image of the first virtual space viewable by the first user or the second user through the first terminal device related to the first user or the second terminal device related to the second user viewing the content; and

A first storage step of storing a first display medium corresponding to the first user,

the image of the first virtual space has a plurality of levels,

the first drawing processing step includes:

a first rendering step of rendering a first image area of a layer related to the first display medium in the plurality of layers; and

a second rendering step of rendering a second image area of the first user interface-related layer of the plurality of hierarchical layers,

in the first drawing step, the state of the first display medium in the first image area is changed based on the state of the first user detected by the first detection step.

25. The information processing method according to claim 24, further comprising:

a second rendering processing step of rendering, by the first terminal device or the second terminal device, an image of a second virtual space that can be viewed by the first user or the second user,

the image of the second virtual space has a plurality of levels,

the second drawing processing step has:

a fourth rendering step of rendering a fourth image area of a second user interface-related layer of the plurality of layers; and

A fifth drawing step of drawing a fifth image area of a layer located on a rear side of the fourth image area when viewed from a viewpoint for the first user or the second user among the plurality of layers,

in the fourth drawing step, a plurality of planar operation regions associated with the second user interface are drawn in a manner of being arranged along a spherical surface or a curved surface.

26. An information processing program that causes a computer to execute:

a first detection function that detects a state of a first user who distributes content of an image containing a first virtual space;

a first rendering processing function of rendering an image of the first virtual space viewable by the first user or the second user through the first terminal device related to the first user or the second terminal device related to the second user viewing the content; and

a first storage function of storing a first display medium corresponding to the first user,

the image of the first virtual space has a plurality of levels,

the first rendering processing function has:

a first rendering function of rendering a first image area of the first display medium-related layer of the plurality of layers; and

A second rendering function rendering a second image area of the first user interface-related layer of the plurality of layers,

the first drawing function changes the state of the first display medium in the first image area based on the state of the first user detected by the first detection function.

27. The information processing program according to claim 26, wherein,

causing the computer to further perform: a second rendering processing function of rendering, by the first terminal device or the second terminal device, an image of a second virtual space that can be viewed by the first user or the second user,

the image of the second virtual space has a plurality of levels,

the second rendering processing function has:

a fourth rendering function of rendering a fourth image area of a second user interface-related layer of the plurality of layers; and

a fifth drawing function of drawing a fifth image area of a layer located on a rear side of the fourth image area when viewed from a viewpoint for the first user or the second user among the plurality of layers,

the fourth drawing function draws a plurality of planar operation areas associated with the second user interface in a manner of being arranged along a spherical surface or a curved surface.

Images