[go: up one dir, main page]

WO2025041486A1 - Dispositif de commande d'affichage, procédé de commande d'affichage et programme de commande d'affichage - Google Patents

Dispositif de commande d'affichage, procédé de commande d'affichage et programme de commande d'affichage Download PDF

Info

Publication number
WO2025041486A1
WO2025041486A1 PCT/JP2024/025553 JP2024025553W WO2025041486A1 WO 2025041486 A1 WO2025041486 A1 WO 2025041486A1 JP 2024025553 W JP2024025553 W JP 2024025553W WO 2025041486 A1 WO2025041486 A1 WO 2025041486A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
content
display control
virtual space
control device
Prior art date
Application number
PCT/JP2024/025553
Other languages
English (en)
Japanese (ja)
Inventor
智彦 後藤
悌二郎 林
浩司 山口
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2025041486A1 publication Critical patent/WO2025041486A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams

Definitions

  • This disclosure relates to a display control device, a display control method, and a display control program for controlling display processing in virtual reality.
  • VR Virtual Reality
  • AR Augmented Reality
  • a technology related to the visual representation of virtual space is known that improves user convenience by avoiding inconveniences that arise when switching viewpoints when a user uses a device such as an HMD (head mounted display) (for example, Patent Document 1).
  • HMD head mounted display
  • An advantage of a virtual space constructed in three dimensions is that the user can move freely around the virtual space. However, if the user moves freely around the virtual space, it becomes impossible for the content provider to predict where the user will go, and the content provider may not be able to provide the user with the video experience they desire. For example, if there is content such as live-action video provided at a certain point in the virtual space, the user may not be able to view the video with an appropriate angle of view and distance, which may cause the user to feel uncomfortable.
  • this disclosure proposes a display control device, a display control method, and a display control program that can provide users with the experience intended by the provider in a virtual space.
  • a display control device includes an acquisition unit that acquires content placement information relating to a user's position in a virtual space and information relating to the placement of content provided in the virtual space, the content placement information including information relating to a first position set for viewing the content and a first area associated with the first position, a determination unit that determines whether the user is located within the first area based on the acquired information, and an adjustment unit that adjusts the relative position of the content and the user in the virtual space based on the result of the determination.
  • FIG. 1 is a diagram showing an overview of a display control system according to an embodiment.
  • FIG. 1 is a diagram for explaining a virtual space according to an embodiment
  • FIG. 2 is a diagram for explaining a virtual space according to the embodiment
  • FIG. 2 is a diagram for explaining coordinate axes of a virtual space.
  • FIG. 1 is a diagram for explaining positional relationships in a virtual space.
  • FIG. 2 is a diagram for explaining positional relationships in a virtual space.
  • FIG. 1 is a diagram for explaining an adjustment process according to an embodiment;
  • FIG. 11 is a diagram (2) for explaining the adjustment process according to the embodiment.
  • FIG. 1 is a diagram illustrating an example of the configuration of a display control device according to an embodiment.
  • FIG. 13 is a diagram showing an example of a data table related to a live-action video content storage unit;
  • FIG. FIG. 4 is a diagram showing an example of a data table related to an audio content storage unit.
  • 1 is a flowchart (1) showing the flow of a display control process according to the embodiment.
  • 11 is a flowchart (2) showing the flow of the display control process according to the embodiment.
  • 11 is a flowchart (3) showing the flow of the display control process according to the embodiment.
  • FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the display control device.
  • Embodiment 1-1 Overview of display control system according to embodiment 1-2. Overview of display control process according to embodiment 1-3. Configuration of display control device according to embodiment 1-4. Processing procedure according to embodiment 1-5. Modification 2. Other embodiments 3. Effects of display control device according to the present disclosure 4. Hardware configuration
  • Fig. 1 is a diagram showing an overview of a display control system 1 according to an embodiment.
  • the display control system 1 includes an HMD 20 and a display control device 100.
  • the display control system 1 is a so-called VR system that constructs a virtual space and reflects various actions of a user 10 wearing the HMD 20 in the virtual space.
  • the display control device 100 is, for example, an information processing terminal for realizing VR or AR technology.
  • the display control device 100 displays a virtual space constructed within a computer or network, called the Metaverse, on a display unit (display, etc.) such as an HMD 20.
  • the display control device 100 also displays a user 10 (for example, an avatar, which is a character that resembles the user) in the virtual space.
  • the display control device 100 is realized by any information processing device, for example, a PC (Personal Computer) or a cloud server.
  • the HMD 20 is an information processing device that allows the user 10 to experience the virtual space constructed by the display control device 100 and handles input to reflect the actions of the user 10 in the virtual space.
  • the HMD 20 has various functions for the user to use the virtual space, such as a self-posture estimation function, various motion sensors (acceleration sensors, gyro sensors, etc.), and a three-dimensional image display function.
  • the display control system 1 can be modified or expanded in various ways.
  • the user 10 may use a specific input device (controller, etc.) that cooperates with the HMD 20 to input to the virtual space.
  • the display control device 100 may display the virtual space on a smartphone, tablet device, or the like, rather than on a display device with a two-screen monitor capable of binocular stereoscopic vision like the HMD 20.
  • the display control system 1 may use a three-dimensional graphic engine on the cloud to display the virtual space, or may use a graphic engine built into the display control device 100.
  • the display control system 1 may store data related to the virtual space in a cloud storage, or may store data in a local device such as the display control device 100.
  • the display control system 1 may also have a content execution function such as a video, a game execution function using a specific game engine, or the like.
  • the display control system 1 shown in FIG. 1 allows a user 10 to move freely around the virtual space, interact with other users in the virtual space, watch content such as videos provided in the virtual space, and play games.
  • the viewpoint and sound source may move suddenly compared to a real space (hereinafter referred to as "real space"), which may cause the user 10 to feel sick or have an uncomfortable experience.
  • real space a real space
  • the video and sound sources are placed at predetermined positions in the virtual space, but the user 10 can move freely around the virtual space, so the user 10 may not be located in a position suitable for viewing as intended by the provider.
  • two-dimensional video content is provided at a certain point in the virtual space, if the user does not view the video from an appropriate angle, the video may be distorted or the user may not understand what kind of video is being shown, and the user may not be able to experience the original experience.
  • a sound source corresponding to the content is placed at a certain point in the virtual space, if the user moves beyond the sound source, the localization of the sound changes suddenly, causing the user to feel uncomfortable.
  • the display control device 100 realizes the above processing by the following configuration.
  • the display control device 100 acquires information on the position of the user 10 in the virtual space and the placement of the content provided in the virtual space (hereinafter referred to as "content placement information").
  • the content placement information includes information on the position set for viewing the content (hereinafter referred to as "optimum viewing position").
  • the optimum viewing position is a position in the virtual space that is set in advance by the content provider as the most suitable position for viewing the content. For example, when the content is a two-dimensional live-action video content, the optimum viewing position is a position that is at an angle directly facing the content and is a predetermined distance away from the content so that the image is displayed at the maximum angle of view for the user 10.
  • the optimum viewing position may be changed as appropriate depending on the performance (display angle of view, etc.) of the device such as the HMD 20 used by the user 10.
  • the content placement information includes information on the area associated with the optimum viewing position (hereinafter referred to as "viewing area").
  • the viewing area is, for example, an area within a specified distance range centered on the optimal viewing position, and may be any shape.
  • the display control device 100 determines whether the user 10 is located within the viewing area based on the position of the user 10 and the content placement information. If the user 10 is located within the viewing area, the display control device 100 adjusts the relative position between the position of the user 10 and the content. Specifically, the display control device 100 performs an adjustment to automatically move the position of the user 10 closer to the optimal viewing position so that the user 10 can view the content in the optimal position. This allows the user 10 to move to the content viewing position without feeling uncomfortable, and therefore allows the user to enjoy the content without feeling sick or unnatural. In other words, the display control device 100 can naturally guide the user 10 to a position and angle of view where the video can be viewed directly in front of them, and therefore allows the user to be provided with an experience in the virtual space that is exactly as intended by the provider.
  • Fig. 2 is a diagram (1) for explaining a virtual space according to an embodiment.
  • FIG. 2 shows a virtual space 30 that mimics a real-world concert hall.
  • a user 10 can experience content that has been created by filming and recording a music concert that was held in the real-world concert hall.
  • a music concert held in advance in real space is filmed and recorded from multiple positions, and the filming and recording locations correspond to, for example, positions 40, 41, 42, 43, 44, 45, and 46 shown in FIG. 2.
  • the content provider Based on the data obtained from such filming and recording, the content provider creates content (a group of video and audio data) for the music concert.
  • the content provider installs a camera and microphone at position 40 and the like to record the music concert.
  • positions 40 to 46 are merely examples, and filming and recording may be performed in real space at positions other than those shown at positions 40 to 46.
  • the recorded data may be two-dimensional images taken with a normal camera, omnidirectional images taken with a camera capable of filming a 360-degree space, or stereo images taken with a twin-lens camera.
  • the recorded data does not necessarily have to be accompanied by video, and may be only audio data picked up by a microphone installed at a predetermined position.
  • FIG. 3 shows how content is provided in the virtual space 30 based on data obtained from the real space.
  • FIG. 3 is a diagram (2) for explaining the virtual space according to the embodiment.
  • content 51A which is an image corresponding to the line of sight
  • content 52A which is an image corresponding to the line of sight
  • content 53A which is an image corresponding to the line of sight
  • content 54A which is an image corresponding to the line of sight
  • content 55A which is an image corresponding to the line of sight
  • content 56A which is an image corresponding to the line of sight
  • content 57A which is an image corresponding to the line of sight
  • omnidirectional content captured by an omnidirectional camera may be placed at position 50.
  • the user 10 can view each piece of content at the position where the content is placed.
  • the position of each piece of content within the virtual space 30 is set according to the x, y and z axes defined in the virtual space.
  • the content is a two-dimensional image
  • information indicating the angle of the image is set in the content.
  • Figure 4 shows the relationship between the coordinate axes and the virtual space.
  • Figure 4 is a diagram for explaining the coordinate axes of the virtual space.
  • virtual space 30 is given an x-axis in the horizontal direction of the space, a y-axis in the depth direction of the space, and a z-axis in the vertical direction of the space.
  • content 60 is placed in virtual space 30, content 60 is given coordinates on the xyz axes.
  • Content 60 is, for example, two-dimensional video content such as content 51A shown in FIG. 3.
  • Figure 5 is a diagram (1) for explaining the positional relationship in virtual space.
  • the optimum viewing position 62 is determined by the coordinate position (x, y, z) in the virtual space 30 and the direction ( ⁇ , ⁇ , ⁇ ) relative to the content.
  • a viewing area 61 is set for content 60.
  • the viewing area 61 is, for example, a space defined with a predetermined size that includes the optimum viewing position 62.
  • the viewing area 61 is defined by the coordinates of eight end points (abcd-efgh), i.e., coordinate data indicating a closed space set to surround the optimal viewing position 62.
  • the size and shape of the viewing area 61 are not limited to the example shown in FIG. 5, and any size and shape may be given.
  • the viewing area 61 does not have to be defined by the content creator or the virtual space provider, and may be set automatically according to predetermined rules.
  • the viewing area 61 may be defined as a space within a predetermined distance centered on the optimal viewing position 62.
  • FIG. 6 shows a plan view (viewed from the z-axis direction) of the viewing area 61 shown in FIG. 5.
  • FIG. 6 is a diagram (2) for explaining the positional relationship in the virtual space.
  • the optimal viewing position 62 is a position directly facing the content 60, and the viewing direction is determined by the pitch, yaw, and roll values. Note that, if a position corresponding to the camera installation position (i.e., the shooting position) when the video corresponding to the content 60 was shot is recorded, the shooting position may be automatically set as the optimal viewing position 62.
  • display control device 100 automatically adsorbs the user's position to optimal viewing position 62. This allows user 10 to view content provided in virtual space 30 at the optimal position intended by the creator of the content.
  • display control device 100 automatically adjusts the field of view (viewing angle) along with the position of user 10 over a predetermined period of time, thereby preventing abrupt changes in viewpoint and the occurrence of 3D motion sickness, etc.
  • FIG. 7 shows content 60 arranged in virtual space 30, an optimal viewing position 62 corresponding to content 60, a viewing area 61, and a user position 65 which is the position of user 10 in virtual space 30.
  • display control device 100 determines whether user position 65 has entered viewing area 61 as a result of the movement of user 10 who moves freely in virtual space 30. For example, display control device 100 performs the determination process for each frame which is the drawing timing of the virtual space.
  • the display control device 100 determines that the user position 65 is included in the viewing area 61 (step S10), it automatically moves the user position 65 to the optimal viewing position 62 and adsorbs the user position 65 to the optimal viewing position 62. At this time, the display control device 100 also adjusts the viewpoint direction of the user 10 (for example, the orientation of the body of the user 10 in the virtual space) so that it coincides with the optimal viewing position 62.
  • the viewpoint direction of the user 10 for example, the orientation of the body of the user 10 in the virtual space
  • the display control device 100 when the user position 65 and the optimal viewing position 62 are farther apart than the maximum speed of movement, the display control device 100 reduces the distance between the user position 65 and the optimal viewing position 62 according to the maximum speed. Then, when the distance between the user position 65 and the optimal viewing position 62 becomes smaller than the minimum speed of movement, the display control device 100 determines that the user position 65 and the optimal viewing position 62 match, and completes the adsorption.
  • the display control device 100 uses an algorithm (so-called adsorption) that gradually increases the moving speed as it approaches the target optimal viewing position 62 and stops when it arrives at the destination, so that natural and stress-free movement for the user can be realized. Note that, although adsorption is given as an example of an algorithm for automatic user movement, automatic movement is not limited to this.
  • the display control device 100 can use any algorithm, such as simple uniform linear motion, as the algorithm for automatic movement.
  • FIG. 8 is a diagram (2) for explaining the adjustment process according to the embodiment.
  • the audio is given sound source position coordinate data set for each sound along the x, y and z axes defined in the virtual space 30. For example, if multiple microphones are installed when recording video, a sound source position corresponding to the sound is given in the virtual space 30 for each position corresponding to the microphone installation position in the real space.
  • the audio data group may be virtual audio that reproduces the sound at a virtual sound source position using three-dimensional audio processing.
  • the direction of the audio may suddenly change when the user moves close to the position of the sound source, which may cause the user to feel uncomfortable or uneasy.
  • the display control device 100 does not automatically move the user 10, but instead automatically moves the sound source. This eliminates the risk of the user 10 feeling uncomfortable, since the audio is localized to the user 10 even if the user 10 moves freely.
  • FIG. 8 shows a user position 65, a sound source 71, and a viewing area 70 that is an area associated with the sound source 71.
  • the sound source 71 and the viewing area 70 may be accompanied by video content (such as the content 60 shown in FIG. 6).
  • the display control device 100 determines whether the user position 65 has entered the viewing area 70 (step S20). If the display control device 100 determines that the user position 65 is included in the viewing area 70, it automatically moves the position of the sound source 71 to coincide with the user position 65.
  • the display control device 100 tracks the movement of the user position 65 (step S21). As long as the user position 65 is included in the viewing area 70, the display control device 100 moves the sound source 71 to coincide with the user position 65. Then, if the user position 65 moves outside the viewing area 70, the display control device 100 returns the sound source 71 to its original position coordinates in the virtual space 30. At this time, the display control device 100 may immediately return the sound source 71 to its original position, or may adjust it so that it gradually returns to its original position from the position immediately after the user position 65 leaves the viewing area 70.
  • the movement of the sound source position can be rephrased as the volume perceived by the user 10.
  • the process shown in FIG. 7 can be expressed, for example, by the following formula, assuming that the coordinate position corresponding to the viewing area 70 is Model(x,y,z), the volume at the sound source position is Svol, and the sound source position is Smin, and that the current position of the user position 65 is Pos and the volume at the current position is Vol as variables for each frame.
  • the display control device 100 makes the user 10 perceive the sound source and the user position 65 as being aligned by matching the volume at the sound source position with the volume at the user position 65. Also, when the user position 65 is no longer included in the viewing area 70, the display control device 100 makes the user 10 perceive the sound from the sound source at a volume calculated based on distance attenuation. In this way, the display control device 100 uses an algorithm that gradually increases the volume from the sound source as the user 10 approaches the sound source, and makes the sound source position the same as the user position 65 when the user 10 has come close to the sound source to a certain extent, thereby realizing natural and stress-free sound playback for the user 10.
  • the above-mentioned sound localization process is more effective, for example, in reproducing stereophonic sound.
  • 3D sound technology stereoophonic technology
  • one or more virtual sound sources are placed in the virtual space 30, and a sound signal at any point is synthesized using the sound signals from those sound sources.
  • the virtual sound source is represented as a point in the virtual space 30, and the volume reproduced at the viewing position differs depending on the relative distance and relative direction from the sound source position.
  • the sound source position is a certain distance away from the viewing position (i.e., the position of the user 10), but if the user 10 is close to the sound source position or tries to pass through the sound source position, the relative distance between the sound source and the user position becomes very small. In that case, the user 10 may perceive a very loud sound or the direction of the sound may change suddenly due to a sudden change in the relative direction, which may cause great discomfort.
  • the display control device 100 when the user 10 approaches the sound source more than a certain distance away (when the user 10 enters the viewing area), the sound source is localized to the user 10, thereby preventing the above-mentioned unnatural sound image localization from occurring.
  • the display control device 100 can allow the user 10 to enjoy the content in an optimal state by automatically adjusting the user's position and the sound source position in the virtual space 30.
  • Fig. 9 is a diagram showing an example of the configuration of the display control device 100 according to an embodiment.
  • the display control device 100 has a communication unit 110, a storage unit 120, and a control unit 130.
  • the display control device 100 may also have an input unit (such as a keyboard or a touch panel) that accepts various operations from a user 10 who uses the display control device 100.
  • the communication unit 110 is realized, for example, by a NIC (Network Interface Card) or a network interface controller.
  • the communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the HMD 20, etc., via the network N.
  • the network N is realized, for example, by a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), or LPWA (Low Power Wide Area).
  • the storage unit 120 is realized, for example, by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.
  • a semiconductor memory element such as a random access memory (RAM) or a flash memory
  • a storage device such as a hard disk or an optical disk.
  • the storage unit 120 stores various information related to the display control process according to the embodiment.
  • the storage unit 120 includes a live-action video content storage unit 121, an audio content storage unit 122, and a 3DCG content storage unit 123.
  • the live-action video content storage unit 121 stores information about live-action video content recorded in a real space corresponding to the virtual space 30. Various live-action video content is placed in the virtual space 30. By viewing the live-action video content within the virtual space 30, a user 10 who uses the virtual space 30 can experience a music concert or the like held in the real space as if it were taking place in the virtual space 30.
  • FIG. 10 is a diagram showing an example of a data table 121A related to the live-action video content storage unit 121.
  • data table 121A stores the optimal viewing position (cam(x, y, z)), the direction of the optimal viewing position ( ⁇ , ⁇ , ⁇ ), and the viewing area (coordinates abcd-efgh) for each live-action video content (video1, video2, etc.).
  • the audio content storage unit 122 stores information about audio content based on audio recorded together with video in a real space corresponding to the virtual space 30, virtually created audio, etc. Various types of audio content are placed in the virtual space 30, just like live-action video content.
  • FIG. 11 is a diagram showing an example of a data table 122A related to the audio content storage unit 122.
  • data table 122A shows that audio content (audio1, etc.) is stored in association with the sound source position (source(x, y, z)) and the viewing area (coordinates abcd-efgh).
  • the 3DCG content storage unit 123 stores various data for constructing the virtual space 30.
  • the control unit 130 is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU, etc., executing a program stored inside the display control device 100 (for example, a display control program related to the present disclosure) using a RAM or the like as a working area.
  • the control unit 130 is also a controller, and may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).
  • control unit 130 has an acquisition unit 131, a determination unit 132, an adjustment unit 133, and an output control unit 134.
  • the acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires the user's position in the virtual space 30. The acquisition unit 131 also acquires content placement information, which is information about the placement of content provided in the virtual space 30 and includes information about an optimal viewing position set for viewing the content and a viewing area associated with the optimal viewing position.
  • content placement information is information about the placement of content provided in the virtual space 30 and includes information about an optimal viewing position set for viewing the content and a viewing area associated with the optimal viewing position.
  • the acquisition unit 131 also acquires, as content, information on video content or audio content captured or recorded in the real world corresponding to the virtual space 30. Specifically, the acquisition unit 131 acquires, as content, content placement information on two-dimensional video. At this time, the content placement information includes, for example, information on the optimal viewing position 62, the viewing area 61, and the coordinates at which the content 60 is placed in the virtual space 30, as shown in FIG. 5. The acquisition unit 131 may also acquire, as content, content placement information on spherical video.
  • the acquisition unit 131 may acquire content placement information including information on the sound source position set for the audio and the viewing area associated with the sound source position. Furthermore, when the content is stereophonic content in which the sound source position includes three-dimensional information, the acquisition unit 131 acquires content placement information in which three-dimensional information (three-dimensional coordinate data) corresponding to the position where the sound source was recorded in the real world is associated as the sound source position. As shown in FIG. 8, the content placement information for such stereophonic content includes information such as volume information of the sound source 71, the viewing area 70, and the coordinates where the sound source 71 is placed in the virtual space 30.
  • the determination unit 132 determines whether or not the user 10 is located within the viewing area based on the information acquired by the acquisition unit 131. For example, when the virtual space 30 is three-dimensional, the determination unit 132 determines whether or not the position coordinates of the user 10 corresponding to the x, y, and z axes belong within the viewing area associated with each content.
  • the adjustment unit 133 adjusts the relative position between the content and the user in the virtual space 30 based on the result of the determination by the determination unit 132.
  • the adjustment unit 133 adjusts the relative position so that the optimal viewing position set for the two-dimensional image, which is the content, and the position of the user 10 are close to each other. For example, the adjustment unit 133 adjusts the relative position so that the user 10 and the two-dimensional image face each other directly.
  • the adjustment unit 133 may also adjust the relative position so that the angle of view at which the two-dimensional image is displayed is maximized with respect to the field of view of the user 10 in the virtual space 30.
  • the adjustment unit 133 may adjust the relative position so that the position corresponding to the position where the celestial sphere video was captured in the real world (optimum viewing position) and the user's position are close to each other.
  • the adjustment unit 133 may adjust the position of the audio in the virtual space 30 relative to the position of the user 10.
  • the adjustment unit 133 determines that the user 10 is located within the viewing area, it adjusts the localization of the sound to be closer to the user 10. Furthermore, when the adjustment unit 133 determines that the user 10 is located within the viewing area, it may move the localization of the sound in accordance with the movement of the user 10, and may adjust the localization of the sound to return to its original position when the user 10 leaves the viewing area. Note that the adjustment of the localization of the sound may be performed by adjusting the volume of the sound source, for example, as shown in FIG. 8.
  • the output control unit 134 controls the information (video, audio, etc.) output from the control unit 130 to be output to the HMD 20, etc.
  • the output control unit 134 outputs the virtual space image rendered as video content to the HMD 20, which is a display.
  • the output destination device such as the HMD 20 may be other information equipment such as a smartphone, a video output device such as a television, a built-in display of the display control device 100, etc.
  • Fig. 12 is a flowchart (1) showing the flow of a display control process according to the embodiment.
  • Fig. 12 shows the flow of the process when the content is a two-dimensional real-life image.
  • the display control device 100 loads content into the virtual space 30 (step S101).
  • the display control device 100 also sets initial values for the user 10 in the virtual space 30 (step S102).
  • the display control device 100 determines whether the user's position has changed (step S103). If there has been a change (step S103; Yes), the display control device 100 calculates the destination of the user 10 (step S104). After calculating the destination, or if there has been no change in the user's position (step S103; No), the display control device 100 determines whether the user's position belongs to an area (viewing area) that is adjusted to automatically move the user's position (step S105).
  • step S105 If the user position is within the viewing area (step S105; Yes), the display control device 100 adjusts the user position and direction as shown in FIG. 7 (step S106).
  • the display control device 100 After adjusting the user position, or after skipping step S106 if the user position is not within the viewing area (step S105; No), the display control device 100 renders live-action 2D video, which is the content (step S107). Furthermore, the display control device 100 renders the surrounding virtual 3D space (step S108). This causes the display control device 100 to finish processing one frame.
  • FIG. 13 is a flowchart (2) showing the flow of the display control process according to the embodiment. Note that steps S201 to S205 are the same as steps S101 to S105 shown in FIG. 12, and therefore a description thereof will be omitted.
  • step S205 If the user 10 is within the viewing area (step S205; Yes), the display control device 100 adjusts the user position to move to an optimal viewing position associated with the omnidirectional video (for example, the shooting position of the omnidirectional video) (step S206). Note that in the case of omnidirectional video, it is not necessary to specify the direction of the viewpoint at the optimal viewing position, and therefore, unlike step S106, the display control device 100 does not need to adjust the direction.
  • the display control device 100 After the automatic movement of the user position, or if the user 10 is outside the viewing area (step S205; No), the display control device 100 renders the spherical video content (step S207). Furthermore, the display control device 100 renders the surrounding virtual 3D space (step S208). This causes the display control device 100 to end the processing of one frame.
  • FIG. 14 is a flowchart (3) showing the flow of the display control process according to the embodiment. Note that steps S301 to S304 are the same as steps S101 to S104 shown in FIG. 12, and therefore a description thereof will be omitted.
  • the display control device 100 determines whether the user's position is within the area in which the sound source position is to be adjusted (step S305). If the user 10 is within the area (step S305; Yes), the display control device 100 automatically moves the sound source position toward the user's position (step S306). Note that if the user 10 is outside the area (step S305; No), the display control device 100 does not move the sound source, but keeps it in its original position.
  • the display control device 100 calculates the volume output from the sound source (the volume perceived by the user 10) based on the position after the sound source has been moved if the sound source position has been moved, or based on the original position if the sound source has not been moved (step S307).
  • the display control device 100 plays back the 3D sound at the calculated volume (step S308).
  • the display control device 100 draws the surrounding virtual 3D space (step S309). This marks the end of the processing for one frame by the display control device 100.
  • the display control device 100 may have sensors that detect various environmental information that the HMD 20 has.
  • the sensors include an outward-facing camera that captures the outside of the display control device 100, an inward-facing camera that captures the user, a microphone that can measure the user's voice and surrounding sounds, and the like.
  • the sensors may also have a function of detecting various information related to the user's movements, such as the orientation, inclination, movement, and movement speed of the user's body.
  • the sensors detect information related to the user's movements, such as information related to the user's head and posture, the movements of the user's head and body (acceleration and angular velocity), the direction of the field of view, and the speed of the viewpoint movement.
  • the sensors function as various motion sensors, such as a three-axis acceleration sensor, a gyro sensor, and a speed sensor, and detect information related to the user's movements. More specifically, the sensors detect at least one change in the position and posture of the user's head by detecting the respective components of yaw, pitch, and roll as the movement of the user's head.
  • content that is captured or recorded in real space is given as an example of content, but the content does not necessarily have to be live-action footage captured in real space.
  • the content may be footage that imitates live-action footage, or creative content such as animation.
  • each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure.
  • the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.
  • the display control device according to the present disclosure includes an acquisition unit (the acquisition unit 131 in the embodiment), a determination unit (the determination unit 132 in the embodiment), and an adjustment unit (the adjustment unit 133 in the embodiment).
  • the acquisition unit acquires content arrangement information, which is information about the position of the user in the virtual space and the arrangement of the content provided in the virtual space, and includes information about a first position (the optimal viewing position in the embodiment) set for viewing the content, and a first region (the viewing region in the embodiment) associated with the first position.
  • the determination unit determines whether or not the user is located in the first region based on the acquired information.
  • the adjustment unit adjusts the relative positions of the content and the user in the virtual space based on the result of the determination.
  • the display control device adjusts the user to automatically move to a position suitable for viewing the content. This allows the display control device to provide the user with the experience originally intended by the content provider.
  • the acquisition unit also acquires information about video content or audio content captured or recorded in the real world corresponding to the virtual space as content.
  • the display control device provides the user with video content and audio content that have been created by shooting from a point in the real world within a virtual 3D space that mimics real space.
  • the display control device can allow the user to view the content from a position and direction that best matches the appearance in the virtual space with the live-action video, thereby further enhancing the user's sense of immersion.
  • the acquisition unit also acquires content placement information regarding the two-dimensional image as the content.
  • the adjustment unit adjusts the relative position so that the first position set in the two-dimensional image and the user's position are close to each other. For example, the adjustment unit adjusts the relative position so that the user and the two-dimensional image face each other directly.
  • the adjustment unit may also adjust the relative position so that the angle of view at which the two-dimensional image is displayed is maximized with respect to the user's field of view in the virtual space.
  • the display control device automatically moves the user to the optimal viewing position. This allows the display control device to prevent the user from feeling discomfort when viewing an image from a position that is shifted from the optimal viewing angle, such as image distortion or a size that does not match the virtual space in the background.
  • the acquisition unit also acquires content placement information related to the omnidirectional video as the content.
  • the adjustment unit adjusts the relative position so that the user's position is close to a first position corresponding to the position where the omnidirectional video is captured in the real world.
  • the display control device automatically moves the user to the center of the celestial sphere (e.g., the shooting position), which is the optimal viewing position.
  • This makes it possible for the display control device to prevent the user from feeling discomfort, such as distortion of the spherical image or a missing part of the spherical image, when the user views the image at a position shifted from the center of the celestial sphere.
  • the display control device even if a user who moves freely in a virtual space recognizes that a spherical image is placed in the virtual space, he or she does not know where the optimal viewing position is.
  • the display control device when the user approaches the spherical image to a certain extent, the user is automatically moved to the optimal viewing position, so that the user can easily obtain a sense of immersion.
  • the acquisition unit acquires content placement information including information on a second position set for the sound (in this embodiment, the original sound source position defined for the sound source) and a second area associated with the second position (in this embodiment, the viewing area 70 related to the sound, etc.).
  • the determination unit determines whether or not the user is located within the second area.
  • the adjustment unit adjusts the position of the sound in the virtual space and the relative position of the user based on the result of the determination.
  • the display control device can adjust the relative position between the user and the sound source not only for video but also for audio, allowing the user to experience audio playback as intended by the content provider.
  • the adjustment unit determines that the user is located within the second area, it adjusts the audio localization to be closer to the user.
  • the adjustment unit determines that the user is located within the second area, it moves the audio localization in accordance with the movement of the user, and when the user leaves the second area, it adjusts the audio localization to return to the second position.
  • the acquisition unit may acquire content placement information regarding stereophonic content in which three-dimensional information corresponding to the position where the audio was recorded in the real world is associated as the second position.
  • the display control device moves the position of the sound source placed in the virtual space according to the distance from the sound source, thereby preventing sudden changes in the relative position and direction between the user and the sound source, thereby realizing natural sound playback for the user.
  • Fig. 15 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the display control device 100.
  • the computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600.
  • Each unit of the computer 1000 is connected by a bus 1050.
  • the CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400, and controls each component. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200, and executes processes corresponding to the various programs.
  • the ROM 1300 stores boot programs such as the Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000.
  • BIOS Basic Input Output System
  • HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs.
  • HDD 1400 is a recording medium that records a display control program related to the present disclosure, which is an example of program data 1450.
  • the communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet).
  • the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.
  • the CPU 1100 of the computer 1000 executes a display control program loaded onto the RAM 1200 to realize the functions of the control unit 130, etc.
  • the display control program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400.
  • the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via the external network 1550.
  • the present technology can also be configured as follows.
  • an acquisition unit that acquires content placement information that includes information regarding a user's position in a virtual space and a placement of content provided in the virtual space, the content placement information including information regarding a first position set for viewing the content and a first area associated with the first position; a determination unit that determines whether or not the user is located within the first area based on the acquired information; an adjustment unit that adjusts a relative position between the content and the user in the virtual space based on a result of the determination;
  • a display control device comprising: (2)
  • the acquisition unit is As the content, information regarding video content or audio content captured or recorded in the real world corresponding to the virtual space is obtained.
  • the display control device according to (1).
  • the acquisition unit is acquiring the content placement information regarding a celestial sphere image as the content;
  • the adjustment unit is when it is determined that the user is located within the first area, the relative position is adjusted so that the first position, which corresponds to a position at which the omnidirectional image is captured in the real world, and the position of the user are close to each other.
  • a display control device according to any one of (2) to (5).
  • the acquisition unit is If the content includes audio, acquiring the content placement information including a second position set for the audio and information about a second area associated with the second position;
  • the determination unit is determining whether the user is located within the second area;
  • the adjustment unit is adjusting a position of the sound in the virtual space relative to the user based on a result of the determination;
  • a display control device according to any one of (2) to (6).
  • the adjustment unit is When it is determined that the user is located within the second area, the localization of the sound is adjusted so as to be closer to the user.
  • the display control device according to (7) above.
  • the adjustment unit is When it is determined that the user is located within the second area, the localization of the sound is moved in accordance with the movement of the user, and when the user leaves the second area, the localization of the sound is adjusted so as to return to the second position.
  • the display control device according to (8).
  • the acquisition unit is acquiring the content placement information relating to the stereophonic content, in which three-dimensional information corresponding to a position where the sound was recorded in the real world is associated as the second position;
  • the display control device according to any one of (7) to (9).
  • the computer Acquire content placement information relating to a user's position in a virtual space and a placement of content provided in the virtual space, the content placement information including a first position set for viewing the content and information relating to a first area associated with the first position; determining whether the user is located within the first area based on the acquired information; adjusting a relative position between the content and the user in the virtual space based on a result of the determination;
  • a display control method comprising: (12) Computer, an acquisition unit that acquires content placement information, the content placement information including information on a user's position in a virtual space and a placement of content provided in the virtual space, the content placement information including information on a first position set for viewing the content and a first area associated with the first position; a determination unit that determines whether or not the user is located within the first area based on the acquired information; an adjustment unit that adjusts a relative position between the content and the user in the virtual space based on a result of the determination;
  • a display control program for causing the display control

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Un dispositif de commande d'affichage selon un mode de réalisation de la présente invention comprend : une unité d'acquisition pour acquérir la position d'un utilisateur dans un espace virtuel, et des informations d'agencement de contenu relatives à l'agencement de contenu fourni dans l'espace virtuel, les informations d'agencement de contenu comprenant des informations relatives à une première position définie pour la visualisation du contenu et à une première région associée à la première position ; une unité de détermination pour déterminer, sur la base des informations acquises, si l'utilisateur est positionné à l'intérieur de la première région ; et une unité d'ajustement pour ajuster la position relative entre le contenu et l'utilisateur à l'intérieur de l'espace virtuel sur la base du résultat de la détermination.
PCT/JP2024/025553 2023-08-23 2024-07-17 Dispositif de commande d'affichage, procédé de commande d'affichage et programme de commande d'affichage WO2025041486A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023135304 2023-08-23
JP2023-135304 2023-08-23

Publications (1)

Publication Number Publication Date
WO2025041486A1 true WO2025041486A1 (fr) 2025-02-27

Family

ID=94731592

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/025553 WO2025041486A1 (fr) 2023-08-23 2024-07-17 Dispositif de commande d'affichage, procédé de commande d'affichage et programme de commande d'affichage

Country Status (1)

Country Link
WO (1) WO2025041486A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009129183A (ja) * 2007-11-22 2009-06-11 Internatl Business Mach Corp <Ibm> 仮想空間を構築するための画像処理方法、装置
US20190057531A1 (en) * 2017-08-16 2019-02-21 Microsoft Technology Licensing, Llc Repositioning user perspectives in virtual reality environments
JP2019192175A (ja) * 2018-04-27 2019-10-31 株式会社コロプラ プログラム、情報処理装置、および方法
US20190377473A1 (en) * 2018-06-06 2019-12-12 Sony Interactive Entertainment Inc. VR Comfort Zones Used to Inform an In-VR GUI Editor
WO2021246183A1 (fr) * 2020-06-03 2021-12-09 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme
WO2023026519A1 (fr) * 2021-08-27 2023-03-02 ソニーグループ株式会社 Dispositif de traitement d'informations, terminal de traitement d'informations, procédé de traitement d'informations et support de stockage

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009129183A (ja) * 2007-11-22 2009-06-11 Internatl Business Mach Corp <Ibm> 仮想空間を構築するための画像処理方法、装置
US20190057531A1 (en) * 2017-08-16 2019-02-21 Microsoft Technology Licensing, Llc Repositioning user perspectives in virtual reality environments
JP2019192175A (ja) * 2018-04-27 2019-10-31 株式会社コロプラ プログラム、情報処理装置、および方法
US20190377473A1 (en) * 2018-06-06 2019-12-12 Sony Interactive Entertainment Inc. VR Comfort Zones Used to Inform an In-VR GUI Editor
WO2021246183A1 (fr) * 2020-06-03 2021-12-09 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme
WO2023026519A1 (fr) * 2021-08-27 2023-03-02 ソニーグループ株式会社 Dispositif de traitement d'informations, terminal de traitement d'informations, procédé de traitement d'informations et support de stockage

Similar Documents

Publication Publication Date Title
US20120207308A1 (en) Interactive sound playback device
JP2013250838A (ja) 情報処理プログラム、情報処理装置、情報処理システム、および情報処理方法
CN116471520A (zh) 音频装置和音频处理的方法
TW201928945A (zh) 音訊場景處理技術
US12299823B2 (en) Audiovisual rendering apparatus and method of operation therefor
JP2025032375A (ja) フィルタパラメータに関するインデックススキーミング
US20220036075A1 (en) A system for controlling audio-capable connected devices in mixed reality environments
JP4789145B2 (ja) コンテンツ再生装置及びコンテンツ再生プログラム
WO2025041486A1 (fr) Dispositif de commande d&#39;affichage, procédé de commande d&#39;affichage et programme de commande d&#39;affichage
CN102568535A (zh) 互动式声音录放装置
US20240406666A1 (en) Sound field capture with headpose compensation
US20250239023A1 (en) Audiovisual rendering apparatus and method of operation therefor
US20250239024A1 (en) Audiovisual rendering apparatus and method of operation therefor
JP2025106335A (ja) 視聴覚レンダリング装置およびその動作方法
BR122025005487A2 (pt) Aparelho de renderização audiovisual e método de renderização de itens audiovisuais

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24856180

Country of ref document: EP

Kind code of ref document: A1