JP2026013594A - Information processing system, program, and information processing method - Google Patents

Abstract

To naturally reproduce an object in a virtual space.
[Solution] An information processing system comprising: an acquisition unit that acquires the distance between the object and a reference plane set in real space, obtained from the movement of the object; and a generation unit that generates a map used to reproduce the object in virtual space, representing the undulations relative to the reference plane, based on the distance acquired by the acquisition unit.
[Selected Figure] Figure 3

Description

This disclosure relates to an information processing system, a program, and an information processing method.

In recent years, technologies have been developed to provide more realistic virtual spaces. For example, Patent Document 1 discloses technology for making images of the background, such as the ground, in a virtual space appear clearer.

In addition, there is also a growing trend to recreate objects in real space in virtual space. For example, objects are being recreated in virtual space based on motion data captured from the movements of objects in real space.

Japanese Patent Application Laid-Open No. 2002-92635

When generating a virtual space, it is sometimes necessary to reduce the processing load of modeling, etc. To reduce the processing load, undulating surfaces in the real world, such as the ground, may be reproduced as flat surfaces in the virtual space.

However, if an object moving on an uneven surface is reproduced exactly on a virtual surface that does not match the real space, such as a flat surface, the object may end up being reproduced unnaturally in the virtual space.

This disclosure therefore proposes new and improved technology that can naturally reproduce objects in virtual space.

According to the present disclosure, an information processing system is provided that includes an acquisition unit that acquires the distance between the object and a reference plane set in real space, obtained from the movement of the object, and a generation unit that generates a map that represents the undulations relative to the reference plane and is used to reproduce the object in virtual space, based on the distance acquired by the acquisition unit.

The present disclosure also provides a program that causes a computer to function as an acquisition unit that acquires the distance between a reference plane set in real space and the object, obtained from the object's movement, and a generation unit that generates a map that represents the undulations relative to the reference plane and is used to reproduce the object in virtual space, based on the distance between the object and the reference plane acquired by the acquisition unit.

The present disclosure also provides a computer-executed information processing method that includes: acquiring a distance between a reference plane set in real space and the object, obtained from the object's movement; and generating a map used to reproduce the object in virtual space, which represents the undulations relative to the reference plane, based on the acquired distance between the object and the reference plane.

10A and 10B are diagrams for explaining correction of the positions of the feet of a player in a virtual space. FIG. 10 is a diagram for explaining the undulations pseudo-represented by the measurement results of characteristic parts of the court and the actual undulations. FIG. 2 is a block diagram illustrating an example of a functional configuration of a server 10 according to an embodiment of the present disclosure. FIG. 10 is a diagram for explaining an example of an operation process of the server 10 according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a height map generated by setting the lowest player height among the player heights obtained on the block. FIG. 6 is a diagram showing an example of a height map generated by resetting the player heights set in a block B including outliers in the height map M1 shown in FIG. 5 . FIG. 10 is a diagram for explaining weights set to adjacent blocks B. FIG. 7 is a diagram illustrating an example of a height map obtained by smoothing the height map M2 shown in FIG. 6 . FIG. 10 is a diagram for explaining a virtual space generated by a spatial processing unit 122. FIG. 9 is a block diagram illustrating an example of a hardware configuration of an information processing device 900 that realizes the server 10 according to an embodiment of the present disclosure.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in this specification and drawings, components having substantially the same functional configuration will be assigned the same reference numerals, and redundant explanations will be omitted.

The explanation will be given in the following order.
1. Overview 2. Example of functional configuration 3. Operational processing 4. Example of hardware configuration 5. Supplementary information

<1. Overview>
The present disclosure relates to an information processing system for reproducing objects in real space in a virtual space. Examples of objects in real space reproduced by the information processing system according to the present disclosure include people (users) such as players playing sports like soccer or skateboarding, performers or spectators at dance, theater, or live music events, and moving objects such as robots or balls. Furthermore, the objects in real space reproduced by the information processing system according to the present disclosure may be parts of the above-mentioned people or objects. In this disclosure, the description will be centered on the assumption that soccer players are reproduced in a virtual space.

Note that there is no particular limit to the number of objects in real space that are reproduced in virtual space. This disclosure will mainly describe an example in which multiple players participating in a match are objects that are reproduced in virtual space.

Players are recreated in virtual space based on motion data captured from the player's movements in real space. The motion data may be obtained, for example, by analyzing images of the player captured by multiple cameras installed in real space.

The motion data may include the position of the player in real space. More specifically, the motion data may include each position of the player's skeleton. The motion data may also include information other than the position of the player in real space and each position of the player's skeleton as described above. Information such as each player's height and shoulder width may also be attached to the motion data as metadata.

The player's position may be on three-dimensional coordinate axes set in real space. The three-dimensional coordinate axes are calibrated and set when capturing an image of the subject or analyzing motion data. More specifically, two of the set three-dimensional coordinate axes may be set to align with a plane in real space (e.g., a surface that serves as a reference in real space, such as the ground, ceiling, or wall), and the remaining axis may be set in the height direction relative to the plane in real space. This disclosure will mainly describe an example in which two of the three-dimensional coordinate axes are set to align with a plane in real space where the player is competing (e.g., the ground of a soccer field, the surface of a baseball stadium, a wall when bouldering, etc.), and the remaining axis is set in the height direction relative to the plane in real space. Note that the plane in real space that serves as the reference for setting the three-dimensional coordinate axes is not limited and may be the ground, ceiling, wall, etc.

The motion data also includes the player's height. The player's height is represented by the distance between the player (more specifically, the player's feet) and a reference plane. The reference plane can be a plane specified by two of the three-dimensional coordinate axes set in real space. More specifically, the reference plane according to the present disclosure is a plane set along real space, specified by two axes set along the ground. Hereinafter, the player's height included in the motion data will also be referred to as "player height."

(Identifying issues)
The real space in which players play may have undulations. Here, a soccer court is taken as an example of a real space. Generally, outdoor soccer courts have a ground shape that rises around the center circle. Furthermore, the court may have undulations due to, for example, the thickness of the turf and the unevenness of the ground. Here, since the motion data is represented based on three-dimensional coordinate axes set in the real space, the player heights included in the motion data are represented taking into account the undulations of the court.

On the other hand, in order to reduce the processing load involved in generating a virtual space, it is possible to recreate undulating surfaces such as outdoor courts as flat surfaces in the virtual space.

In order to naturally recreate players moving around on an uneven court in a virtual space, it may be possible to correct the position of the players in the virtual space, or more specifically, the position of the players' feet.

Figure 1 is a diagram illustrating the correction of the position of a player's feet in virtual space. Figure 1 shows the player's feet F (F1 to F3) and the player's feet Fc (Fc1 to Fc3) after position correction in virtual space V.

The real-space corresponding surface RF shown in Figure 1 is a surface in virtual space V that corresponds to an undulating surface in real space. The reproduction surface VF shown in Figure 1 is a surface for reproducing the real-space corresponding surface as a flat surface in virtual space V. The reproduction surface VF can be a surface in virtual space V that corresponds to a reference surface in real space.

As shown in the top of Figure 1, if the position of a player's foot F is reproduced without correction, even if the player's foot F is in contact with an undulating surface in real space, the player's foot F will float above or sink into the reproduced surface VF due to the difference between the real-space corresponding surface RF and the reproduced surface VF. For example, the player's foot F1, which was in contact with an undulating surface in real space, will be reproduced floating above the reproduced surface VF by the difference h.

On the other hand, as shown in the lower part of Figure 1, after position correction, the player's foot Fc is reproduced with the position of the player's foot F corrected by the difference between the real-space corresponding surface RF and the reproduction surface VF. For example, after position correction, the player's foot Fc1 is reproduced with the position of the player's foot F1 corrected by the difference h between the real-space corresponding surface RF and the reproduction surface VF. As a result, the foot that corresponds to the player's foot F1 and that is in contact with the undulating surface in real space is also reproduced in the virtual space V so that it is in contact with the reproduction surface VF, making it possible to reproduce the player naturally in the virtual space V.

To perform such corrections, it is necessary to obtain the undulations of the undulating surface in real space.

Traditionally, maps showing the undulations of roadways and other areas are generated using depth information obtained by measuring machines equipped with infrared sensors, etc. However, because sports courts have many uneven surfaces, it is difficult to operate the measuring machines. In addition, the condition of the surface can change in a short period of time due to turf maintenance, wind, rain, etc. For example, the turf on soccer courts is generally replaced each season, so the condition of the surface changes with each season. However, operating the measuring machines every short period of time is operationally difficult.

It is also possible to obtain predicted values for the undulations of each point on the court by linearly interpolating other parts of the court based on the results of measuring the height of characteristic points on the court (for example, the corners of the court, the center of the center circle, etc.). However, there will be differences between the undulations represented by the measurement results of characteristic points on the court and the actual undulations due to the influence of local undulations, etc. Figure 2 is a diagram explaining the undulations represented by the measurement results of characteristic points on the court and the actual undulations.

Figure 2 shows the player's feet F (F1-F3), the real-space corresponding surface RF, and the pseudo-undulating surface MF in the virtual space V. The real-space corresponding surface RF is a surface in the virtual space V that corresponds to an undulating surface in real space. The pseudo-undulating surface MF is a surface in the virtual space V that corresponds to an undulating surface represented based on the measurement results of characteristic parts of the court.

As shown in Figure 2, there is a difference between the real-space corresponding surface RF and the pseudo-undulating surface MF, so even if the position of the player's foot F in the virtual space V is corrected according to the height of the pseudo-undulating surface MF, the object may still be reproduced unnaturally.

This disclosure therefore proposes a technology for generating a map that represents the undulations relative to a reference surface, i.e., the height of the court relative to the reference surface, which enables players to be reproduced naturally in virtual space.

<2. Example of functional configuration>
First, a functional configuration example of the server 10 for realizing an embodiment of the present disclosure will be described. Fig. 3 is a block diagram showing a functional configuration example of the server 10 according to an embodiment of the present disclosure. The server 10 is an information processing device that constitutes an information processing system according to the present disclosure. In this embodiment, an example in which the information processing system is constituted by a single server 10 will be described, but the information processing system may include multiple devices, and in this case, the functions of the server 10 described below may be realized by multiple devices.

As shown in FIG. 3, the server 10 in this embodiment includes a communication unit 110, a control unit 120, and a memory unit 130.

(Communication unit 110)
The communication unit 110 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from an external device. The communication unit 110 according to this embodiment may be communicatively connected to an external device or the Internet using, for example, a wired/wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), or the like.

The communication unit 110 functions as an acquisition unit that acquires motion data obtained from the movements of players (an example of a user) from an external device. The motion data can be acquired by analyzing captured images of the players taken by multiple cameras installed in real space. However, the method of acquiring the motion data is not particularly limited as long as it is possible to acquire the player's position and height. For example, a method such as OpenPose, which estimates the position of the player's joints from captured images, may be used, or the data may be acquired by a capture device worn by the player and configured with an inertial sensor or the like.

As explained above, the motion data includes the position of each bone structure of a player in real space and the player's height. The position of each bone structure of a player and the player's height are expressed based on preset three-dimensional coordinate axes. More specifically, the player's height is expressed by the distance between a reference plane, which is a plane set along the court on which the player plays, and the player's feet. The distance between the reference plane and the player's feet is the first distance in this embodiment.

For example, the communication unit 110 may acquire motion data for all players participating in a match at each time during the match. Furthermore, when the technology disclosed herein is applied to the reproduction in a virtual space of dance, theater, live music, or the like, motion data for all performers participating in the performance at each time during the performance may be acquired. The more motion data acquired in an area reproduced in virtual space, the more accurate the heights represented by the map generated by the map generation unit 121 (described below) will be.

(Control unit 120)
The control unit 120 functions as an arithmetic processing unit and a control device, and controls the overall operation of the server 10 in accordance with various programs. The control unit 120 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. The control unit 120 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as needed.

The control unit 120 also functions as a map generation unit 121 and a spatial processing unit 122.

(Map generation unit 121)
The map generation unit 121 is a generation unit that generates a map that represents the undulations relative to a reference plane based on the player heights included in the motion data acquired by the communication unit 110. The map generation unit 121 generates a map for an area that is reproduced in a virtual space.

In this embodiment, an example will be described in which such a map represents the height of each position on the court relative to a reference plane. Here, height represents the distance between the reference plane and each position on the court, and is expressed as a positive or negative value, with a vertical upward direction relative to the reference plane being positive. Hereinafter, such a map will also be referred to as a "height map." The process of generating a height map by the map generation unit 121 will be described in detail below.

(Spatial Processing Section 122)
The spatial processing unit 122 generates an image that reproduces the player in a virtual space based on the height map generated by the map generation unit 121 and the player's motion data acquired by the communication unit 110.

The spatial processing unit 122 functions as a processing unit that corrects the positions of players in the virtual space according to the undulations corresponding to the positions of players placed in the virtual space, as represented by the height map. Details of the player position correction process will be described later.

(Storage unit 130)
The storage unit 130 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 120, and a RAM that temporarily stores parameters that change as appropriate. For example, the storage unit 130 stores a height map generated by the map generation unit 121.

<3. Operation Processing>
Next, an example of the operation process of the server 10 according to an embodiment of the present disclosure will be described.

Figure 4 is a diagram illustrating an example of the operation processing of the server 10 according to an embodiment of the present disclosure. As shown in Figure 4, the operation of the server 10 according to an embodiment of the present disclosure can be divided into the following steps: (S10) obtaining motion data, (S20) generating a height map, and (S30) reproducing a virtual space. Each of these steps will be described below with reference to Figures 5 to 9.

(S10) Obtaining Motion Data The communication unit 110 obtains motion data obtained from the movements of the players from an external device. The communication unit 110 outputs the obtained motion data to the map generation unit 121 and the spatial processing unit 122.

(S20) Generation of Height Map The map generation unit 121 generates a height map based on the player heights included in the motion data acquired by the communication unit 110. The generation of the height map is performed through steps S21 to S23.

The map generation unit 121 first divides the area to be reproduced in virtual space, i.e., the court, into areas of a predetermined size. More specifically, the reference plane corresponding to the court is divided into areas of a predetermined size. The shape of the areas is not particularly limited and may be any shape. For example, it may be a rectangle, i.e., a block. More specifically, the block may be a square with one meter on each side. Here, an example is described in which the block is a square with one meter on each side, but the length of each side may be any length. The shape of the block and the length of each side may be determined automatically based on the size of the area reproduced in the virtual section, or the user may be able to set them as appropriate.

In subsequent processing, the map generation unit 121 generates a map in which the height of the court from the reference plane is represented for each block. Therefore, the smaller the block size, the more detailed the map can be created that shows the height of each position. On the other hand, the more player heights are obtained on a block, the more accurate the height represented for each block will be. Therefore, the size of the block can be changed as appropriate depending on the number of player height data points, etc.

Hereinafter, the height of the court from the reference plane, expressed for each block, will also be referred to as the "block height." The block height is the distance that represents the undulations of the block in this embodiment, and is the second distance.

The map generation unit 121 selects the lowest player height from among the player heights obtained on the blocks included in the motion data obtained by the communication unit 110 (S21). The player height obtained on the block is the player height obtained from the player's feet that are located perpendicular to the face of the block. For example, if three player heights, 10 cm, 0 cm, and -10 cm, are obtained on the block, the map generation unit 121 selects -10 cm.

Here, the foot of the player corresponding to the lowest player height is likely to be in contact with the court. Therefore, the lowest player height is likely to represent the height of the court relative to the reference surface. Therefore, the map generation unit 121 generates a height map by setting the player height selected for each block as the height of each block.

Figure 5 shows an example of a height map generated by setting the lowest player height among the player heights obtained on a block. Height map M1 shown in Figure 5 represents the block heights of each block B (e.g., blocks B1 to B5) into which the court is divided. Note that the illustration is simplified here for ease of explanation, but in reality, blocks B may be divided into more detailed blocks and set on the court.

Legend G is a legend for the block heights represented in height map M1. For example, height map M1 indicates that the court height in block B1 is approximately 20 cm. Also, height map M2 indicates that the court height in block B2 is approximately -20 cm.

It is possible that there may be blocks B within the height map M1 for which no player height is acquired. For example, if there is a block B that no player passes through during a match for which motion data is acquired, no player height will be acquired for that block B.

In such a case, the map generation unit 121 does not need to set the block height of block B. In Figure 5, block B3 in the height map M1 is blank. This indicates that the block height of block B3 has not been set.

Due to errors when generating the motion data, the motion data may contain a player height that is lower or higher than the actual height. Motion data may also be generated based on the capture results of a player who has lifted their feet during a jump, etc. This can result in the block height set for each block B differing from the actual height. For example, block B4 in height map M1 in Figure 5 is significantly lower than the surrounding blocks, and so it is thought that a height different from the actual height has been set as the block height.

Then, the map generation unit 121 generates a height map that more accurately represents the block heights in the subsequent steps S22 and S23.

First, the map generation unit 121 removes outliers (S22). More specifically, the map generation unit 121 determines whether or not an outlier is set in block B by determining whether or not each of the block heights set in each block B satisfies a predetermined condition.

Then, if the block height set for block B satisfies a predetermined condition, the map generation unit 121 removes outliers by changing the block height set for block B.

Changing the block height may involve resetting the set block height, or changing the block height according to the block height set for the surrounding block B. Here, we will explain an example of changing the block height by resetting the set block height.

The specified condition may include, for example, that the block height set for block B is greater than or equal to a first value or less than or equal to a second value.

The first and second values may be set appropriately depending on the expected magnitude of the court's undulations (i.e., the expected block height) and the set position of the reference plane. This allows for the block height to be removed as an outlier if an unexpected block height is set. The first and second values may be the upper and lower limits of the expected block height, respectively, or vice versa. For example, the first and second values may be 20 cm and -20 cm, respectively.

The predetermined condition may also include that the difference between the block height set for block B and the average value of the block heights set for each block B adjacent to block B is equal to or greater than a third value.

The third value may be set appropriately depending on the expected degree of slope of the court's undulations. This allows the block height of a block B that is determined to have been set incorrectly based on its relationship with the block heights set for adjacent blocks B to be removed as an outlier. The third value may be, for example, 5 cm. Here, the first value, second value, and third value may each be a predetermined value determined in advance, or each value may be set arbitrarily by the user.

Figure 6 shows an example of a height map generated by resetting the player heights set in block B containing outliers in height map M1 shown in Figure 5. Height map M2 shown in Figure 6 is a height map generated by resetting the player heights set in block B containing outliers in height map M1. In height map M2, as with height map M1, the heights represented in height map M2 are represented by legend G.

Referring to Figures 5 and 6, blocks B such as block B4 that were not blank in height map M1, i.e., had a block height set, are blank in height map M2. This indicates that such blocks B contain outliers, i.e., the player height set for blocks B that are thought to have a different height from the actual height of block B, has been reset.

Next, the map generation unit 121 smooths the height map M2 using a weighted average (S23). More specifically, for a block B for which a block height has been set, the map generation unit 121 resets the block height to the weighted average of the height of the block and the height of the block B adjacent to the block B.

Figure 7 is a diagram illustrating the weights set for each adjacent block B. Figure 7 shows a block Bs whose height is to be reset, and nine blocks Ba (Ba1, Ba2, etc.) adjacent to block Bs. As shown in Figure 7, a weight of "2" may be used for block Bs whose height is to be reset, and a weight of "1" may be used for block Ba. The map generation unit 121 calculates the height of the block to be reset to block Bs by a weighted average using these weights.

Note that the map generation unit 121 performs weighted averaging by setting the weight of block B, for which no block height is set, to 0.

The weights are not limited to the example shown in Figure 7 and may be set arbitrarily. Furthermore, the weighted average may also use the height of blocks B in a wider range, for example, the height of the block Bs whose height is reset and the height of the block B two blocks away.

In addition, for blocks B for which no block height has been set, the block height is calculated using the block heights set for each adjacent block B. Hereinafter, blocks B for which no block height has been set will be referred to as "unset blocks Bn."

For example, the map generation unit 121 may calculate the average value of the heights of the blocks set in the blocks B adjacent to the unset block Bn as the height of the block to be set in the unset block Bn.

It is also possible that block heights have not been set for all blocks B adjacent to an unset block Bn. In this case, the map generation unit 121 may set the block height set for the block B closest to the unset block Bn, for which a block height has been set, as the block height of the unset block Bn. If there are multiple blocks B closest to the unset block Bn, for which a block height has been set, the average value of the heights of the multiple blocks may be set as the block height of the unset block Bn.

Figure 8 shows an example of a height map obtained by smoothing the height map M2 shown in Figure 6. The height map M3 shown in Figure 8 is a height map generated by performing a weighted average process on each block of the height map M2 and setting the block height for the unassigned blocks Bn. In the height map M3, as with the height maps M1 and M2, the heights represented within the height map M3 are indicated by the legend G.

In height map M3, height map M2 has been smoothed using a weighted average process. For example, in height maps M1 and M2, the block height for block B5 was set to approximately -20 cm. In height map M3, the block height for block B5 has been reset to a higher value and smoothed by taking a weighted average of the set block height and the block height set for adjacent block B.

In addition, in the height map M3, setting the block height for an unset block Bn sets the block height for all blocks B in the height map M3.

For example, in height maps M1 and M2, no player height was obtained for block B3, so no block height was set for block B3 and it was left blank. In height map M3, it can be seen that the block height has been set by calculating the average value of the block heights set for blocks B adjacent to block B3.

Furthermore, in height map M2, due to the removal of outliers, no block height was set for block B4, leaving it blank. In height map M3, it can be seen that the block height was set by calculating the average value of the block heights set for blocks B adjacent to block B4.

The steps S22 and S23 described above correct any discrepancies between the actual block height and the block height in the height map M1 due to errors in generating the motion data, etc.

The block height of each block B indicated by each height map M (height maps M1 to M3) may be adjusted by an administrator of the virtual space. More specifically, the height map M may be transmitted to an external terminal, and an editing screen for the height map M may be displayed on the external terminal. By operating the editing screen, the administrator may specify a block B whose height is to be changed, or may change the height of the specified block B. The height may be changed by inputting a value, or may be changed intuitively by moving the block B shown along with the reference plane in the vertical direction.

The generation of the height map M has been explained above. Up to this point, an example has been explained in which the player height, which represents the distance between the player and the reference plane in the motion data acquired by the communication unit 110, is used to set the height of the blocks in the height map M. In other words, up to this point, an example has been explained in which the reference plane in the height map M where the block height is 0 cm (hereinafter also referred to as the "map reference plane") matches the reference plane in the motion data. However, the map reference plane in the height map M may be set to a plane different from the reference plane in the motion data.

For example, the map reference plane may be set to be based on the height of an object fixed within the court, such as the height of a goal net installed within the court.

If the map reference surface is set to a surface different from the reference surface in the motion data, the player height included in the motion data can be converted to the player height from the map reference surface depending on the positional relationship between the set map reference surface and the reference surface in the motion data. By using the converted player height, the map generation unit 121 can set the block height for each block into which the map reference surface is divided, with the map reference surface being 0 cm.

The map reference plane may be set based on the height of an object set by the user in advance, or may be set based on priority information set by the user in advance. There are no particular restrictions on the type of priority information. For example, it may be expressed as a number such as 0 to 10, or as text such as HIGH, MIDDLE, LOW. Typically, there are multiple candidates for the object in real space that will serve as the map reference plane. Therefore, the user may set priority information for each candidate object in advance, and the object with the highest priority may be set as the map reference plane, thereby better reflecting the user's intentions.

(S30) Reproduction of Virtual Space The spatial processing unit 122 generates an image that reproduces the player in the virtual space, based on the smoothed height map M3 generated by the map generation unit 121 and the player's motion data acquired by the communication unit 110. Note that the spatial processing unit 122 may also generate an image that reproduces the player in the virtual space, based on the height map M1 or height map M2 generated by the map generation unit 121.

For example, when generating the virtual space, the spatial processing unit 122 reproduces an undulating court as a flat surface in the virtual space. Hereinafter, the surface reproduced in the virtual space will also be referred to as the "reproduced surface." The reproduced surface may coincide with the map reference surface. In this case, the height of each block represented by the height map M3 represents the difference between the position of each position on the court in virtual space and the reproduced surface. Below, we will mainly explain an example where the reproduced surface coincides with the map reference surface.

The spatial processing unit 122 corrects the position of the player in the virtual space represented based on the motion data, more specifically, the height of the player in the virtual space, in accordance with the height of the block set in block B corresponding to the position of the player to be placed in the virtual space, as represented by the height map M3.

The spatial processing unit 122 corrects the position of the player's feet by moving them toward the map reference surface in virtual space, i.e., the reproduction surface, by the amount of the undulations corresponding to the position of the player's feet. More specifically, as explained using Figure 1, the spatial processing unit 122 corrects the position of the player's feet F by the amount of the difference between the real-space corresponding surface RF (the surface corresponding to the court in virtual space) and the reproduction surface VF, i.e., by the height of each block represented by the height map M3, and reproduces them in virtual space.

When correcting the position of a player's feet, the positions of other parts of the player (such as hands) may also be corrected by the same amount as the position of the player's feet, depending on the corrected position of the feet. This will generate an even more natural-looking image of the virtual space.

Figure 9 is a diagram illustrating the virtual space generated by the spatial processing unit 122. The upper part of Figure 9 shows image Ib of virtual space V, which reproduces players in real space when no correction is made by the spatial processing unit 122. At the positions corresponding to the players reproduced in image Ib of virtual space V, the height of the court in real space is lower than the map reference surface. Therefore, in image Ib of virtual space V, the feet of player P1 and the hands of player P2, which are actually in contact with the court, are reproduced as being sunken into the court.

The bottom part of Figure 9 shows the image Ic of virtual space V, which reproduces the players in real space after correction by the spatial processing unit 122. The spatial processing unit 122 corrects the position of each part of the player according to the height of the block set for each position where the player reproduced in the image Ic of virtual space V is located, as represented by the height map M3. This generates the image Ic of virtual space V, which reflects the positional relationship between the player and the court in real space, i.e., the height of the player relative to the court in real space. With this configuration, the players are reproduced naturally in the image Ic of virtual space V. For example, in the image Ic of virtual space V, the feet of player P1 and the hands of player P2 are reproduced so that they touch the court without sinking into it.

The above describes an example of correction by the spatial processing unit 122 when the reproduction surface and the map reference surface coincide. However, the spatial processing unit 122 may also use a surface that does not coincide with the map reference surface as the reproduction surface. In this case, the spatial processing unit 122 only needs to correct the position of each part of the player by the amount of the difference between the map reference surface and the reproduction surface at each position on the reproduction surface. The reproduction surface is not limited to a flat surface, but may also be an undulating surface.

Freely setting the reproduction surface increases the degree of freedom in expressing virtual space. For example, when a correction process that uses a surface that does not coincide with the map base surface as the reproduction surface is applied to reproducing performers on stage in a dance, play, or music concert in a virtual space, it becomes possible to express a more complex stage that differs from the stage in real space. This can increase the enjoyment of users watching videos in virtual space. For example, at a music concert, it is expected that complex-shaped objects will be set on the stage where the performers appear, or that there will be objects with steps, such as stairs, connecting the stage to the audience. Even in such cases, by appropriately setting the reproduction surface to the stage or stairs, and correcting the positions of the performers and audience by the amount of the difference between the map base surface and each reproduction surface, it is possible to properly depict the positions of the performers' and audience's feet and hands without them sinking into the surface of the stage or stairs. This makes it possible to provide users with a more natural experience.

Up to this point, we have explained an example in which the spatial processing unit 122 uses the height map M3 to recreate an undulating court as a flat surface in virtual space. However, the spatial processing unit 122 may also recreate an undulating court in virtual space by reflecting the shape of the undulations indicated by the height map M3 on a surface in virtual space that corresponds to the map reference surface of the height map M3. In this case, the spatial processing unit 122 recreates players in virtual space without correcting each part of the player. This allows the appearance in real space to be more faithfully recreated in virtual space.

Up to this point, we have explained an example in which the spatial processing unit 122 generates a virtual space including a player using a height map M generated based on the motion data of the player reproduced in the virtual space. However, the spatial processing unit 122 may also generate a virtual space using a height map M generated using motion data different from the motion data of the player reproduced in the virtual space. For example, when reproducing the state of a match in real time in a virtual space, the height map M may be generated based on player motion data acquired from another match that was previously played on the court where the match is being played.

Other matches played in the past on the court where the match is being played may be, for example, matches played in the same season as the match being recreated in the virtual space. In general, in outdoor sports such as soccer, the turf is often replaced or construction work is carried out between seasons. As a result, there is relatively little change in the topography of the court within the same season. Therefore, a natural virtual space can be created by generating a virtual space based on a height map M generated based on player motion data acquired in such matches.

Also, an average block height may be calculated for each block B using multiple height maps M generated based on motion data acquired from multiple games played in the same season. By using such an average value as the block height, the accuracy of the block height increases, allowing a more natural virtual space to be generated.

However, there may be cases where motion data acquired from a match played in the same season does not exist, such as the first match of the season. In such cases, a height map M generated based on motion data acquired from a match played in the immediately preceding season may be used. Even in this case, a more natural virtual space can be generated compared to using a height map that simulates the undulations of each point on the court based on, for example, the results of measuring the heights of characteristic points on the court.

Up to this point, we have explained an example in which the height map M is generated based on the motion data of players, but the height map M may also be generated based on the motion data of other objects on the court. For example, the height map M may also be generated based on the motion data of a soccer ball. If the soccer ball motion data includes the position of the center of the soccer ball, the height of the soccer ball may be obtained from its size. The height of the soccer ball may then be used to generate the height map M in the same way as the player heights. The motion data of the match referee may also be used in the same way as the player motion data.

Furthermore, the positions of other objects present on the court, as represented by their motion data, may be corrected based on the height map M3, thereby reproducing the other objects in virtual space. For example, the position of a soccer ball may be corrected based on the height map M3 and reproduced in virtual space.

4. Hardware Configuration
An embodiment according to the present disclosure has been described above. Next, an example of the hardware configuration of the server 10 according to the embodiment of the present disclosure will be described with reference to FIG.

The processing by the server 10 described above can be implemented by one or more information processing devices. Figure 10 is a block diagram showing an example hardware configuration of an information processing device 900 that implements the server 10 according to one embodiment of the present disclosure. Note that the information processing device 900 does not necessarily have to have all of the hardware configuration shown in Figure 10. Furthermore, some of the hardware configuration shown in Figure 10 may not be present in the server 10.

As shown in FIG. 10, the information processing device 900 includes a CPU 901, a ROM (Read Only Memory) 903, and a RAM 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of or in addition to the CPU 901, the information processing device 900 may have a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit).

The CPU 901 functions as an arithmetic processing device and control device, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 903, RAM 905, storage device 919, or removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during that execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which is composed of an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect/Interface) bus, via a bridge 909.

The input device 915 is a device operated by the user, such as a button. The input device 915 may include a mouse, keyboard, touch panel, switch, lever, etc. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the CPU 901. By operating this input device 915, the user inputs various data and instructs the information processing device 900 to perform processing operations.

The input device 915 may also include an imaging device and a sensor. The imaging device is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture still images or moving images.

The sensors may be various types of sensors, such as a distance sensor, acceleration sensor, gyro sensor, geomagnetic sensor, vibration sensor, light sensor, or sound sensor. The sensors acquire information about the state of the information processing device 900 itself, such as the orientation of the housing of the information processing device 900, and information about the environment surrounding the information processing device 900, such as the brightness and noise levels around the information processing device 900. The sensors may also include a Global Positioning System (GPS) sensor that receives GPS signals and measures the latitude, longitude, and altitude of the device.

The output device 917 is composed of a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by processing by the information processing device 900 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a lighting device that brightens the surrounding area.

The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, from a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs and various data executed by the CPU 901, as well as various data acquired from external sources.

The drive 921 is a reader/writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is either built into the information processing device 900 or attached externally. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs it to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

The connection port 923 is a port for directly connecting a device to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the network 931. The communication device 925 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi (registered trademark), or WUSB (Wireless USB). The communication device 925 may also be a router for optical communications, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP/IP. The network 931 connected to the communication device 925 is a wired or wireless network, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

<5. Supplementary Information>
Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

For example, in the above embodiment, the map reference surface and the reference surface in the motion data are set along the ground, and a height map representing the undulations of the ground is generated, but the present technology is not limited to this example. For example, the map reference surface and the reference surface in the motion data may be set along a wall. In this way, a map representing the undulations of the wall is generated in place of the height map M in the above embodiment. For example, by generating a map using the motion data of a bouldering athlete, it is possible to generate a map representing the undulations of a climbing wall. Using such a map makes it possible to naturally reproduce a bouldering athlete in a virtual space.

It is also possible to create a computer program to cause the hardware, such as the CPU, ROM, and RAM, built into the server 10 to perform the functions of the server 10. A computer-readable storage medium storing the computer program is also provided.

Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects in addition to or in place of the above-mentioned effects that would be apparent to those skilled in the art from the description herein.

The following configurations also fall within the technical scope of the present disclosure.
(1)
an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object;
a generation unit that generates a map used to reproduce an object in a virtual space, the map representing undulations relative to the reference plane, based on the distance acquired by the acquisition unit; and
Equipped with
Information processing system.
(2)
the distance is a first distance,
The generation unit
Dividing the reference surface into regions of a predetermined size;
setting the shortest first distance among the plurality of first distances acquired on the region as a second distance that is a distance representing the undulations of the region;
The information processing system according to (1) above.
(3)
The generation unit
determining whether each of the plurality of second distances that have been set satisfies a predetermined condition;
The information processing system according to (2), wherein, when the second distance satisfies a predetermined condition, the setting of the second distance for the area corresponding to the second distance is changed.
(4)
The information processing system according to (3), wherein the predetermined condition includes the second distance being equal to or greater than a first value or equal to or less than a second value.
(5)
The information processing system described in (3), wherein the specified condition includes that the difference between the second distance and the average value of the second distances of each area adjacent to the area is greater than or equal to a third value.
(6)
The information processing system according to (5), wherein the third value is set according to the degree of inclination of the undulations relative to the reference surface.
(7)
The information processing system described in any one of (2) to (6), wherein the generation unit resets the second distance for the area in which the second distance is set to the weighted average of the second distance and the second distance of an area adjacent to the area in question.
(8)
The information processing system described in any one of (2) to (7), wherein the generation unit sets the average value of the second distances of the areas adjacent to the area in which the second distance is not set as the second distance of the area in which the second distance is not set.
(9)
The information processing system includes:
a processing unit that corrects a position of the target in the virtual space using the map generated by the generation unit,
The information processing system according to any one of (1) to (8).
(10)
The information processing system described in (9), wherein the processing unit corrects the position of the object in the virtual space according to the undulations corresponding to the position of the object to be placed in the virtual space, as represented by the map.
(11)
The information processing system according to (10), wherein the processing unit performs a correction to move the position of the object toward the reference plane in the virtual space by an amount corresponding to an undulation corresponding to the position of the object.
(12)
The information processing system includes:
a processing unit that reflects the shape of the undulations on a surface in the virtual space corresponding to the reference surface,
The information processing system according to any one of (1) to (11).
(13)
the reference plane is a plane set based on real space,
The distance between the reference plane and the object represents the height of the object from the reference plane.
The information processing system according to any one of (1) to (12).
(14)
The information processing system according to (13), wherein the reference plane is a plane specified by two axes of three-dimensional coordinate axes that are set when obtaining motion data from the movement of the object.
(15)
the target is a part of the user's body existing in the real space,
The reference plane is a plane set based on the location where the user will play.
The information processing system according to any one of (1) to (14).
(16)
the body part is the user's foot;
The information processing system according to (15) above.
(17)
the reference plane is set based on the object in real space that has the highest priority;
The information processing system according to any one of (1) to (16).
(18)
the generation unit generates one or more of the maps based on the distances corresponding to objects in one or more games played in the past at locations in real space where the objects in the virtual space are reproduced, the locations being different from the objects in the virtual space whose positions are corrected by the processing unit;
the processing unit corrects the position of the object in the virtual space according to an undulation corresponding to the position of the object in the virtual space represented by the one map generated by the generation unit, or an average value of undulations corresponding to the position of the object in the virtual space represented by the plurality of maps;
the one or more games are games that took place in the same season or the season immediately preceding the season in which the games played in the real space reproduced by the processing unit are played;

The information processing system according to any one of (9) to (11).
(19)
Computer,
an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object;
a generation unit that generates a map used to reproduce the object in a virtual space, the map representing undulations relative to the reference plane based on the distance between the reference plane and the object acquired by the acquisition unit; and
A program that functions as a
(20)
Obtaining a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object;
generating a map to be used for reproducing the object in a virtual space, the map representing the undulations relative to the reference plane, based on the acquired distance between the reference plane and the object;
2. A computer-implemented information processing method, comprising:

REFERENCE SIGNS LIST 10 Server 110 Communication unit 120 Control unit 121 Map generation unit 122 Spatial processing unit 130 Storage unit B Block M Height map

Claims (20)

an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object;
a generation unit that generates a map used to reproduce an object in a virtual space, the map representing undulations relative to the reference plane, based on the distance acquired by the acquisition unit; and
Equipped with
Information processing system. the distance is a first distance,
The generation unit
Dividing the reference surface into regions of a predetermined size;
setting the shortest first distance among the plurality of first distances acquired over the area as a second distance that is a distance representing the undulations of the area;
The information processing system according to claim 1 . The generation unit
determining whether each of the plurality of second distances that have been set satisfies a predetermined condition;
The information processing system according to claim 2 , wherein, when the second distance satisfies a predetermined condition, a setting of the second distance for the area corresponding to the second distance is changed. The information processing system of claim 3, wherein the predetermined condition includes the second distance being greater than or equal to a first value or less than or equal to a second value. The information processing system of claim 3, wherein the predetermined condition includes the difference between the second distance and the average value of the second distances for each region adjacent to the region being equal to or greater than a third value. The information processing system of claim 5, wherein the third value is set according to the degree of inclination of the undulations relative to the reference surface. The information processing system of claim 2, wherein the generation unit resets the second distance for the region for which the second distance is set to a weighted average of the second distance and the second distance of the region adjacent to the region. The information processing system of claim 2, wherein the generation unit sets the second distance for a region for which the second distance is not set as the average value of the second distances for regions adjacent to the region. The information processing system includes:
a processing unit that corrects a position of the target in the virtual space using the map generated by the generation unit,
The information processing system according to claim 1 . The information processing system of claim 9, wherein the processing unit corrects the position of the object in the virtual space according to the undulations corresponding to the position of the object to be placed in the virtual space, as represented by the map. The information processing system of claim 10, wherein the processing unit performs a correction to move the position of the object toward the reference plane in the virtual space by an amount corresponding to the undulations corresponding to the position of the object. The information processing system includes:
a processing unit that reflects the shape of the undulations on a surface in the virtual space corresponding to the reference surface,
The information processing system according to claim 1 . the reference plane is a plane set based on real space,
The distance between the reference plane and the object represents the height of the object from the reference plane.
The information processing system according to claim 1 . The information processing system of claim 13, wherein the reference plane is a plane specified by two axes of three-dimensional coordinate axes that are set when obtaining motion data from the movement of the subject. the target is a part of the user's body existing in the real space,
The reference plane is a plane set based on the location where the user will play.
The information processing system according to claim 1 . the body part is the user's foot;
16. The information processing system according to claim 15. the reference plane is set based on the object in real space that has the highest priority;
The information processing system according to claim 1 . the generation unit generates one or more of the maps based on the distances corresponding to objects in one or more games played in the past at locations in real space where the objects in the virtual space are reproduced, the locations being different from the objects in the virtual space whose positions are corrected by the processing unit;
the processing unit corrects the position of the object in the virtual space according to an undulation corresponding to the position of the object in the virtual space represented by the one map generated by the generation unit, or an average value of undulations corresponding to the position of the object in the virtual space represented by the plurality of maps;
the one or more games are games that took place in the same season as or the season immediately preceding the season in which the games played in the real space reproduced by the processing unit are played;
The information processing system according to claim 9 . Computer,
an acquisition unit that acquires a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object;
a generation unit that generates a map used to reproduce the object in a virtual space, the map representing undulations relative to the reference plane based on the distance between the reference plane and the object acquired by the acquisition unit; and
A program that functions as a Obtaining a distance between a reference plane set in real space and the object, the distance being obtained from the movement of the object;
generating a map to be used for reproducing the object in a virtual space, the map representing the undulations relative to the reference plane, based on the acquired distance between the reference plane and the object;
2. A computer-implemented information processing method, comprising: