JP2000350859A - Marker placement method and mixed reality device - Google Patents

Abstract

(57) [Problem] To provide a marker arrangement method in a mixed reality space in which a marker for each player can be reliably detected even when a common mixed reality space is shared by a plurality of players. SOLUTION: A marker to be used only by a certain player is arranged at a position invisible to other players. It is also possible to use a real object used in an application that uses a mixed reality space.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for arranging a marker used as a position index when presenting a mixed reality space, and more particularly, to easily detect a marker even when a plurality of players share the mixed reality space. The present invention relates to a marker arrangement method and a mixed reality apparatus using the same.

[0002]

2. Description of the Related Art In recent years, mixed reality (Mixed Reality, M
Research on R) is active. Conventionally, MR aims at coexistence of a virtual reality (VR) world and a real space, which can be experienced only in a state separated from the real space, and is attracting attention as a technology for enhancing the VR.

A typical device for realizing mixed reality is a head mounted display (HMD). That is, mixed reality is realized by combining the real space and the virtual space on the HMD and displaying them. As a method of MR using an HMD, a transflective (see-through) head-mounted display (H
There is an optical see-through method in which an image such as CG is superimposed on the MD), and a video see-through method in which an image such as CG is combined with image data taken by a video camera mounted on the HMD and then displayed on the HMD.

[0004] Applications of MR include medical assistance for presenting to a physician such that the inside of a patient's body is transparent,
New fields that are completely different in quality from conventional VRs are expected, such as the use of work assistance to display the assembly procedure of a product on a real product in a factory.

[0005] A common requirement for these applications is a technique for removing the "shift" between the real space and the virtual space. The "shift" can be classified into a position shift, a time shift, and a qualitative shift, and among them, many efforts have been made to eliminate the position shift, which is the most basic requirement.

In particular, in the case of the video see-through type MR, since an image processing method can be applied relatively easily as a method of correcting a position shift, a positioning method using image processing has been conventionally proposed.

Specifically, a marker marked in a color that is easily detected by image processing is arranged at a predetermined position in the real space, and the viewpoint position is determined by the marker position detected from an image taken by a camera mounted by the player. How to calculate,
For example, there is a method of correcting the output signal of the position and orientation sensor based on the marker position in the image.

[0008]

When detecting markers in image data and estimating the position and orientation of the player based on the results, the markers need to appear in the image at an appropriate size and at substantially uniform intervals. There is. Further, since a sufficient number of markers need to be detected in the image at the same time for the calculation of the viewpoint position, it is necessary to arrange the markers so as to be observed at a certain narrow interval in the image.

On the other hand, in order to improve the tracking accuracy and the identification accuracy of the markers, it is necessary to arrange the markers so that they can be observed at a certain wide interval in the image.

If there is only one player, it is not difficult to arrange markers so as to satisfy such conditions. However, in an application in which a plurality of players share a common mixed reality space, there are certain applications. It is possible that markers arranged so as to be observable at equal intervals from a player are not observed at equal intervals from other players.

[0011] Therefore, in Japanese Patent Application Laid-Open No. 11-84307, in an air hockey game in which two players hit a virtual puck on a table which is a real object, markers of different colors are provided for each player.
It is configured such that the marker can be observed from each player with a suitable arrangement and size.

However, as the number of players sharing the same mixed reality space increases, it becomes more difficult to arrange markers by color coding. That is, in order to detect colors by image processing, the colors of the marker and the background object and the colors of the markers between different users need to be colors that can be easily detected and separated by image processing, respectively. When the number of colors increases, it becomes difficult to satisfy this condition, which causes an extraction error and an identification error between markers.

FIG. 10 is a diagram for explaining the cause of erroneous recognition when using markers of a plurality of colors. In FIG.
The horizontal axis indicates red and the vertical axis indicates green, and the blue axis is omitted to simplify the description. In the drawing, an area A indicates an area defining the color distribution of the marker type A (red marker), and an area B indicates an area defining the color distribution of the marker type B (orange marker). Thus, if the number of players increases and similar colors are used as markers,
For example, despite observing the red marker depending on the position of the light source and the angle of shooting, the area B (that is, the orange marker) is observed.
It may be detected as a color, resulting in erroneous detection.

Further, by arranging a plurality of markers of different colors for each player, a very large number of markers exist in the real space, which complicates the player's field of view and allows the player to experience the mixed reality space. This may cause a loss of realism.

An object of the present invention is to solve the above-mentioned problems, and to provide a marker arrangement method in a mixed reality space in which a marker for each player can be reliably detected even when a plurality of players share a common mixed reality space. Is to do.

Another object of the present invention is to provide a marker arrangement method in a mixed reality space in which the number of markers does not simply increase even if the number of players increases.

Another object of the present invention is to provide a mixed reality apparatus using a marker arrangement method in a mixed reality space according to the present invention.

[0018]

That is, the gist of the present invention is a method of arranging a marker to be arranged in a real space as a position index when generating a mixed reality space, wherein the mixed real space can be moved differently from each other. The present invention resides in a marker arranging method characterized in that when a plurality of players observing within a range observe a mixed reality space, markers are arranged in a positional relationship such that a marker to be used only by another player is not observed.

Another aspect of the present invention resides in a mixed reality apparatus characterized in that position information of a player is calculated and / or corrected using markers arranged by the marker arranging method of the present invention. .

Further, another gist of the present invention resides in a storage medium storing a marker arranging method in a mixed reality space according to the present invention as a program executable by a computer device.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a preferred embodiment of a marker arranging method according to the present invention will be described with reference to the drawings. The present embodiment is a mixed reality game played by three players sharing the same virtual space, but the marker arrangement method in the mixed reality space according to the present invention can be applied to any other applications.

FIG. 1 is a diagram showing a configuration of a mixed reality apparatus for executing a mixed reality game to which a marker arrangement method according to the present invention is applied. FIG. 1 shows a configuration in a case where there are three players. The mixed reality device includes player processing units 100 to 300 provided for each player and a control unit 400 connected to each player processing unit, and is connected to the control unit 400 by increasing or decreasing the number of players. The player processing unit also increases or decreases.

The player processing units 100 to 300 have the same configuration, and include input / output units (102, 103, 1) such as a sensor and a display device mounted on the player's body.
05, 1011, 1021, and 1022) and a circuit portion (101) for generating an image to be displayed on the display device based on the signal processing collected from the sensor and the result of the signal processing and information of another player supplied from the control section. , 104, 10
6, 107).

Hereinafter, the configuration of the player processing unit will be described with reference to FIG. 1 and FIG. 2 showing the types and locations of input / output devices to be mounted by each player in the present embodiment. As shown in FIG. 2, in this embodiment, each player has a head-mounted image display device (hereinafter, referred to as an HMD) 1 which is a display device for combining and displaying a real space and a virtual space on the head.
05 is attached. In the present invention, the HMD may be a video see-through type or an optical see-through type. However, in the following description, an example in which a video see-through type HMD is used will be described.

Two small video cameras 103 are provided near the eyes of the HMD 105. The video captured by the video camera 103 from a viewpoint substantially the same as the viewpoint of the player is supplied to an image synthesizing unit 106 to be described later via the image input unit 104, and is superimposed on the virtual space image and displayed on the HMD 105 of the player. Is done. The image input unit 104 supplies the input image to the head position detection unit 101 in addition to the image synthesis unit 106.

Further, a head position / posture sensor 1011 composed of, for example, a magnetic sensor is mounted on the head of the player. The head position / posture sensor 1011 is, for example, H
It can be mounted using the MD 105. The output signal of the head position / posture sensor 1011 is the head position / posture measurement unit 1
01 is input. The head position / posture measuring unit 101 detects a marker for position correction using the image supplied from the image input unit 104, and corrects the signal supplied from the head position / posture sensor 1011 to obtain the viewpoint of the player. Estimate the position and orientation.

On the other hand, the interactive operation input device 102 is mounted on the arm of the player. Interactive operation input device 102
Has a position / orientation sensor 1021 for detecting the position and orientation of the mounted part, and a switch (trigger) 1022 for turning on / off in accordance with the movement of the mounted part. The command is issued when the player performs a predetermined operation. It is a device for inputting.

In the following description, a plurality of players (three players in the present embodiment) share the same mixed reality space and defeat the enemy while avoiding attacks from the enemy appearing in the mixed reality space. A game in which the number of enemies or points defeated until the time or the damage by the enemy attack reaches a predetermined amount is competed, and the following command input is possible using the interactive operation input device 102. explain.

Command 1 (aiming command) A command for displaying an aiming line indicating an aiming position in a virtual space. With the back of the hand up, the input is made by placing the wrist above the elbow. -Command 2 (shooting command) A command for shooting at the aiming position indicated by the aiming line. While the aiming line is displayed by the aiming command, the input is performed by reciprocating the arm (from the elbow to the palm) back and forth at a certain acceleration or more. -Command 3 (defense command) A command to defend the opponent's attack. With the back of the hand facing the other party, input is performed by pointing the fingertip upward. -Command 4 (reset command) A command that is input when a shooting command is input and then an aiming command is input again. Input by placing the arm down.

That is, as the operation of the player in the game of the present embodiment, command input is normally repeated in a cycle of aiming command → shooting command → reset command → aiming command, and in this cycle, defense commands are issued as necessary. Will be entered.

The command input by the interactive operation input device 102 is supplied to the image generation unit 107.

The image generation unit 107 transfers the player's head position and posture information supplied from the head position / posture measurement unit 101 and command information supplied from the interactive operation input device 102 to the control unit 400. Also, the player's head position / posture information and command information, the other player's head position / posture information and command information and model information received from the control unit 400, the position, moving direction, state information, and space in the enemy character A virtual space image to be displayed on the HMD 105 of the corresponding player is created by using information such as the position and shape of the obstacle arranged in the.

The image synthesizing unit 106 synthesizes the image (real space image) of the video camera 103 attached at a position close to the viewpoint of the player and the virtual space image created by the image generating unit 107, and outputs the synthesized image to the HMD 105 of the player. Supply.

The control unit 400 is a mixed reality space management unit 1 to which the player processing units 100 to 300 described above are connected.
And a real object position measurement unit 2. The mixed reality space management unit 1 includes player processing units 100 to 300
And information on the position and posture of the player's head and the position, posture and command information of the interactive operation input device 102 received from the player, and generate, delete, control, and shoot commands for the enemy character displayed in the virtual space. The game processing such as the hit determination of the shooting is performed. Model, position, moving direction, and state of the enemy character (whether destroyed, etc.)
Is distributed to all connected users together with the information of each player.

Further, as shown in FIG. 3, a real object 31 which may be an obstacle to shooting in order to adjust the difficulty of the game.
To 33, the real objects 31 to 3
Information on the shape and position of the mixed reality space management unit 1
Manages.

FIG. 3A is a perspective view of a game field (mixed reality space) viewed from a certain player, and FIG. 3B is a top view thereof. In FIG. 3, three real space objects 31 to 3 are placed on a table 30 (in the real space).
3 shows a case where 3 is arranged as an obstacle. In this embodiment, since the input operation of the aiming command using the interactive operation input device 102 is an operation of raising the wrist above the elbow as described above, the height of the waist is about the height of the general body type. Although the real space object is arranged on the table 30 having the same, it goes without saying that the table is not necessary depending on the command input operation by the interactive operation input device.

In FIG. 3, real space objects 31, 32 are fixed, and 33 is movable. If there is a movable real space object, the difficulty of the game can be dynamically changed. For example, if the movable real object 33 moves to the left and right at a random speed, shooting of the enemy character as the target becomes more difficult than when only the static real objects 31 and 32 are arranged. The movement control of the movable real object may be performed by the mixed reality space management unit 1 or by another control circuit. In this case, the model of the movable object is managed by the mixed reality space management unit 1, and its position is measured by the object position sensor 331 provided on the movable object 33 by the real object position measurement unit 2.

In the mixed reality apparatus as described above, the control unit 400 includes a server and the player processing units 100 to 300.
It can be realized by a client-server system in which (the circuit part of) is a client. By distributing the processing for each player in the client, it is possible to flexibly cope with an increase or decrease in the number of players. More specifically, the player processing unit is a general-purpose computer device having a video input / output function and a function of receiving signals from various sensors. The control unit 400 also has an interface capable of communicating with each player processing unit, and an object position measurement unit 2. It can be realized by a general-purpose computer device having a function of receiving a measurement signal from a computer.

However, since it is necessary to perform calculations relating to three-dimensional image display in real time, it is preferable to use a relatively high-speed computer device having an accelerator (so-called 3D accelerator) specialized for such calculations. . The control unit 400 and the player processing units 100 to 3
The communication with 00 is also preferably connected by a line having a large capacity such as 100BASE-T. If the capacity of the communication line is small, the processing speed is greatly reduced as the number of players increases.

(Marker Arrangement Method) FIG. 4 is a perspective view showing an example of marker arrangement in this embodiment. In the game of the present embodiment, obstacles due to real objects are arranged. By arranging markers by using the obstacles, while limiting the markers that are in the field of view within the movement range of each player, The condition can be satisfied.

FIGS. 5A to 5C respectively show markers extracted from the players A, B, and C in FIG. In this way, the markers visible from each player are observed at substantially equal intervals and at the number required for calculating the viewpoint position, but the markers for other players are not visually recognized, so there is no need to change the color. FIG.
It is also possible to share the same marker among a plurality of players as shown in FIG.

FIGS. 6A to 6C show the markers observed from each player when no obstacle is used.
It is the figure shown corresponding to (c). It can be clearly understood from the comparison between FIG. 5 and FIG. 6 that the marker arrangement method according to the present invention greatly reduces the number of markers observed by each player and satisfies the above-described conditions.

When the number of players increases, it can be dealt with by changing the shape (cross-sectional shape, height, etc.) of the real object on which the marker is provided, or by adding a color.
Even when the number of colors is increased, one color is not assigned to one player as in the related art, so that it is possible to arrange markers for many players with few colors.

The placement of the marker may be determined manually, but a model of the real object and a model of the movable range of the viewpoint position of each player are generated in advance, and the marker is located within the range visible from the target player. The determination can be made by obtaining a range in which the line of sight of another player is blocked. Further, the position may be calculated using the number of markers to be provided and the arrangement rule. Conversely, it may be configured to calculate an obstacle shape and / or arrangement that satisfies such a condition.

Further, even when there is no object such as an obstacle that can be used for marker placement, the real object is arranged at a position that does not cause a problem in the application, and the existence of the real object is hidden by the virtual space image. Makes it possible to experience a mixed reality space without being conscious of a marker real object. The concealment of the marker and the like by the virtual space image will be described later in detail.

(Detection of Marker) Next, a method of detecting a marker will be described. FIG. 8 is a flowchart illustrating a flow of a process of detecting a marker (a red marker) of type A having a color included in the area A illustrated in FIG. Both the marker detection processing and the use of the detected marker information are performed by the head position / posture measurement unit 101.

First, an image captured by the video camera 103 is obtained via the image input unit 104 (step S).
701). Then, a binarization process is performed (step S70).
2). Specifically, the pixel included in the area A shown in FIG. 7 (illustrated by omitting the blue axis) is 1 and the others are 0. That is, Ii: the i-th pixel Ri, Gi, Bi constituting the input color image I: the value of each color of R, G, B constituting Ii ITHi: the i-th pixel value RminA, GminA, BminA of the binary image: RmaxA, GmaxA, BmaxA: RmaxA, GmaxA, BmaxA: RminA <Ri <Rmax for each Ii, where RminA <Ri <Rmax
A and GminA <Gi <GmaxA and BminA <
A binary image ITH is formed by setting 1 to ITHi corresponding to Ii satisfying Bi <BmaxA and setting 0 to ITHi corresponding to other Ii.

Next, a labeling process is performed on the binary image ITH to extract a marker area (cluster) (step S703). Then, the center of gravity (Xn, Yn) of each cluster
And the area an are calculated (step S704), and output to a viewpoint position / posture estimation module (not shown) inside the head position / posture measurement unit 101 (step S705). In the viewpoint position and orientation estimation module, the head position and orientation sensor 10 is determined based on the absolute coordinates of the markers registered in advance and the marker positions (cluster centroids) detected from the image.
11 to correct the output signal and calculate the viewpoint position and attitude of the player.

In FIG. 8, the processing in the case of detecting a certain one-color (red) marker has been described. However, when the number of players increases and a plurality of color markers exist, a binarized image is generated. The detection process for each color is repeatedly performed while changing the threshold value at the time of execution.

(Erase of Marker) As described above, a marker is essentially unnecessary if the output of the head position / posture sensor 1011 is accurate. Further, it is desirable that the presence of the marker in the mixed reality space is not recognized, for example, the presence of the marker may be impaired by the player's recognition of the marker.

Therefore, the present invention is characterized in that the marker is apparently erased so that the player is not conscious of the presence of the marker. Various methods are conceivable as a method of apparently erasing the marker, but a method of superimposing the virtual image on the marker and displaying it to the player is preferable because the processing load and uncomfortable feeling are small.

FIG. 9 is a view for explaining a marker erasing method. FIG. 9A illustrates markers for the player A arranged by the marker arrangement method according to the present embodiment, and corresponds to FIG. In order to erase a marker placed on a real object such as a table and an obstacle in this way, first, before placing the marker, the place where the marker is to be placed is photographed with a video, digital camera, still camera, etc. Get it as a texture. Then, after arranging the marker, of the image data displayed on the HMD of the player, the image data corresponding to the marker portion is replaced with a previously acquired texture image or is superimposed to display the image data (FIG. 9 ( b)).

By replacing / superimposing the image data in this manner, the player can concentrate on the game without recognizing the presence of the marker. It is preferable to obtain the image data prepared as the texture under the conditions actually used (the light source position and the brightness) in order to reduce a sense of visual discomfort, but they are arranged on a real object of the same material. The texture image from which the marker is deleted may be one common texture.

In an application that does not use obstacles, when a real object is placed only for marker placement, or when it is difficult to replace / superimpose image data on individual markers, etc. As shown in FIG. 9C, an image 91 that covers all the real objects can be used. In this case, what kind of image data is used may be appropriately determined by an application.
For example, in a game application, the image 91 may be used to form a stage or the like in a virtual space, or in an application where the user does not want to feel a difference from the real space. What is necessary is just to use the data of the photographed object. Of course, it is also possible to use three-dimensional object data instead of two-dimensional image data.

[0055]

[Other Embodiments] In the above embodiment, information obtained from a marker is used for error correction of the head position / posture sensor 1011. However, information obtained from the marker can be obtained without using the head position / posture sensor 1011. The viewpoint position and orientation of the player can be obtained from only the information, and the present invention can be applied to such a use. Although the above-described embodiment is a video see-through mixed reality device, it goes without saying that the present invention can be similarly applied to an optical see-through mixed reality device.

Further, in the above-described embodiment, the command is input by detecting the movement of the interactive operation input device without using the trigger 1022, but the player inputs various commands using the trigger 1022. Alternatively, the operation detection and the trigger on / off may be combined. Further, a switch that turns on and off according to the operation of the player may be incorporated in the interactive operation input device and used as a trigger.

An object of the present invention is to supply a storage medium (or a recording medium) storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU) of the system or the apparatus. And MPU) read out and execute the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. By executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written into the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, the program code is read based on the instruction of the program code. Needless to say, the CPU included in the function expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0059]

As described above, according to the present invention,
Even when multiple players share the same mixed reality space, the markers used for alignment in the mixed reality device can be arranged while satisfying the necessary positional and numerical conditions. It becomes possible. Further, since it is not necessary to change the color of the marker for each player, the probability of erroneous detection can be suppressed even when the number of players increases.

Further, since the originally placed real space object can be used, there is an effect that it is not necessary to provide an object for arranging the marker depending on the application.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration example of a mixed reality apparatus to which a marker arrangement method according to the present invention can be applied.

FIG. 2 is a diagram illustrating types of devices to be mounted by a player and mounting portions.

FIG. 3 is a diagram illustrating a game according to the embodiment of the present invention.

FIG. 4 is a diagram illustrating a marker arrangement method according to the present invention.

5 is a diagram showing markers observed from each player according to the arrangement shown in FIG. 4;

FIG. 6 is a diagram showing markers observed from each player when markers similar to those in FIG. 4 are arranged without using an obstacle.

FIG. 7 is a diagram showing a color area of a red marker in the embodiment of the present invention.

FIG. 8 is a flowchart illustrating a procedure of a process for detecting a marker in the color area shown in FIG. 7 from the color image data.

FIG. 9 is a diagram illustrating a marker erasing method.

FIG. 10 is a diagram illustrating erroneous detection when the number of marker colors is increased.

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2C001 AA00 AA07 AA17 BA00 BA06 BB00 BB06 BC00 BC06 CA00 CA09 CC00 CC01 5C061 AA29 AB11 AB12 AB14 AB18 9A001 DD12 HH23 HH29 JJ76 KK62

Claims (8)

[Claims] 1. A method of arranging a marker to be arranged in a real space as a position index when presenting a mixed reality space, wherein a plurality of players observing the mixed reality space within different movable ranges are provided. A marker arrangement method, comprising: arranging markers in such a positional relationship that a marker to be used only by another player is not observed when observing a mixed reality space. 2. The marker arrangement method according to claim 1, wherein the marker used only by the other player is a marker similar in color, shape and size to the marker used by the player. 3. A marker used only by a certain player is:
3. The marker arrangement method according to claim 1, wherein the marker is arranged at a position hidden by a real object from other players. 4. The marker arrangement method according to claim 3, wherein the physical object is arranged for an application using the mixed reality space. 5. The marker arrangement method according to claim 1, wherein the markers arranged include markers shared by a plurality of players. 6. The marker arranging method according to claim 1, wherein said markers have a common color. 7. A mixed reality wherein the position and orientation information of the player is calculated and / or corrected using the markers arranged by the marker arrangement method according to claim 1. Description: apparatus. 8. A storage medium storing the marker arrangement method according to claim 1 as a program executable by a computer device.