KR20100086128A - Mixed Reality Situation Training System - Google Patents

Abstract

The present invention relates to a situation training system that allows a developmentally disabled person to repeat the training safely and without limitations of the surrounding environment by constructing a mixed reality space in which a marker is placed in a real training environment and objects that are difficult to use in a real space are replaced with virtual objects. In other words, the mixed reality training system according to the present invention includes a camera for photographing an image of real reality including one or more markers; A camera input unit for inputting an image captured by the camera, a marker detection unit for detecting all markers included in the image input by the camera input unit, and generating a marker matrix including position coordinates and rotation information, respectively, and three-dimensional model data Using the model data storage unit for storing the, and the marker matrix transmitted from the marker detection unit generates and combines the visibility of a specific marker, the distance between the marker and the marker, and the distance between the camera's viewpoint and the marker to generate the trainer's behavior information, The situation controller which determines the scene information provided to the trainer and whether to transfer to the next scene by applying the behavior information of the trainer to the scene information according to the predefined scenario, the image inputted by the camera input unit as a background image, and the marker detector The marker matrix passed from A training control device including an image matching unit for retrieving and matching a corresponding 3D model from the model data storage unit, and an HMD output unit for outputting an image matched by the image matching unit; And a head mounted display which is integrally coupled with the camera and displays an image output by the HMD output unit.

Description

A situational training system based on the mixed reality}

The present invention relates to a situation training system based on mixed reality, and more particularly, to construct a mixed reality space in which a marker is placed in a real training environment and a virtual object is replaced with objects that are difficult to use in a real space. And a situational training system that allows repetitive training without limitations of the surrounding environment.

In general, people with developmental disabilities are those who need help because their physical and mental abilities are 25% or more behind the normal expectations of their age, and are restricted from performing their functions for daily or social life. Because of their limited intellectual functioning and lack of adaptability, they have a lot of difficulties in adapting to their daily lives and in their work lives. People with developmental disabilities need to be trained to overcome these difficulties. Examples include training in hazardous tools, training in stores or restaurants, and road traffic safety training. However, training with real tools or training in the field is difficult because the possibility of a safety accident is much higher than that of the general public.

The use of mixed reality technology can alleviate the challenges of the developmental disabilities mentioned above. You can learn how to handle difficult-to-handle tools without the help of a teacher, or experience places that are hard to experience in the field through training based on mixed reality.

Researches that use mixed reality technology in education are being actively conducted at home and abroad. In Korea, the Korea Electronics and Telecommunications Research Institute developed a personalized learning system based on realistic e-learning for the purpose of commercialization. According to the system, when a user uses a camera to view a textbook with markers, the user can view virtual text along with the text on a monitor.

Overseas, HIT Lab, Nanyang Technological University, Singapore, developed unique contents for edutainment using AR technology such as 3D Magic Cube and 3D Magic Land using unique marker recognition technology. 3D Rubik's Cube can be viewed by a real user wearing a head-mounted display (HMD) with a camera attached to the marker-based cube. Vienna University of Technology, Austria, is the first to implement mixed reality using PDAs. This content, called an invisible train, shows that when two users grab a PDA and see a rail with markers around it, a virtual train moves over the rail and the user can manipulate it.

These studies provide a user experience-based education and training system based on mixed reality, and use a technique to track a specific location based on images using predefined markers. Since the location is tracked by the information of the image, it is easy to use at low cost by not using special equipment except the camera, and the performance and accuracy of the location tracking are improved by using the marker of the predefined pattern.

On the other hand, education and training systems for people with disabilities require not only these usability, performance and accuracy issues, but also the response to unforeseen unforeseen situations of the disabled. Because of this, the equipment must be available for simple operation. In addition, if the marker is shown to people with disabilities on the training screen, it is necessary to consider this because it may distract concentration or induce confusion.

The present invention was devised to solve the above problems, and when a trainer trains on a preset situation while wearing a HMD with a camera attached and moves in a mixed reality space, it is inserted based on information obtained through a marker. It is a hybrid reality training system that can recognize virtual objects in the same way as real objects and improves the cognitive and situational coping ability of the developmental disabled by moving the virtual objects as if they were moving the real objects. The purpose is to provide.

In order to achieve the above objects, the mixed reality situation training system according to the present invention comprises a camera for photographing an image of real reality including one or more markers; A camera input unit for inputting an image photographed by the camera, a marker detection unit for detecting all markers included in the image input by the camera input unit, and generating a marker matrix including position coordinates and rotation information, respectively; A model data storage unit for storing dimensional model data and a marker matrix transmitted from the marker detector generate and combine the visibility of a specific marker, the distance between the marker and the marker, and the distance between the camera's viewpoint and the marker, and then perform the training of the trainer. A situation controller configured to generate information, determine scene information provided to the trainer by applying the behavior information of the trainer to scene information according to a predefined scenario and whether to transfer to the next scene, and an image input by the camera input unit Is a background image and is transmitted from the marker detection unit. A training control device having an image matching unit for extracting and matching a corresponding 3D model from the model data storage unit at a marker position using a Kerr matrix, and an HMD output unit for outputting an image matched by the image matching unit; And a head mounted display which is integrated with the camera and displays an image output by the HMD output unit.

In the mixed reality training system, the marker detector is configured to binarize, label, and detect the contour of the image input by the camera input unit, and then compare the pattern with a predefined marker to detect the marker. The pattern is determined, and a marker matrix including position coordinates and rotation information of the marker when the camera is at the origin through the size and shape of the detected marker image is generated.

The marker detection unit in the mixed reality training system is characterized in that to apply a Kalman filter to the position coordinates of the marker.

The scene information provided by the context controller in the mixed reality context training system may include at least one of animation, video, voice, and text.

In the mixed reality training system, scene information according to a scenario may be represented by an XML schema in the situation controller.

As described above, according to the present invention, by constructing a scenario based on specific situations that are difficult to train in daily life, and by constructing a mixed reality environment in which a training space can be felt as reality in order to amplify the experience in such a scenario, People with disabilities can train safely with simple equipment, can be interested by using various virtual models, and can improve the training results through repetitive training.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1, the mixed reality training system 1 according to the present invention is a head mounted display (hereinafter referred to as HMD) 30 integrally coupled with a camera 10, a training control device 20, and a camera 10. It is configured to include).

The camera 10 captures an image of the real world and provides it to the training control device 20. One or more markers are attached to the real world captured by the camera 10, and the image captured by the camera 10 is included in the image. An image of the markers is included together.

The training control device 20 may be implemented by an information processing device equipped with software including a corresponding function. In a preferred embodiment of the present invention, a personal computer (PC) is used as the information processing device. The training control device 20 implemented in a personal computer functionally includes a camera input unit 21, a marker detector 22, a model data storage unit 23, a situation control unit 24, an image matching unit 25, and an HMD output unit. (26) is provided.

The camera input unit 21 inputs the image photographed by the camera 10 to the training control device 20, and includes a USB (Universal Serial Bus) port and a driver for controlling the same.

The marker detector 22 detects all the markers included in the image input by the camera input unit 21 and generates a marker matrix including position coordinates and rotation information, respectively.

The process of detecting the marker by the marker detector 22 will now be described with reference to FIG. 2. In FIG. 2, the left side shows the marker detection step by step, and the right side shows the image corresponding to each step.

First, the basic image is an image input by the camera input unit and is a color image of RGB.

In the binarization step, the basic color image is converted into a black and white image. At this time, if the sum of the values of R (red), G (green), and B (blue) is smaller than the predetermined threshold value T, it is black.

In the labeling stage, if the surrounding colors are the same, the area is expanded while the area is defined. For the independently determined area, a separate area ID is assigned, and the area most likely to be a marker is selected.

In the contour detection step, the vertices of the quadrangle are found by detecting the contours based on the amount of change based on the amount of change of the region, which is likely to be a marker, by the partial differential.

In the pattern comparison step, a pattern of the detected marker is determined by comparing the pattern detected by the contour detection with a pattern with predefined markers (pattern file). In the pattern file, the internal shape of the marker is represented by gray image values. The degree of agreement is calculated by comparing the pattern information rotated in the four directions with the rectangles detected in the image, and the one having the highest degree of agreement is determined as the corresponding pattern. In addition, the marker identifier given to each marker can be determined by the pattern of the marker thus determined.

In the coordinate normalization step, a marker matrix including position coordinates and rotation information of the marker when the camera is the origin through the size and shape of the detected marker image is generated.

The marker matrix is a 4x4 matrix and has the following meaning.

The upper left 3x3 matrix in Equation 1 represents rotation information in three dimensions, and is used as a direction vector representing an axis in three dimensions, one row for each.

Next, the last row of Equation 2 means the moving coordinates tx, ty, tz from the origin after the rotation transformation is applied. Since the matrix is a camera origin, the distance between the camera and the marker can be obtained as shown in Equation 3 below.

In addition, the distance between two markers frequently used in the marker comparison is shown in Equation 4.

Since the marker is detected from the camera image to generate the position coordinates of the camera, the result is very dependent on the image. Due to some factors such as camera parameters and ambient light, the positional coordinates of markers in the camera image sometimes show relatively large differences between adjacent frames even when the camera is fixed. In other words, even in the case of calculating the position coordinate value from the marker of the camera image, the position coordinate value has a significant difference, and continuous observation of the position coordinate value generated by the cause causes an unstable shape, that is, The shaking phenomenon occurs. Therefore, in order to improve the reliability of the generated position coordinate values, the present invention solves the problem by applying a Kalman filter.

Kalman filter is an error correction algorithm that is used throughout engineering. The Kalman filter calculates the average of successive input data using the existing mean and standard deviation (general error) and the new input data and standard deviation (general error).

The above method means the most basic method, and the equations used in practice use various transformations that apply them.

The modified Kalman filter algorithm is as follows.

<Time update equations>

( 예측 입력 데이터 )

Predictive Input Data

( 예측 오차 )

(Prediction error)

<Measurement update equations>

( 칼만 게인 : 에러를 최소화하기 위해 사용하는 값 )

(Kalman Gain: Value used to minimize errors)

Calculate using the error and standard deviation of the error. In the present invention, if H is not used and it is set to 1 and the equation is simply expressed, it can be expressed as K = P '/ (P' + R).

( 보정된 데이터를 얻기 위해 예측 데이터와 관측 데이터의 차이에 칼만 게인 값(K)만큼 예측 데이터와 합한다. )

(To obtain the corrected data, the difference between the prediction data and the observation data is added to the prediction data by the Kalman gain value (K).)

Two factors are used to determine the sensitivity (s). It is proportional to the distance difference d between the previous data and the current data, and inversely proportional to the distance f from the camera to the object.

This is to correct the data so that the response speed can be increased by increasing the sensitivity when it is close to the camera or when the motion is large.

The process so far is a data correction step, and the following equation (11) is a step of resetting the error value used in the correction step.

( 민감도와 게인값에 따른 오차 재설정 )

(Reset error according to sensitivity and gain value)

은 최적화 전의 상태변수이고,

는 우리가 얻고자 하는 최적화된 상태변수이다.

는

단계의 오차를 이용해 갱신한 새로운 오차 값이고,

와

은 각각 참값과 오차에 대한 분산, 그리고

는 칼만 게인 값으로 예측된 오차

과

을 이용해 계산해 낸다. In the algorithm

Is the state variable before optimization,

Is the optimized state variable we want to get.

Is

Is the new error value updated using the error of the step,

Wow

Are the variances of the true value and the error, and

Is the error predicted by the Kalman gain value

and

Calculate using

은 구축된 3D 공간의 깊이에 비례하는 계수이다.

는 각각 마커의 움직임에 대한 3차원 공간 좌표이고, 이러한 좌표들을 이용해 기본 칼만 필터 식에 추가적인 입력 s를 계산해 낸다. s는 입력 값의 민감도와 관련이 있는 계수로써 기존 식에서 고정 상수 1 이었으나 본 시스템에서는 마커의 움직임에 비례, 거리에 반비례하는 관계를 가지고 값이 변하게 되며 그 범위가 1 ~ 5 로 정해진다. s가 1에 가까울수록 상태변수

의 값은 안정적이고 잡음에 강하게 되며, 반대로 1과 멀어질수록 반응성이 올라가게 된다.

Is a coefficient proportional to the depth of the constructed 3D space.

Are each three-dimensional space coordinates of the marker's movement, and these coordinates are used to compute additional input s to the basic Kalman filter equation. s is a coefficient that is related to the sensitivity of the input value, which was a fixed constant of 1 in the existing equation. However, in this system, the value changes with the relation proportional to the marker movement and inversely proportional to the distance, and the range is set to 1 to 5. As s approaches 1, the state variable

The value of is stable and noise-resistant. On the contrary, the distance from 1 increases the responsiveness.

을 구해 낸다. In the basic Kalman filter's algorithm, the equation for s in Equation 10 is added, and Equations 6 to 11 are performed repeatedly, using the predicted input values, the actual input values, the error values, and the standard deviations of the true and error values. State variables

Save

In FIG. 1, the model data storage unit 23 stores 3D model data, and nonvolatile storage means such as a hard disk and a flash memory is used.

The situation controller 24 generates and combines the behavior information of the trainer by generating and combining the visibility of a specific marker, the distance between the marker and the marker, and the distance between the camera's viewpoint and the marker using the marker matrix transmitted from the marker detector 22. In addition, the situation controller 24 determines whether to transfer to the next scene and the scene information provided to the trainer by applying the trainer's behavior information to the scene information according to the predefined scenario.

The image matching unit 25 uses the image input by the camera input unit 21 as a background image, and the 3D corresponding to the model position in the model data storage unit 23 at the marker position using the marker matrix transmitted from the marker detection unit 22. Fetch and match the model. At this time, the background image performs texture mapping using an orthogonal projection, and the 3D model renders based on the marker matrix by performing perspective projection.

The HMD output unit 26 outputs the image matched by the image matching unit 25 to the HMD 30, and includes a D-SUB output terminal, a USB port, and a driver for controlling the same.

The HMD 30 is integrally coupled with the camera 10 and displays an image output by the HMD output unit 26.

Training to improve the cognitive abilities of people with developmental disabilities and to help them respond appropriately to their situation includes traffic safety training, learning to use dangerous tools, and using shops and restaurants. Table 1 shows the basic training scenarios based on the situation of serving a restaurant and serving food directly. The training will be progressed to the next stage only if the current stage is successful. If the current stage fails, the training will be repeated.

In situational training scenarios, it is necessary to address the wrong behavior of people with disabilities. However, not all situations can be predicted, so the unexpected situation that is predictable and affects the scenario flow is defined and feedback processing is performed. The accident situation defined in the present invention is defined as an accident situation in which a marker is placed on a wall surface and a distance of the wall marker is closer. When the scenario condition is satisfied while the scenario is in progress, the scenario is entered and the user is provided with a voice comment of wrong behavior.

The mixed reality situation training system 1 supports performing training in a mixed reality environment based on the camera 10. In the training using the mixed reality training system 1, when the trainer wears the HMD 30 and performs the training in the real space, the real world image obtained by photographing the front of the trainer by the camera 10 attached to the HMD 30 is obtained. And a virtual reality in which the virtual 3D models are matched and displayed on the monitor (not shown) of the trainer HMD 30 and the training controller 20 for interaction.

3 schematically shows this training environment.

A virtual food model (not shown) is shown in the kitchen table 50, and a virtual guest model 80 is shown in the guest table 70. The trainer 60 trains serving virtual foods according to the training process defined in the scenario while moving between the kitchen table 50 and the guest table 70.

In the present invention, the HMD 30 and the camera 10 are attached to replace the human vision with the camera image. Therefore, the grasp of the user's behavior is also identified by an image-based marker matrix coming into the camera 10. Information of the marker used in the present invention is as follows.

-Visibility of a specific marker: whether the marker can be found in the current frame

-Distance calculation between markers: Distance between two markers found in the current frame

-Distance between marker and marker: z-axis movement coordinate of marker matrix

The information presented above can be combined to infer what the trainer 60 is doing in the current scenario. For example, if the trainer 60 trains an action that needs to be moved to a specific position, the position of the trainer may be known at present through the coordinates of the marker in the camera image and the information of the surrounding markers. In the case of training using objects, the distance calculation between markers is used as information for inferring the movement or movement of the object. Furthermore, through the matrix comparison of markers, the inclination and direction of the objects, etc. are not simply estimated. Can be used to obtain

In the space where training is prepared, the trainer 60 wears the HMD 30 and waits. When the training starts, the voice 60, the video, the animation, etc. according to the data defined for each scenario will inform the action that the trainer 60 should take at the current stage. 4 illustrates an example of a situation control process performed by the situation controller 24. The situation control process includes a scenario check step S10, a condition check step S20, a condition and exception check step S30, and an exception processing step S40. Trainer 60 must meet the training success conditions defined for each scene, and if the success conditions are met to proceed to the next scene. If you don't meet the conditions for success within a certain amount of time, repeat training.

In the present invention, by setting a common format for representing a scenario and producing a program that accommodates it, a scenario can be easily produced by adding only a data file when adding a new scenario. In other words, scenario contents can be created independently of the program, and scenario creators who don't know programming can easily add scenarios.

In the present invention, the scenario is defined by describing the scene and the flow of the scene, and each scene can find the following features.

-As it is educational content, the user needs to explain the scene and it is expressed by voice or text.

-As it is training content, it requires a specific action from the user.

-Many special effects are used to enhance training effects, including augmented reality, 3D model animation, video and sound effects.

These scenario features are represented in a data structure as shown in Table 2.

In order to define the data of this structure, the present invention uses XML (Extensible Markup Language) format. Using XML, the World Wide Web Consortium (W3C) standard, offers the following advantages:

-It can inherit XML's outstanding extensibility, platform independence, and compatibility, and can shorten development period by reusing existing parser.

XML provides a way to verify that a document is valid for a given schema, thus saving a lot of code by eliminating the part of the program that verifies that the data is valid.

Without knowing the internal program structure or learning additional scripting languages, users who are just familiar with XML documents can easily create scenario documents for the defined Schema format.

Schema proposed in the present invention further describes a <SCENE> tag defining a scene and <MODEL>, <SOUND>, and <SENSORS> tags defining resources to be used in the scene. You can create a scene using resources in the <SCENE> tag, and move to the next scene's ID by referring to the "goto" attribute according to the condition. At this time, for the convenience of the user, a string (self, next, exit) that is used frequently is defined.

Table 3 below shows some examples of actual scenarios using this structure.

<SCENE id = "7">
<ANIMATE> 0 </ ANIMATE>
<VIDEO> 0 </ VIDEO>
<SOUNDMENT> 7 </ SOUNDMENT>
<TEXTMENT> 7 </ TEXTMENT>
<TIMEOUT sec = "20" goto = "self"/>
<SUCCESS goto = "next">
<DIST marker1 = "9" marker2 = "6"> 60 </ DIST>
<DIST marker1 = "9"> 400 </ Dist>
<VISB marker1 = "6" marker2 = "7"/>
</ SUCCESS>
<EXCEPTION goto = "27">
<DIST marker1 = "16"> 200 </ DIST>
</ EXCEPTION>
</ SCENE>

The XML schema used in one embodiment of the present invention is as follows.

<? xml version = "1.0" encoding = "ks_c_5601-1987"?>

<xsd: schema xmlns: xsd = "http://www.w3.org/2001/XMLSchema" xmlns = "Schema.xsd" elementFormDefault = "qualified" targetNamespace = "Schema.xsd">

  <xsd: element name = "SCENARIO">

    <xsd: complexType>

      <xsd: sequence>

        <xsd: element name = "SCENE" maxOccurs = "unbounded" type = "ctScene" />

      </ xsd: sequence>

    </ xsd: complexType>

  </ xsd: element>

  <xsd: complexType name = "ctScene">

    <xsd: sequence>

      <xsd: element name = "ANIMATE" type = "xsd: integer" />

      <xsd: element name = "VIDEO" type = "xsd: integer" />

      <xsd: element name = "SOUNDMENT" type = "xsd: integer" />

      <xsd: element name = "TEXTMENT" type = "xsd: integer" />

      <xsd: element name = "TIMEOUT">

        <xsd: complexType>

          <xsd: attribute name = "minute" use = "required" type = "xsd: integer" />

          <xsd: attribute name = "sec" type = "xsd: integer" />

          <xsd: attribute name = "goto" type = "xsd: integer" />

        </ xsd: complexType>

      </ xsd: element>

      <xsd: element name = "SUCCESS" type = "ctCompare" />

      <xsd: element name = "EXCEPTION" type = "ctCompare" />

    </ xsd: sequence>

    <xsd: attribute name = "id" use = "required" type = "xsd: integer" />

  </ xsd: complexType>

  <xsd: complexType name = "ctCondition">

    <xsd: sequence>

      <xsd: element name = "DIST">

        <xsd: complexType>

          <xsd: attribute name = "marker1" use = "required" type = "xsd: integer" />

          <xsd: attribute name = "marker2" use = "optional" type = "xsd: integer" />

          <xsd: attribute name = "marker3" type = "xsd: integer" />

        </ xsd: complexType>

      </ xsd: element>

      <xsd: element name = "VISB" type = "xsd: integer" />

    </ xsd: sequence>

    <xsd: attribute name = "goto" type = "xsd: integer" />

  </ xsd: complexType>

  <xsd: complexType name = "ctCompare">

    <xsd: sequence>

      <xsd: element name = "condition" type = "ctCondition" />

      <xsd: element name = "success">

        <xsd: complexType>

          <xsd: attribute name = "goto" type = "xsd: integer" />

        </ xsd: complexType>

      </ xsd: element>

      <xsd: element name = "failed">

        <xsd: complexType>

          <xsd: attribute name = "goto" type = "xsd: integer" />

        </ xsd: complexType>

      </ xsd: element>

    </ xsd: sequence>

  </ xsd: complexType>

  <xsd: element name = "SOUNDS">

    <xsd: complexType>

      <xsd: sequence>

        <xsd: element name = "MENT">

          <xsd: complexType>

            <xsd: attribute name = "id" type = "xsd: integer" />

            <xsd: attribute name = "path" type = "xsd: string" />

          </ xsd: complexType>

        </ xsd: element>

      </ xsd: sequence>

      <xsd: attribute name = "base_path" type = "xsd: string" />

    </ xsd: complexType>

  </ xsd: element>

  <xsd: element name = "MODELS">

    <xsd: complexType>

      <xsd: sequence>

        <xsd: element name = "MODEL" type = "ctModel">

        </ xsd: element>

      </ xsd: sequence>

      <xsd: attribute name = "base_path" type = "xsd: string" />

    </ xsd: complexType>

  </ xsd: element>

  <xsd: complexType name = "ctModel">

    <xsd: sequence>

      <xsd: element name = "TRANS">

        <xsd: complexType>

          <xsd: attribute name = "x" type = "xsd: float" />

          <xsd: attribute name = "y" type = "xsd: float" />

          <xsd: attribute name = "z" type = "xsd: float" />

        </ xsd: complexType>

      </ xsd: element>

      <xsd: element name = "ROTATE">

        <xsd: complexType>

          <xsd: attribute name = "x" type = "xsd: float" />

          <xsd: attribute name = "y" type = "xsd: float" />

          <xsd: attribute name = "z" type = "xsd: float" />

        </ xsd: complexType>

      </ xsd: element>

      <xsd: element name = "SCALE">

        <xsd: complexType>

          <xsd: attribute name = "x" type = "xsd: float" />

          <xsd: attribute name = "y" type = "xsd: float" />

          <xsd: attribute name = "z" type = "xsd: float" />

        </ xsd: complexType>

      </ xsd: element>

      <xsd: element name = "Visibility">

        <xsd: complexType>

          <xsd: attribute name = "id" type = "xsd: integer" />

          <xsd: attribute name = "path" type = "xsd: string" />

        </ xsd: complexType>

      </ xsd: element>

    </ xsd: sequence>

  </ xsd: complexType>

  <xsd: element name = "SENSORS">

    <xsd: complexType>

      <xsd: sequence>

        <xsd: element name = "MARKER" type = "ctSensor" />

      </ xsd: sequence>

    </ xsd: complexType>

  </ xsd: element>

  <xsd: complexType name = "ctSensor">

    <xsd: sequence>

      <xsd: element name = "MODEL" type = "xsd: integer" />

    </ xsd: sequence>

    <xsd: attribute name = "id" type = "xsd: integer" />

    <xsd: attribute name = "name" type = "xsd: string" />

  </ xsd: complexType>

</ xsd: schema>

5 shows the HMD 30 with the camera 10 used in the present invention. According to FIG. 5, a small camera 10 is attached to the front of the HMD 30 to obtain a camera image close to the human field of view.

FIG. 6 is a view showing training using the mixed reality training system 1 according to the present invention, and FIG. 7 is a view showing a user interface screen shown to a teacher through a monitor of a training control apparatus. To the trainer, only the 'user training image' portion of FIG. 7 is shown through the HMD 30.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

Figure 1 shows the overall configuration of the present mixed reality situation training system according to the present invention,

2 illustrates a marker detection process by the marker detection unit,

Figure 3 schematically shows a training environment by the situation of serving food,

4 exemplarily illustrates a situation control process by the situation controller;

5 shows a HMD with a camera used in the present invention,

Figure 6 illustrates the training using the mixed reality training system according to the present invention,

7 illustrates a user interface screen displayed to the teacher through the monitor of the training control device.

* Explanation of symbols for the main parts of the drawings

1: Mixed Reality Situation Training System

10: camera 20: training control device

21: camera input unit 22: marker detection unit

23: model data storage unit 24: situation control unit

25: image matching unit 26: HMD output unit

30: HMD

Claims (5)

A camera for photographing an image of real reality including one or more markers; A camera input unit for inputting an image captured by the camera, a marker detector for detecting all markers included in the image input by the camera input unit, and generating a marker matrix including position coordinates and rotation information, respectively; Trainer behavior information by generating and combining a model data storage unit for storing model data and the visibility of a specific marker, the distance between the marker and the marker, and the distance between the camera's viewpoint and the marker using a marker matrix transmitted from the marker detector. And a situation controller configured to determine scene information provided to the trainer and whether to transfer to the next scene by applying the trainer's behavior information to scene information according to a predefined scenario, and the image input by the camera input unit. The background image is to be transferred from the marker detection unit A train control device with the image registration unit for take-off to match the three-dimensional model, HMD output section for outputting the image matching by the image matching portion corresponding in the model data storage section using the matrix in the marker position; And And a head mounted display coupled to the camera integrally and displaying an image output by the HMD output unit. The method of claim 1, The marker detection unit After binarizing, labeling, and detecting the contour of the marker, the image of the image inputted by the camera input unit is compared, and the pattern of the detected marker is determined by comparing the pattern with predefined markers, and the size of the detected marker image and Mixed reality training system, characterized in that for generating a marker matrix containing the position coordinates and rotation information of the marker when the camera is the origin through the shape. The method of claim 2, The marker detection unit Mixed reality training system, characterized by applying a modified Kalman filter algorithm to the position coordinates of the marker. The method of claim 1, The scene information provided by the situation controller includes Mixed reality situation training system comprising at least one of animation, video, voice, text. The method of claim 1, The scene information according to the scenario in the situation controller Mixed reality contextual training system, characterized by an XML schema.

Images