CN100416336C - Calibrate real and virtual views - Google Patents

Abstract

A method for calibrating real and virtual views, comprising: tracking a calibration screen on which the real reference points generated by the real reference point generator are projected; aligning a view of the virtual reference point with a view of a real reference point in the display, wherein the real reference point generator and the display have a fixed relative position; determining point correspondences between the virtual reference points and the real reference points; and determining one or more parameters for rendering the virtual object in the real scene.

Description

Calibrate real and virtual views

technical field

The present invention relates to augmented reality, and more particularly, the present invention relates to systems and methods for augmented reality calibration of transparent head-mounted displays.

Background technique

Augmented vision (also known as augmented reality or augmented reality vision) uses superimposed computer-generated graphical information to augment the user's view of the real world. This information may include, for example, text labels added to some objects in the scene, or a three-dimensional (3D) model of the patient's brain from an MRI scan aligned to a realistic view of the person's head.

A user can observe the real world through his or her eyes and mix additional image information via a translucent display positioned between the observer and the real world. Such a display device may be, for example, an optically transparent head-mounted display.

The display may also be opaque, such as a computer screen or a non-transparent head-mounted display. The display then presents the user with a complete augmented view - a combination of real world view and graphical overlay. The camera captures the real world view in place of the real world observer. Two cameras can be implemented for stereo vision. Computers can be used to combine live video and graphics enhancements. Such a display device is eg a video transparent head mounted display (HMD).

Perspectively position, orient, adjust, and even render graphics for proper alignment with real-world views. In order to obtain precise alignment of real and virtual views, graphics can be fixed as real world objects. For this reason, knowledge of the position and orientation of the user's point of view relative to the object and object orientation is required. Therefore, a relationship needs to be defined between two coordinate systems, one to be added to the user's head and one to be added to the object.

Tracking means the process of keeping track of that relationship. Commercial tracking systems based on optical, magnetic, ultrasonic and mechanical means are available.

Calibration is required to obtain alignment between virtual graphical objects and real objects in the scene. Since real and virtual images are combined in a computer, video-transparent HMDs can be realized in an objective manner independent of the user. In contrast, with optically transparent HMDs, real and virtual images end up in the user's eyes, and the position of the user's eyes behind the translucent screen has a significant impact on alignment.

Different methods for calibrating optically transparent HMDs are considered to be prior art. All known calibration methods require the user to align a virtual structure to a real reference structure. For example, in the SPAAM method, a series of fixed graphic markers are displayed to the user on the screen, and the user moves the head so that these fixed graphic markers are aligned with reference markers in the real scene. Shaking of the user's head interferes with this alignment. Due to the shaking of the head, the positions of the displayed markers are shaken, and it is impossible to precisely align the virtual and real markers.

There are currently no known systems for augmented reality applications such as in operating rooms that require precise measurements and comfortable use. Accordingly, a need exists for a system and method for augmented reality calibration of transparent head-mounted displays.

Contents of the invention

An augmented reality system according to the invention comprising: a real reference generator for displaying a real reference on a calibration screen; an optically transparent display having a fixed position relative to said real reference generator; a virtual reference generator , for displaying a virtual reference on said optically transparent display; an input device, for aligning the view of said virtual reference with the view of said real reference through said optically transparent display, wherein said optically transparent display moving the virtual reference; a processor for determining one or more parameters for rendering a virtual object as part of a real scene seen through the optically transparent display; and a tracking device for implementing the real reference and the virtual reference alignment.

The augmented reality system includes a tracking camera for tracking the pose of the calibration screen relative to the virtual reference.

The augmented reality system includes a tracking camera with a fixed position relative to the reality reference generator for capturing the view of the calibration screen. The augmented reality system also includes a processor, wherein the optical marker configuration is affixed to the calibration screen and imaged by the tracking camera, wherein the processor determines a position between the calibration screen and the head mounted display based on the position of the optical marker configuration in the image captured by the tracking camera relationship, the head-mounted display includes a reality reference generator and an optically transparent display.

The augmented reality system includes: at least one tracking camera for capturing a view of the calibration screen; and a head mounted display including a reality reference generator and an optically transparent display. The augmented reality system also includes a processor, wherein the optical marker configuration is affixed to each of the calibration screen and the head-mounted display and is tracked by at least one tracking camera, wherein the processor is based on the respective optical markers in the view captured by the at least one tracking camera. Mark the position of the configuration to determine the positional relationship between the calibration screen and the HMD.

A system for calibrating real and virtual views according to the invention, comprising: a real reference generator for displaying a real reference on a calibration screen; an optical display having a fixed position relative to said real reference generator a virtual reference generator for generating a virtual reference on the optical display; an input device for aligning the view of the virtual reference with the view of the real reference, wherein on the optical display relative to the view of the real-world reference moves the virtual reference; a processor for determining one or more parameters for rendering a virtual object in a real-world scene seen in the optical display; and a tracking device for implementing a real-world Alignment of reference and virtual reference.

The system also includes a camera that captures a view of the real-world reference displayed in the optical display on which the virtual reference is superimposed. The system also includes a tracking camera having a fixed position relative to the real-world reference generator for capturing a view of the calibration screen; and a processor wherein the optical marker configuration is fixed on the calibration screen and tracked by the tracking camera, wherein the processing The sensor determines the positional relationship between the calibration screen and the head-mounted display, which includes the reality reference generator and the optical display, based on the position of the optical marker configuration in the view captured by the tracking camera.

The system includes a tracking camera connected to a reality reference generator for capturing views of the calibration screen. The system also includes a processor, wherein the optical marker configuration is fixed on the calibration screen and tracked by the tracking camera, wherein the processor determines a position between the calibration screen and the head mounted display based on the position of the optical marker configuration in the view captured by the tracking camera relationship, and the head-mounted display includes a reality reference generator and an optical display.

The system includes: at least one tracking camera for capturing a view of a calibration screen; and a head mounted display including a reality reference generator and an optical display. The system also includes a processor, wherein the optical marker configuration is affixed to each of the calibration screen and the head-mounted display and is tracked by the tracking cameras, wherein the processor determines the position of each optical marker configuration based on the position of the respective optical marker configuration in a view captured by the at least one tracking camera. Determine the positional relationship between the calibration screen and the HMD.

A method according to the invention for calibrating real and virtual views, comprising: tracking a calibration screen on which a real reference generated by a real reference generator is projected; aligning the virtual reference with said real in a display A view of the reference, wherein the real reference generator and the display have fixed relative positions; determining a point correspondence between the virtual reference and the real reference; and determining a point correspondence for rendering the virtual object in the real scene or parameters, wherein the virtual reference is displayed on an optically transparent display through which the real reference is visible.

The method includes capturing a view of a real-world scene including a real-world reference, and displaying the view of the real-world scene augmented with a virtual reference.

Description of drawings

Preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings:

FIG. 1 is an illustration of a calibration system according to an embodiment of the present disclosure;

2A is an illustration of an augmented reality calibration system according to an embodiment of the present disclosure;

2B is an illustration of an augmented reality calibration system according to an embodiment of the present disclosure;

2C is an illustration of a video-transparent augmented reality calibration system in accordance with an embodiment of the present disclosure;

FIG. 3 is a flowchart of a method according to an embodiment disclosed in the present invention;

FIG. 4 is a flowchart of a method according to an embodiment of the present disclosure.

Detailed ways

Systems and methods for calibrating an optically transparent head-mounted display (HMD) implement a real-world reference as a point of light originating from an illuminator attached to the HMD. These points of light "jitter together" with the HMD, and the user does not perceive any jitter between these reference markers and virtual markers displayed at fixed positions on the translucent screen of the HMD.

Referring to FIG. 1 , to calibrate an optically transparent system, a user 100 aligns a virtual reference 101 displayed as a graphic on a translucent screen 103 of the HMD with a real reference structure 102 viewed through the screen 103 . Reality reference structure 102 is implemented as projected light spots and/or images on calibration screen 104 or other photosensitive substrate. Real reference 102 and virtual reference 101 may be, for example, one or more points or shapes.

It should be understood that the present invention can be implemented in various forms of hardware, software, firmware, special purpose processors or combinations thereof. In one embodiment, the invention can be implemented in software as an application program actually implemented on a program storage device. The application program can be uploaded to and executed by a machine comprising any suitable structure.

It should also be understood that since some of the constituent system components and method steps shown in the figures may be implemented in software, the actual connections between system components (or process steps) may vary according to the way the invention is programmed. Given the teachings of the invention provided herein, one skilled in the art will be able to contemplate these and similar implementations or configurations of the invention.

Referring to FIGS. 2A and 2B , reality reference 102 originates from lighting system 200 rigidly attached to HMD 201 and viewed through screen 103 . As the user moves the HMD 201 , the real-world reference 102 moves on the calibration screen 104 and, for small head movements, appears fixed relative to the virtual reference 101 when viewed through the HMD's translucent screen 103 . From the vantage point of the user 100, the alignment process is now easy since the jitter between the real reference 102 and the virtual reference 101 is substantially reduced.

Viewing the reality reference 102 on a flat screen 104, the user may hold the arm's length screen 104 in one hand, place the screen on a table, and so on.

The screen 104 tracks the user's head or the HMD 201 . The tracking arrangement includes an external or (see FIG. 2A ) or helmet (see FIG. 2B ) tracking camera 202 . Screen 104 (in the case of optical tracking) and HMD 201 (in the case of the external tracking camera of FIG. 2A ) include optical markers 203 . Optical marker 203 may be, for example, a retroreflective flat disc or a retroreflective sphere.

The illuminator 200 projects a light image 102 comprising one (or preferably several) points. Illuminator 200 may be constructed using a single light source and an optical system that produces light image 102 . For example, a laser can be used with a lens system and mask, or a diffractive optical component, to create a cluster of spots.

Alternatively, a group of light sources (such as an LED array) can be used, which has the advantage that the light pattern can be made switchable. LEDs can be switched on and off individually. These LEDs can be combined with lens systems or micro-optical components such as lens arrays.

The illuminator 200 may also include scanning means or beam deflecting means to switch between different beam directions.

The screen 103 may be, for example, a monocular arrangement or a binocular arrangement. In a binocular arrangement, the two screens are calibrated individually, preferably one after the other, in combination with appropriate optical components 204 of the translucent display 103 to produce an image of the virtual reference 101 . Since the user sees through the translucent display 103, the virtual reference 101 is visible to the user. Alternatively, a transparent display can be implemented using an image projector and an optical system including a beam splitter.

To perform the calibration, the user 100 moves the virtual reference 101 displayed on the translucent screen 103 to align the reference light image 102 from the user's point of view (eg 205 ). User 100 controls interface 206 (eg, processor 207 and input device 208 ) in order to move virtual reference 101 on screen 103 . The input device 208 may be, for example, a trackball or a mouse. Processor 207 may be a virtual reference generator comprising a processor and a graphics card for rendering a displayed virtual reference on a translucent screen.

To complete the calibration process, the user aligns the virtual reference 101 with several real reference points of light (eg 102). For better calibration accuracy, the user may assume different poses (eg, distance and/or orientation) relative to the calibration screen 104 .

A processor (eg, 207 ) determines the spatial relationship between calibration screen 104 and HMD 201 based on the location of marker 203 and the user-determined alignment of virtual reference 101 and real-world reference 102 . FIG. 2B is an example of an HMD including a tracking camera 202 . As shown, where a tracking camera is affixed to the HMD, only optical markers 203 affixed to the calibration screen 104 can be used to determine the spatial relationship between the calibration screen 104 and the HMD 201 . Furthermore, the pose of the calibration screen 104 can be determined according to the relationship of different optical markers 203 fixed to the screen 104 .

FIG. 2C is a video-transparent augmented reality system in which a camera (eg, 202 ) captures an image of a real-world scene including a real-world reference 102 , according to an embodiment of the present disclosure. The tracking and video functions of camera 202 may be performed by separate cameras. An image of the real world scene is displayed for the user 100 . The virtual view is superimposed on the real view (eg 209 ), and the user 100 perceives both the real and the virtual view. The user can align the virtual reference with the view of the real reference in the real scene.

Referring to Figure 3, consider the case of a helmet tracking camera. The tracking camera and illuminator are mechanically fixed to each other, and the position and orientation of the light beam that creates the reference point is determined relative to the tracking camera's coordinate system. By tracking the calibration screen 301 , the 3D coordinates of the reference point in the tracking camera coordinate system can be determined as the intersection point of the corresponding light beam with the screen plane.

During the calibration process, the user aligns real and virtual references 302 and the system records 303 a set of 3D-2D point correspondences. Each point correspondence is composed of the 3D coordinates of the reference light point and the 2D coordinates of the virtual marker to which the user has aligned the reference. The set of point correspondences enables determination of one or more parameters for rendering a virtual object properly aligned with real world scene 304 . For example, camera parameters determining the user's view of the virtual world displayed on the translucent screen match camera parameters determining the user's view of the real world seen through the screen. These determinations are known in the art, for example, in U.S. Patent Application No. 20020105484, filed September 25, 2001, entitled "System and Method for Calibrating a Monocular Optically Transparent Head-mounted Display System for Augmented Reality" Said, wherein the calibration may include matching the physical environment to the internal representation, matching the computer's internal model to the initial parameter values of the mathematical model of the physical environment. These parameters include, for example, the optical properties of the physical camera, and position and orientation (pose) information of various entities such as the camera, markers used for tracking, and various objects.

After successfully calibrating an optically transparent augmented reality system for a single user, the system can render 3D graphics objects in such a way that they appear firmly fixed in the displayed scene. A tracking system is used to track and account for changes in the user's point of view using corresponding changes in the virtual view of the graphical objects.

As an alternative to the case of a helmet tracking camera, an external tracking device could be used with helmet markers or sensors fixedly fixed relative to the illuminators. The tracking system tracks the HMD and calibrates the screen (401). Likewise, the 3D coordinates of the calibration spot can be aligned (402) and can be determined as the intersection of the beam and the screen plane (403). The virtual reference point is aligned with the real reference point displayed as light on the screen, and the correspondence is recorded (404).

The system includes a head mounted display, a tracking device, a computing and graphics rendering device, a light projection device, and a trackable screen.

The calibration alignment between real and virtual reference structures can be averaged over several measurements corresponding to each point. Note that the virtual and real markers show no jitter relative to each other. Averaging reduces errors in calibration. Averaging is easy to use compared to calibration procedures that use external features. Here, the user can maintain alignment for a second or a few seconds due to reduced jitter between real and virtual markers.

Having described embodiments of systems and methods for calibrating real and virtual views, it is noted that modifications and variations may be implemented by those skilled in the art in light of the above teaching. It is therefore to be understood that changes may be effected in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as defined by the appended claims. Having thus described the invention with such details as are required by the patent laws, what is claimed and desired protected is set forth in the appended claims.

Claims (15)

1. An augmented reality system comprising: A Reality Reference Generator for displaying a Reality Reference on the calibration screen; an optically transparent display having a fixed position relative to said reality reference generator; a virtual reference generator for displaying a virtual reference on said optically transparent display; an input device for aligning the view of the virtual reference with the view of the real reference through the optically transparent display, wherein the virtual reference is moved across the optically transparent display; a processor for determining one or more parameters for rendering a virtual object as part of a real-world scene seen through said optically transparent display; and A tracking device for achieving alignment of real and virtual references. 2. The augmented reality system of claim 1 , further comprising a tracking camera for tracking the pose of the calibration screen relative to the virtual reference. 3. The augmented reality system of claim 1 , further comprising a tracking camera having a fixed position relative to the reality reference generator for capturing a view of the calibration screen. 4. The augmented reality system of claim 3 , further comprising a processor, wherein an optical marker configuration is affixed to the calibration screen and imaged by the tracking camera, wherein the processor is based on images captured by the tracking camera. The position of the optical marker configuration in the image is used to determine the positional relationship between the calibration screen and a head mounted display including the reality reference generator and an optically transparent display. 5. The augmented reality system of claim 1 , further comprising: at least one tracking camera for capturing a view of the calibration screen; and a head mounted display including the reality reference generator and an optically transparent display. 6. The augmented reality system of claim 5 , further comprising a processor, wherein an optical marker configuration is affixed to each of the calibration screen and the head mounted display and is tracked by the at least one tracking camera , wherein the processor determines the positional relationship between the calibration screen and the head mounted display based on the position of each optical marker arrangement in the view of the calibration screen captured by the at least one tracking camera. 7. A system for calibrating real and virtual views comprising: A Reality Reference Generator for displaying a Reality Reference on the calibration screen; an optical display having a fixed position relative to said reality reference generator; a virtual reference generator for generating a virtual reference on said optical display; an input device for aligning the view of the virtual reference with the view of the real reference, wherein the virtual reference is moved relative to the view of the real reference on the optical display; a processor for determining one or more parameters for rendering a virtual object in a real world scene seen in said optical display; and A tracking device for achieving alignment of real and virtual references. 8. The system for calibrating real and virtual views of claim 7, further comprising a camera capturing a view of the real reference, wherein the real reference is displayed in the optical display, the virtual reference being superimposed on said realistic reference. 9. The system for calibrating real and virtual views of claim 8, further comprising: a tracking camera, having a fixed position relative to the reality reference generator, for capturing a view of the calibration screen; and a processor, wherein the optical marker configuration is fixed to the calibration screen and tracked by the tracking camera, wherein the processor determines the calibration screen based on the position of the optical marker configuration in the view captured by the tracking camera and a positional relationship between a head-mounted display including the reality reference generator and an optical display. 10. The system for calibrating real and virtual views of claim 7, further comprising a tracking camera connected to said real reference generator for capturing a view of said calibration screen. 11. The system for calibrating real and virtual views of claim 10, further comprising a processor, wherein an optical marker configuration is fixed to the calibration screen and tracked by the tracking camera, wherein the processor according to The tracking camera captures the position of the optical marker arrangement to determine a positional relationship between the calibration screen and a head mounted display including the reality reference generator and an optical display. 12. The system for calibrating real and virtual views of claim 7, further comprising: at least one tracking camera for capturing a view of said calibration screen; and a head-mounted display including said real-world reference generator and optical displays. 13. The system for calibrating real and virtual views of claim 12, further comprising a processor, wherein optical marker configurations are fixed to each of said calibration screen and said head-mounted display, and are tracked by said camera tracking, wherein the processor determines the positional relationship between the calibration screen and the head mounted display based on the position of each optical marker arrangement in the view of the calibration screen captured by the at least one tracking camera. 14. A method for calibrating real and virtual views, comprising: tracking a calibration screen on which a reality reference generated by a reality reference generator is projected; aligning a virtual reference with a view of said real reference in a display, wherein said real reference generator and said display have fixed relative positions; determining point correspondences between the virtual reference and the real-world reference; and determining one or more parameters for rendering the virtual object in the real world scene, wherein said virtual reference is displayed on an optically transparent display through which said real reference is visible. 15. The method for calibrating real and virtual views of claim 14, further comprising capturing a view of a reality scene including said reality reference; and A view of the real world scene augmented with the virtual reference is displayed.

Images