KR20210042784A - Smart glasses display device based on eye tracking - Google Patents

Abstract

According to the present invention, provided is a device, which includes: a plurality of distance measurement units for measuring a first distance from the fundus and a second distance from the surface of the eyeball by irradiating a laser to the eyeball including the fundus; an eye detection unit detecting an eye direction based on the first distance and the second distance; an image acquisition unit for acquiring an overlay image; a screen synthesizing unit for recognizing an eye-contact object within a predetermined range in an eye direction and calculating a position to display the overlay image at a position designated according to the eye-contact object; and a display unit being in the form of smart glasses and displaying the overlay image at the calculated position. Accordingly, according to the present invention, in the display method and the device thereof, related information can be naturally displayed without obstructing the eyes of a doctor.

Description

Smart glasses display device based on gaze detection {SMART GLASSES DISPLAY DEVICE BASED ON EYE TRACKING}

The present invention relates to a smart glasses display device based on gaze detection, and more particularly, to a gaze detection based smart glasses display device that detects a user's gaze and provides a natural overlay image according to the gaze direction through smart glasses. .

Recently, medical and healthcare are in the spotlight as applications of Virtual Reality (“VR”), Augmented Reality (“AR”), or Mixed Reality (“MR”). VR, AR, and MR technologies using smart glasses are emerging as an alternative to nurturing medical experts and psychological treatment to respond to the increasing demand for medical services such as psychological diseases caused by the advent of the aging age and intensifying competition. Particularly, VR, AR, and MR have been extended to all areas of medical care, such as surgery, treatment, and rehabilitation, away from the conventionally limited use for trauma treatment. With VR, AR, and MR, training training is possible by implementing virtual patients or organs, and medical staff can make appropriate plans before surgery and test them in advance.

Conventional medical equipment that did not use VR, AR, and MR, for example, an ultrasound scanner, has to check the ultrasound image to be displayed on a separate monitor screen, so it is cumbersome for the doctor to alternately check the patient and the monitor for treatment. Existed. Or, even remote surgery equipment or robotic surgery equipments were often unable to efficiently show various information necessary for surgery to medical staff. Therefore, there are cases in which the head is turned to see the necessary information, or the doctor's intuition needs to be treated without the necessary information during surgery.

Due to such hassle and distraction, it acted as a factor that hindered the doctor's concentration at the time of diagnosis. In particular, when a situation in which the concentration of the medical staff is distracted occurs when a situation in which it is necessary to view the related information screen during diagnosis or surgery occurs, the patient may be in danger or miss a part to be observed with concentration.

Accordingly, by introducing VR, AR, and MR to conventional medical equipment, the demand for a display method capable of maximizing the concentration of a doctor during diagnosis and maximizing the convenience of medical staff is gradually increasing.

The technology that is the background of the invention has been prepared to facilitate understanding of the present invention. It should not be understood as an admission that the matters described in the technology underlying the invention exist as prior art.

The present invention provides an innovative display method that prevents the doctor's gaze from dispersing during diagnosis by displaying at a desired point in time and at a desired point in a medical device through equipment such as AR, MR, and VR.

In particular, the problem to be solved by the present invention is to detect the user's gaze and display an image or image containing relevant information in a specific area that intuitively feels comfortable, so that the user can more accurately grasp the patient's affected area or the inside of the body through a virtual object. It is to provide a smart glasses display device based on gaze detection.

The problems of the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above-described problems, the smart glasses display device based on gaze detection according to an embodiment of the present invention irradiates a laser to an eyeball including a fundus to determine a first distance from the fundus and a control from the surface of the eyeball. 2 A plurality of distance measuring units for measuring the distance; A gaze detector configured to detect a gaze direction based on the first distance and the second distance; An image acquisition unit that acquires an overlay image; A screen synthesizing unit for recognizing a gaze matching object within a certain range in the gaze direction and calculating a position to display the overlay image at a designated location according to the gaze matching object; And a display unit in the form of smart glasses to display the overlay image at the calculated position.

In this case, the line of sight detector may sense the position of the pupil based on information on whether the distance measured by each of the plurality of distance measuring units is a first distance or a second distance.

In this case, the gaze detection unit may detect a gaze direction by sensing a direction toward the pupil from the eyeball based on a first distance measured by some of the plurality of distance measurement units.

In addition, at least a portion of the plurality of distance measuring units may be disposed along a support for supporting the display unit.

In addition, the display unit provides a screen including a circular guide area, and the screen combining unit detects the user's gaze direction when the gaze moves along the provided guide area, and based on the gaze direction detected according to the guide area. A reference value for gaze correction can be obtained.

In addition, the line of sight correction reference value may include an eccentricity, flatness, or a focal position of an ellipse.

In addition, the screen synthesizing unit may sense light irradiated by the gaze matching object and extract a reference point for displaying the overlay image.

In addition, the screen synthesizing unit may extract a point spaced apart from the reference point by a predetermined interval as a screen boundary point.

In addition, the display unit may display the overlay image in a 3D stereoscopic shape.

In addition, the display unit may include a microdisplay, a lens, a panel, and a splitter.

On the other hand, the gaze detection-based smart glasses display method according to the present invention irradiates a laser to the eyeball including the fundus by a plurality of distance measuring units to determine a first distance from the fundus and a second distance from the surface of the eyeball. A distance measuring step of measuring; A gaze detection step of detecting a gaze direction based on the first distance and the second distance; An image acquisition step of obtaining an overlay image; A display position calculation step of recognizing a gaze matching object within a certain range in the gaze direction and calculating a position to display the overlay image at a designated position according to the gaze matching object; And a display step of displaying the overlay image at the calculated position through a display unit in the form of smart glasses.

In this case, in the step of detecting the line of sight, the position of the pupil may be sensed based on information on whether a distance measured by each of the plurality of distance measuring units is a first distance or a second distance.

In the step of detecting the gaze, the direction of the eyeball may be detected by sensing a direction in which the pupil is directed from the eyeball based on a first distance measured by some of the plurality of distance measuring units.

In addition, the method for displaying gaze detection-based smart glasses includes the steps of: providing, by the display unit, a screen including a guide area having a circular shape; When the user's gaze moves along the provided guide area, detecting the user's gaze direction and obtaining a gaze correction reference value based on the detected gaze direction according to the guide area.

In addition, the line of sight correction reference value may include an eccentricity, flatness, or a focal position of an ellipse.

In addition, the step of calculating the display position may include sensing light irradiated by the eye-catching object and extracting a reference point for displaying the overlay image.

In addition, the calculating of the display position may further include extracting a point spaced apart from the reference point as a screen boundary point.

In addition, the displaying step may include displaying the overlay image in a 3D stereoscopic shape.

On the other hand, the computer-readable recording medium according to an embodiment of the present invention irradiates a laser to the eyeball including the fundus by a plurality of distance measuring units to determine a first distance from the fundus and a second distance from the surface of the eyeball. It measures, detects a gaze direction based on the first distance and the second distance, acquires an overlay image, recognizes a gaze target within a certain range in the gaze direction, and a designated position according to the gaze target Commands for displaying the overlay image at the calculated position may be stored by calculating a position so that the overlay image can be displayed on a display unit in the form of smart glasses.

The solution to the problem according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.

Basically, the present invention can provide a device capable of displaying necessary information without dispersing the doctor's gaze at a desired viewpoint and a desired point by using VR, AR, and MR.

The present invention has an effect of displaying an augmented reality image that moves based on the gaze movement in a specific area that the user feels intuitively and comfortably, so that the user can grasp the affected part of the patient or the inside of the body through a virtual object with concentration without dispersing the gaze.

The effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is an exemplary view showing a display process of smart glasses according to an embodiment of the present invention.
2A is a schematic diagram of a smart glasses display system based on gaze detection according to an embodiment of the present invention.
2B is a block diagram of a smart glasses display device based on gaze detection according to an embodiment of the present invention.
2C is a block diagram of a smart glasses display device based on gaze detection according to an embodiment of the present invention.
3A and 3B are diagrams for explaining a calibration mode according to line-of-sight detection according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a process of generating an image on a display according to an embodiment of the present invention.
5 is a diagram illustrating a method of displaying smart glasses according to another exemplary embodiment of the present invention.
6 is a diagram illustrating a method of displaying smart glasses according to another exemplary embodiment of the present invention.
7 is a diagram illustrating a method of displaying smart glasses according to an exemplary embodiment of the present invention.

Advantages of the invention and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms. It is provided to completely inform the scope of the invention to the possessor, and the invention is only defined by the scope of the claims.

The shapes, sizes, ratios, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are exemplary, and the present invention is not limited to the illustrated matters. In addition, in describing the present invention, when it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. When'include','have','consists of' and the like mentioned in the present specification are used, other parts may be added unless'only' is used. In the case of expressing the constituent elements in the singular, it includes the case of including the plural unless specifically stated otherwise.

In interpreting the components, even if there is no explicit description, it is interpreted as including an error range.

Each of the features of the various embodiments of the present invention may be partially or entirely combined or combined with each other, and as a person skilled in the art can fully understand, technically various interlocking and driving are possible, and each of the embodiments may be independently implemented with respect to each other. It may be possible to do it together in a related relationship.

1 is an exemplary view showing a process of displaying an overlay image according to an embodiment of the present invention. 2A is a schematic diagram of an augmented reality display system based on gaze detection according to an embodiment of the present invention. 2B is a block diagram of a smart glasses display device based on gaze detection according to an embodiment of the present invention. 2C is a block diagram of a device for displaying smart glasses according to an exemplary embodiment of the present invention. 3A and 3B are diagrams for explaining a gaze direction correction according to gaze detection according to an embodiment of the present invention. 4 is a diagram for explaining a process of displaying an overlay image on a display.

Referring to FIG. 1, in the smart glasses display system based on gaze detection, the image or image acquired from the image acquisition unit 205 is applied to a specific area on the display according to the gaze matching object 200 in augmented reality, mixed reality, or virtual reality. It includes a gaze detection-based smart glasses display device 100 and a gaze detection server 900 that displays an overlay image or an image. However, in the embodiment of FIG. 1, although the gaze detection-based smart glasses display device 100 and the gaze detection server 900 are shown in separate forms, the gaze detection-based smart glasses display device 100 and the gaze detection server ( 900) may be mounted in one housing. When mounted in one housing, the gaze detection-based smart glasses display device 100 and the gaze detection server 900 may be connected by wire.

In addition, in the present embodiment, the case where the line-of-sight object 200 and the image acquisition unit 205 are implemented in one device as an ultrasound scanner is illustrated, but the line-of-sight object 200 and the image acquisition unit 205 are separate devices. May be. For example, in a remote surgical robot, the eye contact object 200 may be a scalpel or a surgical site of a surgical patient. The image acquisition unit 205 may be, for example, a unit that acquires an image indicating an area in which the scalpel should not pass through the nerves of a surgical patient from an image stored in the gaze detection server 900. In other words, the gaze detection server 900 may store an image in which the patient's nerves are displayed based on the image captured by the patient before the surgery.

The gaze detection-based smart glasses display device 100 detects the user's gaze by a plurality of built-in distance measuring units 11, and displays the image acquired from the image acquisition unit 205 on the screen of the display unit 103. It is a device that can be displayed on. Here, the smart glasses display device 100 recognizes the eye contact object 200 and then displays the image acquired from the image acquisition unit 205 in a predetermined area with respect to the recognized eye contact object 200. . In this case, the eye contact object 200 may be an electronic device or a general tool used for surgery or treatment. For example, the eye contact object 200 may be an ultrasound scanner, scissors, clips, scalpels, or the like. In the present embodiment, description will be made on the basis that the line-of-sight object 200 is an ultrasound scanner.

In the present invention, the gaze detection-based smart glasses display device 100 includes the screen 114 projected from the microdisplay 110 (refer to FIG. 2B) of the display unit 103 with both lenses 111, the panel 112, and the splitter. It may be reflected through (113) and visible to the eye. Alternatively, an external screen shot through the external camera 12 of the smart glasses display device 100 or a separate separate camera in the see-through mode while utilizing the HMD display is an overlay screen through the screen of VR. It may be a form that overlaps with and shows.

2A, 2B, and 2C, the smart glasses display device 100 based on gaze detection includes a pupil sensing unit 101, a gaze detection unit 102, a display unit 103, a screen synthesizing unit 104, and It may include a control unit 105. The pupil sensing unit 101, the gaze detection unit 102, and the display unit 103, the screen synthesis unit 104 and the control unit 105 may be composed of software, hardware, or a combination of software and hardware, those skilled in the art These components can be implemented using appropriate means.

The pupil sensing unit 101 is configured to sense the pupil of the user, and includes a plurality of distance measuring units 11. In this case, a plurality of distance measuring units 11 may be disposed along the edge of the eyeglasses of the smart glasses display device 100 based on gaze detection.

The distance measuring unit 11 senses the position of the pupil 15 in order to determine the movement of the user's gaze, and is a sensor including a laser light emitting unit and a light receiving unit, and preferably irradiates an infrared laser in a wavelength band without damage to the eyesight of the pupil. do. In FIG. 2A, for convenience of explanation, a laser irradiation line LL irradiated from the distance measuring unit 11 is illustrated.

4, the plurality of distance measuring units 11 irradiate infrared rays to the eyeball 10, and then the distance reflected from the fundus of the eyeball through the pupil 15 (d1 + d2) or directly reflected from the surface of the eyeball. It is possible to measure the distance (d2). That is, the reflection distance may be divided into a distance d1 from the fundus to the eyeball surface (or pupil) and a distance d2 from the eyeball surface to the sensor unit 114. In this case, the position of the pupil is sensed by measuring the distances d1 and d2 obtained through the plurality of sensors.

Specifically, the laser light irradiated by some of the user's distance measuring unit 11 passes through the pupil 15 and then is reflected from the fundus inside the user's eyeball 10, and again through the pupil 15, the distance measuring unit 11 ) Is sensed.

At this time, only some of the plurality of laser lights irradiated by the plurality of distance measuring units 11 may pass through the pupil. The laser light that has not passed through the pupil is reflected from the surface of the eyeball 10, but is incident on the light receiving unit of the distance measuring unit 11.

The length d1 from the fundus to the eyeball surface and the length d2 from the eyeball surface 10 to the display unit 103 may be measured using lasers irradiated by the plurality of distance measuring units 11. That is, since the distance (d1+d2) passing through the pupil 15 is relatively shorter than the distance (d1 + d2) not passing through the pupil 15, the location of the user's pupil 15 is determined through length comparison. I can grasp it. In addition, based on whether the laser light irradiated by one distance measuring unit 11 passes through the pupil and/or the distance measured from the passed light according to the distance information of the plurality of distance measuring units 11, the pupil from the eyeball You can accurately determine this direction.

Accordingly, the present invention measures distances to the fundus from various locations where the plurality of distance measuring units 11 are located, as shown in FIG. 4, so that it is possible to derive a very detailed gaze direction.

In this case, the plurality of distance measuring units 11 are upper (L1) and lower ( It is placed along L2). However, the location of the distance measuring unit 11 is not limited thereto, and is disposed on the side of the rear side of the eyeglass frame of the smart glasses display device 100 based on gaze detection, or the display unit 103 of the smart glasses display device 100 It may be disposed directly on the lens 111 or the panel 112.

The gaze detection unit 102 is configured to sense a user's gaze direction for an augmented reality image to be displayed on the display unit 103, and the distance (d1 + d2) reflected from the fundus of the eyeball through the pupil, and the surface of the eyeball The gaze direction may be detected based on the distance d2 directly reflected from and the position of the pupil. A more detailed description of the pupil sensing unit 101 and the line of sight detection unit 102 will be described later with reference to FIGS. 3A to 3B.

The display unit 103 is a component that displays the image acquired from the gaze matching object 200 on the display unit 103 as an augmented reality, virtual reality, or mixed reality image. For example, referring to FIG. 2B, the display unit 103 may include a microdisplay 10, a lens 111, a panel 112, and a splitter 13. In this case, when the microdisplay 10 irradiates an image through the lens 111 disposed in front, the irradiated image reaches a region of the panel 112 through the lens 111 and then the display panel 12 The image is totally reflected by the splitter 13 attached to one side, and the image 114 can be seen by the user's eyes again by the splitter 13 in front of the eye.

In addition, the display unit 103 may display the image or image acquired by the image acquisition unit 205 as an augmented reality, virtual reality, or mixed reality image, as well as various information related to the acquired image. For example, when a medical staff cuts a blood vessel of a patient during surgery, the remaining time until safely connecting the blood vessels remains approximately 60.0 seconds may be displayed by overlapping with the first image 210 as the second image 220. .

The augmented reality, virtual reality, or mixed image (hereinafter referred to as an overlay image) acquired by the image acquisition unit 205, for example, an image scanned by an ultrasound scanner, displays the display 103 at a specific position designated according to the eye contact object 200. Overlaid through. For example, the image acquired by the image acquisition unit 205 may be an ultrasound image of a mother undergoing ultrasound examination. In this case, the smart glasses display device 100 recognizes whether the eye-catching object 200 (the ultrasound scanner in the embodiment of FIG. 4) is within a specific range 250 from the eye line direction. In addition, the overlay image acquired by the acquisition unit 205 is superimposed on a specific position (eg, an ultrasound irradiation surface of the ultrasound scanner in the case of an ultrasound scanner) according to the eye-catching object 200.

The image is displayed on the left and right lenses of the smart glasses display device 100 based on gaze detection. In this case, the left lens and/or the right lens may display the same image. As illustrated in FIG. 1B, the left-eye image T1 ′ and the right-eye image T1 ′ of the same object are displayed so that the user may recognize the virtual object T1 in a three-dimensional shape (3D).

In addition, the image may move within the display unit 103 based on the user's gaze movement. A detailed description of this will be described later.

The screen synthesizing unit 104 may arrange and synthesize the image or image acquired by the image acquisition unit 205 in an appropriate position. The screen synthesizing unit 104 checks the position of the gaze matching object 200 through the image acquired from the external camera 12, and calculates the position at which the overlay image should be displayed in conjunction with the gaze direction. In the ultrasound scanner embodiment of FIG. 4, the position where the overlay image is to be displayed is an ultrasound irradiation surface of the ultrasound scanner. The screen synthesizing unit 104 may control the display 103 to display the ultrasound image at the calculated position. In this case, when scanning the mother's belly with ultrasound, the doctor can check whether the scanned screen is well displayed while looking at the ultrasound irradiation surface. Therefore, the doctor can perform an ultrasound examination while adjusting the position of the fetus and the desired part to be scanned well without distracting the eyes.

When the smart glasses display device 100 is an HMD showing virtual reality, the screen synthesizing unit 104 may display an image of the external camera 12 and an image of the image acquisition unit 205 by overlapping the image. However, even in this case, after calculating the position where the overlay image should be displayed in conjunction with the gaze direction, the image of the external camera 12 and the image of the image acquisition unit 205 are superimposed and displayed.

The control unit 105 may perform a function of controlling overall operations of the pupil sensing unit 101, the gaze detection unit 102, the display unit 103, and the screen combining unit 104.

Hereinafter, a method of correcting the position of the pupil and the distance estimated from the measured distances d1 and d2 by the screen synthesizing unit 104 will be described in detail with reference to FIGS. 3A and 3B.

Specifically, as shown in FIG. 3A, the screen synthesizing unit 104 provides an arbitrary screen 300 displaying a circular guide area TR to the user through the display 103 in the calibration mode. Here, the calibration mode is a mode for measuring the user's gaze direction, and is. The user gazes clockwise (based on the user, on the other hand, may be counterclockwise) along the edge of the guide area TR from the initial starting point SP along the guide area TR displayed on an arbitrary screen 300 You are guided to move.

At this time, when the user moves the gaze direction along the guide area TR, the gaze detection unit 102 detects the gaze direction of the user. In this case, as shown in FIG. 3B, the ideal user's ideal has an ideal circular trajectory, but the actual user's actual eyes are narrower in width and wider in vertical width than the ideal user's ideal. It is formed to have an elliptical orbit. In other words, the gaze direction in which the user can comfortably gaze generally has an elliptical shape rather than an exact circular shape. In this case, the screen synthesizing unit 104 calculates a gaze correction reference value based on the actual gaze direction measured in the calibration mode. In this case, the gaze correction reference value may include values representing the ellipse, such as eccentricity, flatness, and a focal position of the ellipse.

Accordingly, the smart glasses display device 100 according to the present invention may correct the gaze direction detected by the gaze detection unit 102 based on the gaze correction reference value. Accordingly, it is possible to accurately estimate the gaze direction that the user actually intends to look at, and the accuracy of the display position of the overlay image is remarkably increased in screen processing.

In the present embodiment, the position of the eye contact object 200 is determined based on the image acquired through the external camera 12, but the present invention is not limited thereto. For example, the screen synthesizing unit 104 may grasp the position of the eye aligning object 200 by using beacons 260 mounted on at least three positions of the eye aligning object 200. That is, the screen synthesizing unit 104 may periodically detect the signal of the beacon 260 mounted on the gaze matching object 200 to determine the position of the gaze matching object 200.

Hereinafter, another embodiment according to the present invention will be described with reference to FIG. 5.

In the embodiment of FIG. 5, the line-of-sight object 200 may be light irradiated by a light irradiation unit 260 mounted on one surface of an ultrasound scanner. For example, when a doctor wants to view an entire ultrasound image at a desired location while examining through an ultrasound scanner, the doctor turns on the light irradiation unit 260 mounted on the ultrasound scanner to display the desired location.

In this case, when the light irradiated by the light irradiation unit 260 is sensed through the external camera 12, the screen boundary points P1-P4 and P2 are centered on the irradiated point as shown in FIG. 5. -P3) is calculated. Here, assuming that the point where the infrared laser is irradiated is the reference point, four points P1, P2, P3, and P4 spaced at predetermined intervals in a diagonal direction around the reference point may be extracted as a screen boundary point. Accordingly, the screen synthesizing unit 104 may calculate the position of the overlay image 210 based on the light irradiated by the light irradiation unit 260, which is the line-of-sight object 200.

Meanwhile, as illustrated in FIG. 2A, as an exemplary embodiment, some components may be deleted or new components may be added according to the needs of a person skilled in the art. For example, it may further include a controller connected to the smart glasses display device 100 based on gaze detection by wired or wirelessly.

Hereinafter, another embodiment according to the present invention will be described with reference to FIG. 6.

In the embodiment of FIG. 6, the eye contact object 200 may be a scalpel and a face of a person to be operated on. In addition, in this case, the overlay image 210 may be a predetermined operating line. For example, when a doctor holds a scalpel and performs an operation in which a patient's face is incised, an image of the operating line may be overlaid on the patient's face. In this case, when the patient's face and the scalpel are detected among the images captured by the external camera 12 within a certain range of the operator's gaze direction, the surgical line, which is an overlay image, is overlapped and displayed on the patient's face. That is, there may be a plurality of gaze-matching objects 200, and when several conditions (when at least a part of a scalpel and a person's face is detected) are satisfied, the screen synthesizing unit 104 is a surgery that is the gaze-matching target 200 The position of the overlay image 210 may be calculated based on the subject's face. Alternatively, in this case, the overlay image 210 may be information about a situation for reference (information on whether the depth at which the knife to be picked enters the skin is appropriate). Alternatively, positions of nerves/blood vessels that should never be touched during surgery may be displayed as the overlay image 210. In this case, the overlay image 210 to be overlaid (or overlapped) may be displayed by a known image processing method including a block search or edge detection method of an image.

Accordingly, the present invention displays an overlay image that moves based on the gaze direction on a specific area of the smart glasses, so that the gaze of the medical staff is not distracted and only the original tasks such as surgery or examination can be concentrated.

Hereinafter, a method of displaying gaze detection-based smart glasses according to an embodiment of the present invention will be described in detail with reference to FIG. 7.

First, in the display method of smart glasses according to an embodiment of the present invention, by irradiating a laser to the eyeball including the fundus by a plurality of distance measuring units 11, the first distance from the fundus and from the surface of the eyeball. It includes a distance measurement step (S110) of measuring the second distance.

In this case, the gaze detection unit 102 is based on whether the distance measured by each distance measuring unit 11 is reflected from the fundus or reflected from the eyeball surface, that is, based on the first distance and the second distance. The direction of the gaze can be detected. (Gaze detection step, S120)

Meanwhile, the display method of smart glasses acquires an overlay image. As described above, the overlay image can be obtained from an ultrasound scanner, an external camera, a separate connected camera, or a server storing an image/image. (Image acquisition step, S130) In this case, the line of sight detection unit 102 may sense the position of the pupil based on information on whether the distance measured by each of the plurality of distance measuring units is a first distance or a second distance. . Alternatively, the gaze detection unit 102 may detect the direction of the eyeball by sensing a direction in which the pupil is directed from the eyeball based on a first distance measured by some of the distance measurement units among the plurality of distance measurement units.

The screen synthesizing unit 104 may calculate a position so as to recognize the gaze matching object within a certain range in the gaze direction and display the overlay image at a designated position according to the gaze matching object. (Display position calculation step, S140) At this time, after the screen combining unit 104 provides a screen including a circular guide area by the display unit 103, when the user moves the line of sight along the provided guide area , The user's gaze direction may be detected, and a gaze correction reference value may be obtained based on the gaze direction detected according to the guide area.

Meanwhile, the operation of calculating the display position (S140) may include sensing light irradiated by the eye-catching object and extracting a reference point for displaying the overlay image. In this case, the step of calculating the display position may further include extracting a point spaced apart from the reference point by a predetermined interval as a screen boundary point.

Finally, the display unit 103 in the form of smart glasses may include a display step of displaying the overlay image at the calculated position.

Accordingly, according to the present invention, it is possible to provide a device capable of displaying necessary information without dispersing the doctor's gaze at a desired viewpoint and a desired point by using VR, AR, and MR.

In addition, the present invention displays an augmented reality image that moves based on gaze movement in a specific area that the user feels intuitively and comfortably, so that the user can grasp the affected part of the patient or the inside of the body through a virtual object with concentration without dispersing the gaze. have.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made without departing from the spirit of the present invention. . Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: gaze direction-based smart glasses display device
10: pupil 11: distance measuring unit
12: photographing unit LL: laser irradiation
200: eye catching object 210: overlay image

Claims (19)

A plurality of distance measuring units configured to measure a first distance from the fundus and a second distance from the surface of the eye by irradiating a laser to the eyeball including the fundus;
A gaze detector configured to detect a gaze direction based on the first distance and the second distance;
An image acquisition unit that acquires an overlay image;
A screen synthesizing unit for recognizing a gaze matching object within a certain range in the gaze direction and calculating a position to display the overlay image at a designated location according to the gaze matching object; And
Displaying the overlay image at the calculated position and including a display unit in the form of smart glasses, gaze detection-based smart glasses display device. The method of claim 1,
The gaze detection unit senses the position of the pupil based on information on whether a distance measured by each of the plurality of distance measurement units is a first distance or a second distance. The method of claim 1,
The gaze detection unit detects a gaze direction by sensing a direction of a pupil from the eyeball based on a first distance measured by a distance measurement unit of some of the plurality of distance measurement units. The method of claim 1,
At least a portion of the plurality of distance measuring units is disposed along a support for supporting the display unit. The method of claim 1,
The display unit provides a screen including a circular guide area, and the screen combining unit detects a user's gaze direction when the gaze moves along the provided guide area, and corrects gaze based on the gaze direction detected according to the guide area. A smart glasses display device based on gaze detection to obtain a reference value. The method of claim 5,
The gaze correction reference value includes an eccentricity, a flatness, or a focal position of an ellipse. The method of claim 1,
The screen synthesizing unit senses the light irradiated by the gaze matching object and extracts a reference point for displaying the overlay image. The method of claim 7,
The screen synthesizing unit extracts a point spaced apart from the reference point by a predetermined interval as a screen boundary point. The method of claim 1,
The display unit displays the overlay image in a 3D stereoscopic shape, based on gaze detection smart glasses display. The method of claim 1,
The display unit comprises a micro-display, a lens, a panel, and a splitter, gaze detection-based smart glasses display device. A distance measuring step of measuring a first distance from the fundus and a second distance from the surface of the eyeball by irradiating a laser to the eyeball including the fundus by a plurality of distance measuring units;
A gaze detection step of detecting a gaze direction based on the first distance and the second distance;
An image acquisition step of obtaining an overlay image;
A display position calculation step of recognizing a gaze matching object within a certain range in the gaze direction and calculating a position to display the overlay image at a designated position according to the gaze matching object; And
A display step of displaying the overlay image at the calculated position through a display unit in the form of smart glasses, based on gaze detection. The method of claim 11,
In the gaze detection step, the position of the pupil is sensed based on information on whether a distance measured by each of the plurality of distance measuring units is a first distance or a second distance. The method of claim 11,
The gaze detection step detects the gaze direction by sensing the direction of the pupil from the eyeball based on a first distance measured by some of the distance measurement units among the plurality of distance measurement units. The method of claim 11,
Providing a screen including a guide area having a circular shape by the display unit;
When the user's gaze moves along the provided guide area, detecting the user's gaze direction and obtaining a gaze correction reference value based on the detected gaze direction according to the guide area; further comprising,
Smart glasses display method based on gaze detection. The method of claim 14,
The gaze correction reference value includes an eccentricity, a flatness, or a focal position of an ellipse. The method of claim 11,
The display position calculation step includes sensing light irradiated by the gaze matching object and extracting a reference point for displaying the overlay image. The method of claim 16,
The display position calculation step further comprises extracting a point spaced apart from the reference point by a predetermined interval as a screen boundary point. The method of claim 11,
The displaying step includes displaying the overlay image in a 3D stereoscopic shape. A laser is irradiated to the eyeball including the fundus by a plurality of distance measuring units to measure a first distance from the fundus and a second distance from the surface of the eyeball,
Detecting a gaze direction based on the first distance and the second distance,
Acquire an overlay image,
Recognizing the eye-catching object within a certain range in the eye-gaze direction, calculating a position so that the overlay image can be displayed at a designated position according to the eye-catching object,
A computer-readable recording medium that stores instructions for displaying the overlay image at the calculated position through a display unit in the form of smart glasses.

Images