CN116830014A - Augmented reality device and method for detecting user gaze - Google Patents

Abstract

An augmented reality device and an augmented reality method for detecting a user's gaze are provided. The augmented reality device includes: a light guide plate; a light reflection part including a pattern; a support for securing the augmented reality device to a face of a user of the augmented reality device; a light emitting part and a light receiving part mounted to the supporting part; and at least one processor, wherein: the processor controls the light emitting section to emit light to the light reflecting section, recognizes a pattern based on the light received through the light receiving section, and obtains gaze information of a user of the augmented reality apparatus based on the recognized pattern; the light emitted toward the light reflecting portion is reflected by the light reflecting portion and then directed toward the eyes of the user; and the light received by the light receiving portion includes light that has been directed to and then reflected by the eyes of the user.

Description

Augmented reality device and method for detecting user gaze

Technical field

The present disclosure relates to an augmented reality (AR) device and method for detecting a user's gaze, and more particularly, to an AR device that detects a user's gaze by using a light emitter and a light receiver located in a support of the AR device Apparatus and methods thereof.

Background technique

Augmented reality (AR) is a technology that projects virtual images onto a real-world physical environment space or real-world objects and displays the virtual image as a single image. When worn on the user's face or head, AR devices allow the user to see real-life scenes and virtual images through a glasses-type device that uses a see-through display (e.g., a waveguide in front of the user's eyes). As research into such AR devices is actively underway, various types of wearable devices have been released or are expected to be released. In related art glasses-type AR devices, cameras are usually arranged on the edge portion surrounding the waveguide to track the user's gaze, which causes the edge portion of the AR device to be magnified and further causes the user wearing the AR device to feel uncomfortable.

Contents of the invention

technical problem

An augmented reality (AR) device and method are provided that are capable of detecting a user's gaze by using light reflectors and light receivers located in a support extending from a frame of the AR device.

An AR device and method capable of detecting a user's gaze by using light reflected by a light reflector formed on a waveguide are provided.

Provided are an AR device and a method capable of more accurately detecting a user's gaze by calculating the degree of offset of a support of the AR device based on a pattern formed on a light reflector.

Technical solutions

According to an aspect of the present disclosure, an augmented reality (AR) device is provided, including: a waveguide; a light reflector including a pattern; a support configured to secure the AR device to a face of a user of the AR device; and a light emitter and a light receiver mounted on the support; and at least one processor configured to: control the light emitter to emit light to the light reflector, identify the pattern based on the light received by the light receiver, and based on the identified pattern to obtain the gaze information of the user of the AR device, wherein the light emitted to the light reflector is reflected by the light reflector and directed to the user's eyes, and wherein the light received by the light receiver includes the light that is directed to the user's eyes. Light is reflected by the user's eyes.

The support may include temples extending from a frame surrounding the waveguide for positioning on the user's ears, and a nose support extending from the frame and positioning on the user's nose.

Light reflectors can be coated on the waveguide.

Light reflectors can be formed on the waveguide.

At least one processor may also be configured to analyze the identified pattern and identify the degree of offset of the support relative to the frame from which the support extends.

The at least one processor may be further configured to: generate a mapping function for calculating the position of the user's gaze point based on the degree of offset of the support relative to the frame, and based on the mapping function and the degree of offset of the support relative to the frame, Obtain the user's gaze information.

At least one processor may be further configured to obtain the position of one or more feature points corresponding to the user's eyes based on the light received by the light receiver.

At least one processor may be configured to input the position of one or more feature points corresponding to the user's eyes and the degree of offset of the support relative to the frame into the mapping function and calculate the position of the user's gaze point.

The position of the one or more feature points may include the position of the pupil feature point of the user's eye and the position of the flicker feature point of the user's eye.

The at least one processor may be further configured to: display a target point at a specific location on the waveguide to calibrate the mapping function, receive through the light receiver light reflected by the eye of a user looking at the displayed target point, and based on the light reflected by the gaze. The target point displayed is the light reflected by the user's eye to calibrate the mapping function.

The at least one processor may be further configured to: identify a pattern of the light reflector based on light reflected by an eye of a user looking at the displayed target point; identify a degree of bias of the support based on the identified pattern; and based on The position of one or more feature points corresponding to the eyes of the user looking at the displayed target point is obtained from the light reflected by the user's eyes looking at the displayed target point.

At least one processor may be further configured to: input the offset degree of the temple and the position of one or more feature points into the mapping function; and calibrate the mapping function such that the position value of the target point is output from the mapping function.

The light emitter may be an infrared light emitting diode (IR LED), and the light receiver may be an IR camera.

The light emitter may be an infrared (IR) scanner and the light receiver an IR detector.

At least one processor may be further configured to: obtain, based on the IR light obtained from the IR detector, the position of one or more feature points corresponding to the user's eyes calibrated according to the degree of offset of the support relative to the frame; and based on The user's gaze information is obtained from the calibrated positions of one or more feature points corresponding to the user's eyes.

According to another aspect of the present disclosure, there is provided a method of detecting user gaze performed by an augmented reality (AR) device, the method comprising: reflecting light including a pattern from a light emitter installed in a support of the AR device The light emitted by the light emitter is directed to the eyes of the user wearing the AR device; the light reflected by the user's eyes is received by the light receiver installed on the support; based on the light received through the light receiver Recognize the pattern; and obtain the user's gaze information based on the recognized pattern.

The method may further include analyzing the recognized pattern and identifying a degree of offset of the support relative to the frame based on the recognized pattern, wherein the support extends from the frame, wherein obtaining the gaze information may include based on the support relative to the frame The degree of offset is used to determine the user's gaze direction.

The method may further include generating a mapping function for calculating the position of the user's gaze point based on the degree of offset of the support relative to the frame, wherein obtaining the gaze information may include based on the mapping function and the degree of offset of the support relative to the frame to calculate the location of the user's gaze point.

The method may further include: obtaining the position of one or more feature points corresponding to the user's eyes based on the light received by the light receiver, wherein obtaining the gaze information includes converting one or more feature points corresponding to the user's eyes. The position of each feature point and the offset degree of the support relative to the frame are input into the mapping function and the position of the user's gaze point is calculated.

According to another aspect of the present disclosure, there is provided a computer-readable recording medium having recorded thereon a program for executing a method, which method includes: emitting light including a pattern through a light emitter installed in a support of an AR device. The reflector emits light, and the light emitted by the light emitter is directed to the eyes of the user wearing the AR device; the light reflected by the user's eyes is received through the light receiver installed on the support; based on the light received through the light receiver to recognize the pattern; and to obtain the user's gaze information based on the recognized pattern.

Description of the drawings

1 is a diagram illustrating an example in which an augmented reality (AR) device detects a user's gaze using a gaze detector located in a temple portion of the AR device according to an example embodiment of the present disclosure.

FIG. 2 is a diagram illustrating an example of an AR device according to example embodiments of the present disclosure.

3 is a block diagram of an AR device according to an example embodiment of the present disclosure.

4 is a diagram illustrating an example of operations of a light transmitter and a light receiver of an AR device according to an example embodiment of the present disclosure.

FIG. 5a is a diagram illustrating an example of a light emitter that emits planar light according to an example embodiment of the present disclosure.

5b is a diagram illustrating an example of a light emitter that emits point light according to an example embodiment of the present disclosure.

FIG. 5c is a diagram illustrating an example of a light emitter that emits line light according to an example embodiment of the present disclosure.

6a is a diagram illustrating an example in which a light emitter and a light receiver are arranged in a temple of an AR device according to an example embodiment of the present disclosure.

6b is a diagram illustrating an example in which a light emitter and a light receiver are arranged in a nose support of an AR device according to an example embodiment of the present disclosure.

6c is a diagram illustrating an example in which a light emitter and a light receiver are arranged in temples and a nose support of an AR device according to an example embodiment of the present disclosure.

6d is a diagram illustrating an example in which a light emitter and a light receiver are arranged in temples and a nose support of an AR device according to an example embodiment of the present disclosure.

7a is a diagram illustrating an example of a dot pattern formed on a light reflector of an AR device according to an example embodiment of the present disclosure.

7b is a diagram illustrating an example of a grid pattern formed on a light reflector of an AR device according to an example embodiment of the present disclosure.

Figure 7c is a diagram illustrating an example of a pattern in the form of a 2D marker according to an example embodiment of the present disclosure.

Figure 7d is a diagram illustrating an example of a light reflector covering a portion of a waveguide according to an example embodiment of the present disclosure.

8a is a diagram illustrating a light emission angle and pattern before the temples of the AR device are biased according to an example embodiment of the present disclosure.

8b is a diagram illustrating a light emission angle and pattern after the temples of the AR device are biased according to an example embodiment of the present disclosure.

9 is a diagram illustrating an example of a pattern recognized from an array of light received by a light receiver when a light emitter of an AR device is an infrared (IR) scanner according to an example embodiment of the present disclosure.

10 is a diagram illustrating an example of eye characteristics recognized from an array of light received through a light receiver of an AR device when the light emitter of the AR device is an IR scanner according to an example embodiment of the present disclosure.

11 is a diagram illustrating an example of a function used by an AR device to calculate an eyeball center and calculate a user's gaze point according to an example embodiment of the present disclosure.

12 is a flowchart of a method of detecting user gaze performed by an AR device according to an example embodiment of the present disclosure.

Detailed ways

Hereinafter, embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings to enable those skilled in the art to carry out the present disclosure without difficulty. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments of the disclosure set forth herein. In order to clearly describe the present disclosure, parts irrelevant to the description of the present disclosure are omitted, and similar reference numerals are assigned to similar elements throughout this specification.

Throughout this specification, it will be understood that when an element is referred to as being "connected to" another element, it can be "directly connected" to the other element or "electrically connected" to the other element through intervening elements. Furthermore, when an element is referred to as "comprising" a constituent element, other constituent elements may also be included but not excluded unless there is any other specific reference thereto.

The term "augmented reality (AR)" as used herein refers to a technology that provides viewing of virtual images over a real-world physical environment space or together with real objects.

In addition, the term "AR device" means a device capable of creating "AR" and includes not only AR glasses that are usually worn on the user's face, but also head-mounted display (HMD) devices and AR helmets worn on the user's head, etc. .

Meanwhile, the term "real-world scene" refers to the real-world scene that the user sees through the AR device, and can include real-world objects. Furthermore, the term "virtual image" means an image generated by an optical engine, and may include both static images and dynamic images. The virtual image can be observed together with the real scene, and can be an image representing information about real objects in the real scene, information about the operation of the AR device, a control menu, and the like.

Therefore, the AR device may be equipped with: an optical engine for generating a virtual image including light generated by the light source; and a waveguide formed of a transparent material for guiding the virtual image generated by the optical engine to the user's eyes and allowing the user to look together to real-world scenes and virtual images. Furthermore, as mentioned above, AR devices need to be able to allow users to observe real-world scenes, and therefore, optical elements for redirecting light paths with substantially straightness are needed to guide the light generated by the optical engine to the user's eyes through waveguides. Here, the path of the light may be redirected by using reflection by, for example, a mirror or by using diffraction by a diffractive element, such as a diffractive optical element (DOE) or a holographic optical element (HOE), but the present disclosure is not limited thereto. this.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an example in which an augmented reality (AR) device 1000 detects a user's gaze using a gaze detector 1500 located in a temple portion of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 1 , the AR device 1000 may detect a user's gaze by using a light emitter 1510 and a light receiver 1520 . The light emitter 1510 and the light receiver 1520 for detecting the user's gaze may be provided in, for example, a temple portion of the AR device 1000, and the AR device 1000 may be highly efficient by using the light emitter 1510 and the light receiver 1520 provided in the temple portion. to identify the user's eyes. Infrared (IR) light may be emitted from the light emitter 1510 disposed in the temple portion toward the waveguide of the AR device 1000, reflected by the light reflector, and received from the user's eyes through the light receiver 1520 disposed in the temple portion. Furthermore, the AR device 1000 may obtain information about the user's eyes based on the received IR light and detect the user's gaze direction by using the obtained information about the eyes.

The AR device 1000 represents a device capable of creating “AR” and may include, for example, AR glasses worn on a user's face, but the present disclosure is not limited thereto. For example, the AR device 1000 may include a head-mounted display (HMD) device worn on the user's head, an AR helmet, and the like. In this case, the gaze detector 1500 may be provided on an inner portion of the side of the HMD device facing the user's eyes in the HMD device, or on the inner portion of the side of the AR helmet facing the user's eyes in the AR helmet.

FIG. 2 is a diagram illustrating an example of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 2 , the AR device 1000 may include a glasses-type body configured to be worn by a user as a glasses-type display device.

The glasses-type body may include a frame 110 and a support 190 . Support 190 may extend from frame 110 and be used to position AR device 1000 on the user's head. The support 190 may include temples 191 and a nose support 192 . The temples 191 may extend from the frame 110 and may be used to secure the AR device 1000 to the user's head on the side surfaces of the glasses-type body. The nose support 192 may extend from the frame 110 and may be used to position the AR device 1000 on the user's nose, and may include, for example, a nose bridge and a nose pad, although the disclosure is not limited thereto.

Additionally, the waveguide 170 to which the light reflector 1400 is attached may be located on the frame 110 . The frame 110 may be formed around the outer peripheral surface of the waveguide 170 . Waveguide 170 may be configured to receive projected light in an input region and to output at least a portion of the input light in an output region. Waveguide 170 may include left eye waveguide 170L and right eye waveguide 170R.

The left eye reflector 1400L and the left eye waveguide 170L may be disposed at a position corresponding to the user's left eye, and the right eye reflector 1400R and the right eye waveguide 170R may be disposed at a position corresponding to the user's right eye. For example, the left eye reflector 1400L may be attached to the left eye waveguide 170L, or the right eye reflector 1400R may be attached to the right eye waveguide 170R, but the disclosure is not limited thereto. Additionally, for example, the left eye reflector 1400L may be coated on the inside of the left eye waveguide 170L to attach to the left eye waveguide 170L, or the right eye reflector 1400R may be coated on the inside of the right eye waveguide 170R to attach to the left eye waveguide 170L. Right eye waveguide 170R.

In addition, the optical engine 120 of the projector that projects display light including an image may include a left-eye optical engine 120L and a right-eye optical engine 120R. The left eye optical engine 120L and the right eye optical engine 120R may be located on both sides of the AR device 1000 . Alternatively, an optical engine 120 may be included in a central portion of the AR device 1000 around the nose support 192 . Light emitted from optical engine 120 may be displayed through waveguide 170 .

The light emitter 1510 and the light receiver 1520 of the gaze detector 1500 may be disposed on an inner portion of the support 190 of the AR device 1000, which is a position between the support 190 and the user's eyes. The light emitter 1510 and the light receiver 1520 may be disposed in the support 190 of the AR device 1000 facing the light reflector 1400. For example, the light emitter 1510 and the light receiver 1520 may be disposed on the inside of the temple 191 of the AR device 1000 at a position about 10 mm to 15 mm apart from the frame 110, so as to respectively emit and Receive IR light.

3 is a block diagram of an AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 3 , an AR device 1000 according to an example embodiment of the present disclosure may include a user input 1100, a microphone 1200, a display 1300, a light reflector 1400, a gaze detector 1500, a communication interface 1600, a storage device 1700, and a processor 1800. Additionally, gaze detector 1500 may include a light emitter 1510 and a light receiver 1520.

The user inputter 1100 refers to a device through which the user inputs data for controlling the AR device 1000 . For example, the user input 1100 may include a keyboard, a dome switch, a touch pad (e.g., a touch-type capacitive touch pad, a pressure-type resistive overlay touch pad, an infrared sensor-type touch pad, a surface acoustic wave conduction touch pad, an integrated tension measurement touch pad , piezoelectric effect type touch panel), scroll wheel, push switch, but the present disclosure is not limited thereto.

The microphone 1200 may receive external audio signals and process the received audio signals into electrical voice data. For example, microphone 1200 may receive audio signals from an external device or speaker. The microphone 1200 may use various noise reduction algorithms to remove noise generated during the process of receiving external audio signals. The microphone 1200 may receive a user's voice input for controlling the AR device 1000 .

Display 1300 may display information processed by AR device 1000. For example, the display 1300 may display a user interface for capturing an image of the surrounding environment of the AR device 1000, and information related to a service provided based on the captured image of the surrounding environment of the AR device 1000.

According to example embodiments of the present disclosure, the display 1300 may provide AR images. The display 1300 according to example embodiments of the present disclosure may include a waveguide 170 and an optical engine 120. Waveguide 170 may include a transparent material through which a portion of the rear surface is visible when a user wears AR device 1000 . Waveguide 1320 may be configured as a flat plate including a single layer or a multi-layer structure of transparent material through which light can be internally reflected and propagated. The waveguide 1320 may face the exit surface of the optical engine 120 to receive the light of the virtual image projected from the optical engine 120 . Here, a transparent material is a material through which light can pass, its transparency may not be 100%, and it may have a specific color. According to an example embodiment of the present disclosure, the waveguide 170 includes a transparent material, so the user can view not only the virtual object of the virtual image, but also the external real scene, and therefore the waveguide 170 can be called a see-through display. The display 1300 may output a virtual object of a virtual image through the waveguide 170, thereby providing an AR image. When the AR device 1000 is a glasses-type device, the display 1300 may include a left display and a right display.

The light reflector 1400 may reflect light emitted from the light emitter 1510 which will be described later. The light reflector 1400 and the waveguide 170 may be disposed at a position facing the user's eyes, and may be attached to each other. For example, light reflector 1400 may be coated on at least a portion of waveguide 170 . Furthermore, in addition to the waveguide 170, the light reflector 1400 may be attached to or coated on other elements included in the glasses-type AR device 1000, such as a vision correction lens for vision correction or a lens mounted to protect the waveguide. Cover glass 170. Light reflector 1400 may include a material capable of reflecting IR light emitted from light emitter 1510 . The light reflector 1400 may be, for example, silver, gold, copper, or a material including one or more of these metals, but the disclosure is not limited thereto. Accordingly, IR light emitted from the light emitter 1510 may be reflected by the light reflector 1400 and directed to the user's eyes, and IR light reflected back from the user's eyes may be reflected by the light reflector 1400 and directed to the light receiver 1520 .

Light reflector 1400 may be coated on waveguide 170 to have a specific pattern. Patterns formed on the light reflector 1400 may include, for example, dot patterns, line patterns, grid patterns, 2D markers, etc., but the present disclosure is not limited thereto. Furthermore, the pattern formed on the light reflector 1400 may be formed on, for example, a portion of the waveguide 170 where the user's gaze is less frequently directed. The pattern formed on the light reflector 1400 may be formed, for example, on a portion of the waveguide 170 that does not interfere with capturing or scanning the user's eyes. For example, in the light reflector 1400, the specific pattern may indicate a pattern formed by portions where light emitted from the light emitter 1510 is reflected and portions where the light is not reflected. Since the light emitted toward the portion where the light is not reflected in the light reflector 1400 is not reflected by the light reflector 1400, the light receiver 1520 does not receive the light emitted toward the portion where the light is not reflected. Therefore, the pattern of the photo reflector 1400 formed by the portion where the light is reflected and the portion where the light is not reflected can be detected from the light received by the light receiver 1520 . Furthermore, when the light emitted from the light emitter 1510 is IR light, the specific pattern may include a material for reflecting the IR light, and the material for reflecting the IR light may be invisible to the user's eyes. Since most of the real-world light or real-world scene observed by the user through the AR device 1000 includes visible light, the user may not be disturbed by the light reflector 1400 formed with a specific pattern, and may observe the real-world light or real-world scene.

The gaze detector 1500 may include a light emitter 1510 that emits IR light to detect the user's gaze and a light receiver 1520 that receives the IR light, and may detect data related to the user's gaze of the user wearing the AR device 1000 .

The light emitter 1510 of the gaze detector 1500 may emit IR light toward the light reflector 1400 so that the IR light reflected by the light reflector 1400 may be directed to the user's eyes. The light emitter 1510 may emit IR light toward the light reflector 1400, the emitted IR light may be reflected by the light reflector 1400, and the reflected IR light may be directed to the user's eyes. The light emitter 1510 may be disposed in the AR device 1000 at a position where the IR light may be emitted to the light reflector 1400. The light emitter 1510 may be located on the support 190 of Figure 2 (eg, temples 191 and nose support 192 of Figure 2) that supports the AR device 1000 on the user's face.

Furthermore, IR light reflected from the user's eyes may be reflected by the light reflector 1400 and received by the light receiver 1520 of the gaze detector 1500 . The IR light directed to the user's eyes may be reflected from the user's eyes, the IR light reflected from the user's eyes may be reflected by the light reflector 1400 , and the light receiver 1520 may receive the IR light reflected by the light reflector 1400 . The light receiver 1520 may be disposed in the AR device 1000 at a position where the IR light reflected from the light reflector 1400 can be received. The light receiver 1520 may be located on the support 190 of Figure 2 (eg, temples 191 and nose support 192 of Figure 2) that supports the AR device 1000 on the user's face. Additionally, nose support 192 of FIG. 2 may include a nose bridge and a nose pad, for example. In addition, the nose bridge and the nose pad may be integrated, but the present disclosure is not limited to this.

For example, light emitter 1510 may be an IR light emitting diode (LED) that emits IR light, and light receiver 1520 may be an IR camera that captures IR light. In this case, the IR camera may use the IR light reflected by the light reflector 1400 to capture the user's eyes. When the light emitter 1510 is an IR LED and the light receiver 1520 is an IR camera, the light emitter 1510 may emit the IR light of the planar light to the light reflector 1400 , and the light receiver 1520 may receive the planar light reflected from the light reflector 1400 Light IR light. The planar light may be light emitted in a planar form, and the planar light emitted from the light emitter 1510 may be directed to at least a portion of the entire area of the light reflector 1400 . At least a portion of the entire area of the light reflector 1400 may be disposed such that the plane light reflected from at least a portion of the entire area of the light reflector 1400 may cover the user's eyes.

Alternatively, for example, the light emitter 1510 may be an IR scanner that emits IR light, and the light receiver 1520 may be an IR detector that detects the IR light. In this case, the IR scanner may emit IR light such that the IR light for scanning the user's eyes is directed to the user's eyes, and the IR detector may detect the IR light reflected from the user's eyes. When the light emitter 1510 is an IR scanner that emits IR light and the light receiver 1520 is an IR detector that detects IR light, the light emitter 1510 may emit line light in the form of a line, and the line emitted from the light emitter 1510 Light may be directed to a portion of the entire area of light reflector 1400. At least part of the entire area of the light reflector 1400 may be disposed so that line light reflected from at least part of the entire area of the light reflector 1400 may cover the user's eyes. When the light emitter 1510 is an IR scanner that emits IR light and the light receiver 1520 is an IR detector that detects IR light, the light emitter 1510 may emit point light in the form of points, and the point light emitted from the light emitter 1510 may be directed to a portion of the entire area of light reflector 1400. At least a portion of the entire area of the light reflector 1400 may be disposed such that point light reflected from at least a portion of the entire area of the light reflector 1400 may cover the user's eyes.

When the light emitter 1510 is an IR scanner and the light receiver 1520 is an IR detector, the light emitter 151 may emit IR light of point light or line light to the light reflector 1400, and the light receiver 1520 may receive light reflected from the light reflector 1400. The point light or line light IR light reflected by the detector 1400. In this case, the light emitter 1510 may sequentially emit IR light while moving the light emission direction of the light emitter 1510 so that the IR light of point light or line light may cover the space where the user's eyes are. Although an IR scanner typically includes an IR LED and a microelectromechanical system (MEMS) mirror capable of controlling the direction of IR light emitted from the IR LED and reflecting the IR light, in the following, IR scanner, IR LED, and MEMS mirror are collectively referred to as is and is described as an IR scanner. Furthermore, although an IR detector generally includes several photodiodes installed in a part requiring light detection, in the following, the IR detector and the photodiode are described as IR detectors.

When the AR device 1000 is a glasses-type device, the light emitter 1510 and the light receiver 1520 may be provided on the temples 191 of the AR device 1000 . For example, referring to FIG. 2, the light emitter 1510 and the light receiver 1520 may be disposed on the inner part of the temple 191 of the AR device 1000, which is the position between the temple 191 and the user's eyes. For example, referring to FIG. 2 , the light emitter 1510 and the light receiver 1520 may be disposed on the inner side of the temple 191 of the AR device 1000 at a position spaced approximately 10 mm to 15 mm from the frame 110 . The light emitter 1510 and the light receiver 1520 may be disposed in the temple 191 of the AR device 1000 facing the light reflector 1400.

Furthermore, for example, referring to FIG. 2 , the light emitter 1510 and the light receiver 1520 may be provided on the nose support 192 of the AR device 1000 . The light emitter 1510 and the light receiver 1520 may be disposed on an inner portion of the nose support 192 of the AR device 1000, which is a position between the nose support 192 and the user's eyes. For example, referring to FIG. 2 , the light emitter 1510 and the light receiver 1520 may be disposed on the inner side of the nose support 192 of the AR device 1000 at a position spaced approximately 10 mm to 15 mm from the frame 110 . The light emitter 1510 and the light receiver 1520 may be disposed in the nose support 192 of the AR device 1000 facing the light reflector 1400.

The gaze detector 1500 may provide the processor 1800 with data related to the gaze of the user's eyes, and the processor 1800 may obtain the user's gaze information based on the data related to the gaze of the user's eyes. Data related to the gaze of the user's eyes is data obtained by the gaze detector 1500 and may include the type of IR light emitted from the light emitter 1510 (eg, point light, line light, or plane light), the type of IR light emitted from the light emitter 1510 Characteristics of the emitted IR light, data regarding the emission area of the IR light emitted from the light emitter 1510 , and data indicating characteristics of the IR light received from the light receiver 1520 . In addition, the user's gaze information is information related to the user's gaze, which can be generated by analyzing data related to the gaze of the user's eyes, and can include information about, for example, the position of the user's pupil, the position of the pupil center point, the position of the user's iris, the user's Information such as the center of the eye, the position of the flickering characteristic point of the user's eye, the user's gaze point, the user's gaze direction, etc., but the present disclosure is not limited thereto. The user's gaze direction may be, for example, the direction of the user's gaze from the center of the user's eyes toward a gaze point of the user's gaze. For example, the user's gaze direction may be represented by a vector value from the center of the user's left eye toward the gaze point and a vector value from the center of the user's right eye toward the gaze point, but the present disclosure is not limited thereto. According to an example embodiment of the present disclosure, the gaze detector 1500 may detect data related to the gaze of the user wearing the AR device 1000 at a previously determined time interval.

The communication interface 1600 may transmit/receive data for receiving services related to the AR device 1000 to/from external devices and servers.

The storage device 1700 may store a program to be executed by the processor 1800 which will be described later, and may store data input to or output from the AR device 1000 .

The storage device 1700 may include at least one of internal memory or external memory.

The internal memory may include at least one of the following: for example, volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.), non-volatile memory (e.g., one-time removable memory) Programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, etc.), hard disk drive (HDD) or Solid State Drive (SSD). According to example embodiments of the present disclosure, the processor 1800 may load a command or data received from at least one of a non-volatile memory or another element into the volatile memory and process the command or data. Additionally, processor 1800 may store data received or generated from another element in non-volatile memory. The external memory may include at least one of the following: Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure Digital (Mini-SD), Extreme Digital (xD), or a Memory Stick.

Programs stored in the storage device 1700 may be classified into a plurality of modules according to their functions. According to an example embodiment, each of the plurality of modules may include one or more computer readable codes, which may include, for example, light illumination code 1710, light reception code 1720, eye feature detection code 1730, pattern detection code 1740, bias Determination code 1750, pupil position detection code 1760, gaze determination code 1770, and calibration code 1780. For example, a memory may be included in the gaze detector 1500, and in this case, the light irradiation code 1710 and the light reception code 1720 may be stored in the memory included in the gaze detector 1500 as firmware.

The processor 1800 controls the overall operation of the AR device 1000. For example, the processor 1800 may execute a program stored in the storage device 1700 to generally control the user input 1100, the microphone 1200, the display 1300, the light reflector 1400, the gaze detector 1500, the communication interface 1600, the storage device 1700, and the like.

The processor 1800 may execute the light irradiation code 1710, the light reception code 1720, the eye feature detection code 1730, the pattern detection code 1740, the offset determination code 1750, the pupil position detection code 1760, the gaze determination code 1770, and Calibrate code 1780 to determine the user's gaze point and gaze direction.

According to an example embodiment of the present disclosure, the AR device 1000 may include a plurality of processors 1800, and a light illumination code 1710, a light reception code 1720, an eye feature detection code 1730, a pattern detection code 1740, an offset determination code 1750, a pupil position detection code Code 1760 , gaze determination code 1770 , and calibration code 1780 may be executed by multiple processors 1800 .

For example, some of the light exposure code 1710, the light reception code 1720, the eye feature detection code 1730, the pattern detection code 1740, the offset determination code 1750, the pupil position detection code 1760, the gaze determination code 1770, and the calibration code 1780 may be determined by A processor executes, and others of light exposure code 1710 , light reception code 1720 , eye feature detection code 1730 , pattern detection code 1740 , offset determination code 1750 , pupil position detection code 1760 , gaze determination code 1770 , and calibration code 1780 It may be executed by the second processor, but the present disclosure is not limited thereto.

For example, the gaze detector 1500 may include another processor and memory, and the other processor may execute the light illumination code 1710 and the light reception code 1720 stored in the memory, and the processor 1800 may execute the eye-level code 1720 stored in the storage device 1700 . Feature detection code 1730, pattern detection code 1740, offset determination code 1750, pupil position detection code 1760, gaze determination code 1770, and calibration code 1780.

The processor 1800 may execute the light illumination code 1710 stored in the storage device 1700 so that the light emitter 1510 may emit IR light to the light reflector 1400. The processor 1800 may control the light emitter 1510 by executing the light illumination code 1710, and the light emitter 1510 controlled by the processor 1800 may emit IR light to at least a partial area of the light reflector 1400 such that the IR light reflected by the reflector 1400 The light can cover the user's eyes.

For example, when the light receiver 1520 is an IR camera, the light emitter 1510 may be an IR LED, and the processor 1800 may control the IR LED such that the IR light emitted from the IR LED may be reflected by the light reflector 1400 and irradiated to a target including a user. The eye area for the IR camera to capture the user's eyes. For example, in order to reflect the light emitted from the IR LED by using the light reflector 1400 and irradiate the light to an area including the user's eyes, the processor 1800 may control the irradiation direction of the IR light emitted from the IR LED and supply power to the IR LED, Thereby controlling the emission of IR light from the IR LED.

According to an example embodiment of the present disclosure, the IR camera and the IR LED may be installed toward the light reflector 1400 of the AR device 1000 so that the IR camera may capture the entire area of the user's eyes, and the processor 1800 may control the IR camera installed toward the light reflector 1400 IR LED to emit IR light. An example of controlling the illumination direction of IR light emitted from the IR LED will be described in more detail with reference to FIG. 5a.

According to another example embodiment of the present disclosure, when the light receiver 1520 is an IR detector, the light emitter 1510 may be an IR scanner, and the processor 1800 may control the IR scanner to reflect from the IR scan using the light reflector 1400 The IR light emitted by the device scans the user's eyes so that the IR detector can detect the user's eyes. For example, in order to scan the user's eyes by using the light reflector 1400 to reflect the light emitted from the IR scanner, the processor 1800 may control the irradiation direction of the IR light emitted from the IR scanner and supply power to the IR scanner, thereby controlling the light emitted from the IR scanner. IR scanners emit IR light. An example of controlling the illumination direction of IR light emitted from the IR scanner will be described in more detail with reference to FIGS. 5b and 5c.

The processor 1800 may execute the light receiving code 1720 stored in the storage device 1700 so that the light receiver 1520 may receive the light reflected by the light reflector 1400 from the user's eyes. The processor 1800 may control the light receiver 1520 by executing the light receiving code 1720, and the light receiver 1520 controlled by the processor 1800 may receive the light reflected by the light reflector 1400 from the user's eyes.

For example, when the light emitter 1510 is an IR LED, the light receiver 1520 may be an IR camera, and the processor 1800 may control the IR camera to capture the user's eyes through light from the user's eyes reflected by the reflector 1400.

Alternatively, for example, when the light emitter 1510 is an IR scanner, the light receiver 1520 may be an IR detector, and the processor 1800 may control the IR detector to detect IR light from the user's eyes that is reflected by the light reflector 1400 , so that the IR detector can detect the user's eyes.

The processor 1800 may execute the eye feature detection code 1730 stored in the storage device 1700 to detect features related to the gaze of the user's eyes. For example, the processor 1800 may execute the eye feature detection code 1730 to detect the position of the pupil feature point of the user's eye and the position of the flicker feature point of the user's eye. The pupil feature point may be, for example, a pupil center point, and the flicker feature point of the eye may be a portion of the detected eye area where the brightness is greater than or equal to a specific value. The position of the pupil feature point and the position of the eye's flicker feature point can be identified, for example, by coordinate values indicating the position in the coordinate system of the light receiver 1520 . For example, the coordinate system of the light receiver 1520 may be the coordinate system of the IR camera or the coordinate system of the IR detector, and the coordinate values in the coordinate system of the light receiver 1520 may be 2D coordinate values.

Processor 1800 may detect features related to eye gaze by analyzing light received by light receiver 1520 . For example, when the light receiver 1520 is an IR camera, the processor 1800 may identify the location of the pupil feature point and the location of the eye's flicker feature point in the image captured by the IR camera. Alternatively, for example, when the light receiver 1520 is an IR detector, the processor 1800 may analyze the IR light detected by the IR detector to identify the location of the pupil feature point and the location of the eye's flicker feature point. When the position of the feature point is identified based on the IR light detected by the IR detector, the position of the pupil feature point and the position of the flicker feature point may have values calibrated by reflecting the offset of the support 190 . The positions of the pupil feature points and the positions of the flicker feature points calibrated by reflecting the offset of the support 190 will be described in more detail with reference to FIG. 11 .

In addition, the processor 1800 may analyze the light received by the light receiver 1520, thereby obtaining coordinate values indicating the position of the pupil feature point and coordinate values indicating the position of the flash feature point of the eye. For example, when the light receiver 1520 is an IR camera, the processor 1800 may obtain the coordinate values of the pupil feature point and the coordinate values of the flicker feature point of the eye from the coordinate system of the IR camera. The coordinate system of the IR camera may be used to indicate the position of the pupil feature point of the eye and the position of the flicker feature point, and for example, coordinate values on the coordinate system of the IR camera corresponding to the pixels of the image captured by the IR camera may be set in advance . Furthermore, based on the properties (for example, brightness) of the IR light received through the IR camera, coordinate values corresponding to the feature points of the eye can be identified.

For example, when light receiver 1520 is an IR camera, processor 1800 may identify the location of the pupil center point in an image captured by the IR camera. The processor 1800 may identify brightness of IR light received through an image sensor of an IR camera including a plurality of photodiodes, and identify at least one pixel receiving IR light indicative of a pupil among pixels of an image captured by the IR camera, thereby identifying The position of the pupil center point. For example, the location of a pixel in an image captured by an IR camera may be identified by the coordinate system of the IR camera, and the location of the pupil center point may have coordinate values in the coordinate system of the IR camera, such as at least one corresponding to the pupil center point The position value of a pixel.

For example, the processor 1800 may identify the location of the brightest point in an image captured by an IR camera to identify flickering characteristic points of the eye. The processor 1800 may identify a brightness of IR light received through an image sensor of an IR camera including a plurality of photodiodes, and may identify among pixels of an image captured by the IR camera corresponding to bright IR light that is equal to or greater than a specific reference At least one pixel of the image is used to identify the location of the flashing feature point of the eye. For example, the processor 1800 may identify the pixels corresponding to the brightest IR light among the pixels of the image captured by the IR camera, thereby identifying the location of the flash characteristic point of the eye. For example, the position of a pixel in an image captured by an IR camera can be identified by the coordinate system of the IR camera, and the position of the flicker feature point of the eye can have coordinate values in the coordinate system of the IR camera, such as corresponding to the flicker feature point The position value of the pixel.

Alternatively, for example, when the light receiver 1520 is an IR detector, the processor 1800 may calculate coordinate values of the pupil feature point and coordinate values of the flicker feature point of the eye in the coordinate system of the IR detector.

When the light emitter 1510 is an IR scanner, the processor 1800 can control the IR scanner to sequentially illuminate the point light source or the line light source to cover the area where the user's eyes are, and sequentially receive the light reflected from the user's eyes through the IR detector. to scan the area where the user's eyes are. Additionally, the processor 1800 may analyze the array of light sequentially received through the IR detector to identify pupil feature points and flicker feature points of the eye.

The coordinate system of the IR detector may be used to indicate the position of the pupil feature point of the pupil and the position of the flicker feature point of the eye, and for example, the coordinate system of the IR detector may be preset to correspond to the light sequentially received through the IR detector The coordinate values corresponding to the light in the array. For example, the irradiation direction and irradiation time of the light emitted from the IR scanner may be determined according to the operation setting value of the IR scanner, and an array of lights may be formed from the light emitted from the IR scanner. For example, based on the irradiation direction and irradiation time of the light emitted from the IR scanner and the reception time of the light received from the IR detector, coordinate values on the coordinate system of the IR detector corresponding to the light in the light array can be identified . Furthermore, based on the properties (for example, brightness) of the light in the array of light sequentially received through the IR detector, the light corresponding to the feature point of the eye and the coordinate values of the light can be identified.

For example, the processor 1800 may identify light whose brightness is equal to or less than a specific value in the received light array, thereby identifying the pupil feature point based on coordinate values on the coordinate system of the IR detector corresponding to the identified light. Location.

For example, the processor 1800 may identify light whose brightness is equal to or greater than a specific value in the received light array, thereby identifying the coordinate value corresponding to the identified light on the coordinate system of the IR detector as a flicker characteristic of the eye. The coordinate value of the point.

Furthermore, for example, when the light receiver 1520 is an IR detector, the coordinate value of the pupil feature point and the coordinate value of the flicker feature point of the eye may be values calibrated by reflecting the offset degree of the support 190 of the AR device 1000, This is described below. In this case, the processor 1800 may calculate the coordinate values of the pupil feature points and the flicker feature points of the eyes calibrated by reflecting the offset degree of the temples 191 of the AR device 1000 and/or the offset degree of the nose support 192 coordinate value. Calibrated coordinate values can be input to a mapping function described below.

The processor 1800 may execute the pattern detection code 1740 stored in the storage device 1700 to detect the pattern of the light reflector 1400 . The light reflector 1400 may be coated on one surface of the waveguide 170 of the AR device 1000 to have a specific pattern. The processor 1800 may receive the IR light reflected by the user's eyes and reflected by the light reflector 1400 through the light receiver 1520 and identify the shape of the pattern based on the received IR light. Patterns formed on the light reflector 1400 may include, for example, dot patterns, line patterns, grid patterns, 2D markers, etc., but the present disclosure is not limited thereto. Examples of patterns formed on the light reflector 1400 and identified by the pattern detection code 1740 will be described in more detail with reference to FIGS. 7a to 7d. When temples 191 are offset relative to frame 110, the pattern identified by pattern detection code 1740 may have a distorted shape.

For example, when the light receiver 1520 is an IR camera, the IR camera may capture the user's eyes based on the IR light reflected by the light reflector 1400, and the processor 1800 may identify a pattern within the image from the image obtained by capturing the user's eyes.

For example, when the light receiver 1520 is an IR detector, the IR detector may sequentially receive the IR light reflected by the light reflector 1400, and the processor 1800 may identify an array of sequentially received IR light that is consistent with the light reflector. 1400 pattern related parts.

The processor 1800 may execute the offset determination code 1750 stored in the storage device 1700 to determine the degree of offset of the support 190 of the AR device 1000 relative to the frame 110 . The support 190 may include, for example, temples 191 and a nose support 192 . When the user wears the AR device 1000, the temples 191 may become wider or narrower based on the size of the user's head and face. At this time, when IR light is received after the support 190 is biased relative to the frame 110 , the bias determination code 1750 may identify a pattern having a deformed shape from the received IR light. Additionally, the offset determination code 1750 may compare a pattern with a deformed shape to an undeformed pattern to estimate the degree of offset of the support 190 . For example, the offset determination code 1750 may compare a pattern with a deformed shape to an undeformed pattern to identify the degree of deformation of the pattern, and may determine the degree of offset of the support 190 based on the degree of deformation of the pattern. Alternatively, the offset determination code 1750 may estimate the degree of offset of the support 190 by analyzing only patterns with deformed shapes without comparing the patterns with deformed shapes to undeformed patterns. For example, when the pattern is a pattern of dots, the offset determination code 1750 may identify differences between the spacing between dots in the pattern having a deformed shape, thereby estimating the degree of offset of the support 190 .

For example, the degree of offset of the temple 191 may be expressed as an offset angle indicating the difference between the default angle of the temple 191 relative to the frame 110 and the offset angle of the temple 191 relative to the frame 110, but The present disclosure is not limited thereto. Additionally, for example, the degree of offset of nose support 192 may be expressed as an offset angle indicating the default angle of nose support 192 relative to frame 110 and the offset angle of nose support 192 relative to frame 110 . difference between, but the present disclosure is not limited to this.

In the above, it has been described that the pattern detection code 1740 detects the deformation of the pattern formed on the light reflector 1400, and the offset determination code 1750 analyzes the pattern having a deformed shape to estimate the offset degree of the support 190, but the present disclosure does not Limited to this.

For example, a specific pattern may be formed on a portion of the AR device 1000 that may reflect light emitted from the light emitter 1510 to direct the reflected light to the light receiver 1520 . For example, the pattern may be formed on a portion of the frame 110 of the AR device 1000 facing the light emitter 1510 and the light receiver 1520 . For example, the pattern may be formed by forming specific curves in portions of the frame 110 . Alternatively, for example, the pattern may be formed by forming a material capable of reflecting light on a partial area of the frame 110 . Furthermore, the pattern may be attached or coated on other elements included in the glasses-type AR device 1000, such as a vision correction lens for vision correction or a cover glass installed to protect the waveguide.

Alternatively, for example, the pattern may be formed on a portion of the nose support 192 of the AR device 1000 facing the light emitter 1510 and the light receiver 1520 . For example, the pattern may be formed by forming specific curves in portions of nose support 192 . Alternatively, for example, the pattern may be formed by forming a light-reflective material on a partial area of the nose support 192 . In this case, the light emitter 1510 and the light receiver 1520 are preferably located in the temple 191 of the AR device 1000, but the present disclosure is not limited thereto.

Alternatively, for example, the pattern may be formed in a partial area of the temple 191 of the AR device 1000 facing the light emitter 1510 and the light receiver 1520 . For example, the pattern may be formed by forming specific curves in partial areas of the temples 191 . Alternatively, for example, the pattern may be formed by forming a material capable of reflecting light on a partial area of the temple 191 . In this case, the light emitter 1510 and the light receiver 1520 are preferably located on the nose support 192 of the AR device 1000, but the present disclosure is not limited thereto.

The processor 1800 may execute the pupil position detection code 1760 stored in the storage device 1700, thereby detecting the pupil position of the user's eye. The pupil position detection code 1760 may identify the pupil position of the user's eye based on the IR light reflected from the light reflector 1400 .

For example, when light receiver 1520 is an IR camera, pupil position detection code 1760 may identify the pupil position of the user's eye within the image from the image captured by the IR camera. Alternatively, for example, when the light receiver 1520 is an IR detector, the pupil position detection code 1760 may analyze the IR light sequentially obtained by the IR detector to calculate the pupil position of the user's eye.

The pupil position detection code 1760 can identify the pupil center point of the user's eye, thereby identifying the pupil position of the user's eye.

The processor 1800 may execute the gaze determination code 1770 stored in the storage device 1700 to obtain information about the user's gaze. Processor 1800 may execute gaze determination code 1770 to calculate the location of the center of the user's eyes. The center of the user's eye may be the center of the user's eyeball. The processor 1800 may calculate the position of the center of the user's eye based on the pupil position of the user's eye obtained through the pupil position detection code 1760 and the offset degree of the support 190 obtained through the offset determination code 1750 . For example, the processor 1800 may calculate the position of the center of the user's eye based on a matrix for calibrating the degree of offset of the support 190 , a value indicating the position of the center of the user's eye, and an axis of an image obtained by capturing the user's eye. The value calculated for the offset may be the value of the pupil position of the user's eye obtained by the pupil position detection code 1760 . For example, the center of the eye may be the center of the eyeball, and the position of the center of the user's eye may have a 3D coordinate value in the coordinate system of the real space.

Processor 1800 may execute gaze determination code 1770 to calculate the location of the user's gaze point. In order to calculate the position of the user's gaze point, the processor 1800 may pre-generate a mapping function for calculating the position of the gaze point from the characteristics of the user's eyes. The mapping function is a function used to calculate the position of the user's gaze point taking into account the characteristics of the user's eyes and the offset information of the support 190, and may be generated during the calibration process of the calibration code 1780, which will be described below. For example, the position of the gaze point may have a 3D coordinate value in a coordinate system in real space, but the present disclosure is not limited thereto. For example, the position of the gaze point may have coordinate values in the coordinate system of the waveguide 170, but the present disclosure is not limited thereto.

The processor 1800 may execute the gaze determination code 1770 to calibrate the features related to the user's gaze obtained from the eye feature detection code 1730 based on the degree of bias obtained from the bias determination code 1750 . In addition, the processor 1800 may apply the features related to the user's gaze calibrated based on the degree of offset to the mapping function, thereby calculating the position of the user's gaze point. Furthermore, the user's gaze direction may be determined based on the position of the center point of the user's eyes and the user's gaze point calculated by the gaze determination code 1770 . A method of obtaining the user's gaze direction by using the gaze determination code 1770 will be described in more detail with reference to FIG. 11 .

Alternatively, the processor 1800 may calculate the user's gaze point without using the mapping function described above. For example, when the light receiver 1520 is an IR camera, the gaze determination code 1770 may calculate the gaze direction of the user's eyes from an image obtained by capturing the user's eyes by using a specific algorithm. In this case, the obtained gaze direction may be a vector value in the camera coordinate system indicating the gaze direction of the user's eyes. The algorithm for obtaining the gaze direction of the user's eyes may be an algorithm for fitting a 3D eye model. The algorithm for fitting the 3D eye model may be an algorithm for obtaining a vector value indicating the user's gaze direction by comparing an eye image corresponding to a reference vector value indicating the user's gaze direction with an image captured by the IR camera . Furthermore, the gaze determination code 1770 may convert a vector value indicating the gaze direction in the camera coordinate system into a vector value indicating the gaze direction in the coordinate system of the waveguide 170 by using the offset angle of the temple 191 . Thereafter, the gaze determination code 1770 may calculate the intersection point between the vector indicating the gaze direction in the coordinate system of the waveguide 170 and the waveguide 170 to obtain the user's gaze point.

The processor 1800 may execute the calibration code 1780 stored in the storage device 1700 to calibrate the mapping function based on the offset angle of the support 190 . The processor 1800 may execute the calibration code 1780 to calibrate the mapping function to obtain the user's gaze point based on the preset default offset angle and eye characteristics.

For example, when the light receiver 1520 is an IR camera, the processor 1800 may display a target point for calibration through the waveguide 170 and capture the user's eyes looking at the target point by using the IR camera. In addition, the processor 1800 may identify and analyze patterns within an image obtained by capturing the user's eyes, thereby obtaining the offset angle of the support 190 . In addition, the processor 1800 may detect the position of a feature point related to the user's eye from an image obtained by capturing the user's eye, and input the position of the feature point of the user's eye and the offset angle of the support 190 into the mapping function. The processor 1800 may calibrate the mapping function so that the position value of the target point may be output from the mapping function in which the position of the feature point of the user's eye and the offset angle of the support 190 are input.

For example, when the light receiver 1520 is an IR detector, the processor 1800 may display a target point for calibration through the waveguide 170 and control the IR scanner to emit IR light for scanning the user's eyes looking at the target point. In addition, the processor 1800 can receive and analyze the IR light reflected from the user's eyes through the IR detector, identify the pattern of the light reflector 1400, and estimate the offset angle of the temple 191. The processor 1800 may analyze the IR light based on the offset angle of the temple 191 to identify the location of the calibrated feature point of the eye. For example, when the processor 1800 estimates the position of the feature point of the eye by using the IR scanner and the IR detector, since the result of estimating the position of the feature point of the eye is affected by the operating angle of the IR scanner, the feature point of the eye The position may be calibrated based on the offset of the support 190 by calculating a value obtained by subtracting the offset angle of the support 190 from the operating angle of the IR scanner.

In addition, the processor 1800 may input the position of the calibrated feature point of the eye into the mapping function and calibrate the mapping function such that the position of the target point may be output from the mapping function in which the position of the calibrated feature point of the eye is input. value.

FIG. 4 is a diagram illustrating an example of operations of the light emitter 1510 and the light receiver 1520 of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 4 , the light emitter 1510 may emit IR light toward the light reflector 1400 , and the emitted IR light may be reflected by the light reflector 1400 and directed to the user's eyes. In addition, the IR light directed to the user's eyes may be reflected back by the user's eyes and directed to the light reflector 1400 , and the IR light reflected by the user's eyes may be reflected back by the light reflector 1400 and directed to the light receiver 1520 . In addition, the light receiver 1520 may receive the IR light reflected by the light reflector 1400 and directed to the light receiver 1520 from the user's eye.

In FIG. 4 , for convenience of explanation, the AR device 1000 as a glasses-type display device has been described as emitting IR light to the user's left eye and receiving IR light reflected from the user's left eye, but the present disclosure is not limited thereto. The AR device 1000 may emit IR light to the user's right eye and receive IR light reflected from the user's right eye in the same manner as shown in FIG. 4 .

FIG. 5a is a diagram illustrating an example of a light emitter 1510 that emits planar light according to an example embodiment of the present disclosure.

Referring to Figure 5a, the light emitter 1510 of Figure 2 can be an IR LED, and the light receiver 1520 of Figure 2 can be an IR camera. In this case, the light emitter 1510 may emit the IR light of the plane light toward the light reflector 1400, and the emitted IR light may be reflected by the light reflector 1400 and directed to the user's eyes. For example, in order to reflect the light emitted from the IR LED by using the light reflector 1400 and irradiate the light to an area including the user's eyes, the processor 1800 may control the irradiation direction of the IR light emitted from the IR LED and supply power to the IR LED, Thereby controlling the emission of IR light from the IR LED. In addition, the IR light of the plane light reflected by the light reflector 1400 can cover the entire user's eyes. In this case, the light receiver 1520 may be an IR camera, and the IR camera may receive IR light reflected by the user's eyes, thereby capturing the user's eyes.

For example, coordinate values on the coordinate system of the IR camera corresponding to pixels of the image captured by the IR camera may be set in advance. In addition, the processor 1800 may identify coordinate values corresponding to the feature points of the eye based on properties (eg, brightness) of the IR light received through the IR camera. For example, processor 1800 may identify the brightness of IR light received by an image sensor of an IR camera including a plurality of photodiodes, and identify at least one pixel among the pixels of an image captured by the IR camera that receives IR light indicative of a pupil, Thus, the coordinate value 51 corresponding to the pupil center point on the IR coordinate system is identified. In this case, the IR light indicating the pupil may be IR light having a brightness lower than a specific value, but the present disclosure is not limited thereto.

Furthermore, for example, the processor 1800 may identify the brightness of the IR light received through the image sensor of the IR camera including a plurality of photodiodes, and may identify at least one flicker characteristic point representing the eye among the pixels of the image captured by the IR camera. One pixel, thereby identifying the coordinate value 52 corresponding to the flicker feature point on the IR coordinate system. In this case, the IR light representing the flicker characteristic point of the eye may be IR light having a brightness greater than a specific value, but the present disclosure is not limited thereto.

According to an example embodiment of the present disclosure, the light emitter 1510 may be an IR LED that emits flash light, and in this case, the light reflector 1400 may be an IR event camera. An IR event camera can be an IR camera that is activated when a specific event occurs and automatically captures objects. For example, an IR event camera can be activated to automatically capture the user's eyes when the pattern of flashing light is different.

FIG. 5b is a diagram illustrating an example of a light emitter 1510 that emits point light according to an example embodiment of the present disclosure.

Referring to Figure 5b, the light emitter 1510 of Figure 2 may be an IR scanner, and the light receiver 1520 of Figure 2 may be an IR detector. In this case, the light emitter 1510 may emit point-light IR light toward the light reflector 1400, and the emitted IR light may be reflected by the light reflector 1400 and directed to the user's eyes. In this case, the light emitter 1510 may sequentially emit the IR light of the point light to the light reflector 1400 while changing the emission direction to the vertical direction or the horizontal direction, and the sequentially emitted IR light of the point light may be The light reflector 1400 reflects to cover the entire user's eyes. For example, in order to reflect the IR light of the spot light sequentially emitted from the IR scanner by using the light reflector 1400 and to irradiate the IR light of the spot light to an area including the user's eyes, the processor 1800 may control the spot light emitted from the IR scanner. The irradiation direction of the IR light. In this case, the light emitter 1510 may be a two-dimensional (2D) scanner, and the light receiver 1520 may be at least one photodiode.

For example, coordinate values on the coordinate system of the IR detector corresponding to the IR light in the array of light sequentially received through the IR detector may be set in advance, and the processor 1800 may based on the IR light sequentially received through the IR detector. The properties (eg, brightness) of the IR light in the light array are used to identify coordinate values corresponding to feature points of the eye. For example, the processor 1800 may identify coordinate values on the coordinate system of the IR detector corresponding to IR light in the array of received IR lights whose brightness is equal to or less than a specific value, thereby identifying the coordinate value on the IR coordinate system corresponding to the pupil. The coordinate value corresponding to the center point is 53. Furthermore, for example, the processor 1800 may identify a coordinate value on the coordinate system of the IR detector corresponding to the IR light whose brightness is equal to or less than a specific value in the array of received IR light, thereby identifying the coordinate value on the IR coordinate system. The coordinate value 54 corresponding to the flicker feature point of the eye.

FIG. 5c is a diagram illustrating an example of a light emitter 1510 that emits line light according to an example embodiment of the present disclosure.

Referring to Figure 5c, the light emitter 1510 of Figure 2 may be an IR scanner, and the light receiver 1520 of Figure 2 may be an IR detector. In this case, the light emitter 1510 may emit linear IR light to the light reflector 1400, and the emitted IR light may be reflected by the light reflector 1400 and directed to the user's eyes. For example, in order to reflect the IR light of the line light sequentially emitted from the IR scanner and to irradiate the IR light of the point light to an area including the user's eyes by using the light reflector 1400, the processor 1800 may control the point light emitted from the IR scanner. The irradiation direction of the IR light. In this case, the light emitter 1510 may sequentially emit the IR light of the line light to the light reflector 1400 while changing the emission direction, and the sequentially emitted IR light of the line light may be reflected by the light reflector 1400 to cover the entire User eyes. In this case, the light emitter 1510 may be a one-dimensional (ID) scanner, and the light receiver 1520 may be a photodiode array including a plurality of photodiodes. When the light receiver 1520 is a photodiode array, more preferably, the light receiver 1520 is disposed in the temple 191 . For example, when a horizontal line of light is emitted from the light emitter 1510, the light emitter 1510 may change the emission direction to a vertical direction to cover an area including the user's eyes, and when a vertical line of light is emitted from the light emitter 1510, the light emitter The transmitter 1510 may change the emission direction to a horizontal direction.

For example, coordinate values on the coordinate system of the IR detector corresponding to the IR light in the array of light sequentially received through the IR detector may be set in advance, and the processor 1800 may based on the IR light sequentially received through the IR detector. The properties (eg, brightness) of the IR light in the light array are used to identify coordinate values corresponding to feature points of the eye. For example, the processor 1800 may identify a line light of IR light having a brightness greater than or equal to a specific value, and identify, based on each line light, an IR light phase with a brightness less than or equal to a specific value among the plurality of photodiodes that have received the line light. The corresponding photodiode is used to identify the coordinate value 55 corresponding to the pupil center point on the IR coordinate system. Furthermore, for example, the processor 1800 may identify line lights with IR lights having a brightness greater than or equal to a specific value, and identify IR lights with brightness less than or equal to a specific value among the plurality of photodiodes that have received the line lights based on each line light. The photodiode corresponds to the light, thereby identifying the coordinate value 56 on the IR coordinate system corresponding to the flashing characteristic point of the eye.

In FIGS. 5a to 5c , for convenience of description, an example in which the AR device 1000 as a glasses-type display device emits IR light to one eye of the user has been described, but the present disclosure is not limited thereto. The AR device 1000 may emit IR light to the user's other eye in the same manner as shown in FIGS. 5a to 5c.

FIG. 6a is a diagram illustrating an example in which the light emitter 1510 and the light receiver 1520 are provided in the temple 191 of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 6a, the AR device 1000 of FIG. 2 may be a glasses-type device, and the light emitter 1510 and the light receiver 1520 may be provided on the temples 191 of the AR device 1000. The light emitter 1510 and the light receiver 1520 may be disposed on an inner portion of the temple 191 of the AR device 1000, which is a position between the temple 191 and the user's eyes. For example, the light emitter 1510 and the light receiver 1520 may be disposed on the inner side of the temple 191 of the AR device 1000 at a position spaced approximately 10 mm to 15 mm from the frame. The light emitter 1510 and the light receiver 1520 may be disposed in the temple 191 of the AR device 1000 facing the light reflector 1400.

FIG. 6b is a diagram illustrating an example in which the light emitter 1510 and the light receiver 1520 are provided in the nose support 192 of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 6b, the AR device 1000 of FIG. 2 may be a glasses-type device, and the light emitter 1510 and the light receiver 1520 may be provided on the nose support 192 of the AR device 1000. The light emitter 1510 and the light receiver 1520 may be disposed on an inner portion of the nose support 192 of the AR device 1000, which is a position between the nose support 192 and the user's eyes. The light emitter 1510 and the light receiver 1520 may be disposed in the nose support 192 of the AR device 1000 facing the light reflector 1400.

FIG. 6c is a diagram illustrating an example in which the light emitter 1510 and the light receiver 1520 are provided in the temples 191 and the nose support 192 of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 6c, the AR device 1000 of FIG. 2 may be a glasses-type device, the light emitter 1510 may be disposed on the temples 191 of the AR device 1000, and the light receiver 1520 may be disposed on the nose support 192 of the AR device 1000. The light emitter 1510 may be disposed on an inner portion of the temple 191 of the AR device 1000, which is a position between the temple 191 and the user's eyes. The light receiver 1520 may be disposed on an inner portion of the nose support 192 of the AR device 1000, which is a position between the nose support 192 and the user's eyes. The light emitter 1510 and the light receiver 1520 may be disposed facing the light reflector 1400 in the AR device 1000 .

6d is a diagram illustrating an example in which the light emitter 1510 and the light receiver 1520 are provided in the temples 191 and the nose support 192 of the AR device 1000 according to an example embodiment of the present disclosure.

Referring to FIG. 6d, the AR device 1000 of FIG. 2 may be a glasses-type device, the light emitter 1510 may be disposed on the nose support 192 of the AR device 1000, and the light receiver 1520 may be disposed on the temples 191 of the AR device 1000. The light emitter 1510 may be disposed on an inner portion of the nose support 192 of the AR device 1000, which is a position between the nose support 192 and the user's eyes. The light receiver 1520 may be disposed on an inner portion of the temple 191 of the AR device 1000, which is a position between the temple 191 and the user's eyes. The light emitter 1510 and the light receiver 1520 may be disposed facing the light reflector 1400 in the AR device 1000 .

In FIGS. 6a to 6d , it has been described that one light transmitter 1510 and one light receiver 1520 are provided in the AR device 1000, but the present disclosure is not limited thereto. For example, a plurality of light emitters 1510 may be provided in the AR device 1000. In this case, the plurality of light emitters 1510 may be provided on the temples 191 , or the plurality of light emitters 1510 may be provided on the nose support 192 . Alternatively, a plurality of light emitters 1510 may be separately provided on the temples 191 and the nose support 192 .

Furthermore, for example, a plurality of light receivers 1520 may be provided in the AR device 1000. In this case, the plurality of light receivers 1520 may be provided on the temples 191 , or the plurality of light receivers 1520 may be provided on the nose support 192 . Alternatively, a plurality of light receivers 1520 may be separately provided on the temples 191 and the nose support 192 .

In FIGS. 6a to 6d , for convenience of description, it has been described that the light emitter 1510 and the light receiver 1520 are provided in the left eye portion of the AR device 1000 as a glasses-type display device, but the present disclosure is not limited thereto. In the AR device 1000, the light emitter 1510 and the light receiver 1520 may be disposed in the right eye portion of the AR device 1000 in the same manner as shown in FIGS. 6a to 6d.

7 a is a diagram illustrating an example of a dot pattern formed on the light reflector 1400 of the AR device 1000 according to an exemplary embodiment of the present disclosure, and FIG. 7 b is a diagram illustrating a dot pattern formed on the light reflector 1400 of the AR device 1000 according to an exemplary embodiment of the present disclosure. Figure 7c is a diagram illustrating an example of a pattern in the form of a 2D marker according to an example embodiment of the present disclosure, and Figure 7d is a diagram illustrating an example of a grid pattern on the light reflector 1400 of the device 1000 according to an example embodiment of the present disclosure. Diagram of an example of light reflector 1400 covering a portion of waveguide 170 of an example embodiment of the present disclosure.

Referring to FIG. 7a, a dot pattern may be formed on the light reflector 1400 of the AR device 1000 of FIG. 2. Referring to FIG. 7b, a grid-like pattern may be formed on the light reflector 1400 of the AR device 1000 of FIG. 2. According to example embodiments, IR light may not be reflected from the patterned portion. At this time, since the dot pattern or grid pattern is used to detect the offset of the support or temples of the AR device, it is preferable to have a regular shape.

Referring to Figure 7c, this pattern is formed on the portion of the light reflector 1400 of the AR device 1000 of Figure 2 where the user's gaze is less frequently directed. The formed pattern may be, for example, a pattern in the form of a 2D marker, but the disclosure is not limited thereto. The pattern may be formed, for example, on a portion of the light reflector 1400 that does not interfere with capturing or scanning the user's eyes.

Referring to FIG. 7d , a light reflector 1400 may be formed on a portion of the waveguide 170 of the AR device 1000 of FIG. 2 . For example, light reflector 1400 may not be located in a portion of waveguide 170 that has little to do with reflection of IR light.

For example, when the light receiver 1520 is an IR camera, the IR camera may capture the user's eyes based on the IR light reflected by the light reflector 1400, and may not reflect the IR light from the patterned portion. For example, a portion in an image obtained by capturing the user's eyes that does not reflect IR light may be in black, and the processor 1800 may identify the black portion in the image obtained by capturing the user's eyes, thereby identifying a pattern in the image.

For example, when the light receiver 1520 is an IR detector, the IR detector may sequentially receive the IR light reflected by the light reflector 1400 and may not reflect the IR light from the pattern-formed portion. For example, processor 1800 may identify portions of an IR light array formed by sequentially received IR light that do not reflect IR light, thereby identifying the pattern of light reflector 1400 .

8a is a diagram illustrating light emission angles and patterns before the temples 191 of the AR device 1000 are biased according to an example embodiment of the present disclosure, and FIG. 8b is a diagram illustrating a light emission angle according to an example embodiment of the present disclosure. Illustration of light emission angles and patterns after the temples of an AR device are offset.

Referring to FIG. 8a, in the glasses-type AR device 1000 as shown in FIG. 2, before the temples 191 are biased, the light emitter 1510 may emit IR light to the light reflector 1400 on which a dot pattern is formed. According to an example embodiment, the light emitter 1510 may emit IR light to the light reflector 1400 at a first light emission angle 80, and the AR device 1000 may identify the first pattern 82 based on the IR light received by the light receiver 1520.

In addition, referring to FIG. 8b, in the glasses-type AR device 1000 as shown in FIG. 2, after the temples 191 are biased, the light emitter 1510 may emit IR light to the light reflector 1400 at a second light emission angle 90, And the AR device 1000 can recognize the second pattern 92 based on the IR light received by the light receiver 1520 .

As shown in FIGS. 8a and 8b , the first pattern 82 and the second pattern 92 may include points with different distances from each other, and the AR device 1000 may compare the first pattern 82 and the second pattern 92 to identify the temple 191 degree of bias. For example, the offset angle with respect to the first pattern 82 may be set to "0", and the offset angle with respect to the second pattern 92 may be based on a position difference between points in the first pattern 82 and points in the second pattern 92 to calculate. For example, the offset angle with respect to the second pattern 92 may be the difference between the first light emission angle 80 and the second light emission angle 90 . For example, when the pattern corresponding to the first light emission angle 80 is the first pattern 82 and the pattern corresponding to the first light emission angle 80 is the second pattern 92, the processor 1800 may convert the dots in the first pattern 82 to The spacing is compared with the spacing of the dots in the second pattern 92, thereby identifying a difference in the spacing of the dots in the second pattern 92 relative to the spacing of the dots in the first pattern 82, and the identified difference can be input to In at least one function for calculating the degree of offset of the temple 191, thereby calculating the degree of offset of the temple 191, the offset degree indicating the first light emission angle 80 corresponding to the first pattern 82 and the second pattern 92 and the corresponding second light emission angle 90 .

Furthermore, for example, when the pattern corresponding to the first light emission angle 80 is the first pattern 82 and the pattern corresponding to the first light emission angle 80 is the second pattern 92 , the processor 1800 may change the pattern in the first pattern 82 The position of the point is compared with the position of the point in the second pattern 92, thereby identifying a difference in the position of the point in the second pattern 92 relative to the position of the point in the first pattern 82, and the identified difference can be The value is input into at least one function for calculating the degree of offset of the temple 191, thereby calculating the degree of offset of the temple 191, the degree of offset being indicative of the first light emission angle 80 corresponding to the first pattern 82 and the The difference between the second light emission angles 90 corresponding to the second pattern 92 .

According to another example embodiment, the processor 1800 may identify the degree of offset of the temple 191 based on the second pattern 92 without comparing the first pattern 82 and the second pattern 92 . In this case, the processor 1800 may identify the difference in the spacing of the points in the second pattern 92 and input the identified difference into at least one function for calculating the offset degree of the temple 191, thereby Calculate the degree of offset of the temple 191.

9 is a diagram illustrating patterns recognized from an array of light received through the light receiver 1520 when the light emitter of the AR device 1000 is the IR scanner 1520-1 or 1520-2 according to an example embodiment of the present disclosure. Diagram of an example.

Referring to FIG. 9 , the IR scanner 1520 - 1 may represent the IR scanner before the support 190 of the glasses-type AR device 1000 shown in FIG. 2 is biased. The IR scanner 1520 - 1 may sequentially emit point-light IR light toward an area where the user's eyes are located by using the light reflector 1400 , and the light receiver 1520 may receive the first light array 90 .

Furthermore, the IR scanner 1520-2 may represent the IR scanner after the support 190 of the AR device 1000 is biased. The IR scanner 1520 - 2 may sequentially emit point-light IR light toward an area where the user's eyes are located by using the light reflector 1400 , and the light receiver 1520 may receive the second light array 92 .

In the first light array 90 and the second light array 92 , the positions and intervals of the light signals corresponding to the dot patterns of the light reflector 1400 may be different from each other, and the AR device 1000 may combine the light signals in the first light array 90 with the light signals. The portion corresponding to the dot pattern of the reflector 1400 is compared with the portion of the second light array 92 corresponding to the dot pattern of the light reflector 1400 . In addition, the AR device 1000 may identify the degree of offset of the support 190 of the AR device 1000 based on the comparison result.

For example, since the IR light emitted toward the dot pattern is not reflected by the light reflector 1400, the portions 90-1, 90-2, 90-3, and 90-4 of the first light array 90 corresponding to the dot pattern and the third No optical signal may be received in the portions 92-1, 92-2, 92-3, and 92-4 of the two-light array 92 corresponding to the dot patterns. The processor 1800 may identify portions of the first light array 90 that do not receive the optical signal, thereby identifying portions 90-1, 90-2, 90-3, and 90-4 of the first light array 90 that correspond to the dot patterns. , and identify the coordinate values 90-5, 90-6, and 90- of the portions 90-1, 90-2, 90-3, and 90-4 corresponding to the dot pattern on the coordinate system of the IR scanner 1520-1, respectively. 7 and 90-8. Additionally, for example, the processor 1800 may identify portions of the second light array 92 that do not receive optical signals, thereby identifying portions 92-1, 92-2, 92-3 of the second light array 92 that correspond to the dot patterns. and 92-4, and identify the coordinate values 92-5, 92- of the portions 92-1, 92-2, 92-3 and 92-4 corresponding to the dot pattern on the coordinate system of the IR scanner 1520-2, respectively. 6, 92-7 and 92-8.

The processor 1800 may respectively indicate the coordinate values 90-1, 90-2, 90-3 and 90-4 of the first light array 90 corresponding to the dot pattern on the coordinate system of the IR scanner 1520-1. 5, 90-6, 90-7 and 90-8 respectively indicate portions 92-1, 92-2 and 92-3 of the second light array 92 corresponding to the dot pattern on the coordinate system of the IR scanner 1520-2. Compare with the coordinate values 92-5, 92-6, 92-7 and 92-8 of 92-4 to identify the coordinate values 90-5, 90-6, 90-7 and 90-8 and the coordinate value 92-5 , the difference between 92-6, 92-7 and 92-8. For example, the processor 1800 may be based on the difference between coordinate value 90-5 and coordinate value 92-5, the difference between coordinate value 90-6 and coordinate value 92-6, the difference between coordinate value 90-7 and coordinate value 92 The difference between -7 and the difference between the coordinate value 90-8 and the coordinate value 92-8 is used to calculate the offset degree of the support 190 of the AR device 1000.

In FIG. 9 , for convenience of description, the IR scanner 1520 - 1 before the support 190 of the AR device 1000 is biased and after the support 190 of the AR device 1000 is biased are separately shown in FIG. 9 IR scanner 1520-2 to distinguish between IR scanner 1520-1 before support 190 of AR device 1000 is biased and IR scanner 1520-2 after support 190 of AR device 1000 is biased open. However, the IR scanner 1520-1 before the support 190 of the AR device 1000 is biased and the IR scanner 1520-2 after the support 190 of the AR device 1000 is biased may be installed in the AR device 1000 Same IR scanner.

Furthermore, in FIG. 9 , for convenience of description, a first light array 90 has been described for identifying the dot pattern using the IR scanner 1520 - 1 before the support 190 is biased, and a second light array 92 The dot pattern is recognized using the IR scanner 1520-2 after the support 190 of the AR device 1000 is biased, but the present disclosure is not limited thereto. Multiple light arrays for overlaying the dot pattern may be used to identify the dot pattern using the IR scanner 1520-1 before the support 190 is biased, and multiple light arrays for overlaying the dot pattern may be used to identify the dot pattern using the IR scanner 1520-1 before the support 190 is biased. The support 190 is biased followed by the IR scanner 1520-2 to identify the dot pattern.

10 is a diagram illustrating eyes recognized from an array of light received through the light receiver 1520 when the light emitter of the AR device 1000 is the IR scanner 1520-1 or 1520-2 according to an example embodiment of the present disclosure. Diagram of an example of a feature.

Referring to FIG. 10 , the IR scanner 1520 - 1 before the support 190 of the glasses-type AR device 1000 as shown in FIG. 2 is biased may sequentially emit point light to the area where the user's eyes are located by using the light reflector 1400 . IR light, and the light receiver 1520 can receive the third light array 100 .

In addition, after the support 190 of the AR device 1000 is biased, the IR scanner 1520-2 may sequentially emit point-light IR light to an area where the user's eyes are located by using the light reflector 1400, and the light receiver 1520 may receive Fourth light array 102.

In the third light array 100 and the fourth light array 102, the positions and intervals of the light signals corresponding to the characteristic points of the eyes may be different from each other. Considering the offset angle of the support 190, the AR device 1000 may calibrate the position of the portion in the fourth light array 102 corresponding to the feature point of the eye.

For example, the processor 1800 may identify the portion 102-1 in the fourth light array 102 corresponding to the flicker feature point of the eye based on the brightness of the light in the fourth light array 102, and identify the portion 102-1 on the coordinate system of the IR scanner. The coordinate value 102-2 of the portion 102-1 corresponding to the flicker characteristic point of the eye is indicated. Thereafter, the processor 1800 may calibrate the coordinate value 102-2 of the flicker feature point indicating the eye by using the offset angle calculated in FIG. 9. For example, the processor 1800 may multiply the coordinate value 102-2 representing the flicker characteristic point of the eye by the compensation matrix 18 of FIG. 11 (to be described below), thereby obtaining the calibrated coordinate value.

In FIG. 10 , for convenience of description, the IR scanner 1520 - 1 before the support 190 of the AR device 1000 is biased and after the support 190 of the AR device 1000 is biased are separately shown in FIG. 10 IR scanner 1520-2 to distinguish between IR scanner 1520-1 before support 190 of AR device 1000 is biased and IR scanner 1520-2 after support 190 of AR device 1000 is biased open. However, the IR scanner 1520-1 before the support 190 of the AR device 1000 is biased and the IR scanner 1520-2 after the support 190 of the AR device 1000 is biased are the same ones installed in the AR device 1000. IR scanner.

In addition, in FIG. 10 , for convenience of description, it has been described that the third light array 100 is used to identify feature points of the eye using the IR scanner 1520 - 1 before the support 190 is biased, and a fourth light array 102 is used to identify feature points of the eyes using the IR scanner 1520-2 after the support 190 of the AR device 1000 is biased, but the present disclosure is not limited thereto. Multiple light arrays for covering the user's eyes may be used to identify feature points of the eyes using the IR scanner 1520-1 before the support 190 is biased, and multiple light arrays for covering the user's eyes may be used to Characteristic points of the eye are identified using the IR scanner 1520-2 after the support 190 is biased.

11 is a diagram illustrating an example of a function used by the AR device 1000 to calculate the center of the eyeball and calculate the user's gaze point 16 according to an example embodiment of the present disclosure.

Equation 11 represents the relationship between the coordinate value 12 of the pupil center of the eye in the coordinate system of the IR camera and the coordinate value 13 representing the center of the eyeball in real space. In an example embodiment of the present disclosure, the coordinate value 12 of the coordinate system of the IR camera may have a 2D coordinate value, and the coordinate value 13 of the real space may have a 2D coordinate value or a 3D coordinate value.

For example, when the coordinate value 13 representing the center of the eyeball is multiplied by the camera rotation matrix 20 and the scaling factor and a value representing the offset in the image is added, the coordinate value 12 of the pupil center of the eye can be calculated. The camera rotation matrix 20 is a matrix that converts coordinate values in the real space into coordinate values in the coordinate system of the camera in consideration of the position where the camera is installed. By multiplying the coordinate value 13 representing the center of the eyeball by the camera rotation matrix 20, the coordinate value 13 representing the center of the eyeball can be converted into a coordinate value of the coordinate system of the IR camera, and the size of the converted coordinate value can be determined by converting Coordinate values are normalized by multiplying them by a scaling factor. Furthermore, by adding a value indicating the offset in the image to the magnitude of the normalized coordinate value, the normalized 2D coordinate value can be corrected by reflecting the offset in the image, allowing the pupil center of the eye to be calculated The coordinate value is 12. Furthermore, the value representing the offset in the image may be used to place the eye position in the image captured by the IR camera at a reference position. For example, a value representing an offset in an image may be a value for moving the center point of the eye in the captured image based on a difference between the center point of the image captured by the IR camera and the center point of the eye in the captured image. The value to the center point of the captured image.

In an example embodiment of the present disclosure, the camera rotation matrix 20, the scaling factor, and the value representing the bias in the image may take into account, for example, where the IR LED is set, where the IR camera is set, the capture direction of the IR camera, the IR camera's The viewing angle, images captured by the IR camera, information related to human eyes (e.g., eyeball size, pupil size, etc.), previously captured eye images, etc., may be determined in advance on the AR device 1000 during manufacturing of the AR device 1000 middle.

In an example embodiment of the present disclosure, the AR device 1000 may input the coordinate value 12 of the pupil center of the eye into Equation 11, thereby obtaining the coordinate value 13 representing the center of the eyeball.

Equation 15 can represent the relationship between the value 17, which represents the characteristic point of the eye, and the user's gaze point 16. For example, the feature points calibrated by reflecting the offset of the support 190 can be obtained by multiplying the value 17 of the feature point representing the eye by the compensation matrix 18 . Furthermore, the value 19 output by inputting a value 17 representing the characteristic point of the eye by the compensation matrix 18 into the mapping function F may be the coordinate value 16 representing the user's gaze point. . The compensation matrix 18 may be a matrix for compensating the degree of offset of the support 190 . It can be determined by comparing the image captured from the IR camera in a state where the support 190 of the AR device 1000 is not biased and the image captured by the IR camera in a state where the support 190 of the AR device 1000 is biased. The compensation matrix 18 is determined by comparison, and the compensation matrix 18 may be preset in the AR device 1000 during manufacturing of the AR device 1000 . For example, taking into account the degree of bias of the support 190 , the compensation matrix 18 may be determined such that an image captured from an IR camera in a state in which the support 190 is biased may be converted to a state in which the support 190 is not biased. Below is an image captured by an IR camera. However, the example of determining the compensation matrix 18 is not limited to this.

For example, the compensation matrix 18 may be obtained by comparing information obtained from an image captured by an IR camera in a state in which the support 190 is not biased with information obtained from an image captured by an IR camera in a state in which the support 190 of the AR device 1000 is biased. Determined by comparison between the information obtained from the images. The data obtained from the image may include, for example, the size of the image, the position of the eyes in the image, the position of the pupil in the image, the position of the pattern in the image, etc., but the present disclosure is not limited thereto.

Alternatively, for example, the compensation matrix 18 may be obtained by comparing an image captured from an IR camera in a state in which the support 190 is not biased with an image captured from an IR camera in a state in which the support 190 of the AR device 1000 is biased. The information obtained from the images is compared to determine. Alternatively, for example, the compensation matrix 18 may be obtained by comparing information obtained from an image captured from an IR camera in a state where the support 190 is not biased with a state in which the support 190 of the AR device 1000 is biased. The images captured by the camera are compared to determine.

Furthermore, the mapping function F that is a function for calculating the user's gaze point from the feature points of the eyes may be determined such that the user's gaze point is calculated from the reward matrix 18 and the feature points of the eyes, and may be used in the AR device 1000 It is previously set in the AR device 1000 during manufacturing.

The AR device 1000 may obtain the coordinate value 16 representing the user's gaze point from the position of the feature point of the eye using Equation 15.

FIG. 12 is a flowchart of a method of detecting user gaze performed by the AR device 1000 of FIGS. 2 and 3 according to an example embodiment of the present disclosure.

In operation S1200, the AR device 1000 may emit IR light to the light reflector 1400 through the light emitter 1510, which is installed on the support 190 extending from the frame 110 of the AR device 1000. The AR device 1000 may emit IR light to at least a partial area of the light reflector 1400 so that the IR light reflected by the light reflector 1400 may cover the user's eyes.

For example, when the light receiver 1520 is an IR camera, the light emitter 1510 may be an IR LED, and the AR device 1000 may control the IR LED such that the IR light emitted from the IR LED may be reflected by the light reflector 1400 and may cover the user's eyes , so that the IR camera captures the user's eyes. Alternatively, for example, when the light receiver 1520 is an IR detector, the light emitter 1510 may be an IR scanner, and the AR device 1000 may control the IR scanner to reflect the IR emitted from the IR scanner using the light reflector 1400 The light is used to scan the user's eyes so that the IR detector can detect the user's eyes.

In operation S1205, the AR device 1000 may receive the IR light reflected by the user's eyes and reflected back by the light reflector 1400 through the light receiver 1520 installed on the support 190 extending from the frame 110 of the AR device 1000.

For example, when the light emitter 1510 is an IR LED, the light receiver 1520 may be an IR camera, and the AR device 1000 may control the IR camera to capture the user's eyes through light from the user's eyes reflected by the reflector 1400. Alternatively, for example, when the light emitter 1510 is an IR scanner, the light receiver 1520 may be an IR detector, and the AR device 1000 may control the IR detector to detect the light reflected by the user's eyes and reflected back by the light reflector 1400 IR light, allowing the IR detector to detect the user's eyes.

In operation S1210, the AR device 1000 may detect preset characteristics related to the gaze of the user's eyes based on the received IR light. For example, the AR device 1000 may detect the position of the pupil feature point of the user's eye and the position of the flicker feature point of the eye. The pupil feature point may be, for example, a pupil center point, and the flicker feature point of the eye may be a portion of the detected eye area where the brightness is greater than or equal to a specific value. The position of the pupil feature point and the position of the eye's flicker feature point can be identified, for example, by coordinate values indicating the position in the coordinate system of the light receiver 1520 . For example, the coordinate system of the light receiver 1520 may be the coordinate system of the IR camera or the coordinate system of the IR detector, and the coordinate values in the coordinate system of the light receiver 1520 may be 2D coordinate values.

The AR device 1000 may detect preset characteristics related to the gaze of the eye by analyzing light received by the light receiver 1520 . For example, when the light receiver 1520 is an IR camera, the AR device 1000 can identify the position of the pupil feature point and the position of the blink feature point of the eye in an image captured by the IR camera. Alternatively, for example, when the light receiver 1520 is an IR detector, the AR device 1000 may analyze the IR light detected by the IR detector, thereby identifying the position of the pupil feature point and the position of the blink feature point of the eye.

In addition, the AR device 1000 may analyze the light received by the light receiver 1520, thereby obtaining coordinate values indicating the position of the pupil feature point and coordinate values indicating the position of the flash feature point of the eye. For example, when the light receiver 1520 is an IR camera, the AR device 1000 may obtain the coordinate values of the pupil feature point and the coordinate value of the blink feature point of the eye from the coordinate system of the IR camera. For example, when the light receiver 1520 is an IR camera, the AR device 1000 can identify the location of the pupil center point in an image captured by the IR camera. For example, the position of the pupil center point may have coordinate values in the coordinate system of the IR camera.

For example, the AR device 1000 can identify the position of the brightest point in an image captured by an IR camera to identify a flicker characteristic point of the eye. The AR device 1000 can identify the brightness of IR light received through an image sensor of an IR camera including a plurality of photodiodes, and can identify in pixels of an image captured by the IR camera corresponding to bright IR light that is equal to or greater than a specific reference At least one pixel of the image is used to identify the location of the flashing feature point of the eye. For example, the AR device 1000 can identify pixels corresponding to the brightest IR light among the pixels of an image captured by an IR camera, thereby identifying the location of a flash feature point of the eye. For example, the position of the blink feature point of the eye may have coordinate values in the coordinate system of the IR camera.

Alternatively, for example, when the light receiver 1520 is an IR detector, the AR device 1000 may calculate coordinate values of the pupil feature point and coordinate values of the flicker feature point of the eye in the coordinate system of the IR detector.

When the light emitter 1510 is an IR scanner, the AR device 1000 can control the IR scanner to sequentially illuminate the point light source or the line light source to cover the area where the user's eyes are, and sequentially receive the light reflected from the user's eyes through the IR detector. , to scan the area where the user's eyes are. Furthermore, the AR device 1000 can analyze an array of light sequentially received through the IR detector, thereby identifying pupil feature points and flicker feature points of the eye. For example, the AR device 1000 can identify light whose brightness is equal to or greater than a specific value in the received light array, thereby identifying the coordinates of the blinking feature point of the eye. For example, the position of the eye's flicker feature point may have coordinate values in the coordinate system of the IR detector.

In operation S1215, the AR device 1000 may detect the pattern of the light reflector 1400 based on the received IR light. The light reflector 1400 may be coated on one surface of the waveguide 170 of the AR device 1000 to have a specific pattern. The AR device 1000 may receive the IR light reflected by the user's eyes and reflected by the light reflector 1400 through the light receiver 1520 and recognize the shape of the pattern based on the received IR light. Patterns formed on the light reflector 1400 may include, for example, dot patterns, line patterns, grid patterns, 2D markers, etc., but the present disclosure is not limited thereto. When temples 191 are offset relative to frame 110, the pattern identified by pattern detection code 1740 may have a distorted shape.

For example, when the light receiver 1520 is an IR camera, the IR camera can capture the user's eyes based on the IR light reflected by the light reflector 1400, and the AR device 1000 can recognize a pattern within the image from the image obtained by capturing the user's eyes. For example, when the light receiver 1520 is an IR detector, the IR detector may sequentially receive the IR light reflected by the light reflector 1400, and the AR device 1000 may identify an array of the sequentially received IR light that is consistent with the light reflector. 1400 pattern related parts.

In operation S1220, the AR device 1000 may determine the degree to which the support 190 of the AR device 1000 is offset relative to the frame 110. When receiving IR light after the support 190 is biased relative to the frame 110, the AR device 1000 may recognize a pattern having a deformed shape from the received IR light. Furthermore, for example, as in FIGS. 8a and 8b , the AR device 1000 may compare a pattern with a deformed shape with an undeformed pattern, thereby estimating the degree of bias of the support 190 . For example, the degree of offset of the temple 191 may be expressed as an offset angle indicating the difference between the default angle of the temple 191 relative to the frame 110 and the offset angle of the temple 191 relative to the frame 110, but The present disclosure is not limited thereto. Additionally, for example, the degree of offset of nose support 192 may be expressed as an offset angle indicating the default angle of nose support 192 relative to frame 110 and the offset angle of nose support 192 relative to frame 110 . difference between, but the present disclosure is not limited to this.

Furthermore, for example, when the light receiver 1520 is an IR detector, the coordinate value of the pupil feature point and the coordinate value of the flicker feature point of the eye may be values calibrated by reflecting the offset degree of the support 190 of the AR device 1000 . When the light receiver 1520 is an IR detector, for example, when calculating the coordinate value 102-2 corresponding to the IR light 102-1 corresponding to the feature point of the eye in the light array 102 of FIG. 10, the support may be reflected 190 degree of bias. For example, by reflecting the degree of offset of the support 190, the AR device 1000 can calibrate the position of the light 102-1 in the light array 102 received by the IR detector that corresponds to the characteristic point of the eye in the light array 102. . The AR device 1000 may directly calculate the coordinate value 102 corresponding to the feature point of the eye in the coordinate system of the IR detector based on the calibrated position of the light 102 - 1 in the light array 102 corresponding to the feature point of the eye. In this case, the AR device 1000 may calculate the coordinate values of the pupil feature points and the flicker feature points of the eyes calibrated by reflecting the offset degree of the temples 191 of the AR device 1000 and/or the offset degree of the nose support 192 coordinate value. Calibrated coordinate values can be input into the mapping function.

In operation S1225, the AR device 1000 may identify the pupil position of the user's eye based on the IR light reflected from the light reflector 1400. For example, when the light receiver 1520 is an IR camera, the AR device 1000 can identify the pupil position of the user's eye within the image from the image captured by the IR camera. Alternatively, for example, when the light receiver 1520 is an IR detector, the AR device 1000 may analyze the IR light sequentially obtained by the IR detector, thereby calculating the pupil position of the user's eye. The AR device 1000 can identify the pupil center point of the user's eye, thereby identifying the pupil position of the user's eye.

In operation S1230, the AR device 1000 can obtain the user's gaze direction. The AR device 1000 can calculate the position of the center of the user's eyes. The center of the user's eye may be the center of the user's eyeball. The AR device 1000 may calculate the position of the center of the user's eye based on the pupil position of the user's eye and the offset degree of the support 190 . For example, the processor 1800 may calculate the position of the center of the user's eye based on a matrix for calibrating the degree of offset of the support 190 , a value indicating the position of the center of the user's eye, and an axis of an image obtained by capturing the user's eye. The value calculated for the offset may be the value of the pupil position of the user's eye obtained by the pupil position detection code 1760 . For example, the center of the eye may be the center of the eyeball, and the position of the center of the user's eye may have a 3D coordinate value in the coordinate system of the real space.

The AR device 1000 can calculate the position of the user's gaze point. In order to calculate the position of the user's gaze point, the AR device 1000 may pre-generate a mapping function for calculating the position of the gaze point from the characteristics of the user's eyes. The mapping function is a function used to calculate the location of the user's gaze point taking into account the characteristics of the user's eyes and the offset information of the support 190 , and may be generated during the calibration process of the calibration code 1780 . For example, the position of the gaze point may have a 3D coordinate value in a coordinate system in real space, but the present disclosure is not limited thereto. For example, the position of the gaze point may have coordinate values in the coordinate system of the waveguide 170, but is not limited thereto.

The AR device 1000 may calibrate features related to the user's gaze based on the degree of offset obtained from the offset determination code 1750 . In addition, the AR device 1000 may apply the features related to the user's gaze calibrated based on the degree of offset to the mapping function, thereby calculating the position of the user's gaze point. Furthermore, the user's gaze direction may be determined based on the position of the center point of the user's eyes and the user's gaze point calculated by the gaze determination code 1770 .

Meanwhile, the AR device 1000 may calibrate the mapping function based on the offset angle of the support 190 . The AR device 1000 may calibrate the mapping function for obtaining the user's gaze point based on the default offset angle and the characteristics of the eye.

For example, when the light receiver 1520 is an IR camera, the AR device 1000 can display a target point for calibration through the waveguide 170 and capture the user's eyes looking at the target point by using the IR camera. In addition, the processor 1800 may identify and analyze patterns within an image obtained by capturing the user's eyes, thereby obtaining the offset angle of the support 190 . In addition, the AR device 1000 may detect the position of a feature point related to the user's eye from an image obtained by capturing the user's eye, and input the position of the feature point of the user's eye and the offset angle of the support 190 into the mapping function. The AR device 1000 can calibrate the mapping function so that the position value of the target point can be output from the mapping function in which the position of the feature point of the user's eye and the offset angle of the support 190 are input.

For example, when the light receiver 1520 is an IR detector, the AR device 1000 may display a target point for calibration on the waveguide 170 and control the IR scanner to emit IR light for scanning the user's eyes looking at the target point. In addition, the AR device 1000 can receive and analyze the IR light reflected from the user's eyes through the IR detector, identify the pattern of the light reflector 1400, and estimate the offset angle of the temples 191. The AR device 1000 may analyze the IR light based on the offset angle of the temple 191 to identify the position of the calibrated feature point of the eye. For example, when the AR device 1000 estimates the position of the feature point of the eye by using the IR scanner and the IR detector, since the result of estimating the position of the feature point of the eye is affected by the operating angle of the IR scanner, the feature point of the eye The position may be calibrated based on the offset of the support 190 by calculating a value obtained by subtracting the offset angle of the support 190 from the operating angle of the IR scanner.

Furthermore, the AR device 1000 may input the position of the calibrated feature point of the eye into the mapping function and calibrate the mapping function such that the position of the target point may be output from the mapping function in which the position of the calibrated feature point of the eye is input. value.

Example embodiments of the present disclosure may be implemented as a recording medium including computer-readable instructions (eg, computer-executable program modules). According to example embodiments, computer-readable instructions may be computer code implemented as one or more computer-executable program modules. Computer-readable media can be any available media that can be accessed by the computer and can include volatile or nonvolatile media and removable or non-removable media. In addition, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-volatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program code, program modules or other data. Remove the media. Communication media typically may include computer readable instructions, data structures, or other data (eg, program modules) that modulate a data signal.

Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage medium" refers to a tangible device and does not include signals (eg, electromagnetic waves), and does not distinguish between a case where data is semi-permanently stored in the storage medium and a case where data is temporarily stored. For example, non-transitory storage media may include buffers that temporarily store data.

According to example embodiments of the present disclosure, methods according to various embodiments of the present disclosure disclosed herein may be included in and provided in a computer program product. Computer program products may be traded as commodities between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., PlayStore ^™ ), or directly Distributed between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable application) may be temporarily stored in a machine-readable storage medium (eg, the memory of a manufacturer's server, an application store's server, or a relay server).

Furthermore, in this specification, the term "unit" may be a hardware component such as a processor or a circuit and/or a software component executed by a hardware component such as a processor.

Furthermore, in the specification, the expression "including at least one of a, b or c" means "including only a", "including only b", "including only c", "including a and b", "including b and c", "including a and c", or "including a, b and c".

The above description of the present disclosure is provided for illustrative purposes only, and it will be understood by those skilled in the art that the present disclosure can be easily modified to other detailed configurations without modifying the technical aspects and essential features of the present disclosure. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive. For example, an element described as a single entity may be distributed in an implementation, and similarly, elements described as distributed may be combined in an implementation.

The scope of the disclosure is defined not by the detailed description of the disclosure but by the appended claims, and all modifications or substitutions derived from the scope and spirit of the claims and their equivalents are within the scope of the disclosure.

Claims (15)

1. An augmented reality AR device comprising:

a waveguide;

a light reflector including a pattern;

a support configured to secure the AR device to a face of a user of the AR device;

a light emitter and a light receiver mounted on the support; and

at least one processor configured to:

controlling the light emitter to emit light to the light reflector,

identifying the pattern based on the light received by the light receiver, and

based on the identified pattern, gaze information of a user of the AR device is obtained, wherein light emitted towards the light reflector is reflected by the light reflector and directed towards the user's eye, and

wherein the light received by the light receiver includes light reflected by the user's eye that is directed toward the user's eye.

2. The AR device of claim 1, wherein the support comprises:

a temple extending from a frame surrounding the waveguide to be positioned over the user's ear; and

a nose support extends from the frame and is positioned over the nose of the user.

3. The AR device of claim 2, wherein the light reflector is formed on the waveguide.

4. The AR device of claim 3, wherein the at least one processor is further configured to: the degree of offset of the support relative to the frame from which the support extends is identified by analyzing the identified pattern.

5. The AR device of claim 3, wherein the at least one processor is further configured to:

generating a mapping function for calculating the position of the gaze point of the user based on the degree of bias of the support relative to the frame, and

the gaze information of the user is obtained based on the mapping function and a degree of bias of the support relative to the frame.

6. The AR device of claim 5, wherein the at least one processor is further configured to: based on the light received by the light receiver, the position of one or more feature points corresponding to the eyes of the user is obtained.

7. The AR device of claim 6, wherein the at least one processor is configured to: the position of the one or more feature points corresponding to the eyes of the user and the degree of offset of the support relative to the frame are input into the mapping function, and the position of the gaze point of the user is calculated.

8. The AR device of claim 5, wherein the at least one processor is further configured to:

displaying a target point at a specific location on the waveguide, to calibrate the mapping function,

receiving, by the light receiver, light reflected by the eyes of the user looking at the displayed target point, and

the mapping function is calibrated based on light reflected by the eyes of the user looking at the displayed target point.

9. The AR device of claim 8, wherein to calibrate the mapping function, the at least one processor is further configured to:

based on light reflected by the eyes of the user looking at the displayed target point, a pattern of the light reflector is identified,

identifying a degree of bias of the support based on the identified pattern, and

based on light reflected by the eyes of the user looking at the displayed target point, a position of one or more feature points corresponding to the eyes of the user looking at the displayed target point is obtained.

10. The AR device of claim 9, wherein the at least one processor is further configured to:

inputting the bias degree of the glasses leg and the positions of the one or more feature points into the mapping function, and

The mapping function is calibrated such that the position value of the target point is output from the mapping function.

11. A method performed by an augmented reality AR device of detecting a gaze of a user, the method comprising:

transmitting light to a light reflector comprising a pattern through a light transmitter mounted in a support of the AR device, the light transmitted by the light transmitter being directed to an eye of a user wearing the AR device;

receiving light reflected by the eyes of the user through a light receiver mounted on the support;

identifying the pattern based on the light received by the light receiver; and

based on the identified pattern, gaze information of the user is obtained.

12. The method of claim 11, further comprising: analyzing the identified pattern and, based on the identified pattern, identifying a degree of bias of the support relative to the frame,

wherein the support extends from the frame,

wherein obtaining the gaze information comprises: a gaze direction of the user is determined based on a degree of bias of the support relative to the frame.

13. The method of claim 12, further comprising: based on the degree of bias of the support relative to the frame, a mapping function is generated for calculating the position of the gaze point of the user,

Wherein obtaining the gaze information comprises: based on the mapping function and the degree of bias of the support relative to the frame, a position of a gaze point of the user is calculated.

14. The method of claim 12, further comprising: based on the light received by the light receiver, obtaining the position of one or more feature points corresponding to the eyes of the user,

wherein obtaining the gaze information comprises: the position of the one or more feature points corresponding to the eyes of the user and the degree of offset of the support relative to the frame are input into the mapping function, and the position of the gaze point of the user is calculated.

15. A computer-readable recording medium having recorded thereon a program for executing the method according to claim 11 on a computer.