KR20240124152A - Wearable device for performing rendering regarding virtual object based on external light and method thereof - Google Patents

Abstract

In one embodiment, a processor of a wearable device may be configured to obtain information related to at least one real light source using an image acquired from a camera. The processor may be configured to receive an input for displaying a virtual space on the display while displaying at least a portion of the image acquired from the camera on the display. The processor may be configured to determine the location of at least one virtual light source in the virtual space using the information based on the input. The processor may be configured to display a virtual object having a visual effect applied to the at least one virtual light source on the display based on a distance between a virtual object included in the virtual space and a location of the virtual space corresponding to the wearable device.

Description

{WEARABLE DEVICE FOR PERFORMING RENDERING REGARDING VIRTUAL OBJECT BASED ON EXTERNAL LIGHT AND METHOD THEREOF}

The present disclosure relates to a wearable device and method for performing rendering of a virtual object based on external light.

In order to provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services that display computer-generated information in conjunction with external objects in the real world. The electronic devices may be wearable devices that can be worn by a user. For example, the electronic devices may be AR glasses and/or a head-mounted device (HMD).

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

In one embodiment, a wearable device may include a camera, a display, and a processor. The processor may be configured to acquire information related to at least one real light source using an image acquired from the camera. The processor may be configured to receive an input for displaying a virtual space on the display while displaying at least a portion of the image acquired from the camera on the display. The processor may be configured to determine a location of at least one virtual light source in the virtual space using the information based on the input. The processor may be configured to display a virtual object to which a visual effect for the at least one virtual light source is applied on the display based on a distance between a virtual object included in the virtual space and a location of the virtual space corresponding to the wearable device.

In one embodiment, a method of a wearable device may include an operation of acquiring information related to at least one real light source using an image acquired from a camera of the wearable device. The method may include an operation of receiving an input for displaying a virtual space on a display of the wearable device while displaying at least a portion of the image acquired from the camera on the display of the wearable device. The method may include an operation of determining a location of at least one virtual light source in the virtual space using the information based on the input. The method may include an operation of displaying a virtual object to which a visual effect for the at least one virtual light source is applied on the display based on a distance between a virtual object included in the virtual space and a location of the virtual space corresponding to the wearable device.

According to one embodiment, a wearable device may include a camera, a display, and a processor. The processor may be configured to obtain information related to a real light source disposed in an external space including the wearable device using the camera. The processor may be configured to identify a first area of a virtual space linked with a virtual light source corresponding to the real light source based on the information. The processor may be configured to perform rendering for a first virtual object included in the first area among a plurality of virtual objects included in the virtual space, while the display displays a virtual space distinct from the external space, based on a visual effect related to the virtual light source. The processor may be configured to perform rendering for a second virtual object included in a second area different from the first area among the plurality of virtual objects, independently of the visual effect, on the display.

In one embodiment, a method of a wearable device may include an operation of acquiring information related to a real light source disposed in an external space including the wearable device by using a camera of the wearable device. The method may include an operation of identifying a first area of a virtual space linked with a virtual light source corresponding to the real light source based on the information. The method may include an operation of performing rendering for a first virtual object included in the first area among a plurality of virtual objects included in the virtual space, while displaying a virtual space distinct from the external space on a display of the wearable device, based on a visual effect related to the virtual light source. The method may include an operation of performing rendering for a second virtual object included in a second area different from the first area among the plurality of virtual objects, on the display, independently of the visual effect.

FIGS. 1A and 1B illustrate examples of screens displayed by a wearable device according to one embodiment.
FIG. 2 illustrates an example block diagram of a wearable device, according to one embodiment.
FIGS. 3A and 3B illustrate an example of a flow diagram of a wearable device according to one embodiment.
Figure 4 illustrates an example of the operation of a wearable device that determines the location of an actual light source based on multiple images.
FIG. 5 illustrates an example of a flow diagram of a wearable device according to one embodiment.
FIG. 6 illustrates an example of a flow diagram of a wearable device according to one embodiment.
FIG. 7 illustrates an example of a flow diagram of a wearable device according to one embodiment.
Figure 8 illustrates an example of the operation of a wearable device that identifies an actual light source in external space.
FIG. 9 illustrates an example of a flow diagram of a wearable device according to one embodiment.
Figures 10a and 10b illustrate an example of the operation of a wearable device that performs rendering for a virtual space based on information related to actual light sources in external space.
FIGS. 11A and 11B illustrate an example of the operation of a wearable device that performs rendering for a virtual object included in a virtual space based on information related to an actual light source in an external space.
FIG. 12 illustrates an example of a flow diagram of a wearable device according to one embodiment.
Figure 13 illustrates an example of the operation of a wearable device that performs rendering of a virtual space based on the movement of an actual light source.
Figure 14 illustrates an example of the operation of a wearable device that performs rendering for a virtual space based on the brightness distribution of an external space.
FIG. 15 illustrates an example of a flow diagram of a wearable device according to one embodiment.
FIG. 16A illustrates an example of a perspective view of a wearable device, according to one embodiment.
FIG. 16b illustrates an example of one or more hardware elements positioned within a wearable device, according to one embodiment.
FIGS. 17A and 17B illustrate examples of the appearance of a wearable device according to one embodiment.
Figure 18 is an example diagram of a network environment related to a metaverse service.

Below, various embodiments of this document are described with reference to the attached drawings.

The various embodiments of this document and the terminology used herein are not intended to limit the technology described in this document to the specific embodiments, but should be understood to encompass various modifications, equivalents, and/or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and/or B", "A, B, or C" or "at least one of A, B and/or C" can include all possible combinations of the items listed together. Expressions such as "first", "second", "first" or "second" can modify the corresponding components, regardless of order or importance, and are only used to distinguish one component from another and do not limit the corresponding components. When it is said that a certain (e.g., a first) component is "(functionally or communicatively) connected" or "connected" to another (e.g., a second) component, said certain component may be directly connected to said other component, or may be connected through another component (e.g., a third component).

The term "module" as used in this document includes a unit composed of hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally composed component or a minimum unit or part thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

FIGS. 1A to 1B illustrate an example of a screen displayed by a wearable device (101) according to an embodiment. In an embodiment, the wearable device (101) may include a head-mounted display (HMD) wearable on a head of a user (110). Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of one or more hardwares included in the wearable device (101) is exemplarily described with reference to FIG. 2. An example of a structure of the wearable device (101) wearable on the head of a user (110) is described with reference to FIGS. 16A to 16B and/or FIGS. 17A to 17B. The wearable device (101) may be referred to as an electronic device. For example, an electronic device may be combined with an accessory to attach to a user's head to form an HMD.

In one embodiment, a wearable device (101) can perform functions related to video see-through (VST) and/or virtual reality (VR). In a state where a user (110) wears the wearable device (101), in one embodiment, the wearable device (101) can include a housing that covers an eye of the user (110). In the state, the wearable device (101) can include a display disposed on a first side of the housing facing the eye. The wearable device (101) can include a camera disposed on a second side opposite to the first side. Using the camera, the wearable device (101) can acquire images including ambient light. The wearable device (101) can sequentially output the images within the display arranged on the first surface, so that the user (110) can recognize the ambient light through the display. The display area of the display arranged on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object within frames output through the display, so that the user (110) can recognize the virtual object together with a real object recognized by the ambient light.

In one embodiment, the wearable device (101) may perform functions related to augmented reality (AR) and/or mixed reality (MR). As shown in FIGS. 1A and 1B , when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to an eye of the user (110). The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of the display may be formed within the lens through which the ambient light passes. Since the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display.

Referring to FIG. 1A, screens (131, 132) displayed by the wearable device (101) in different modes are illustrated. Referring to FIG. 1A, the wearable device (101) can display the screen (131) in a first designated mode referred to as the VST mode. In the VST mode, the wearable device (101) can display the screen (131) including images and/or videos acquired through a camera. In the VST mode, a user (110) wearing the wearable device (101) can recognize an external space including the wearable device (101) through the screen (131), independently of the housing covering both eyes of the user (110). In VST mode, the wearable device (101) can display one or more virtual objects (e.g., virtual objects (180, 181)) on the screen (131) together with images and/or videos of external space. For example, the wearable device (101) can display a virtual object (181) having a form of an icon representing an application on a virtual object (180) having a form of a panel. In one embodiment, the virtual object (180) can be referred to as an application tray (or app tray). In response to an input for selecting the virtual object (181), the wearable device (101) can execute the application and display a screen (132) provided from the application.

Referring to FIG. 1A, the wearable device (101) can display a screen (132) in a second designated mode referred to as a VR mode. In the VR mode, the wearable device (101) can display a screen (132) representing at least a portion of a virtual space (140). In the VR mode, the wearable device (101) according to one embodiment can display the screen (132) based on a field-of-view (FoV) formed in the virtual space (140). Referring to FIG. 1A, the wearable device (101) can display virtual objects (161, 162) included in the FoV of the virtual space (140) on the screen (132). On the screen (132), the wearable device (101) can display virtual objects (161, 162) having a sense of perspective by utilizing binocular disparity. The virtual objects (161, 162) included in the screen (132) may include graphical objects, windows (e.g., activities), and/or widgets (or gadgets) provided from a program (e.g., software application) executed by the wearable device (101). Referring to FIG. 1A, the wearable device (101) is exemplarily illustrated as a virtual object (161) having a three-dimensional shape and a virtual object (162) having a window shape, but the embodiment is not limited thereto.

In one embodiment, the wearable device (101) can recognize at least one real light source (e.g., a floor lamp (120)) in an external space. The wearable device (101) recognizing the real light source can include an operation of identifying at least one of a location of the real light source in the external space, a color or brightness of light radiating from the real light source. Referring to FIG. 1A, a wearable device (101) adjacent to a floor lamp (120), which is an example of a real light source, is exemplarily illustrated. The wearable device (101) operating in a VST mode can identify the floor lamp (120) using a camera facing a front direction of a user (110) wearing the wearable device (101). In one embodiment, the wearable device (101) can obtain information related to at least one real light source using an image obtained from the camera. According to one embodiment, an operation of a wearable device (101) recognizing an actual light source is described with reference to FIGS. 4 to 9.

According to one embodiment, the wearable device (101) may identify an input for displaying a virtual space on the display while displaying at least a portion of an image acquired from a camera on a display, such as a screen (131). The input may include an input for selecting a virtual object (181) having a form of an icon representing an application for providing the virtual space (140). The embodiment is not limited thereto, and the input may include at least one of a remote controller connected to the wearable device (101), a button included in the wearable device (101), or a voice command of a user (110) (e.g., an utterance triggering the display of the virtual space (140).

According to one embodiment, the wearable device (101) may determine or identify a location of at least one virtual light source (150) in the virtual space (140) based on the input and using the information. In the exemplary case of FIG. 1A in which the wearable device (101) identifies a floor lamp (120), the wearable device (101) may determine a location (P1) of a virtual light source (150) corresponding to the floor lamp (120), which is an actual light source, in the virtual space (140) based on a positional relationship between the floor lamp (120) and the wearable device (101). The positional relationship may include at least one of a distance between the floor lamp (120) and the wearable device (101), or an azimuth angle of the floor lamp (120) with respect to the wearable device (101). In one embodiment, based on the input for switching from the first designated mode for VST to the second designated mode for VR associated with the virtual space (140), the wearable device (101) can determine the location in the virtual space (140) of a virtual light source (150) corresponding to the actual light source, the floor lamp (120).

According to one embodiment, the wearable device (101) may perform rendering for a virtual space (140) including a virtual light source (150) corresponding to an actual light source (e.g., a floor lamp (120)). The embodiment is not limited thereto, and the wearable device (101) may adjust the illuminance in the virtual space (140) based on the actual light source. For example, independently of the virtual light source (150) corresponding to the actual light source, the electronic device (101) may perform rendering for a virtual space (140) having illuminance and/or color related to the actual light source. Hereinafter, the rendering may include one or more functions for displaying an image and/or video (e.g., a screen (132)) representing the virtual space (140). When the position (P2) of the virtual space (140) corresponds to the wearable device (101), the positional relationship between the wearable device (101) and the floor lamp (120) may match the positional relationship between the position (P2) of the virtual space (140) and the position (P1) of the virtual light source (150). For example, the position of an actual light source (e.g., a floor lamp (120)) seen through the screen (131) of the VST mode may be mapped to the position of a virtual light source (150) rendered on the screen (132) of the VR mode. According to one embodiment, the wearable device (101) may perform rendering related to light (hereinafter, virtual light) emitted from the virtual light source (150) based on the color, brightness, and/or intensity (or strength) of light (hereinafter, actual light) emitted from the actual light source (e.g., the floor lamp (120)). The operation of a wearable device (101) performing rendering based on at least one virtual light source corresponding to at least one real light source according to one embodiment is described with reference to FIGS. 10A to 10B, FIGS. 11A to 11B, and FIGS. 12 to 15.

According to one embodiment, the wearable device (101) may apply a visual effect to at least one virtual light source (150) to a virtual object included in a virtual space (140) based on a distance between the virtual object (e.g., virtual objects (161, 162)) and a location (P2) of the virtual space (140) corresponding to the wearable device (101). The wearable device (101) displaying the virtual object may include an operation of performing rendering on the virtual object and/or the virtual space (140) including the virtual object. The wearable device (101) applying the visual effect to the virtual light source (150) may include at least one of an operation of at least partially changing a color and/or brightness of the virtual object based on virtual light radiated from the virtual light source (150), or an operation of displaying a shadow extended from the virtual object. Referring to FIG. 1A, the wearable device (101) may display a visual object (170) that represents a shadow extending from a virtual object (161) along a direction (D1) from a virtual light source (150) to the virtual object (161). According to one embodiment, the wearable device (101) may perform rendering for a virtual object (161) that is spaced apart from a location (P2) by a distance exceeding a specified threshold, using a visual effect for the virtual light source (150).

According to one embodiment, the wearable device (101) may apply a visual effect related to the virtual light source (150) to the virtual object by using at least one of the position (P1) of the virtual light source (150) in the virtual space (140), the position, category of the virtual object, or the position (P2) corresponding to the wearable device (101) in the virtual space (140). For example, the wearable device (101) may identify and/or determine whether to apply the visual effect to the virtual object based on whether the virtual object is included in a designated category for interaction with a user (110) wearing the wearable device (101). The designated category may include a panel having a form of a two-dimensional plane referred to as a window. The embodiments are not limited thereto, and the above-mentioned categories may include virtual objects deployable within the screen (132) and/or virtual space (140) for transmitting and/or interacting with information, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, time pickers, progress bars, and/or tables. Referring to FIG. 1A, while displaying a virtual object (162) as a window for interacting with a user (110), the wearable device (101) may stop applying visual effects associated with the virtual light source (150) to the virtual object (162). For example, the wearable device (101) may perform rendering for the virtual object (162) independently from the visual effects associated with the virtual light source (150).

As described above, according to one embodiment, the wearable device (101) may perform rendering for a virtual space (140) for VR based on information about an environment including the wearable device (101) (e.g., an external space including the wearable device (101)). For example, the wearable device (101) may place a virtual light source (150) corresponding to an actual light source included in the external space (e.g., a floor lamp (120)) in the virtual space (140). Based on the virtual light source (150) placed in the virtual space (140), the wearable device (101) may perform rendering for one or more virtual objects (161, 162) included in the virtual space (140). For example, a wearable device (101) displaying a screen (131) including a floor lamp (120), which is a real light source, based on a VST mode can display a screen (132) rendered based on a virtual light source (150) corresponding to the floor lamp (120) after switching from the VST mode to a VR mode. By using the virtual light source (150) corresponding to the floor lamp (120), the wearable device (101) can provide a continuous user experience while switching from the VST mode to the VR mode.

Referring to FIG. 1b, screens (131, 133, 134) displayed by the wearable device (101) in the VST mode and/or the VR mode are exemplarily illustrated. Referring to FIG. 1b, in a state of displaying the screen (131) based on the VST mode, the wearable device (101) may display a virtual object (180) including options for changing the mode. Within the virtual object (180), the wearable device (101) may display an icon (e.g., virtual objects (181, 182)) representing an application running in a mode (e.g., VR mode) different from the VST mode.

Referring to FIG. 1B, in response to an input indicating display of a screen (133) based on a VR mode, the wearable device (101) may display a screen (133) provided from the application. For example, based on identifying an input indicating selection of a virtual object (181), the wearable device (101) may display the screen (133). The screen (133) displayed by the wearable device (101) may include a designated screen (e.g., a home screen) based on the VR mode. The designated screen may include an object (e.g., an icon) for executing at least one of one or more functions supported by the wearable device (101). In a state of displaying the screen (133), the wearable device (101) may place a virtual light source (150) corresponding to an actual light source, such as a floor lamp (120), in the virtual space. Based on the virtual light source (150) placed in the above virtual space, the wearable device (101) can display a screen (133) to which a visual effect related to the virtual light source (150) is applied.

Referring to FIG. 1b, in response to an input indicating execution of an application for VR mode, the wearable device (101) may display a screen (133) provided from the application. For example, based on identifying an input indicating selection of a virtual object (182), the wearable device (101) may display the screen (133). The virtual object (182) may include an icon of an application for providing immersive virtual reality. The wearable device (101) may execute an application corresponding to the virtual object (182) to display a screen (134) based on immersive virtual reality. While displaying the screen (134) including a virtual object (185) for playing a video, the wearable device (101) may at least temporarily suspend rendering based on a virtual light source corresponding to an actual light source. For example, to improve user experience based on immersive virtual reality, the wearable device (101) may limit rendering based on the virtual light source.

Hereinafter, with reference to FIG. 2, an example of one or more hardware and/or software included in the wearable device (101) of FIGS. 1a and 1b is described.

FIG. 2 illustrates an example of a block diagram of a wearable device (101) according to one embodiment. The wearable device (101) of FIG. 2 may include the wearable device (101) of FIGS. 1A to 1B.

According to one embodiment, the wearable device (101) may include at least one of a processor (210), a memory (215), a display (220), a camera (225), a sensor (230), or a communication circuit (240). The processor (210), the memory (215), the display (220), the camera (225), the sensor (230), and the communication circuit (240) may be electronically and/or operably coupled with each other by an electronic component such as a communication bus (202). Hereinafter, operably coupled hardware may mean that a direct connection or an indirect connection is established between the hardwares, either wired or wireless, such that a second hardware is controlled by a first hardware among the hardwares. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least a portion of the processor (210), the memory (215), and the communication circuit (240)) may be included in a single integrated circuit, such as a system on a chip (SoC). The type and/or number of hardware included in the wearable device (101) is not limited to that illustrated in FIG. 2. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 2.

In one embodiment, the processor (210) of the wearable device (101) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and/or an application processor (AP). The processor (210) may have a structure of a single-core processor, or a structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

In one embodiment, the memory (215) of the wearable device (101) may include hardware components for storing data and/or instructions input and/or output to the processor (210) of the wearable device (101). The memory (215) may include, for example, volatile memory, such as random-access memory (RAM), and/or non-volatile memory, such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), Cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC).

In one embodiment, a display (220) of a wearable device (101) can output visualized information (e.g., the screens of FIGS. 1A-1B , 10A-10B-11A-11B , and/or 13-14 ) to a user (e.g., the user (110) of FIGS. 1A-1B ). For example, the display (220) can be controlled by a processor (210) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (220) can include a flat panel display (FPD) and/or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and/or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The display (220) of FIG. 2 may include at least one display (1650, 1750), which will be described later with reference to FIGS. 16a to 16b and/or FIGS. 17a to 17b.

In one embodiment, the camera (225) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and/or brightness of light. The plurality of optical sensors included in the camera (225) may be arranged in the form of a two-dimensional array. The camera (225) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (225) may mean one (a) two-dimensional frame data acquired from the camera (225). For example, video data captured using the camera (225) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (225) according to a frame rate. The camera (225) may further include a flash light positioned toward the direction in which the camera (225) receives light and outputs light toward the direction.

According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information that may be processed by a processor (210) and/or a memory (215) of the wearable device (101) from non-electronic information related to the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and/or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (101). In one embodiment, the IMU may include at least one of an acceleration sensor, a gyro sensor, and a gravity sensor. Using the above IMU, the processor (210) of the wearable device (101) can identify the motion of the wearable device (101) based on six degrees of freedom (DoF). The motion of the wearable device (101) based on six degrees of freedom can include translation and rotation (e.g., roll, pitch, and yaw) of the wearable device (101) along three axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other.

In one embodiment, the communication circuit (240) of the wearable device (101) may include hardware components for supporting transmission and/or reception of electrical signals between the wearable device (101) and an external electronic device (e.g., a remote controller connected to the wearable device (101)). The communication circuit (240) may include, for example, at least one of a modem (MODEM), an antenna, and an optic/electronic (O/E) converter. The communication circuit (240) may support transmission and/or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and/or 6G.

Although not shown, according to one embodiment, the wearable device (101) may include output means for outputting information in a form other than a visualized form. For example, the wearable device (101) may include a speaker for outputting an acoustic signal. For example, the wearable device (101) may include a motor for providing haptic feedback based on vibration.

Referring to FIG. 2, according to one embodiment, one or more instructions (or commands) representing operations and/or actions to be performed on data by a processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and/or an application. Hereinafter, when an application is installed in an electronic device (e.g., the wearable device (101)), it may mean that one or more instructions provided in the form of an application are stored in the memory (215), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension specified by the operating system of the wearable device (101)). According to one embodiment, the wearable device (101) may execute one or more instructions stored in the memory (215) to perform the operations of FIGS. 3A to 3B, FIGS. 5 to 7, FIG. 9, FIG. 12, and/or FIG. 15.

Referring to FIG. 2, programs installed in the wearable device (101) may be classified into one of different layers, including an application layer (260), a framework layer (270), and/or a hardware abstraction layer (HAL) (250), based on a target. For example, programs (e.g., drivers) designed to target the hardware (e.g., display (220), camera (225), sensor (230), and/or communication circuit (240)) of the wearable device (101) may be classified within the hardware abstraction layer (250). For example, within the framework layer (270), programs designed to target at least one of the hardware abstraction layer (250) and/or the application layer (260) (e.g., an eye tracker (271), a gesture tracker (272), a motion tracker (273), a real light source identifier (274), a virtual light source generator (275), and/or a virtual space manager (276)) may be classified. The programs classified into the framework layer (270) may provide an executable API (application programming interface) based on other programs.

Referring to FIG. 2, a program designed to target a user (e.g., a user (110) of FIGS. 1A to 1B) controlling a wearable device (101) may be classified within the application layer (260). For example, a program classified within the application layer (260) may include at least one of an application (261) for playing and/or streaming a video, an application (262) for a video conference, an application (263) for viewing media content (e.g., images and/or videos) of a memory (215), or an application (264) for a call connection. The embodiment is not limited thereto. For example, a program classified within the application layer (260) may call an API to cause execution of a function supported by programs classified within the framework layer (270).

Referring to FIG. 2, according to one embodiment, the wearable device (101) may process information related to the gaze of a user wearing the wearable device (101) based on the execution of the gaze tracker (271) in the framework layer (270). For example, the wearable device (101), when worn by the user, may obtain an image including the user's eye from a first camera positioned toward the user's eye. Based on the position and/or direction of the pupil included in the image, the wearable device (101) may identify the direction of the user's gaze.

Referring to FIG. 2, according to one embodiment, the wearable device (101) can identify a motion of a designated body part including a hand based on the execution of a gesture tracker (272) within the framework layer (270). For example, the wearable device (101) can obtain an image and/or video including the body part from a second camera. Based on the motion and/or posture of the designated body part represented by the image and/or video, the wearable device (101) can identify a gesture performed by the designated body part.

Referring to FIG. 2, according to one embodiment, the wearable device (101) can identify the motion of the wearable device (101) based on the execution of the motion tracker (273) within the framework layer (270). Within a state where the wearable device (101) is worn by a user, the motion of the wearable device (101) can be related to the motion of the user's head. For example, the wearable device (101) can identify the direction of the wearable device (101) that substantially coincides with the direction of the head. The wearable device (101) can identify the motion of the wearable device (101) based on sensor data of a sensor (230) including an IMU.

Referring to FIG. 2, according to one embodiment, the wearable device (101) may obtain information about an external space in which the wearable device (101) is included or adjacent to the wearable device (101) based on the execution of the actual light source identifier (274) in the framework layer (270). For example, the wearable device (101) may obtain information related to an actual light source (e.g., a floor lamp (120) of FIGS. 1A and 1B) disposed in an external space including the wearable device (101) by using the camera (225). The embodiment is not limited thereto, and the wearable device (101) may obtain the information by using a sensor (230) (e.g., a light sensor). The information obtained based on the execution of the real light source identifier (274) may include a location (e.g., a three-dimensional location of the real light source in external space), brightness, and/or color of at least one real light source. In one embodiment, the processor (210) of the wearable device (101) may store the information obtained based on the execution of the real light source identifier (274) in the memory (215). In one embodiment, the wearable device (101) may monitor at least one real light source included in external space based on the execution of the real light source identifier (274).

Referring to FIG. 2, according to one embodiment, the wearable device (101) may place at least one virtual light source in a virtual space (e.g., the virtual space (140) of FIG. 1A) based on the information acquired by the real light source identifier (274) based on the execution of the virtual light source generator (275) in the framework layer (270). The embodiment is not limited thereto, and the wearable device (101) may indirectly place at least one virtual light source based on the brightness distribution in the virtual space. In a state where the virtual light source generator (275) is executed, the wearable device (101) may place a virtual light source having a color and brightness of an real light source represented by the information in the virtual space based on the information acquired by the real light source identifier (274).

Referring to FIG. 2, according to one embodiment, the wearable device (101) may perform rendering for a virtual space based on the execution of the virtual space manager (276) in the framework layer (270). With the virtual space manager (276) executed, the wearable device (101) may perform rendering for a virtual space including at least one virtual light source arranged based on the virtual light source generator (275). In one embodiment, the wearable device (101) may identify a virtual space mapped to an external space based on the virtual space manager (276). The wearable device (101) may determine a reference position (e.g., position (P2) of FIG. 1A) of the virtual space for forming a FoV (e.g., FoV of FIGS. 1A to 1B) based on the position and/or direction of the wearable device (101) in the external space identified based on the data of the sensor (230). The above reference position may correspond to the position of the wearable device (101) in the virtual space. In one embodiment, the wearable device (101) may perform SLAM (simultaneous localization and mapping) to recognize external space and the position of the wearable device (101) within the external space.

As described above, according to one embodiment, the wearable device (101) can identify at least one real light source in an external space including the wearable device (101) by using the real light source identifier (274). The wearable device (101) can obtain information about the at least one real light source based on the real light source identifier (274) in a VST mode (or an AR mode). According to one embodiment, the wearable device (101) can place at least one virtual light source in a virtual space based on the information, and perform rendering on the virtual space by using the at least one virtual light source, based on switching from the VST mode to the VR mode. According to one embodiment, the wearable device (101) can selectively apply a visual effect based on the at least one virtual light source to a plurality of virtual objects included in the virtual space. For example, the wearable device (101) may restrict application of the visual effect depending on the position of the virtual object included in the FoV when the virtual light source is included in the FoV of the virtual space. For example, in order to prevent backlighting of the virtual object by the virtual light source included in the FoV and/or a shadow caused by the virtual light source on the virtual object, the wearable device (101) may restrict application of the visual effect. The wearable device (101) may determine whether to apply the visual effect to the virtual object, depending on the area in the virtual space linked with the virtual light source and/or the category of the virtual object to which the visual effect is to be applied.

Hereinafter, the operation of the wearable device (101) of FIGS. 1A to 1B and/or FIG. 2 is described with reference to FIGS. 3A to 3B.

FIGS. 3A and 3B illustrate an example of a flow diagram of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1A and 1B and/or 2 may include the wearable device of FIGS. 3A and 3B. At least one of the operations of FIGS. 3A and 3B may be performed by the wearable device (101) of FIG. 2 and/or the processor (210).

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 3A, in operation (310), the wearable device according to one embodiment may identify at least one real light source included in an external space. According to one embodiment, the wearable device may identify at least one real light source based on the execution of the real light source identifier (274) of FIG. 2. The wearable device may combine multiple images acquired from a camera (e.g., the camera (225) of FIG. 2) to identify a real light source commonly captured in the multiple images. An operation of the wearable device determining or identifying a location of a real light source included in all of the multiple images is described with reference to FIG. 4.

Referring to FIG. 3A, in operation (320), according to one embodiment, a wearable device may store information about at least one real light source. The wearable device may store the information in a memory (e.g., memory (215) of FIG. 2). The information stored by the wearable device may include at least one of a location of the real light source identified by the wearable device in an external space, a color, illuminance, brightness, or intensity of the real light source. The information stored in the wearable device based on operation (320) may include one or more parameters for reconstructing a virtual light source corresponding to the real light source in a virtual space. The information of operation (320) may be referred to as characteristic information about at least one real light source.

Referring to FIG. 3A, in operation (330), according to one embodiment, a wearable device may identify an event for displaying a virtual space for VR (virtual reality). The event may be generated by an input for executing an application (e.g., an input indicating selection of a virtual object (181) of FIGS. 1A and 1B). The event may be generated by a designated gesture (e.g., a gesture of tapping the wearable device a designated number of times) identified by the eye tracker (271), the gesture tracker (272), and/or the motion tracker (273) of FIG. 2. The event may be generated based on the identification of a voice command including a designated utterance (e.g., "Switch to VR mode"). The embodiment is not limited thereto.

Referring to FIG. 3A, in operation (340), according to one embodiment, the wearable device may generate at least one virtual light source in a virtual space based on the stored information, and perform rendering on at least a portion of the virtual space based on the generated at least one virtual light source. In response to identifying the event of operation (330), the wearable device may perform operation (340). According to one embodiment, the wearable device may generate a virtual space including at least one virtual light source corresponding to at least one real light source based on the stored information based on operation (320). The wearable device may perform operation (340) based on execution of the virtual space manager (276) of FIG. 2. According to one embodiment, the wearable device may determine a location of at least one virtual light source in the virtual space based on a location of the at least one real light source in an external space indicated by the information. For example, the at least one virtual light source may have fixed coordinates in the virtual space based on world-lock.

According to one embodiment, the wearable device can perform rendering of the virtual space based on at least one virtual light source arranged in the virtual space based on the information. Based on the rendering, the wearable device can obtain a screen (e.g., screen (132) of FIG. 1A) representing virtual light emitted from the at least one virtual light source. The wearable device can display the obtained screen on a display (e.g., display (220) of FIG. 2) to provide a VR including a virtual light source corresponding to an actual light source to a user wearing the wearable device. The wearable device can obtain the screen representing the virtual light based on an algorithm for tracing a path of the virtual light, such as ray tracing.

In one embodiment, the wearable device can adjust the illumination of the virtual space using information about at least one real light source stored based on the operation (320). For example, the wearable device can obtain a virtual space having an illumination, color, and/or brightness associated with at least one real light source without generating a virtual light source (e.g., at least one virtual light source of operation (340)). In one embodiment that generates a virtual space that does not include a virtual light source corresponding to a real light source, the wearable device can obtain the virtual space based on the appearance of an external space associated with the real light source using the information of operation (320). Based on the virtual space, the wearable device can provide a virtual reality based on the appearance of the external space to the user independently of the virtual light source. For example, the wearable device can perform rendering on the virtual space using the information of operation (320) to obtain the virtual space based on the location, color, illumination, and/or brightness of the real light source. In one embodiment, the wearable device's acquisition of the virtual space may be performed independently of at least one virtual light source of operation (340).

As described above, according to one embodiment, the wearable device can simulate a real light source in a virtual space by using a virtual light source corresponding to a real light source. Using the virtual light source of the virtual space, the wearable device can display at least one virtual object included in the virtual space based on the real light of the real light source. According to one embodiment, the wearable device can make a user wearing the wearable device recognize a virtual space having a similar color tone to an external space including a real light source by using a virtual space in which a real light source is simulated.

Referring to FIG. 3B, in operation (350), the wearable device according to one embodiment may display a first screen based on the VST. The wearable device may display the first screen including an image and/or video acquired through a camera of the wearable device (e.g., the camera (225) of FIG. 2), such as the screen (131) of FIGS. 1A and 1B.

Referring to FIG. 3B , in operation (352), the wearable device according to one embodiment may determine positions of one or more virtual light sources in the virtual space using one or more real light sources included in an external space including the wearable device. The wearable device may perform operation (352) of FIG. 3B based on operations (310, 320, 330) of FIG. 3A . For example, the wearable device may generate and/or place one or more virtual light sources in the virtual space based on the position, color, illuminance, and/or brightness of the one or more real light sources.

Referring to FIG. 3B, at operation (354), according to one embodiment, the wearable device may receive an input for switching from VST to VR. The input may include an input indicating selection of an application designed based on the VR mode. The input may include an input indicating display of a designated screen (e.g., a home screen) based on the VR mode. The input may be identified by motions of the user's head and/or hands tracked by the wearable device. The input may be identified by an external electronic device (e.g., a remote controller) connected to the wearable device. The input may be identified by buttons and/or sensors (e.g., touch sensors) of the wearable device.

Referring to FIG. 3b, in operation (356), the wearable device according to one embodiment may identify whether the input indicates a transition to a designated screen related to VR (e.g., a home screen based on a VR mode). The designated screen of operation (356) may include screen (133) of FIG. 1b. Based on operation (354), if an input indicating a transition to the designated screen is received (356-Yes), the wearable device may perform operation (358). If another input independent of the designated screen is received (356-No), the wearable device may perform operation (360).

Referring to FIG. 3b, in operation (358), the wearable device according to one embodiment may display a designated screen based on a virtual space including one or more virtual light sources. The wearable device may apply a visual effect based on the one or more virtual light sources to one or more virtual objects included in the virtual space. The designated screen of operation (358) may include the screen (133) of FIG. 1b.

Referring to FIG. 3B, in operation (360), according to an embodiment, the wearable device may display a second screen related to VR based on an application identified or selected by an input and execution of an application related to VR. The second screen may be provided from an application for providing immersive VR. The second screen of operation (360) may include the screen (134) of FIG. 1B. In a state of displaying the second screen of operation (360), the wearable device may limit or stop applying a visual effect based on one or more virtual light sources. The embodiment is not limited thereto, and the wearable device may receive an input indicating to apply the visual effect while executing the application of operation (360). In response to the input, the wearable device may display a virtual space and/or a second screen including at least one virtual object to which a visual effect based on the one or more virtual light sources is applied. In order to receive the input for applying the above visual effect, the wearable device may display a visual object in the form of a pop-up window in response to the input of the action (354). For example, in the visual object, the wearable device may display a specified text such as "Render the virtual space based on the actual light source?"

Below, with reference to FIG. 4, an exemplary operation of a wearable device for obtaining information about an actual light source based on multiple images is described.

FIG. 4 illustrates an example of an operation of a wearable device (101) that determines the location of an actual light source based on a plurality of images. The wearable devices of FIGS. 1A to 1B and FIGS. 2 to 3A to 3B may include the wearable device (101) of FIG. 4. The operation of the wearable device (101) described with reference to FIG. 4 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210). The operation of the wearable device (101) described with reference to FIG. 4 may be related to at least one of the operations of FIGS. 3A to 3B (e.g., operation (310) of FIG. 3A).

Referring to FIG. 4, according to one embodiment, the wearable device (101) can determine or identify a position of the wearable device (101) in external space based on a spatial coordinate system formed by three axes (e.g., the x-axis, the y-axis, and the z-axis of FIG. 4) that are perpendicular to each other. The position of the wearable device (101) can correspond to an origin (O) of the spatial coordinate system. A specific point of the spatial coordinate system can be matched to a specific point of the external space that includes the wearable device (101). According to one embodiment, the wearable device (101) can identify a position and/or motion of the wearable device (101) in the spatial coordinate system using a sensor (230) of FIG. 2 (e.g., an IMU).

Referring to FIG. 4, according to one embodiment, the wearable device (101) can obtain images (410, 420) for different directions (D1, D2) of the wearable device (101) using a camera (e.g., camera (225) of FIG. 2). The camera can be placed on one side of the wearable device (101) opposite to the other side on which the display (e.g., display (220) of FIG. 2) is placed. The camera can have the other direction opposite to the one direction of the display. The wearable device (101) can identify the direction of the wearable device (101) using data of a sensor such as an IMU (e.g., sensor (230) of FIG. 2). Based on data indicating the direction of the wearable device (101) acquired from the sensor, the wearable device (101) can acquire an image (410) corresponding to the direction (D1) and an image (420) corresponding to the direction (D2). For example, in a state where the image (410) corresponding to the direction (D1) has been acquired, in response to identifying a motion of the wearable device (101) rotating toward the direction (D2) based on the sensor, the wearable device (101) can acquire an image (420) corresponding to the direction (D2) from the camera.

Referring to FIG. 4, according to one embodiment, the wearable device (101) can perform object recognition for images (410, 420) for different directions (D1, D2) using a spatial coordinate system. For example, it is assumed that a floor lamp (e.g., floor lamp (120) of FIGS. 1A to 1B) is captured in all of the images (410, 420). The wearable device (101) can position the images (410, 420) based on the directions (D1, D2) in the spatial coordinate system. The wearable device (101) can identify the intersection (PL) of lines extended from points (A, B) based on identifying the floor lamp (120) captured at point (A) of the image (410) and point (B) of the image (420). For example, the wearable device (101) can identify whether an external object (e.g., an actual light source), such as a floor lamp (120), is commonly captured in the images (410, 420) by comparing the color distributions of the images (410, 420). The wearable device (101) can determine the coordinates of the intersection point (PL) in the spatial coordinate system as the coordinates of the floor lamp (120) commonly captured in the images (410, 420). Since the spatial coordinate system matches the external space, the wearable device (101) can determine the location of the floor lamp (120) in the external space based on the coordinates of the intersection point (PL).

As described above, according to one embodiment, the wearable device (101) can identify the location of the actual light source in the external space based on the locations of the actual light sources identified in each of the plurality of images (410, 420). The wearable device (101) can determine the location of the virtual light source in the virtual space mapped to the external space based on the location of the actual light source. For example, the wearable device (101) can place the virtual light source related to the floor lamp (120) at a point in the virtual space corresponding to the intersection point (PL) in the space coordinate system of FIG. 4. For example, the three-dimensional xyz coordinates of the intersection point (PL) can be used to obtain the three-dimensional xyz coordinates of the virtual light source placed in the virtual space by the wearable device (101). The wearable device (101) can provide a user experience (or mood) related to the reflection of light in the virtual space by using the virtual light source placed in the virtual space.

Below, with reference to FIGS. 5 to 7, an operation of a wearable device (101) identifying an actual light source according to one embodiment is described.

FIG. 5 illustrates an example of a flow diagram of a wearable device according to one embodiment. The wearable devices of FIGS. 1A to 1B and FIGS. 2 to 3A to 3B may include the wearable device of FIG. 5. The operations of the wearable device described with reference to FIG. 5 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210). At least one of the operations of FIG. 5 may be related to the operations of the wearable device (101) of FIG. 4.

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 5, in operation (510), according to one embodiment, the wearable device can identify an actual light source from a plurality of images (e.g., images (410, 420) of FIG. 4) acquired from a camera (e.g., camera (225) of FIG. 2). For example, the wearable device can identify an actual light source commonly included in all of the plurality of images. The wearable device can perform operation (510) by comparing one or more feature points included in the plurality of images or by using an artificial neural network. The artificial neural network used to identify the actual light source of operation (510) can be formed based on a structure for analyzing images and/or videos, such as a convolutional neural network (CNN).

Referring to FIG. 5, in operation (520), according to one embodiment, the wearable device can identify a location of an actual light source in external space based on the locations of the actual light source in the plurality of images. As in points (A, B) of FIG. 4, the wearable device can determine or identify, in each of the plurality of images, the locations of the actual light source included in all of the plurality of images. Using the locations of the plurality of images and the directions of the plurality of images (e.g., directions (D1, D2) of FIG. 4), the wearable device can identify the location of the actual light source in three dimensions. In one embodiment, the wearable device can obtain information about the brightness and/or color of the actual light source based on operation (320) of FIG. 3a, together with the location of the actual light source in operation (520).

Referring to FIG. 5, in operation (530), according to one embodiment, the wearable device can identify an event for displaying a virtual space for VR (virtual reality). In one embodiment, the wearable device can identify the event of operation (530) similar to operation (330) of FIG. 3A. In one embodiment, the wearable device can identify the event of operation (530) while performing operations (510, 520) based on the VST mode.

Referring to FIG. 5, in operation (540), according to an embodiment, the wearable device may place a virtual light source in a virtual space using the location of the actual light source identified based on operation (520), and perform rendering for at least a portion of the virtual space based on the placed virtual light source. The wearable device may perform operation (540) in response to an event of operation (530). The wearable device may place a virtual light source corresponding to the actual light source at a location in the virtual space mapped to the location of operation (520). The wearable device may perform rendering for at least a portion of the virtual space based on the virtual light source. For example, the wearable device may perform rendering for the virtual space based on a visual effect based on the virtual light source.

As described above, according to one embodiment, the wearable device can perform rendering of a virtual space based on the result of identifying the location, color, brightness, and/or intensity of a real light source (or real light) existing in an external space. Since the virtual space is rendered based on the result, the wearable device can provide a user experience similar to viewing the real light source within a VR-based screen. While the wearable device switches from a VST mode to a VR mode, the wearable device can provide a user experience in which the real light source is maintained in the virtual space.

According to one embodiment, the actual light sources that the wearable device identifies based on the operations of FIG. 5 may have different shapes. Hereinafter, the operation of the wearable device for identifying actual light sources having different shapes is described with reference to FIGS. 6 to 8.

FIG. 6 illustrates an example of a flow diagram of a wearable device according to one embodiment. The wearable devices of FIGS. 1A to 1B and FIGS. 2 to 3A to 3B may include the wearable device of FIG. 6. The operations of the wearable device described with reference to FIG. 6 may be performed by the wearable device (101) and/or the processor (210) of FIG. 2. At least one of the operations of FIG. 6 may be related to the operations of FIG. 5.

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 6, in operation (610), the wearable device according to one embodiment can identify an actual light source that is contained in an external space and radiates directional light. The wearable device can identify an actual light source that is designed to radiate light toward a specific direction, such as the floor lamp (120) and/or a spot light of FIGS. 1A and 1B. According to one embodiment, the wearable device can perform operation (610) of FIG. 6 based on operation (510) of FIG. 5.

Referring to FIG. 6, in operation (620), the wearable device according to one embodiment may determine a position in a virtual space corresponding to a position of an actual light source in external space. According to one embodiment, the wearable device may determine or identify a position of an actual light source in external space by using a plurality of images (e.g., images (410, 420) of FIG. 4) acquired using a camera (e.g., camera (225) of FIG. 2). For example, the wearable device may perform operation (620) of FIG. 6 similarly to operation (520) of FIG. 5. In one embodiment, the wearable device may store information about an actual light source. The wearable device may store the information in a memory (e.g., memory (215) of FIG. 2). The wearable device may store, in the information, a direction, a color, and/or a brightness of the directional light transmitted from the actual light source. In one embodiment, the wearable device may store elements of a three-dimensional vector (e.g., a vector based on the spatial coordinate system of FIG. 4) representing the direction of said directional light, together with a parameter indicating that the actual light source is a light source emitting directional light.

Referring to FIG. 6, in operation (630), a wearable device according to an embodiment may identify an event for displaying a virtual space for VR (virtual reality). The wearable device may perform operation (630) of FIG. 6 similarly to operation (330) of FIG. 3A and/or operation (530) of FIG. 5. For example, a wearable device that has identified an event of operation (630) may execute one or more applications (e.g., VR applications) for providing a virtual space based on the event.

Referring to FIG. 6, in operation (640), according to one embodiment, the wearable device may perform rendering for at least a portion of the virtual space based on a location of a virtual light source in the virtual space. Having identified an actual light source emitting directional light in operation (610), the wearable device may place a virtual light source emitting directional light in the virtual space based on operation (640). Since the wearable device performs rendering for the virtual space based on the virtual light source, the wearable device may obtain and/or display a screen representing the directional light emitted from the virtual light source. The direction in which light output from the virtual light source is propagated in the virtual space may substantially coincide with the direction in which light output from the actual light source is propagated in external space.

Although the operation of the wearable device for an actual light source emitting directional light has been described, the embodiment is not limited thereto. For example, the wearable device can identify an actual light source emitting light through a surface formed in two dimensions or three dimensions, such as a surface light source. Hereinafter, with reference to FIG. 7, the operation of the wearable device identifying a surface light source according to one embodiment is described.

FIG. 7 illustrates an example of a flow diagram of a wearable device according to one embodiment. The wearable devices of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device of FIG. 7. The operations of the wearable device described with reference to FIG. 7 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210). At least one of the operations of FIG. 7 may be related to the operations of FIGS. 5 to 6.

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 7, in operation (710), according to one embodiment, the wearable device may identify a surface emitting or reflecting light from at least one image acquired from a camera (e.g., camera (225) of FIG. 2). Operation (710) of FIG. 7 may be performed similarly to operation (510) of FIG. 5 and/or operation (610) of FIG. 6. According to one embodiment, the wearable device may identify the surface of operation (710) based on a brightness (or luminance) distribution in the image. The embodiment is not limited thereto, and the wearable device may identify the surface of operation (710) based on a color distribution in the image. The color distribution may be related to the intensities of three primary colors (e.g., red, green, and blue) in each of the pixels included in the image.

For example, if light from an actual light source is reflected from a surface captured by a camera, in an image acquired from the camera, a portion corresponding to the surface may have a brightness that is brighter than another portion. In the example, the wearable device may identify a virtual light source having a shape of the surface based on a location of the portion in the image. For example, if a real light source including a shape of a plane and/or a curved surface, such as a surface light source, is adjacent to the wearable device, the wearable device may identify the shape of the plane and/or the curved surface of the real light source using at least one image.

Referring to FIG. 7, in operation (720), the wearable device according to one embodiment may store information related to a surface identified in external space based on operation (710). The wearable device may store the information in a memory (e.g., memory (215) of FIG. 2 ). According to one embodiment, the wearable device may perform operation (720) similar to operation (320) of FIG. 3A , operation (520) of FIG. 5 , and/or operation (620) of FIG. 6 . According to one embodiment, the wearable device may store, in the information, a color, brightness, and/or direction of actual light emitted and/or reflected from the surface. The wearable device may store, in the information, a shape of the surface that emits and/or reflects light. For example, the wearable device may store, in the information, coordinates of vertices of the surface based on a spatial coordinate system (e.g., the spatial coordinate system of FIG. 4 ).

Referring to FIG. 7, in operation (730), according to one embodiment, the wearable device may identify an event for displaying a virtual space for VR. The wearable device may perform operation (730) of FIG. 7 similarly to operation (330) of FIG. 3A, operation (530) of FIG. 5, and/or operation (630) of FIG. 6. Based on the event of operation (730), the wearable device may initiate rendering of a virtual space corresponding to the event.

Referring to FIG. 7, in operation (740), according to an embodiment, the wearable device may perform rendering of a virtual space including a virtual light source or a surface reflecting virtual light emitted from a virtual light source based on the information stored by operation (720). Having identified an actual light source including the surface of operation (710), the wearable device may place a virtual light source having the shape of the surface in the virtual space based on operation (740). Since the wearable device performs the rendering of operation (740) based on the virtual light source, the wearable device may acquire and/or display a screen expressing virtual light emitted from the virtual light source in the shape of a surface.

Below, the operation of a wearable device that identifies one or more real light sources by the operations of FIGS. 4 to 7 is described based on the exemplary case of FIG. 8.

FIG. 8 illustrates an example of an operation of a wearable device (101) that identifies an actual light source in an external space. The wearable devices of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device (101) of FIG. 8. The operation of the wearable device (101) described with reference to FIG. 8 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210). The operation of the wearable device (101) of FIG. 8 may be related to the operations of FIGS. 3A to 3B, and FIGS. 4 to 7.

Referring to FIG. 8, an example of a screen (810) displayed by a wearable device (101) operating in VST mode is illustrated. The screen (810) may include an image and/or a video including a front view of a user (110) wearing the wearable device (101) (e.g., a direction in which both eyes of the user (110) are facing). According to one embodiment, the wearable device (101) may identify one or more real light sources (820, 830) from the image and/or the video.

For example, the wearable device (101) can identify a spot light source (820), which is an actual light source emitting directional light, based on an image acquired from a camera (e.g., the camera (225) of FIG. 2) for display on the screen (810). The wearable device (101) can determine a location of the spot light source (820) in external space based on the operation described with reference to FIGS. 4 to 6. Based on determining the location of the spot light source (820), the wearable device (101) can store information used to place a virtual light source corresponding to the spot light source (820) in virtual space. In the information, the wearable device (101) can store at least one of the location, color, brightness, or intensity of the spot light source (820).

For example, the wearable device (101) can identify the sun (830), which is an actual light source, from an image corresponding to the screen (810) while displaying the screen (810) based on the VST mode. The wearable device (101) can identify the sun (830) in the image based on the operation described with reference to FIGS. 4 and 5. Based on identifying the sun (830), the wearable device (101) can store information used to place a virtual light source corresponding to the sun (830) in a virtual space. The wearable device (101) can store at least one of the location of the sun (830) in external space, the color, brightness, or intensity of the sun (830) in the information. Based on identifying the sun (830), the wearable device (101) can store the path of the sun (830) in the information. The path stored in the above information can represent the position of the sun (830) at a point in time after the point in time when the sun (830) was identified.

For example, the wearable device (101) can identify surfaces that emit and/or reflect light from an image that includes a portion of an external space. Referring to FIG. 8, in an image corresponding to a screen (810), the wearable device (101) can identify surfaces (840, 850) that reflect light transmitted from the sun (830). In the image, the brightness of the portion corresponding to the surfaces (840, 850) may be greater than the brightness of other portions. According to one embodiment, the wearable device (101) can classify the surfaces (840, 850) that reflect light as indirect light sources. The wearable device (101) can store information for rendering the light reflected from the surfaces (840, 850) in a virtual space. The wearable device (101) can store at least one of the shape, position, or size of the surfaces (840, 850) in the information. The wearable device (101) can store at least one of the color, direction, brightness, or intensity of light reflected from the surfaces (840, 850) in the information. The wearable device (101) can store information about an actual light source (the sun (830) in the exemplary case of FIG. 8) corresponding to the light reflected from the surfaces (840, 850) in the information based on identifying the surfaces (840, 850).

As described above, according to one embodiment, the wearable device (101) can store information based on characteristics of an actual light source. The wearable device (101) can store information on the location, color, brightness, and/or intensity of an actual light source existing in an external space, as well as characteristics of light output from the actual light source (e.g., directional light). Based on the information, the wearable device (101) can perform rendering of a virtual object based on at least one of the color or brightness of the actual light source.

Below, with reference to FIG. 9, the operation of a wearable device (101) that stores information corresponding to an actual light source based on the brightness of the actual light source is described.

FIG. 9 illustrates an example of a flow diagram of a wearable device according to one embodiment. The wearable devices (101) of FIGS. 1A to 1B and FIGS. 2 to 3A to 3B may include the wearable device of FIG. 9. The operations of the wearable device of FIG. 9 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210). At least one of the operations of FIG. 9 may be related to the operations of FIGS. 3A to 3B and/or FIGS. 5 to 7.

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 9, in operation (910), the wearable device according to one embodiment may identify a plurality of real light sources from at least one image. The wearable device may perform operation (910) of FIG. 9 similarly to operation (510) of FIG. 5, operation (610) of FIG. 6, and/or operation (710) of FIG. 7. According to one embodiment, the wearable device may identify portions of the image corresponding to each of the plurality of real light sources based on a brightness distribution and/or a color distribution of an image acquired through a camera (e.g., camera (225) of FIG. 2).

Referring to FIG. 9, in operation (920), according to one embodiment, the wearable device may perform filtering on a plurality of real light sources based on a threshold brightness. The wearable device may identify brightnesses of the plurality of real light sources based on colors and/or brightnesses of portions corresponding to each of the plurality of real light sources in an image. The wearable device may identify whether each of the identified brightnesses is equal to or greater than the threshold brightness. The wearable device may selectively store information on a real light source having a brightness equal to or greater than the threshold brightness among the plurality of real light sources. For example, the wearable device may refrain from storing information on a real light source having a brightness less than the threshold brightness among the plurality of light sources. The filtering of operation (920) may include selecting a real light source having a brightness equal to or greater than the threshold brightness and storing information on the selected real light source.

Referring to FIG. 9, in operation (930), a wearable device according to one embodiment may identify an event for displaying a virtual space for VR. The wearable device may perform operation (930) of FIG. 9 similarly to operation (330) of FIG. 3A, operation (530) of FIG. 5, operation (630) of FIG. 6, and/or operation (730) of FIG. 7.

Referring to FIG. 9, in operation (940), according to one embodiment, the wearable device may perform rendering for at least a portion of the virtual space based on an actual light source having a brightness higher than a threshold brightness. For example, the wearable device may place a virtual light source corresponding to the actual light source filtered by operation (920) (e.g., an actual light source having a brightness higher than a threshold brightness) in the virtual space. Based on the rendering for the virtual space in which the virtual light source is placed, the wearable device may display a screen (e.g., screen (132) of FIG. 1A) including at least a portion of the virtual space. According to one embodiment, the wearable device may perform operation (940) of FIG. 9 similarly to operation (340) of FIG. 3A, operation (540) of FIG. 5, operation (640) of FIG. 6, and/or operation (740) of FIG. 7.

Hereinafter, with reference to FIGS. 10A to 10B, 11A to 11B, and 12 to 15, the operation of a wearable device that performs rendering for at least one virtual light source placed in a virtual space based on information of at least one real light source is described.

FIGS. 10A to 10B illustrate an example of an operation of a wearable device (101) that performs rendering for a virtual space based on information related to an actual light source in an external space. The wearable devices (101) of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device (101) of FIGS. 10A to 10B. The operation of the wearable device (101) of FIGS. 10A to 10B may be performed by the wearable device (101) of FIG. 2 and/or the processor (210).

Referring to FIGS. 10A and 10B , according to one embodiment, the wearable device (101) may acquire an image and/or video for a specified direction by using a camera (e.g., the camera (225) of FIG. 2 ) positioned toward the specified direction. The specified direction may include a direction (e.g., a normal direction) of one side of the wearable device (101) opposite to the other side on which the display is positioned. The wearable device (101) may identify at least one actual light source included in an external space from the image and/or the video. In the exemplary case of FIG. 10A , the wearable device (101) may identify, by using the image and/or the video, a floor lamp (120) and a surface light source (1020), which are examples of actual light sources.

According to one embodiment, the wearable device (101) can identify an input for entering and/or switching to a VR mode while identifying at least one real light source. Based on the input, the wearable device (101) can display a screen (1010). The wearable device (101) can display at least a portion of a virtual space for the VR mode through the screen (1010). The wearable device (101) can place a virtual light source corresponding to the real light source in the virtual space based on the result of identifying the real light source in the external space based on the operations described above with reference to FIGS. 1A to 1B and FIGS. 2 to 9. Based on the virtual light source placed in the virtual space, the wearable device (101) can perform rendering for the virtual space.

Referring to the exemplary case of FIG. 10A, based on identifying a floor lamp (120) in an external space, the wearable device (101) according to one embodiment may place a virtual light source (1031) corresponding to the floor lamp (120) in the virtual space. Based on identifying a surface light source (1020) in the external space, the wearable device (101) may place a virtual light source (1032) corresponding to the surface light source (1020) in the virtual space. The wearable device (101) may place the virtual light source (1031) in the virtual space based on the position of the floor lamp (120) in the external space. The direction of the virtual light radiated from the virtual light source (1031) may match the direction of the actual light radiated from the floor lamp (120) in the external space. The shape of a virtual light source (1032) placed in a virtual space based on a surface light source (1020) may match the shape of one side (1022) of the surface light source (1020) from which actual light is output.

Referring to FIG. 10A, the wearable device (101) may display a virtual object (1040) included in the virtual space on a screen (1010) displayed based on rendering of the virtual space. Referring to FIG. 10A, based on positional relationships between virtual light sources (1031, 1032) and the virtual object (1040) in the virtual space, the wearable device (101) may apply a visual effect based on the virtual light sources (1031, 1032) to the virtual object (1040). For example, the wearable device (101) may display a visual object (1051) having a form of a shadow that starts from the virtual object (1040) and extends along a direction from the virtual light source (1031) toward the virtual object (1040). For example, the wearable device (101) may display a visual object (1052) in the form of a shadow that originates from a virtual object (1040) and extends in a direction from a virtual light source (1032) toward the virtual object (1040).

In one embodiment, the wearable device (101) can adjust the brightness of the first faces of the virtual object (1040) facing each of the virtual light sources (1031, 1032) to be greater than the brightness of the second faces opposite the first faces. The wearable device (101) can adjust the color and/or brightness of the face of the virtual object (1040) facing the virtual light source (1031) based on the color and/or brightness of the virtual light directed from the virtual light source (1031) to the virtual object (1040). Similarly, the wearable device (101) can adjust the color and/or brightness of the face of the virtual object (1040) facing the virtual light source (1032) based on the color and/or brightness of the virtual light directed from the virtual light source (1032) to the virtual object (1040).

In one embodiment, the wearable device (101) can adjust the shape of the virtual light source (1031) based on an actual light source (e.g., a floor lamp (120)) and/or a virtual space corresponding to the virtual light source (1031). Referring to FIG. 11B, screens (1091, 1092) that display the shape of the virtual light source (1031) corresponding to an actual light source (e.g., a floor lamp (120)) in a shape corresponding to a virtual space displayed through the display are exemplarily illustrated.

Referring to FIG. 11b, in a state where a screen (1091) based on a virtual space having a shape (or context) related to indoors (e.g., an office) is displayed, the wearable device (101) can display a virtual light source (1031-1) having a shape suitable for the shape of the virtual space. In the state, the wearable device (101) that has identified an actual light source such as a floor lamp (120) can place a virtual light source (1031-1) in the virtual space corresponding to the identified actual light source, and determine the shape of the virtual light source (1031-1) based on the shape, type, and/or context of the virtual space.

For example, in a state of displaying a virtual space including virtual objects related to an indoor environment, such as a virtual object (1072) having a shape of a desk, the wearable device (101) may display a virtual light source (1031-1) having a shape (e.g., a lamp) that can be placed on one side of the virtual object (1072). Since a visual effect based on the virtual light source (1031-1) is applied to a virtual object (1040) on one side of the virtual object (1072), the wearable device (101) may display a virtual object (1051) that casts a shadow on the one side of the virtual object (1072). The embodiment is not limited thereto, and the wearable device (101) may display a floating virtual object (1073) on the screen (1091), and/or display an image (1074) of at least a portion of an external space of the wearable device (101). In one embodiment, the area within the screen (1091) where the image (1074) is displayed may be referred to as a pass-through (PT) area.

Referring to FIG. 10b, depending on the virtual space provided through the screen (1092), the wearable device (101) can change the shape of a virtual light source (1031-2) corresponding to an actual light source (e.g., a floor lamp (120)). In a state where the screen (1092) related to a virtual space related to space is displayed, the wearable device (101) can display a virtual light source (1031-2) having a shape related to space (e.g., a virtual light source having a shape of a star such as the sun) together with a virtual object related to space (e.g., a virtual object (1081) having a shape of Saturn). The wearable device (101) can adjust the shade of the virtual object (1081) based on the positional relationship between the virtual light source (1031-2) and the virtual object (1081) within the virtual space. In one embodiment, the wearable device (101) may apply a visual effect based on a virtual light source (1031-2) to some of the virtual objects included in the virtual space. For example, when the visual effect is applied to a virtual object (1081), the wearable device (101) may limit the application of the visual effect to a virtual object (1082) floating on the screen (1092). For example, the wearable device (101) may not apply the visual effect to a virtual object (1082) including icons (1083) for interacting with a user wearing the wearable device (101).

As described above, according to one embodiment, the wearable device (101) can perform rendering for a virtual space based on a virtual light source corresponding to an actual light source. The wearable device (101) can provide a user experience based on an actual light source moved into a virtual space to a user (110) wearing the wearable device (101) by using a virtual light source placed in the virtual space. According to one embodiment, the wearable device (101) can perform rendering based on a virtual light source based on the visibility of a virtual object. For example, if a visual effect applied to the virtual object by the virtual light source reduces or eliminates the visibility of the virtual object, such as a shadow directed at the user (110), the wearable device (101) can stop applying the visual effect to the virtual object.

Hereinafter, with reference to FIGS. 11A and 11B, an embodiment of a wearable device (101) that selectively applies visual effects based on virtual light sources to virtual objects is described.

FIGS. 11A and 11B illustrate an example of an operation of a wearable device (101) that performs rendering for a virtual object included in a virtual space (140) based on information related to an actual light source in an external space. The wearable devices of FIGS. 1A and 1B, FIG. 2, and FIGS. 3A and 3B may include the wearable device (101) of FIGS. 11A and 11B. The operation of the wearable device (101) described with reference to FIGS. 11A and 11B may be performed by the wearable device (101) of FIG. 2 and/or the processor (210).

Referring to FIG. 11A, a screen (1110) displayed by a wearable device (101) that identifies an actual light source is exemplarily illustrated. The wearable device (101) may display a virtual light source (1120) corresponding to the actual light source on the screen (1110). The position (P2) of the virtual light source (1120) in the virtual space (140) may correspond to the position of the actual light source with respect to the wearable device (101) in an external space including the wearable device (101). The wearable device (101) may perform rendering for at least one virtual object included in the virtual space (140) based on the virtual light source (1120). For example, the wearable device (101) may perform rendering for a virtual object (1130) based on virtual light output from the virtual light source (1120). Referring to FIG. 11a, the wearable device (101) may perform rendering on the virtual object (1130) to display a visual object (1132) representing a shadow extended from the virtual object (1130) on the screen (1110).

Referring to FIG. 11A, according to one embodiment, the wearable device (101) may determine whether to apply a visual effect based on the virtual light source (1120) to the virtual object based on at least one of a position (P1) of the wearable device (101) in a virtual space (140), a distance between the position (P1) and a virtual object, a category of the virtual object, or a positional relationship between a virtual light source (1120) and the virtual object. For example, the application of a visual effect based on a virtual light source (1120) to a virtual object (1140) that supports a function of interacting with a user (110) wearing the wearable device (101) may be refrained from.

In one embodiment, the wearable device (101) performing rendering for a virtual object (1140) included in an area spaced apart from a specified distance (P1) of the wearable device (101) in the virtual space (140) and included in a specified category for interaction may stop applying a visual effect based on a virtual light source (1120) to the virtual object (1140). The specified distance may include a reachable distance (e.g., about 50 cm) of a hand of a user (110) of the wearable device (101) in external space. In one embodiment, the wearable device (101) may perform rendering for another virtual object spaced apart from the area (e.g., the virtual object (1130)) based on the virtual light source (1120).

In one embodiment, a virtual object (1140) that supports a function of interacting with a user (110) may be more likely to be positioned toward a location (P1) of a wearable device (101) in a virtual space (140). When the wearable device (101) combines a shadow formed by a virtual light source (e.g., a virtual light source (1120)) positioned beyond the virtual object (1140) with respect to the location (P1) with respect to the virtual object (1140) positioned toward the location (P1), the shadow may extend toward the location (P1). For example, the possibility that a user (110) wearing the wearable device (101) recognizes the virtual object (1140) may be reduced by the shadow. According to one embodiment, the wearable device (101) may stop applying a visual effect based on a virtual light source (1120) to a virtual object (1140) supporting the function based on a category of the virtual object (e.g., a designated category for classifying a virtual object for interaction and/or a visual object).

In one embodiment, the application of a visual effect related to a virtual light source (1120) may be stopped based on whether a virtual object (1140) is floating in the virtual space (140). For example, the wearable device (101) may stop applying a visual effect based on a virtual light source (1120) to a virtual object (1140) floating in the virtual space (140). A virtual object (1140) floating in the virtual space (140) may mean having a fixed location and/or coordinates within the virtual space (140). For example, the virtual object (1140) may have a location within the virtual space (140) that is not associated with another virtual object and/or a real object mapped to the virtual space (140). A virtual object (1140) floating in a virtual space (140) may include a window (or activity) provided from an application executed by a wearable device (101). In a state of displaying a virtual object (1140) corresponding to a window provided from an application, the wearable device (101) may restrict and/or stop applying a visual effect based on a virtual light source (1120) to the virtual object (1140) in order to maintain the visibility of information included in the virtual object (1140).

In one embodiment, the application of a visual effect related to a virtual light source (1120) may be stopped based on the location and/or category of the virtual object. For example, the wearable device (101) may apply a visual effect based on a virtual light source (1120) to a virtual object (e.g., the virtual object (1150)) placed on a reference plane (1155) of the virtual space (140) (e.g., a plane having a z-axis coordinate of 0). The application of the visual effect to the virtual object placed on the reference plane (1155) may be performed independently of the location (P1) of the wearable device (101) and the distance between the virtual object and the virtual object within the virtual space (140). For example, even if the distance between the virtual object and the location (P1) placed on the reference plane (1155) is less than a specified distance for stopping the visual effect based on the virtual light source (1120), the wearable device (101) may apply the visual effect to the virtual object. For example, for a virtual object (e.g., virtual object (1140)) displayed for interaction with the user (110), such as a window, widget, and/or icon provided from an application, the wearable device (101) may not apply the visual effect based on the virtual light source (1120).

In one embodiment, the wearable device (101) may stop applying a visual effect based on a virtual light source (1120) to the virtual object (1150) if the distance between the reference position (P3) of the virtual object (1150) and the position (P1) of the wearable device (101) in the virtual space (140) is less than or equal to the specified distance. For example, for a virtual object (1150) displayed relatively close to a user (110) wearing the wearable device (101), the wearable device (101) may stop performing rendering for the virtual object (1150) based on the virtual light source (1120) in order to prevent a shadow extended from the virtual object (1150) from darkening the screen (1110). For example, the wearable device (101) may stop performing ray casting for a ray radiating from a virtual light source (1120) and directed toward a virtual object (1150). For example, the wearable device (101) may display a visual object (1132) representing a shadow formed by the virtual object (1130) based on the ray casting. Ray casting may include an algorithm that traces a trajectory of virtual light propagating in the virtual space (140) (e.g., a trajectory of virtual light reflected from one surface of the virtual object) and renders the virtual space (140) including the virtual light.

As described above, according to one embodiment, the wearable device (101) can identify a first area of the virtual space (140) linked with a virtual light source (1120) corresponding to an actual light source. The first area can include a portion of the virtual space (140) that exceeds a specified distance from a location (P1) in the virtual space (140) of the wearable device (101). The wearable device (101) can perform rendering for a first virtual object (e.g., virtual object (1130)) included in the first area among a plurality of virtual objects (1130, 1140, 1150) included in the virtual space (140) while displaying the virtual space on the display, which is distinct from the external space, based on a visual effect related to the virtual light source.

According to one embodiment, the wearable device (101) may perform rendering on the display, independently of the visual effect, for a second virtual object (e.g., virtual objects (1140, 1150)) included in a second area different from the first area, among a plurality of virtual objects (1130, 1140, 1150). For example, the wearable device (101) may refrain from applying a visual effect based on a virtual light source (1120) to the virtual object (1150) included in the second area. In one embodiment, the wearable device (101) may perform rendering on a virtual object (e.g., virtual object (1140)) included in the second area and included in a category for interacting with the user (110), independently of the virtual light source (1120). The wearable device (101) may determine whether to apply a visual effect based on a virtual light source (1120) to a virtual object based on a virtual object positioned relatively close to a location (P1) of the wearable device (101) in the virtual space (140) and/or the type of the virtual object. For example, in order to prevent a shadow caused by a virtual light source (1120) from reducing visibility of virtual objects (1130, 1140, 1150) on a screen (1110) visible to a user (110) wearing the wearable device (101), the wearable device (101) may perform rendering for a virtual object included in the second area independently of the visual effect.

According to one embodiment, the wearable device (101) can change one or more parameters used for a visual effect based on a virtual light source (1120) according to an input received from a user (110). Referring to FIG. 11B , a virtual object (1170) displayed by the wearable device (101) for adjusting the one or more parameters according to one embodiment is exemplarily illustrated. Although the virtual object (1170) is illustrated as having a form of a pop-up window, the embodiment is not limited thereto. In terms of setting the one or more parameters, the virtual object (1170) may be referred to as a setting window. Referring to FIG. 11B , for convenience of explanation, the virtual object (1170) is illustrated outside an area of a sheet on which the screen (1110) is illustrated. According to one embodiment, the wearable device (101) may display a virtual object (1170) in a virtual space (140) and/or a screen (1110).

According to one embodiment, the wearable device (101) may display the virtual object (1170) based on identifying a gesture of the user (110). For example, the wearable device (101) may display the virtual object (1170) based on identifying a direction of the user's (110) gaze toward the virtual light source (1120) within the screen (1110) for a specified period of time. For example, the wearable device (101) may display the virtual object (1170) based on a hand gesture of the user (110) related to the virtual light source (1120). The hand gesture may include a pinch gesture performed by contact of the fingertips of two fingers of the user (110) adjacent to the virtual light source (1120) and/or a pointing gesture performed by one or more designated fingers (e.g., an index finger) straightened toward the virtual light source (1120). Similarly, in one embodiment displaying a virtual object (1160) having the form of a button displayed at a designated location on the screen (1110), the wearable device (101) may display a virtual object (1170) based on a gesture of the user (110) selecting the virtual object (1160). The virtual object (1160) may include an icon representing an adjustment of one or more parameters associated with the virtual light source (1120). The embodiment is not limited thereto, and the wearable device (101) may display the virtual object (1170) based on an utterance related to the virtual object (1170) (e.g., a natural language sentence such as “I want to change the virtual light source”).

According to one embodiment, the wearable device (101) may provide options for controlling a visual effect (or rendering) related to a virtual light source (1120) through a virtual object (1170). Based on the options, the wearable device (101) may change one or more parameters used for a visual effect based on the virtual light source (1120). Referring to FIG. 11B, the wearable device (101) may display a visual object (1171) within the virtual object (1170) for controlling whether to place the virtual light source (1120) in the virtual space (140) based on an actual light source in an external space. Referring to FIG. 11B, a visual object (1171) having a form of a switch is illustrated as an example, but the embodiment is not limited thereto. For example, the wearable device (101) may identify an input for controlling whether to generate a virtual light source corresponding to a real light source using a visual object in the form of a radio button. The wearable device (101) may use an input associated with the visual object (1171) to toggle placement of a virtual light source (1120) in the virtual space (140) based on the location, color, and/or brightness of the real light source.

Referring to FIG. 11b, the wearable device (101) may provide options related to an area for limiting a visual effect based on a virtual light source (1120) within a virtual object (1170). For example, the wearable device (101) may display a visual object (1172) within the virtual object (1170) for controlling whether to generate the area. Although the visual object (1172) having the form of a switch is exemplarily illustrated, the embodiment is not limited thereto, and the wearable device (101) may display a visual object having the form of a radio button. In the exemplary state of FIG. 11b where an input for creating a region based on a visual object (1172) is identified, the wearable device (101) can identify an input for adjusting the size and/or reference position of the region using the visual objects (1173, 1174, 1175) displayed on the virtual object (1170).

According to one embodiment, the wearable device (101) can identify an input for adjusting the size of an area for limiting a visual effect based on a virtual light source (1120) through a visual object (1173) having a form of a slider. The wearable device (101) can display, on the visual object (1173) having a form of a line extending in a reference direction (e.g., a horizontal direction), a handle (1174) that can be moved on the line by a direction of a user's (110) gaze and/or a hand gesture (e.g., a pinch gesture). Based on the position of the handle (1174) superimposed on the visual object (1173), the wearable device (101) can change the size of the area.

According to one embodiment, the wearable device (101) can identify an input for adjusting a reference position of an area for limiting a visual effect based on a virtual light source (1120) through a visual object (1175) in the form of a button. Based on an input indicating selection of a visual object (1175), the wearable device (101) can provide options for the reference position in the form of a pop-up window and/or a list. For example, the wearable device (101) can provide a first option for forming the area based on a location (P1) of the wearable device (101) in the virtual space (140) and/or a second option for forming the area based on a fixed location in the virtual space (140). Based on an input for selecting the first option, the wearable device (101) can display text (e.g., “User”) on the visual object (1175) indicating that the location (P1) has been selected as a reference position. Based on an input selecting the second option, the wearable device (101) may display text (e.g., “fixed location”) on the visual object (1175) indicating that the region is formed based on the fixed location. In a state where the second option is selected, the wearable device (101) may form a region centered on the fixed location (e.g., a reference location on the z-axis) within the virtual space (140) and stop applying visual effects based on the virtual light source (1120) to virtual objects included in the region.

According to one embodiment, the wearable device (101) may create an area in the virtual space (140) in which application of a visual effect related to a virtual light source (1120) is restricted, using a size set based on a visual object (1173) and a reference position set based on a visual object (1175). In the exemplary state of FIG. 11b, the wearable device (101) may create an area centered on a location (P1) of the wearable device (101) in the virtual space (140) set by the visual object (1175) and having a size set by a location of a handle (1174) on the visual object (1173). The wearable device (101) may apply a visual effect based on the virtual light source (1120) to another virtual object that is different from at least one virtual object included in the area among virtual objects (1130, 1140, 1150) included in the virtual space (140).

In one embodiment, the wearable device (101) may identify an input for adjusting a parameter related to a shadow (e.g., a visual object (1132)) displayed by a visual effect based on a virtual light source (1120) within a virtual object (1170). The parameter may include a size, a color, and/or an opacity of the shadow. In one embodiment of FIG. 11B, the wearable device (101) may display a visual object (1176) within the virtual object (1170) that is related to a size of the shadow and has a form of a slider. Through a handle (1177) displayed on the visual object (1176), the wearable device (101) may identify an input for adjusting the size of the shadow. In one embodiment of FIG. 11b, the wearable device (101) may display a visual object (1178) in the form of a slider, which is related to the transparency of the shadow, within the virtual object (1170). Through a handle (1179) displayed on the visual object (1178), the wearable device (101) may identify an input for adjusting the transparency of the shadow.

According to one embodiment, the wearable device (101) can perform rendering of a shadow of a virtual object based on a size set based on the visual object (1176) and a transparency set based on the visual object (1178). In the exemplary state of FIG. 11b, a size of a visual object (1132) that expresses a shadow extending in a direction from a virtual light source (1120) toward the virtual object (1130) can be related to a position of a handle (1177) on the visual object (1176). The wearable device (101) can change the transparency of the visual object (1132) based on a position of a handle (1179) on the visual object (1178).

Below, with reference to FIG. 12, the operation of the wearable device (101) described with reference to FIGS. 11a and 11b is described.

FIG. 12 illustrates an example of a flow diagram of a wearable device according to one embodiment. The wearable devices of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device of FIG. 12. The operation of the wearable device described with reference to FIG. 12 may be performed by the wearable device (101) and/or the processor (210) of FIG. 2.

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 12, in operation (1210), a wearable device according to an embodiment may identify a virtual space including a virtual light source. The virtual light source of operation (1210) may include the virtual light source (150) of FIGS. 1A to 1B , the virtual light sources (1031, 1032) of FIGS. 10A to 10B , and/or the virtual light source (1120) of FIGS. 11A to 11B . The virtual space of operation (1210) may include the virtual space (140) of FIG. 1A and/or FIGS. 11A to 11B . According to an embodiment, the wearable device may perform operation (1210) based on the execution of the virtual space manager (276) of FIG. 2 . The wearable device can identify a virtual space in which a virtual light source corresponding to the real light source is arranged based on information including a result of identifying a real light source from an external space.

Referring to FIG. 12, in operation (1220), according to an embodiment, a wearable device may identify a virtual light source included in a FoV of a virtual space to be displayed through a display. The FoV of operation (1220) may include the FoV of FIGS. 1A to 1B. For example, the wearable device may select the FoV in the virtual space based on a position and/or orientation of the wearable device. In a state where at least one virtual light source corresponding to at least one real light source included in an external space is placed in the virtual space, the wearable device may identify at least one virtual light source included in the FoV. According to an embodiment, based on identifying at least one virtual light source from the FoV by operation (1220), the wearable device may perform operation (1230).

Referring to FIG. 12, in operation (1230), according to one embodiment, the wearable device may perform rendering on a virtual object by using an area based on a reference position of the wearable device in the virtual space and/or a category including a virtual object. For example, the wearable device may apply a visual effect related to a virtual light source of operation (1220) to a virtual object included in an area spaced apart from a reference position of the wearable device in the virtual space by a specified distance or more. The wearable device may refrain from applying a visual effect related to a virtual light source of operation (1220) to a virtual object spaced apart from the reference position by a specified distance or less. For example, the wearable device may apply a visual effect related to a virtual light source of operation (1220) to a virtual object different from a specified category for interacting with a user (e.g., the user (110) of FIGS. 1A and 1B ). The wearable device may apply a visual effect related to a virtual light source of an operation (1220) to a virtual object included in the above-mentioned category. Although an exemplary operation in which a visual effect related to a virtual light source is applied based on a distance between a reference position and a virtual object, and/or a type of a virtual object has been described, the embodiment is not limited thereto.

In one embodiment, the real light source identified by the wearable device may include a real light source (e.g., the sun and/or the moon) that moves regularly. Hereinafter, with reference to FIG. 13, the operation of the wearable device with respect to a virtual light source that moves in virtual space based on the regularity of the real light source is described.

FIG. 13 illustrates an example of an operation of a wearable device (101) that performs rendering for a virtual space based on the movement of an actual light source (1320). The wearable devices of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device (101) of FIG. 13. The operation of the wearable device (101) described with reference to FIG. 13 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210).

Referring to FIG. 13, an exemplary state of a wearable device (101) that identifies an external space including a sun (1320), which is an example of an actual light source, is illustrated. According to one embodiment, the wearable device (101) may identify the sun (1320) from an image and/or video of the external space, based on image recognition and/or object recognition. Based on identifying the sun (1320), the wearable device (101) may generate and/or store information for placing a virtual light source corresponding to the sun (1320) in a virtual space (e.g., the virtual space (140) of FIG. 1A). The wearable device (101) may store, in the information, one or more parameters for moving the virtual light source in the virtual space, based on the movement of the sun (1320) in the external space. The wearable device (101) may store one or more parameters for changing the brightness and/or color of the virtual light source in the virtual space based on changes in brightness and/or color of the sun (1320) over time in the information.

Referring to FIG. 13, the wearable device (101) that identifies the sun (1320), which is an actual light source, in an external space can display a screen (1310) representing at least a portion of a virtual space in response to an input for switching to a VR mode. Based on displaying the screen (1310), the wearable device (101) can stop displaying at least a portion of the external space to a user (110) wearing the wearable device (101). Referring to FIG. 13, the wearable device (101) can perform rendering of the virtual space using a virtual light source based on the sun (1320), which is an actual light source.

According to one embodiment, the wearable device (101) may perform rendering for a virtual space including a virtual light source corresponding to an actual light source based on a movement path of an actual light source (e.g., a movement path of the sun (1320)). Referring to FIG. 13, a sun (1320-1) at a first position at a first time point and a sun (1320-2) at a second position at a second time point after the first time point are illustrated. It is assumed that a first part (1330) formed on a wall surface of an external space is a part where light emitted from the sun (1320-1) at the first position is reflected. It is assumed that a second part (1340) formed on a wall surface of an external space is a part where light emitted from the sun (1320-2) at the second position is reflected.

In an exemplary case of FIG. 13, according to one embodiment, the wearable device (101) may perform rendering of a virtual object (1350) based on a virtual light source in a virtual space corresponding to the sun (1320-1) at a first location at the first time point. Based on the virtual light directed from the virtual light source corresponding to the sun (1320-1) at the first location to the virtual object (1350), the wearable device (101) may display a first visual object (1360) expressing a shadow of the virtual object (1350) on the screen (1310). The location of the virtual light source in the virtual space may correspond to the first location in the external space. Similarly, at a second point in time after the first point in time, the wearable device (101) can display a visual object (1370) representing a shadow of the virtual object (1350) based on a virtual light source of the virtual space corresponding to the sun (1320-2) at the second location. At another point in time between the first point in time and the second point in time, the wearable device (101) can visualize the shadow of the virtual object (1350) by using a visual object of an intermediate form of the visual objects (1360, 1370).

In an exemplary case of FIG. 13, the wearable device (101) can adjust the color, brightness, and/or pattern of the planes (1381, 1382) of the virtual space based on the brightness distribution and/or the color distribution in the image of the external space. Referring to FIG. 13, at a first point in time, based on identifying a portion (1330) having a relatively large size in the image, the wearable device (101) can display the plane (1381) based on a first brightness. At a second point in time after the first point in time, based on identifying a portion (1340) having a relatively small size, the wearable device (101) can display the plane (1381) based on a second brightness that is less than the first brightness.

As described above, according to one embodiment, the wearable device (101) can apply a color sense of an external space including the wearable device (101) to the virtual space by using a virtual light source corresponding to an actual light source. For example, the wearable device (101) can apply a weather condition included in the external space, such as the sun (1320), to the virtual space. For example, the wearable device (101) can place a virtual light source having a color, brightness, and/or intensity corresponding to weather information based on a location of the wearable device (101) in the virtual space. By using the virtual light source, the wearable device (101) can provide a user experience based on connectivity between the VST mode and the VR mode to the user (110).

According to one embodiment, the wearable device (101) may apply a brightness distribution and/or a color distribution of an image of an external space to the virtual space by using a texture and/or a pattern applied to one side of the virtual space instead of a virtual light source. Hereinafter, with reference to FIG. 14, one embodiment of the wearable device (101) that changes the texture and/or the pattern will be described.

FIG. 14 illustrates an example of an operation of a wearable device (101) that performs rendering for a virtual space (1420) based on a brightness distribution of an external space. The wearable devices of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device (101) of FIG. 14. The operation of the wearable device (101) described with reference to FIG. 14 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210).

Referring to FIG. 14, according to one embodiment, a wearable device (101) may perform rendering on a virtual space (1420) having a hemisphere shape, and display a screen (1410) representing at least a portion of the virtual space (1420). The shape of the virtual space (1420) is not limited to the hemisphere shape illustrated in FIG. 14. The wearable device (101) may visualize a background of the screen (1410) by using a curved surface having a hemisphere shape in the virtual space (1420). For example, the wearable device (101) may visualize the background by using a texture and/or pattern of the curved surface. In one embodiment, the curved surface may be referred to as a boundary surface of the virtual space (1420).

According to one embodiment, the wearable device (101) can obtain an image (1430) of an external space using a camera (e.g., the camera (225) of FIG. 2). The wearable device (101) can perform rendering on a boundary surface of a virtual space (1420) based on a brightness distribution of the image (1430). Referring to FIG. 14, the wearable device (101) facing three walls can identify parts (1431, 1432, 1433) corresponding to each of the walls from the image (1430). Based on the brightness distribution of the image (1430), the wearable device (101) can identify portions (1431-2, 1433-2) having relatively dark brightness and portions (1431-1, 1432, 1433-1) having relatively bright brightness. The wearable device (101) can store information representing the brightness distribution of the image (1430) in a memory (e.g., memory (215) of FIG. 2). Based on the information, the wearable device (101) can apply a texture and/or pattern expressing the brightness distribution to a boundary surface of the virtual space (1420). For example, the wearable device (101) can apply a gradient pattern based on the brightness distribution to the boundary surface.

Referring to FIG. 14, a wearable device (101) according to one embodiment may change the brightness and/or color of a boundary surface of a virtual space (1420) based on a brightness distribution of an image (1430). For example, the wearable device (101) may set the brightness of a portion (1422) of the boundary surface corresponding to a portion (1431-2) of the image (1430) having a relatively dark brightness to a first brightness, and may set the brightness of a portion (1421) of the boundary surface corresponding to a portion (1431-1) of the image (1430) having a relatively bright brightness to a second brightness exceeding the first brightness. Similarly, the wearable device (101) can set a portion (1424) of the boundary surface corresponding to a portion (1433-2) of the image (1430) to the first brightness, and can set a portion (1423) of the boundary surface corresponding to a portion (1433-1) of the image (1430) to the second brightness. Based on the rendering of the virtual space (1420) including the boundary surface, the wearable device (101) can display a portion (1422) having the first brightness and a portion (1421) having the second brightness on the screen (1410).

In one embodiment, the wearable device (101) may perform rendering for virtual objects (1441, 1442) included in the virtual space (1420) based on a brightness distribution at a boundary surface of the virtual space (1420). For example, the wearable device (101) may perform rendering for a virtual object (1442) adjacent to portions (1422, 1424) of the boundary surface set to a first brightness based on the first brightness. The wearable device (101) may perform rendering for a virtual object (1441) adjacent to portions (1421, 1423) of the boundary surface set to a second brightness based on the second brightness. In an exemplary state where the second brightness exceeds the first brightness, the virtual object (1441) displayed on the screen (1410) may be displayed based on a brightness that exceeds the brightness of the virtual object (1442).

As described above, according to one embodiment, the wearable device (101) can apply a brightness distribution identified from an image (1430) of an external space to one side (e.g., a boundary side) of a virtual space (1420). By applying the brightness distribution to the virtual space, the wearable device (101) can provide a continuous user experience of the color sensation of the external space between the VST mode and the VR mode. Although the operation of the wearable device (101) providing a virtual space (1420) based on an actual light source using a texture and/or a pattern has been described as an example, the embodiment is not limited thereto. According to one embodiment, the wearable device (101) can identify brightness, color, and/or illuminance of the actual light source based on information related to the actual light source identified from the image (1430). The wearable device (101) can perform rendering of the virtual space (1420) based on the brightness, the color, and/or the illuminance. For example, the wearable device (101) can perform rendering of the virtual space (1420) and/or one or more virtual objects (e.g., virtual objects (1441, 1442)) included in the virtual space (1420) based on the brightness, the color, and/or the illuminance. Since the wearable device (101) renders the virtual space (1420) using the actual light source identified from the image (1430), the wearable device (101) can provide the user with the virtual space (1420) in which the brightness, the color, and/or the illuminance of the actual light source is at least partially maintained.

Below, with reference to FIG. 15, the operation of the wearable device (101) described with reference to FIG. 14 is described.

FIG. 15 illustrates an example of a flow diagram of a wearable device according to one embodiment. The wearable devices of FIGS. 1A to 1B, FIG. 2, and FIGS. 3A to 3B may include the wearable device of FIG. 15. The operation of the wearable device (101) described with reference to FIG. 15 may be performed by the wearable device (101) of FIG. 2 and/or the processor (210).

In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 15, in operation (1510), a wearable device according to one embodiment may identify a brightness distribution of an external space from at least one image. The at least one image of operation (1510) may include image (1430) of FIG. 14. In one embodiment, the wearable device may perform operation (1510) based on a brightness distribution and/or a color distribution of pixels included in an image acquired through a camera (e.g., camera (225) of FIG. 2).

Referring to FIG. 15, in operation (1520), according to one embodiment, the wearable device may identify a pattern to be applied to a boundary surface of a virtual space based on the identified brightness distribution. The pattern of operation (1520) may be referred to as a texture of the boundary surface. The wearable device may identify a pattern having a brightness distribution corresponding to the brightness distribution. According to one embodiment, the wearable device may store the pattern identified based on operation (1520) in a memory (e.g., the memory (215) of FIG. 2).

Referring to FIG. 15, in operation (1530), a wearable device according to an embodiment may identify an event for displaying a virtual space for VR (virtual reality). The wearable device may perform operation (1530) of FIG. 15 similarly to operation (330) of FIG. 3A, operation (530) of FIG. 5, operation (630) of FIG. 6, operation (730) of FIG. 7, and/or operation (930) of FIG. 9. In response to the event, the wearable device may perform operation (1540).

Referring to FIG. 15, in operation (1540), according to one embodiment, a wearable device may perform rendering for at least a portion of a virtual space to which an identified pattern is applied. In one embodiment, a boundary surface of the virtual space to which the pattern of operation (1520) is applied may have a shape surrounding an inner space of the virtual space. For example, the pattern may be recognized as a background in at least a portion of the virtual space rendered by operation (1540). The screen (1410) of FIG. 14 may correspond to at least a portion of the virtual space rendered by operation (1540).

As described above, according to one embodiment, the wearable device can place virtual light sources corresponding to real light sources in a virtual space to continuously provide a user experience based on real light sources between the VST mode and the VR mode. The wearable device can apply a visual effect based on the virtual light source to at least one virtual object placed on the virtual object. The application of the visual effect by the wearable device to the virtual object can be conditionally performed by a location of the virtual object in the virtual space and/or a category of the virtual object. For example, the wearable device can refrain from applying the visual effect to a virtual object that is placed adjacent to the wearable device in the virtual space and/or is intended to interact with a user wearing the wearable device.

Below, with reference to FIGS. 16A to 16B and/or FIGS. 17A to 17B, an example of a form factor of a wearable device (101) according to one embodiment is described.

FIG. 16A illustrates an example of a perspective view of a wearable device, according to one embodiment. FIG. 16B illustrates an example of one or more hardware elements positioned within a wearable device (1600), according to one embodiment. The wearable device (101) of FIGS. 1A-1B and 2 may include the wearable device (1600) of FIGS. 16A-16B. As shown in FIG. 16A , the wearable device (1600), according to one embodiment, may include at least one display (1650) and a frame supporting the at least one display (1650).

According to one embodiment, a wearable device (1600) may be worn on a part of a user's body. The wearable device (1600) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (1600). For example, the wearable device (1600) may output a virtual reality image to the user through at least one display (1650) in response to a designated gesture of the user acquired through the motion recognition camera (1640-2) of FIG. 16B.

According to one embodiment, at least one display (1650) in a wearable device (1600) can provide visual information to a user. The at least one display (1650) can include the display (220) of FIG. 2. For example, the at least one display (1650) can include a transparent or translucent lens. The at least one display (1650) can include a first display (1650-1) and/or a second display (1650-2) spaced apart from the first display (1650-1). For example, the first display (1650-1) and the second display (1650-2) can be arranged at positions corresponding to the user's left and right eyes, respectively.

Referring to FIG. 16B, at least one display (1650) may form a display area on a lens to provide a user wearing the wearable device (1600) with visual information included in external light passing through the lens, together with other visual information that is distinct from the visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. The display area formed by at least one display (1650) may be formed on a second surface (1632) among a first surface (1631) and a second surface (1632) of the lens. When a user wears the wearable device (1600), external light may be incident on the first surface (1631) and transmitted through the second surface (1632) to the user. As another example, at least one display (1650) can display a virtual reality image to be combined with a real screen transmitted via external light. The virtual reality image output from the at least one display (1650) can be transmitted to the user's eyes via one or more hardware included in the wearable device (1600) (e.g., optical devices (1682, 1684), and/or at least one waveguide (1633, 1634)).

According to one embodiment, a wearable device (1600) may include waveguides (1633, 1634) that diffract light transmitted from at least one display (1650) and relayed by optical devices (1682, 1684) and transmit the diffracted light to a user. The waveguides (1633, 1634) may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of an exterior or interior of the waveguides (1633, 1634). The nanopatterns may be formed based on a grating structure having a polygonal and/or curved shape. Light incident on one end of the waveguides (1633, 1634) may be propagated to the other end of the waveguides (1633, 1634) by the nanopatterns. The waveguides (1633, 1634) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)), and at least one reflective element (e.g., a reflective mirror). For example, the waveguides (1633, 1634) may be arranged in the wearable device (1600) to guide a screen displayed by at least one display (1650) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) occurring within the waveguides (1633, 1634).

According to one embodiment, a wearable device (1600) may analyze an object included in a real image collected through a shooting camera (1640-1), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1650). The virtual object may include at least one of text and image for various information related to the object included in the real image. The wearable device (1600) may analyze the object based on a multi-camera, such as a stereo camera. For the object analysis, the wearable device (1600) may execute ToF (time-of-flight) and/or SLAM (simultaneous localization and mapping) supported by the multi-camera. A user wearing the wearable device (1600) may view an image displayed on at least one display (1650).

According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (1600) to be worn on the user's body. According to one embodiment, the frame may be configured such that, when the user wears the wearable device (1600), the first display (1650-1) and the second display (1650-2) may be positioned corresponding to the user's left and right eyes. The frame may support at least one display (1650). For example, the frame may support the first display (1650-1) and the second display (1650-2) to be positioned corresponding to the user's left and right eyes.

Referring to FIG. 16A, the frame may include an area (1620) that at least partially contacts a portion of the user's body when the user wears the wearable device (1600). For example, the area (1620) of the frame that contacts a portion of the user's body may include an area that contacts a portion of the user's nose, a portion of the user's ear, and a portion of a side of the user's face that the wearable device (1600) makes contact with. According to one embodiment, the frame may include a nose pad (1610) that contacts a portion of the user's body. When the wearable device (1600) is worn by the user, the nose pad (1610) may contact a portion of the user's nose. The frame may include a first temple (1604) and a second temple (1605) that contact another portion of the user's body that is distinct from the portion of the user's body.

In one embodiment, the frame may include a first rim (1601) surrounding at least a portion of the first display (1650-1), a second rim (1602) surrounding at least a portion of the second display (1650-2), a bridge (1603) disposed between the first rim (1601) and the second rim (1602), a first pad (1611) disposed along a portion of an edge of the first rim (1601) from one end of the bridge (1603), a second pad (1612) disposed along a portion of an edge of the second rim (1602) from the other end of the bridge (1603), a first temple (1604) extending from the first rim (1601) and secured to a portion of an ear of the wearer, and a second temple (1605) extending from the second rim (1602) and secured to a portion of an ear opposite the ear. The first pad (1611) and the second pad (1612) can be brought into contact with a portion of the user's nose, and the first temple (1604) and the second temple (1605) can be brought into contact with a portion of the user's face and a portion of the user's ear. The temples (1604, 1605) can be rotatably connected to the rim via the hinge units (1606, 1607) of FIG. 16B. The first temple (1604) can be rotatably connected to the first rim (1601) via the first hinge unit (1606) disposed between the first rim (1601) and the first temple (1604). The second temple (1605) can be rotatably connected to the second rim (1602) via a second hinge unit (1607) disposed between the second rim (1602) and the second temple (1605). In one embodiment, the wearable device (1600) can identify an external object (e.g., a user's fingertip) touching the frame and/or a gesture performed by the external object by using a touch sensor, a grip sensor, and/or a proximity sensor formed on at least a portion of a surface of the frame.

According to one embodiment, the wearable device (1600) may include hardwares (e.g., the hardwares described above based on the block diagram of FIG. 2) that perform various functions. For example, the hardwares may include a battery module (1670), an antenna module (1675), optical devices (1682, 1684), speakers (1692-1, 1692-2), microphones (1694-1, 1694-2, 1694-3), a light-emitting module (not shown), and/or a printed circuit board (1690). The various hardwares may be arranged within the frame.

According to one embodiment, microphones (1694-1, 1694-2, 1694-3) of the wearable device (1600) may be arranged on at least a portion of the frame to acquire a sound signal. A first microphone (1694-1) arranged on the nose pad (1610), a second microphone (1694-2) arranged on the second rim (1602), and a third microphone (1694-3) arranged on the first rim (1601) are illustrated in FIG. 16B , but the number and arrangement of the microphones (1694) are not limited to the embodiment of FIG. 16B . When the number of microphones (1694) included in the wearable device (1600) is two or more, the wearable device (1600) may identify a direction of a sound signal by using a plurality of microphones arranged on different portions of the frame.

In one embodiment, the optical devices (1682, 1684) can transmit virtual objects transmitted from at least one display (1650) to the wave guides (1633, 1634). For example, the optical devices (1682, 1684) can be projectors. The optical devices (1682, 1684) can be positioned adjacent to the at least one display (1650) or can be included within the at least one display (1650) as a part of the at least one display (1650). The first optical device (1682) can correspond to the first display (1650-1), and the second optical device (1684) can correspond to the second display (1650-2). The first optical device (1682) can transmit light output from the first display (1650-1) to the first waveguide (1633), and the second optical device (1684) can transmit light output from the second display (1650-2) to the second waveguide (1634).

In one embodiment, the camera (1640) may include an eye tracking camera (ET CAM) (1640-1), a motion recognition camera (1640-2), and/or a recording camera (1640-3). The recording camera (1640-3), the eye tracking camera (1640-1), and the motion recognition camera (1640-2) may be positioned at different locations on the frame and may perform different functions. The recording camera (1640-3), the eye tracking camera (1640-1), and the motion recognition camera (1640-2) may be examples of the camera (225) of FIG. 2. The eye tracking camera (1640-1) may output data representing a gaze of a user wearing the wearable device (1600). For example, the wearable device (1600) can detect the gaze from an image including the user's pupil obtained through the gaze tracking camera (1640-1). An example in which the gaze tracking camera (1640-1) is positioned toward the user's right eye is illustrated in FIG. 16B, but the embodiment is not limited thereto, and the gaze tracking camera (1640-1) may be positioned solely toward the user's left eye, or toward both eyes.

In one embodiment, the capturing camera (1640-3) can capture an actual image or background to be aligned with a virtual image in order to implement augmented reality or mixed reality content. The capturing camera can capture an image of a specific object existing at a location where a user is looking and provide the image to at least one display (1650). The at least one display (1650) can display a single image in which information about a real image or background including an image of the specific object acquired using the capturing camera and a virtual image provided through optical devices (1682, 1684) are superimposed. In one embodiment, the capturing camera can be placed on a bridge (1603) disposed between the first rim (1601) and the second rim (1602).

In one embodiment, the gaze tracking camera (1640-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (1600) and matching the user's gaze with visual information provided to at least one display (1650). For example, when the wearable device (1600) looks straight ahead, the wearable device (1600) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1650). The gaze tracking camera (1640-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1640-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1640-1) can be placed at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1640-1) may be positioned within the first rim (1601) and/or the second rim (1602) to face the direction in which the user wearing the wearable device (1600) is positioned.

In one embodiment, the gesture recognition camera (1640-2) may provide a specific event on a screen provided on at least one display (1650) by recognizing a movement of the user's entire body or a part of the user's body, such as the user's torso, hands, or face. The gesture recognition camera (1640-2) may recognize a user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide an indication corresponding to the signal on at least one display (1650). The processor may identify a signal corresponding to the gesture, and perform a designated function based on the identification. In one embodiment, the gesture recognition camera (1640-2) may be disposed on the first rim (1601) and/or the second rim (1602).

In one embodiment, the camera (1640) included in the wearable device (1600) is not limited to the gaze tracking camera (1640-1) and the motion recognition camera (1640-2) described above. For example, the wearable device (1600) can identify an external object included in the FoV by using the photographing camera (1640-3) positioned toward the user's FoV. The wearable device (1600) identifying the external object can be performed based on a sensor for identifying the distance between the wearable device (1600) and the external object, such as a depth sensor and/or a time of flight (ToF) sensor. The camera (1640) positioned toward the FoV can support an autofocus function and/or an optical image stabilization (OIS) function. For example, the wearable device (1600) may include a camera (1640) (e.g., a face tracking (FT) camera) positioned toward the face to acquire an image including the face of a user wearing the wearable device (1600).

Although not shown, in one embodiment, the wearable device (1600) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., the user's eyes, face, and/or an external object within the FoV) being imaged using the camera (1640). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (1606, 1607).

In one embodiment, the battery module (1670) can supply power to the electronic components of the wearable device (1600). In one embodiment, the battery module (1670) can be disposed within the first temple (1604) and/or the second temple (1605). For example, the battery module (1670) can be a plurality of battery modules (1670). The plurality of battery modules (1670) can be disposed within each of the first temple (1604) and the second temple (1605). In one embodiment, the battery module (1670) can be disposed at an end of the first temple (1604) and/or the second temple (1605).

In one embodiment, the antenna module (1675) can transmit signals or power to, or receive signals or power from, the wearable device (1600). The antenna module (1675) can be electrically and/or operatively connected to communication circuitry within the wearable device (1600) (e.g., communication circuitry (240) of FIG. 2 ). In one embodiment, the antenna module (1675) can be positioned within the first temple (1604) and/or the second temple (1605). For example, the antenna module (1675) can be positioned proximate one surface of the first temple (1604) and/or the second temple (1605).

In one embodiment, the speakers (1692-1, 1692-2) can output audio signals to the outside of the wearable device (1600). The audio output module may be referred to as a speaker. In one embodiment, the speakers (1692-1, 1692-2) can be positioned within the first temple (1604) and/or the second temple (1605) so as to be positioned adjacent to the ears of a user wearing the wearable device (1600). For example, the wearable device (1600) may include the second speaker (1692-2) positioned within the first temple (1604) and thus adjacent to the user's left ear, and the first speaker (1692-1) positioned within the second temple (1605) and thus adjacent to the user's right ear.

In one embodiment, the light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or may emit light with an action corresponding to a specific state in order to visually provide information about a specific state of the wearable device (1600) to a user. For example, when the wearable device (1600) requires charging, the wearable device (1600) may repeatedly emit red light at a designated time. In one embodiment, the light-emitting module may be disposed on the first rim (1601) and/or the second rim (1602).

Referring to FIG. 16B, according to one embodiment, a wearable device (1600) may include a printed circuit board (PCB) (1690). The PCB (1690) may be included in at least one of the first temple (1604) or the second temple (1605). The PCB (1690) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (1600) (e.g., hardwares illustrated by the blocks described above with reference to FIG. 2) may be positioned on the PCB (1690). The wearable device (1600) may include a flexible PCB (FPCB) for interconnecting the hardwares.

According to one embodiment, a wearable device (1600) may include at least one of a gyro sensor, a gravity sensor, and/or an acceleration sensor for detecting a posture of the wearable device (1600) and/or a posture of a body part (e.g., a head) of a user wearing the wearable device (1600). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and/or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyro sensor may measure an angular velocity about each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1600) may identify a user's motion and/or gesture performed to execute or terminate a specific function of the wearable device (1600) based on the IMU.

FIGS. 17A and 17B illustrate an example of an exterior appearance of a wearable device (1700) according to one embodiment. The wearable device (101) of FIGS. 1A and 1B and FIG. 2 may include the wearable device (1700) of FIGS. 17A and 17B. An example of an exterior appearance of a first side (1710) of a housing of the wearable device (1700) according to one embodiment is illustrated in FIG. 17A, and an example of an exterior appearance of a second side (1720) opposite to the first side (1710) is illustrated in FIG. 17B.

Referring to FIG. 17A, a first surface (1710) of a wearable device (1700) according to one embodiment may have a form attachable on a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1700) may further include a strap for being fixed on a body part of a user, and/or one or more temples (e.g., the first temple (1604) and/or the second temple (1605) of FIGS. 16A and 16B). A first display (1750-1) for outputting an image to a left eye among the user's two eyes, and a second display (1750-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (1710). The wearable device (1700) may be formed on the first surface (1710) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from light radiated from the first display (1750-1) and the second display (1750-2).

According to one embodiment, a wearable device (1700) may include cameras (1740-3, 1740-4) for photographing and/or tracking both eyes of a user adjacent to each of the first display (1750-1) and the second display (1750-2). The cameras (1740-3, 1740-4) may be referred to as ET cameras. According to one embodiment, a wearable device (1700) may include cameras (1740-1, 1740-2) for photographing and/or recognizing a face of a user. The cameras (1740-1, 1740-2) may be referred to as FT cameras.

Referring to FIG. 17b, cameras (e.g., cameras (1740-5, 1740-6, 1740-7, 1740-8, 1740-9, 1740-10)) and/or sensors (e.g., depth sensors (1730)) for obtaining information related to an external environment of the wearable device (1700) may be disposed on a second surface (1720) opposite to the first surface (1710) of FIG. 17a. For example, the cameras (1740-5, 1740-6, 1740-7, 1740-8, 1740-9, 1740-10) may be disposed on the second surface (1720) to recognize external objects different from the wearable device (1700). For example, using cameras (1740-9, 1740-10), the wearable device (1700) can acquire images and/or media to be transmitted to each of the user's two eyes. The camera (1740-9) can be positioned on the second face (1720) of the wearable device (1700) to acquire a frame to be displayed through the second display (1750-2) corresponding to the right eye among the two eyes. The camera (1740-10) can be positioned on the second face (1720) of the wearable device (1700) to acquire a frame to be displayed through the first display (1750-1) corresponding to the left eye among the two eyes.

In one embodiment, the wearable device (1700) may include a depth sensor (1730) disposed on the second surface (1720) to identify a distance between the wearable device (1700) and an external object. Using the depth sensor (1730), the wearable device (1700) may obtain spatial information (e.g., a depth map) for at least a portion of a FoV of a user wearing the wearable device (1700).

Although not shown, a microphone may be placed on the second side (1720) of the wearable device (1700) to acquire sound output from an external object. The number of microphones may be one or more depending on the embodiment.

As described above, according to one embodiment, the wearable device (1700) may have a form factor for being worn on a user's head. The wearable device (1700) may provide a user experience based on a VST mode and/or a VR mode within the head-worn state. Using the first display (1750-1) and the second display (1750-2), the wearable device (1700) may display any one of the screens (131, 132) of FIG. 1A. Using the cameras (1740-5, 1740-6, 1740-7, 1740-8, 1740-9, 1740-10), the wearable device (1700) may identify at least one actual light source in an external space including the wearable device (1700). The wearable device (1700) can perform rendering of a virtual space to be displayed through the first display (1750-1) and/or the second display (1750-2) using a virtual light source corresponding to the actual light source. Based on the rendering, the wearable device (1700) can provide a user experience as if the actual light source recognized through the VST mode has been moved to the VR mode.

In one embodiment, the operation of the wearable device described above may be related to a metaverse service provided through a network. Hereinafter, with reference to FIG. 18, an example of a metaverse service provided to a user based on a wearable device according to one embodiment is described.

Metaverse is a compound word of the English word 'meta' meaning 'virtual' or 'transcendence' and 'universe' meaning universe, and refers to a three-dimensional virtual world where social, economic, and cultural activities like the real world take place. The metaverse is a concept that is one step more advanced than virtual reality (VR, a cutting-edge technology that allows people to have real-life experiences in a computer-created virtual world), and has the characteristic of allowing people to engage in social and cultural activities like real reality, rather than simply enjoying games or virtual reality by utilizing avatars. The metaverse service can provide media content to enhance immersion in the virtual world based on augmented reality (AR), virtual reality environment (VR), mixed environment (MR), and/or extended reality (XR).

For example, media content provided by the metaverse service may include social interaction content including avatar-based games, concerts, parties, and/or conferences. For example, the media content may include advertisements, user created content, and/or information for economic activities such as selling and/or shopping of products. Ownership of the user created content may be proven by a blockchain-based non-fungible token (NFT). The metaverse service may support economic activities based on real currency and/or cryptocurrency. Virtual content linked to the real world, such as a digital twin or life logging, may be provided by the metaverse service.

Figure 18 is an example diagram of a network environment (1801) that provides metaverse services through a server (1810).

Referring to FIG. 18, a network environment (1801) may include a server (1810), a user terminal (1820) (e.g., a first terminal (1820-1) and a second terminal (1820-2)), and a network connecting the server (1810) and the user terminal (1820). Within the network environment (1801), the server (1810) may provide a metaverse service to the user terminal (1820). The network may be formed by at least one intermediate node (1830) including an access point (AP) and/or a base station. The user terminal (1820) may connect to the server (1810) through the network and output a user interface (UI) related to the metaverse service to the user of the user terminal (1820). Based on the above UI, the user terminal (1820) can obtain information to be input from the user to the metaverse service, or output information related to the metaverse service (e.g., multimedia content) to the user.

At this time, the server (1810) provides a virtual space so that the user terminal (1820) can be active in the virtual space. In addition, the user terminal (1820) installs a S/W agent for accessing the virtual space provided by the server (1810) to express information that the server (1810) provides to the user, or transmits information that the user wishes to express in the virtual space to the server. The S/W agent may be provided directly through the server (1810), downloaded from a public server, or embedded and provided when purchasing the terminal.

In one embodiment, the metaverse service may be provided to a user terminal (1820) and/or a user by using a server (1810). The embodiment is not limited thereto, and the metaverse service may be provided through individual contact between users. For example, within a network environment (1801), the metaverse service may be provided independently from the server (1810) by a direct connection between a first terminal (1820-1) and a second terminal (1820-2). Referring to FIG. 18, within a network environment (1801), the first terminal (1820-1) and the second terminal (1820-2) may be connected to each other through a network formed by at least one intermediate node (1830). In one embodiment where the first terminal (1820-1) and the second terminal (1820-2) are directly connected, either of the user terminals of the first terminal (1820-1) and the second terminal (1820-2) may perform the role of the server (1810). For example, a metaverse environment may be configured only by connecting devices (e.g., peer-to-peer (P2P) connections) to each other.

In one embodiment, the user terminal (1820) (or the user terminal (1820) including the first terminal (1820-1) and the second terminal (1820-2)) may be made of various form factors and is characterized by including an output device for providing images and/or sounds to the user and an input device for inputting information to a metaverse service. Examples of various form factors of the user terminal (1820) include a smartphone (e.g., the second terminal (1820-2)), an AR device (e.g., the first terminal (1820-1)), a VR device, an MR device, a VST (video see through) device, an OST (optical see through) device, a smart lens, a smart mirror, an input/output capable TV or projector.

A network (e.g., a network formed by at least one intermediate node (1830)) includes various broadband networks including 3G, 4G, and 5G, and short-range networks including WiFi and BT (e.g., wired networks or wireless networks directly connecting the first terminal (1820-1) and the second terminal (1820-2)). In one embodiment, the user terminal (1820) of FIG. 18 may include a wearable device (101) of FIGS. 1A to 1B , FIGS. 2 to 14 , a wearable device of FIG. 15 , a wearable device (1600) of FIGS. 16A to 16B , and/or a wearable device (1700) of FIGS. 17A to 17B .

In one embodiment, a method for a wearable device to provide VR based on real light sources may be required. As described above, in one embodiment, a wearable device (e.g., the wearable device (101) of FIGS. 1A-1B and 2) may include a camera (e.g., the camera (225) of FIG. 2), a display (e.g., the display (220) of FIG. 2), and a processor (e.g., the processor (210) of FIG. 2). The processor may be configured to acquire information related to at least one real light source using an image acquired from the camera (e.g., images (410, 420) of FIG. 4 and/or image (1430) of FIG. 14). The processor may be configured to receive an input for displaying a virtual space (e.g., the virtual space (140) of FIG. 1A and/or the virtual space (1420) of FIG. 14) on the display while displaying at least a portion of an image acquired from the camera on the display. The processor may be configured to, based on the input, use the information to determine a location of at least one virtual light source (e.g., the virtual light source (150) of FIGS. 1A-1B, the virtual light sources (1031, 1032) of FIGS. 10A-10B, and/or the virtual light source (1120) of FIGS. 11A-11B) in the virtual space. The processor may be configured to display a virtual object to which a visual effect for at least one virtual light source is applied on the display based on a distance between a virtual object included in the virtual space (e.g., virtual objects (161, 162) of FIG. 1A, virtual objects (1040) of FIGS. 10A to 10B, virtual objects (1130, 1140, 1150) of FIGS. 11A to 11B, virtual objects (1350) of FIG. 13, virtual objects (1441, 1442) of FIG. 14) and a location in the virtual space corresponding to the wearable device.

For example, the processor may be configured to perform rendering of the virtual object using the visual effect based on the distance exceeding a specified threshold.

For example, the processor may be configured to identify whether to apply the visual effect to the virtual object based on whether the virtual object falls into a designated category for interaction with a user wearing the wearable device.

For example, the processor may be configured to determine a position of the at least one virtual light source based on the input for switching from a first designated mode for video see-through (VST) to a second designated mode for virtual reality (VR) associated with the virtual space.

For example, the processor may be configured to display a visual object on the display, based on the visual effect, that represents a shadow formed along a direction of virtual light from the location to the virtual object.

For example, the processor may be configured to perform rendering of the virtual object based on at least one of a color or brightness of the at least one real light source included in the information.

For example, the processor may be configured to perform rendering on a boundary surface of the virtual space based on a brightness distribution of the image.

For example, the wearable device may include a sensor (e.g., sensor (230) of FIG. 2). The image may be a first image. The processor may be configured to obtain data indicating a direction of the wearable device from the sensor. The processor may be configured to obtain a second image corresponding to the second direction from the camera, while identifying a direction of the wearable device facing a second direction different from a first direction of the wearable device corresponding to the first image based on the data. The processor may be configured to determine a location of the at least one virtual light source in the virtual space based on a first location of the at least one real light source in the first image and a second location of the at least one real light source in the second image.

According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of acquiring information related to at least one real light source by using an image acquired from a camera of the wearable device. The method may include an operation of receiving an input for displaying a virtual space on a display of the wearable device while displaying at least a portion of the image acquired from the camera on the display of the wearable device. The method may include an operation of determining a position of at least one virtual light source in the virtual space by using the information based on the input. The method may include an operation of displaying a virtual object to which a visual effect for the at least one virtual light source is applied on the display based on a distance between a virtual object included in the virtual space and a position of the virtual space corresponding to the wearable device.

For example, the displaying action may include performing rendering on the virtual object using the visual effect based on the distance exceeding a specified threshold.

For example, the displaying action may include an action of identifying whether to apply the visual effect to the virtual object based on whether the virtual object falls into a designated category for interacting with a user wearing the wearable device.

For example, the identifying action may include determining a position of the at least one virtual light source based on the input for switching from a first designated mode for VST to a second designated mode for VR associated with the virtual space.

For example, the displaying action may include displaying a visual object representing a shadow formed along a direction of virtual light from the location to the virtual object on the display based on the visual effect.

For example, the displaying action may include performing rendering on the virtual object based on at least one of a color or brightness of the at least one real light source included in the information.

For example, the indicated action may include an action of performing rendering on a boundary surface of the virtual space based on a brightness distribution of the image.

For example, the image may be a first image. The acquiring operation may include an operation of acquiring a second image corresponding to the second direction from the camera while identifying a direction of the wearable device facing a second direction different from the first direction of the wearable device using a sensor of the wearable device. The identifying operation may include an operation of determining a position of the at least one virtual light source in the virtual space based on a first position of the at least one real light source in the first image and a second position of the at least one real light source in the second image.

As described above, according to one embodiment, a wearable device (e.g., the wearable device (101) of FIGS. 1A to 1B and FIG. 2) may include a camera (e.g., the camera (225) of FIG. 2), a display (e.g., the display (220) of FIG. 2), and a processor (e.g., the processor (210) of FIG. 2). The processor may be configured to obtain information related to an actual light source disposed in an external space including the wearable device by using the camera. The processor may be configured to identify a first area of a virtual space (e.g., the virtual space (140) of FIG. 1A and/or the virtual space (1420) of FIG. 14) linked to a virtual light source (e.g., the virtual light source (150) of FIGS. 1A to 1B, the virtual light sources (1031, 1032) of FIGS. 10A to 10B, and/or the virtual light source (1120) of FIGS. 11A to 11B) corresponding to the actual light source based on the information. The processor may be configured to perform rendering for a first virtual object (e.g., the virtual object (1130) of FIGS. 11A to 11B) included in the first area among a plurality of virtual objects included in the virtual space, while displaying the virtual space on the display, which is distinct from the external space, based on a visual effect related to the virtual light source. The above processor may be configured to perform rendering on the display, independently of the visual effect, of a second virtual object (e.g., virtual objects (1140, 1150) of FIGS. 11A and 11B) included in a second area different from the first area among the plurality of virtual objects.

For example, the processor may be configured to perform rendering, independently of the visual effects, of the second virtual object included in a designated category for interaction, in the second area spaced a specified distance or less from a reference position in the virtual space.

For example, the processor may be configured to display, based on the visual effect, a first plane of the first virtual object, which faces the virtual light source, with a higher brightness than a second plane that is different from the first plane.

For example, the processor may be configured to display a shadow extending from the first virtual object along a direction from the virtual light source toward the first virtual object based on the visual effect.

As described above, according to one embodiment, the method of the wearable device may include an operation of obtaining information related to an actual light source disposed in an external space including the wearable device by using a camera of the wearable device. The method may include an operation of identifying a first area of a virtual space linked with a virtual light source corresponding to the actual light source based on the information. The method may include an operation of performing rendering for a first virtual object included in the first area among a plurality of virtual objects included in the virtual space, while displaying a virtual space distinct from the external space on a display of the wearable device, based on a visual effect related to the virtual light source. The method may include an operation of performing rendering for a second virtual object included in a second area different from the first area among the plurality of virtual objects on the display, independently of the visual effect.

For example, the operation of performing rendering for the second virtual object may include an operation of performing rendering for the second virtual object included in a designated category for interaction, independently of the visual effect, in the second area spaced apart from a reference position of the virtual space by a designated distance or less.

For example, the operation of performing rendering for the first virtual object may include an operation of displaying, based on the visual effect, a first plane of the first virtual object facing the virtual light source, with a brightness higher than that of a second plane that is different from the first plane.

For example, the act of performing rendering on the first virtual object may include an act of displaying a shadow extended from the first virtual object along a direction from the virtual light source toward the first virtual object based on the visual effect.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, the processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may be one that continuously stores a program executable by a computer, or one that temporarily stores it for execution or downloading. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, and may also be distributed on a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, and flash memories configured to store program commands. In addition, examples of other media may include an app store that distributes applications, a site that supplies or distributes various software, and a recording or storage medium managed by a server.

Although the embodiments have been described above by way of limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and/or components such as the described systems, structures, devices, and circuits are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

Claims (20)

In a wearable device (101),
Camera (225);
display (220); and
It comprises a processor (210), and the processor comprises:
Using the images (410, 420; 1430) acquired from the above camera, information related to at least one actual light source is acquired;
While displaying at least a portion of an image acquired from the camera on the display, receiving an input for displaying a virtual space (140; 1420) on the display;
Based on the above input, using the above information, determining the position of at least one virtual light source (150; 1031, 1032; 1120) in the virtual space; and
A virtual object (161, 162; 1040; 1130, 1140, 1150; 1350; 1441, 1442) included in the virtual space and a location of the virtual space corresponding to the wearable device, configured to display the virtual object to which a visual effect for at least one virtual light source is applied on the display.
Wearable devices.
In claim 1, the processor,
configured to perform rendering on the virtual object using the visual effect based on the distance exceeding the specified threshold.
Wearable devices.
In claim 1, the processor,
configured to identify whether to apply the visual effect to the virtual object based on whether the virtual object is included in a designated category for interaction with a user wearing the wearable device.
Wearable devices.
In claim 1, the processor,
configured to determine the position of the at least one virtual light source based on the input for switching from a first designated mode for VST (video see-through) to a second designated mode for VR (virtual reality) related to the virtual space.
Wearable devices.
In claim 1, the processor,
A visual object configured to display a visual object representing a shadow formed along the direction of virtual light from the location to the virtual object on the display based on the visual effect described above.
Wearable devices.
In claim 1, the processor,
configured to perform rendering of the virtual object based on at least one of the color or brightness of the at least one real light source included in the above information;
Wearable devices.
In claim 1, the processor,
configured to perform rendering on the boundary surface of the virtual space based on the brightness distribution of the image.
Wearable devices.
In claim 1, a sensor (230) is further included,
The above image is the first image,
The above processor,
Obtaining data indicating the direction of the wearable device from the above sensor,
Based on the above data, a direction of the wearable device facing a second direction different from a first direction of the wearable device corresponding to the first image is identified, and a second image corresponding to the second direction is acquired from the camera;
configured to determine a position of the at least one virtual light source in the virtual space based on a first position of the at least one real light source in the first image, and a second position of the at least one real light source in the second image;
Wearable devices.
In a method of a wearable device,
An operation of obtaining information related to at least one actual light source using an image obtained from a camera of the wearable device;
An action of receiving an input for displaying a virtual space on the display while displaying at least a portion of an image acquired from the camera on the display of the wearable device;
An operation of determining a position of at least one virtual light source in the virtual space based on the input and using the information; and
An operation of displaying a virtual object to which a visual effect for at least one virtual light source is applied on the display based on a distance between a virtual object included in the virtual space and a location of the virtual space corresponding to the wearable device,
method.
In claim 9, the indicated action is:
An action comprising performing rendering on the virtual object using the visual effect based on the distance exceeding the specified threshold.
method.
In claim 9, the indicated action is:
An operation for identifying whether to apply the visual effect to the virtual object based on whether the virtual object falls into a designated category for interacting with a user wearing the wearable device.
method.
In claim 9, the identifying action comprises:
An operation for determining a position of at least one virtual light source based on the input for switching from a first designated mode for VST to a second designated mode for VR associated with the virtual space,
method.
In claim 9, the indicated action is:
An action including displaying a visual object representing a shadow formed along the direction of virtual light from the location to the virtual object on the display based on the visual effect.
method.
In claim 9, the indicated action is:
An operation for performing rendering on the virtual object based on at least one of a color or brightness of the at least one real light source included in the above information,
method.
In claim 9, the indicated action is:
An operation for performing rendering on a boundary surface of the virtual space based on the brightness distribution of the image,
method.
In claim 9, the image is a first image;
The above acquisition action is,
An operation of identifying a direction of the wearable device facing a second direction different from a first direction of the wearable device by using a sensor of the wearable device, and acquiring a second image corresponding to the second direction from the camera,
The above identifying action is,
An operation of identifying a position of the at least one virtual light source in the virtual space based on a first position of the at least one real light source in the first image and a second position of the at least one real light source in the second image,
method.
In a wearable device (101),
Camera (225);
display (220); and
It comprises a processor (210), and the processor comprises:
Using the above camera, information related to an actual light source placed in an external space including the wearable device is acquired;
Based on the above information, a first area of a virtual space (140; 1420) linked to a virtual light source (150; 1031, 1032; 1120) corresponding to the actual light source is identified;
In the above display, in a state where a virtual space distinct from the external space is displayed, among a plurality of virtual objects included in the virtual space, rendering is performed for a first virtual object (1130) included in the first area based on a visual effect related to the virtual light source; and
In the above display, the rendering of a second virtual object (1140, 1150) included in a second area different from the first area among the plurality of virtual objects is performed independently from the visual effect.
Wearable devices.
In claim 17, the processor,
In the second area spaced apart from the reference position of the virtual space by a specified distance or less, rendering of the second virtual object included in the specified category for interaction is performed independently of the visual effect.
Wearable devices.
In claim 17, the processor,
Based on the above visual effect, among the different planes of the first virtual object, a first plane facing the virtual light source is configured to be displayed with a brightness higher than that of a second plane different from the first plane.
Wearable devices.
In claim 17, the processor,
Based on the above visual effect, configured to display a shadow extending from the first virtual object along a direction from the virtual light source to the first virtual object,
Wearable devices.

Images