KR102858186B1 - Camera module - Google Patents

Abstract

A camera module according to an embodiment of the present invention includes a light emitting unit that outputs an optical signal to an object; a light receiving unit that collects an optical signal output from the light emitting unit and reflected by the object; a sensor unit that receives the optical signal collected by the light receiving unit; and a control unit that controls at least one of the light emitting unit and the sensor unit to shift at least one of an output phase of the light emitting unit and a reception phase of the sensor unit.

Description

Camera Module {CAMERA MODULE}

The embodiment relates to a camera module.

3D content is being applied in many fields, including gaming, culture, education, manufacturing, and autonomous driving. Depth information (depth maps) are required to acquire 3D content. Depth information represents spatial distance and provides perspective information for one point in a 2D image relative to another.

Methods for acquiring depth information include projecting IR (Infrared) structured light onto an object, using stereo cameras, and the Time of Flight (TOF) method. According to the TOF method, the distance to an object is calculated by measuring the time of flight, or the time it takes for light to be emitted and reflected. The greatest advantage of the ToF method is that it provides distance information for three-dimensional space quickly and in real time. Furthermore, users can obtain accurate distance information without applying separate algorithms or hardware corrections. Furthermore, accurate depth information can be obtained even when measuring very close or moving subjects.

Accordingly, there are attempts to utilize the TOF method for biometric authentication. For example, it is known that the shape of the veins on the fingers, etc., does not change from fetal age throughout life and varies from person to person. Therefore, vein patterns can be identified using a camera device equipped with TOF functionality. To do this, after photographing a finger, the background can be removed based on the color and shape of the finger to detect each finger. The vein pattern for each finger can be extracted from the color information of each finger. Specifically, the average finger color, the color of the veins distributed on the finger, and the color of the wrinkles on the finger can differ. For example, the color of the veins distributed on the finger may be less red than the average finger color, and the color of the wrinkles on the finger may be darker than the average finger color. These characteristics can be used to calculate a value approximating the veins for each pixel, and the calculated results can be used to extract the vein pattern. Furthermore, the extracted vein pattern for each finger can be compared with pre-registered data to identify the individual.

However, camera devices equipped with TOF functionality require additional light sources, resulting in significant power consumption. This issue can be particularly pronounced in devices with limited battery capacity, such as Shift devices. Therefore, the development of ToF camera devices that can improve power efficiency is required.

The technical problem to be solved by the present invention is to provide a camera module with high power efficiency.

A camera module according to an embodiment of the present invention includes a light emitting unit that outputs an optical signal to an object; a light receiving unit that collects an optical signal output from the light emitting unit and reflected by the object; a sensor unit that receives the optical signal collected by the light receiving unit; and a control unit that controls at least one of the light emitting unit and the sensor unit to shift at least one of an output phase of the light emitting unit and a reception phase of the sensor unit.

The duty ratio of the above optical signal can be set within a range greater than 0% and less than 25%.

The control unit controls the light emitting unit or the sensor unit according to a plurality of sequences, and the plurality of sequences may include a first sequence corresponding to a reference phase and a second sequence shifted by a predetermined phase from the reference phase.

Each of the above plurality of sequences can correspond to a different phase.

The exposure period of the optical signal corresponding to the first sequence may be shorter than the exposure period of the optical signal corresponding to the second sequence.

The control unit can set the phase of the second sequence based on the image signal generated corresponding to the first sequence.

The above control unit can be configured to perform each of the plurality of sequences during an exposure cycle corresponding to one image frame.

The control unit can control the sensor unit to shift the phase of the demodulation signal for the optical signal for each pixel of the sensor unit based on the image signal generated in response to the first sequence.

It may further include an image processing unit that generates a depth image using a plurality of image signals corresponding to the plurality of sequences.

According to one embodiment of the present invention, the power efficiency of a camera module can be increased.

Additionally, it can measure accurate distance information about objects even when using little power.

Figure 1 is a configuration diagram of a camera module according to an embodiment of the present invention.
Figure 2 is a configuration diagram of a light emitting unit according to an embodiment of the present invention.
Fig. 3 is a drawing for explaining a light receiving unit according to an embodiment of the present invention.
Figure 4 is a drawing for explaining a sensor unit according to an embodiment of the present invention.
FIG. 5 is a drawing for explaining an electric signal generation process according to an embodiment of the present invention.
FIG. 6 is a drawing for explaining a sub-frame image according to an embodiment of the present invention.
FIG. 7 is a drawing for explaining a depth image according to an embodiment of the present invention.
FIG. 8 is a drawing for explaining a ToF IR image according to an embodiment of the present invention.
Fig. 9 is a drawing for explaining an optical signal of a light emitting unit according to an embodiment of the present invention.
FIG. 10 is a diagram showing a measurable phase region according to an optical signal of a light emitting unit according to an embodiment of the present invention.
Fig. 11 is a drawing for explaining the output phase shift of a light emitting unit according to an embodiment of the present invention.
Fig. 12 is a drawing for explaining the reception phase shift of the sensor unit according to an embodiment of the present invention.
Fig. 13 is a drawing for explaining a phase-specific measurable area according to an embodiment of the present invention.
FIG. 14 is a drawing for explaining one embodiment of shift phase setting of a control unit according to an embodiment of the present invention.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

However, the technical idea of the present invention is not limited to some of the embodiments described, but can be implemented in various different forms, and within the scope of the technical idea of the present invention, one or more of the components between the embodiments can be selectively combined or substituted for use.

In addition, terms (including technical and scientific terms) used in the embodiments of the present invention may be interpreted as having a meaning that can be generally understood by a person of ordinary skill in the technical field to which the present invention belongs, unless explicitly and specifically defined and described, and terms that are commonly used, such as terms defined in a dictionary, may be interpreted in consideration of the contextual meaning of the relevant technology.

Additionally, the terms used in the embodiments of the present invention are intended to describe the embodiments and are not intended to limit the present invention.

In this specification, the singular may also include the plural unless specifically stated otherwise in the phrase, and when it is described as “A and/or at least one (or more) of B, C”, it may include one or more of all combinations that can be combined with A, B, C.

Additionally, in describing components of embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used.

These terms are intended only to distinguish one component from another, and are not intended to limit the nature, order, or sequence of the component.

And, when a component is described as being 'connected', 'coupled' or 'connected' to another component, it may include not only cases where the component is directly connected, coupled or connected to the other component, but also cases where the component is 'connected', 'coupled' or 'connected' by another component between the component and the other component.

Additionally, when described as being formed or arranged "above or below" each component, "above" or "below" includes not only cases where the two components are in direct contact with each other, but also cases where one or more other components are formed or arranged between the two components. Furthermore, when expressed as "above" or "below", it can include the meaning of a downward direction as well as an upward direction based on one component.

Figure 1 is a configuration diagram of a camera module according to an embodiment of the present invention.

The camera module (100) according to an embodiment of the present invention may be named a camera device, a ToF (Time of Flight) camera module, a ToF camera device, etc.

The camera module (100) according to an embodiment of the present invention may be included in an optical device. The optical device may include any one of a mobile phone, a cell phone, a smart phone, a portable smart device, a digital camera, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), and a navigation device. However, the type of the optical device is not limited thereto, and any device for taking images or photographs may be included in the optical device.

Referring to FIG. 1, a camera module (100) according to an embodiment of the present invention includes a light emitting unit (110), a light receiving unit (120), a sensor unit (130), and a control unit (140), and may further include an image processing unit (150) and a tilt unit (160).

The light emitting unit (110) may be a light emitting module, a light emitting unit, a light emitting assembly, or a light emitting device. The light emitting unit (110) may output an optical signal. The light emitting unit (110) may generate an optical signal and then output, i.e., irradiate, the optical signal to an object. At this time, the light emitting unit (110) may generate and output an optical signal in the form of a pulse wave or a continuous wave. The continuous wave may be in the form of a sinusoid wave or a square wave. In the present specification, the optical signal output by the light emitting unit (110) may refer to an optical signal incident on an object. The optical signal output by the light emitting unit (110) may be referred to as output light, an output optical signal, etc. with respect to the camera module (100). The light output by the light emitting unit (110) may be referred to as incident light, an incident optical signal, etc. with respect to the object.

The light emitting unit (110) can output, or irradiate, light to an object for a predetermined exposure cycle (integration time). Here, the exposure cycle may mean one frame cycle, or one image frame cycle. When generating multiple frames, the set exposure cycle is repeated. For example, when the camera module (100) captures an object at 20 FPS, the exposure cycle becomes 1/20 [sec]. And when generating 100 frames, the exposure cycle can be repeated 100 times.

The light emitting unit (110) can output multiple optical signals having different frequencies. The light emitting unit (110) can sequentially and repeatedly output multiple optical signals having different frequencies. Alternatively, the light emitting unit (110) can simultaneously output multiple optical signals having different frequencies.

The light emitting unit (110) can set the duty ratio of the optical signal within a preset range. According to an embodiment of the present invention, the duty ratio of the optical signal output by the light emitting unit (110) can be set within a range greater than 0% and less than 25%. For example, the duty ratio of the optical signal can be set to 10% or 20%. The duty ratio of the optical signal can be preset or set by the control unit (140).

The light receiving unit (120) may be a light receiving module, a light receiving unit, a light receiving assembly, or a light receiving device. The light receiving unit (120) may collect an optical signal output from the light emitting unit (110) and reflected from an object. The light receiving unit (120) may be arranged parallel to the light emitting unit (110). The light receiving unit (120) may be arranged next to the light emitting unit (110). The light receiving unit (120) may be arranged in the same direction as the light emitting unit (110). The light receiving unit (120) may include a filter for passing an optical signal reflected from an object.

In this specification, the light signal collected by the light receiving unit (120) may refer to a light signal reflected after the light signal output from the light emitting unit (110) reaches the object. The light signal collected by the light receiving unit (120) may be referred to as input light, input light signal, etc. based on the camera module (100). The light output by the light receiving unit (120) may be referred to as reflected light, reflected light signal, etc. based on the object.

The sensor unit (130) can sense the optical signal collected by the light receiving unit (120). The sensor unit (130) may be an image sensor that senses the optical signal. The sensor unit (130) may be used interchangeably with a sensor, an image sensor, an image sensor unit, a ToF sensor, a ToF image sensor, a ToF image sensor unit, etc.

The sensor unit (130) can detect light and generate an electrical signal. That is, the sensor unit (130) can generate an electrical signal through the light signal collected by the light receiving unit (120). The generated electrical signal may be in analog form. The sensor unit (130) can generate an image signal based on the generated electrical signal and transmit the generated image signal to the image processing unit (150). At this time, the image signal may be an analog electrical signal or a signal obtained by converting an analog electrical signal into a digital signal. When transmitting an analog electrical signal as an image signal, the image processing unit (150) can convert the electrical signal into a digital signal through a device such as an analog-digital converter (ADC).

The sensor unit (130) can detect light having a wavelength corresponding to the wavelength of light output from the light emitting unit (110). For example, the sensor unit (130) can detect infrared rays. Alternatively, the sensor unit (130) can detect visible light.

The sensor unit (130) may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. In addition, the sensor unit (130) may include a ToF sensor that receives an infrared light signal reflected from a subject and then measures the distance using a time or phase difference.

The control unit (140) can control each component included in the camera module (100).

According to an embodiment of the present invention, the control unit (140) can control at least one of the light emitting unit (110) and the sensor unit (130). According to one embodiment of the present invention, the control unit (140) can control at least one of the light emitting unit and the sensor unit to shift at least one of the output phase of the light emitting unit and the reception phase of the sensor unit. For example, the control unit (140) can control the light emitting unit (110) to shift the output phase of an optical signal according to a plurality of sequences. As another example, the control unit (140) can control the sensor unit (130) to shift the reception phase of an optical signal according to a plurality of sequences. As another example, the control unit (140) can control the light emitting unit (110) and the sensor unit (130) to shift the output phase of an optical signal and the reception phase of an optical signal according to a plurality of sequences.

In addition, the control unit (140) can control the sensing cycle of the light signal collected by the light receiving unit (120) of the sensor unit (130) in conjunction with the exposure cycle of the light emitting unit (110).

Additionally, the control unit (140) can control the tilt unit (160). For example, the control unit (140) can control the tilt operation of the tilt unit (160) according to a predetermined rule.

The image processing unit (150) can receive an image signal from the sensor unit (130) and process the image signal (e.g., digital conversion, interpolation, frame synthesis, etc.) to generate an image.

According to one embodiment, the image processing unit (150) can synthesize a single frame (high resolution) using a plurality of frames (low resolution). That is, the image processing unit (150) can synthesize a plurality of image frames corresponding to an image signal received from the sensor unit (130) and generate a synthesized result as a composite image. The composite image generated by the image processing unit (150) can have a higher resolution than the plurality of image frames corresponding to the image signal. That is, the image processing unit (150) can generate a high-resolution image through a super resolution (SR) technique.

The image processing unit (150) may include a processor that processes an image signal to generate an image. Multiple processors may be implemented depending on the function of the image processing unit (150), and some of the multiple processors may be implemented in conjunction with the sensor unit (130). For example, a processor that converts an analog electrical signal into a digital image signal may be implemented in conjunction with a sensor. As another example, the multiple processors included in the image processing unit (150) may be implemented separately from the sensor unit (130).

The tilt unit (160) can tilt at least one of the filters and lenses so that the optical path of light passing through at least one of the filters and lenses repeatedly shifts according to a predetermined rule. To this end, the tilt unit (160) can include a tilt driver and a tilt actuator.

The lens may be a variable lens capable of changing the optical path. The variable lens may be a focus-variable lens. Additionally, the variable lens may be a lens with an adjustable focus. The variable lens may be at least one of a liquid lens, a polymer lens, a liquid crystal lens, a VCM type, and an SMA type. The liquid lens may include a liquid lens containing one liquid and a liquid lens containing two liquids. The liquid lens containing one liquid can vary its focus by adjusting a membrane positioned corresponding to the liquid, and for example, the focus can be varied by pressurizing the membrane using the electromagnetic force of a magnet and a coil. The liquid lens containing two liquids can include a conductive liquid and a non-conductive liquid and control the interface formed between the conductive liquid and the non-conductive liquid using a voltage applied to the liquid lens. The polymer lens can vary its focus by using a driving unit such as a piezoelectric element to actuate the polymer material. The liquid crystal lens can vary its focus by controlling the liquid crystal using an electromagnetic force. The VCM type can vary the focus by controlling the electromagnetic force between a magnet and a coil of a solid lens or a lens assembly including a solid lens. The SMA type can vary the focus by controlling the solid lens or a lens assembly including a solid lens using a shape memory alloy.

The tilt unit (160) can tilt at least one of the filter and the lens so that the path of light passing through the filter after tilting is shifted by a unit that is greater than 0 pixel and less than 1 pixel of the sensor unit (130) based on the path of light passing through at least one of the filter and the lens before tilting. The tilt unit (160) can tilt at least one of the filter and the lens so that the path of light passing through at least one of the filter and the lens is shifted at least once from a preset reference path.

Hereinafter, with reference to the drawings, each configuration of the camera module (100) according to the embodiment of the present invention illustrated in FIG. 1 will be examined in detail.

Figure 2 is a configuration diagram of a light emitting unit according to an embodiment of the present invention.

As previously discussed with reference to FIG. 1, the light emitting unit (110) may refer to a component that generates an optical signal and then outputs it to an object. To implement this function, the light emitting unit (110) may include a light emitting element (111) and an optical element (112), and may include a light modulation unit (112).

First, the light-emitting element (111) may refer to an element that receives electricity and generates light. The light generated by the light-emitting element (111) may be infrared rays with a wavelength of 770 to 3000 nm. Alternatively, the light generated by the light-emitting element (111) may be visible light with a wavelength of 380 to 770 nm.

The light emitting element (111) may include a light emitting diode (LED). In addition, the light emitting element (111) may include an organic light emitting diode (OLED) or a laser diode (LD).

The light emitting element (111) can be implemented in a form arranged according to a certain pattern. Accordingly, the light emitting element (111) can be configured in plurality. The plurality of light emitting elements (111) can be arranged according to rows and columns on the substrate. The plurality of light emitting elements (111) can be mounted on the substrate. The substrate can be a printed circuit board (PCB) on which a circuit pattern is formed. The substrate can be implemented as a flexible printed circuit board (FPCB) to ensure a certain flexibility. In addition, the substrate can be implemented as any one of a resin-based printed circuit board, a metal core PCB, a ceramic PCB, and an FR-4 substrate. In addition, the plurality of light emitting elements (111) can also be implemented in the form of a chip.

The optical modulation unit (112) can control the blinking of the light-emitting element (111) so that the light-emitting element (111) generates an optical signal in the form of a continuous wave or pulse wave. The optical modulation unit (112) can control the light-emitting element (111) to generate light in the form of a continuous wave or pulse wave through frequency modulation, pulse modulation, or the like. For example, the optical modulation unit (112) can control the light-emitting element (111) to generate light in the form of a pulse wave or continuous wave by repeating the blinking (on/off) of the light-emitting element (111) at regular time intervals. The regular time interval may be the frequency of the optical signal.

Fig. 3 is a drawing for explaining a light receiving unit according to an embodiment of the present invention.

Referring to FIG. 3, the light receiving unit (120) includes a lens assembly (121) and a filter (125). The lens assembly (121) may include a lens (122), a lens barrel (123), and a lens holder (124).

The lens (122) may be composed of multiple lenses, or may be composed of a single lens. The lens (122) may include the variable lens described above. When the lens (122) is composed of multiple lenses, each lens may be aligned with respect to a central axis to form an optical system. Here, the central axis may be identical to the optical axis of the optical system.

The lens barrel (123) is coupled with a lens holder (124), and a space for accommodating a lens may be provided inside. The lens barrel (123) may be rotatably coupled with one or more lenses, but this is exemplary, and the lens barrel may be coupled in other ways, such as using an adhesive (e.g., an adhesive resin such as epoxy).

The lens holder (124) is coupled with the lens barrel (123) to support the lens barrel (123) and can be coupled to a printed circuit board (126) on which a sensor (130) is mounted. Here, the sensor may correspond to the sensor unit (130) of FIG. 1. A space to which a filter (125) can be attached may be formed at the bottom of the lens barrel (123) by the lens holder (124). A spiral pattern may be formed on the inner surface of the lens holder (124), and similarly, the lens holder (124) may be rotationally coupled with the lens barrel (123) having a spiral pattern formed on the outer surface. However, this is exemplary, and the lens holder (124) and the lens barrel (123) may be coupled using an adhesive, or the lens holder (124) and the lens barrel (123) may be formed as an integral part.

The lens holder (124) can be divided into an upper holder (124-1) coupled with a lens barrel (123) and a lower holder (124-2) coupled with a printed circuit board (126) on which a sensor (130) is mounted. The upper holder (124-1) and the lower holder (124-2) may be formed as an integral body, or may be formed as separate structures and then fastened or coupled, or may have a structure in which they are separated and spaced apart from each other. In this case, the diameter of the upper holder (124-1) may be formed to be smaller than the diameter of the lower holder (124-2).

The filter (125) can be coupled to the lens holder (124). The filter (125) can be positioned between the lens assembly (121) and the sensor. The filter (125) can be positioned on the optical path between the object and the sensor. The filter (125) can filter light having a predetermined wavelength range. The filter (125) can pass light of a specific wavelength. That is, the filter (125) can block light other than the specific wavelength by reflecting or absorbing it. The filter (125) can pass infrared rays and block light of wavelengths other than infrared rays. Alternatively, the filter (125) can pass visible light and block light of wavelengths other than visible light. The filter (125) can be shifted. The filter (125) can be shifted integrally with the lens holder (124). The filter (125) can be tilted. The filter (125) can be shifted to adjust the optical path. The filter (125) can change the path of light incident on the sensor unit (130) through shifting. The filter (125) can change the FOV (Field of View) angle of the incident light or the direction of the FOV.

Although not shown in FIG. 3, the image processing unit (150) and the like may be implemented within a printed circuit board. In addition, the light emitting unit (110) of FIG. 1 may be placed on the side of the sensor (130) on the printed circuit board (126), or may be placed outside the camera module (100), for example, on the side of the camera module (100).

The above example is only one embodiment, and the light receiving unit (120) may be configured with another structure capable of collecting light incident on the camera module (100) and transmitting it to the sensor.

Figure 4 is a drawing for explaining a sensor unit according to an embodiment of the present invention.

As illustrated in FIG. 4, the sensor unit (130) according to an embodiment of the present invention may have a plurality of cell areas (P1, P2, ...) arranged in a grid shape. For example, as illustrated in FIG. 4, a sensor unit (130) with a resolution of 320x240 may have 76,800 cell areas arranged in a grid shape.

And, a certain gap (L) can be formed between each cell area, and wires or the like for electrically connecting multiple cells can be placed in the gap (L). The width (dL) of this gap (L) can be very small compared to the width of the cell area.

The cell region (P1, P2, …) may refer to a region that converts an input light signal into electrical energy. That is, the cell region (P1, P2, …) may refer to a cell region equipped with a photodiode that converts light into electrical energy, or may refer to a cell region in which the equipped photodiode operates.

According to one embodiment, a plurality of cell regions (P1, P2, …) may be provided with two photodiodes. Each cell region (P1, P2, …) may include a first light receiving unit (132-1) including a first photodiode and a first transistor, and a second light receiving unit (132-2) including a second photodiode and a second transistor.

The first light receiving unit (132-1) and the second light receiving unit (132-2) can receive an optical signal with a phase difference of 180 degrees from each other. That is, when the first photodiode is turned on, absorbs an optical signal, and then turns off, the second photodiode can be turned on, absorbs an optical signal, and then turns off. The first light receiving unit (132-1) can be referred to as an In Phase receiving unit, and the second light receiving unit (132-2) can be referred to as an Out Phase receiving unit. In this way, when the first light receiving unit (132-1) and the second light receiving unit (132-2) are activated with a time difference, a difference occurs in the amount of light received depending on the distance from the object. For example, when an object is located directly in front of the camera module (100) (i.e., when the distance = 0), the time it takes for the light signal to be reflected from the object after being output from the light emitting unit (110) is 0, so the blinking cycle of the light source becomes the light reception cycle. Accordingly, only the first light receiving unit (132-1) receives light, and the second light receiving unit (132-2) does not receive light. As another example, when an object is located a predetermined distance away from the camera module (100), it takes time for the light to be reflected from the object after being output from the light emitting unit (110), so the blinking cycle of the light source becomes different from the light reception cycle. Accordingly, a difference occurs in the amount of light received by the first light receiving unit (132-1) and the second light receiving unit (132-2). That is, the distance to the object can be calculated using the difference in the amount of light input to the first light receiving unit (132-1) and the second light receiving unit (132-2).

FIG. 5 is a drawing for explaining an electric signal generation process according to an embodiment of the present invention.

As shown in FIG. 5, there may be four reference signals (C ₁ to C ₄ ), that is, demodulation signals, according to an embodiment of the present invention. Each demodulation signal (C ₁ to C ₄ ) may have the same frequency as the output light (the light output by the light-emitting unit (110)), that is, the incident light from the object's perspective, but may have a 90 degree phase difference with each other. One of the four demodulation signals (C ₁ ) may have the same phase as the output light. The input light (the light input by the light-receiving unit (120)), that is, the reflected light from the object's perspective, is delayed in phase by the distance that the output light reflects back after being incident on the object. The sensor unit (130) mixes the input light and each demodulation signal. Then, the sensor unit (130) can generate an electric signal corresponding to the shaded portion of FIG. 3 for each demodulation signal. The electrical signal generated for each demodulation signal may be transmitted to the image processing unit (150) as an image signal, or the digitally converted electrical signal may be transmitted to the image processing unit (150) as an image signal.

In another embodiment, when output light is generated with multiple frequencies during the exposure time, the sensor absorbs input light according to the multiple frequencies. For example, assume that output light is generated with frequencies f ₁ and f ₂ , and multiple demodulation signals have a phase difference of 90 degrees. Then, since the incident light also has frequencies f ₁ and f ₂ , four electrical signals can be generated through the input light with frequency f ₁ and the four demodulation signals corresponding thereto. In addition, four electrical signals can be generated through the input light with frequency f ₂ and the four demodulation signals corresponding thereto. Therefore, a total of eight electrical signals can be generated.

FIG. 6 is a drawing for explaining a sub-frame image according to an embodiment of the present invention.

As previously discussed, electrical signals can be generated corresponding to the phases of the four demodulation signals. Therefore, as illustrated in FIG. 6, the image processing unit (150) can obtain sub-frame images corresponding to the four phases. Here, the four phases can be 0°, 90°, 180°, and 270°, and the sub-frame images can be used interchangeably with phase images, phase IR images, etc.

Additionally, the image processing unit (150) can generate a depth image based on multiple sub-frame images.

FIG. 7 is a drawing for explaining a depth image according to an embodiment of the present invention.

The depth image of Fig. 7 represents an image generated based on the sub-frame image of Fig. 4. The image processing unit (150) can generate a depth image using multiple sub-frame images, and this can be implemented through the following mathematical expressions 1 and 2.

Here, Raq(x ₀ ) refers to a sub-frame image corresponding to a 0-degree phase. Raq(x ₉₀ ) refers to a sub-frame image corresponding to a 90-degree phase. Raq(x ₁₈₀ ) refers to a sub-frame image corresponding to a 180-degree phase. Raq(x ₂₇₀ ) refers to a sub-frame image corresponding to a 270-degree phase.

That is, the image processing unit (150) can calculate the phase difference between the light signal output by the light emitting unit (110) and the light signal input by the light receiving unit (120) for each pixel through mathematical expression 1.

Here, f represents the frequency of the optical signal. c represents the speed of light.

That is, the image processing unit (150) can calculate the distance between the camera module (100) and the object for each pixel through mathematical expression 2.

Meanwhile, the image processing unit (150) may also generate a ToF IR image based on multiple sub-frame images.

FIG. 8 is a drawing for explaining a ToF IR image according to an embodiment of the present invention.

Figure 8 shows an amplitude image, which is a type of ToF IR image generated through the four sub-frame images of Figure 4.

In order to generate an amplitude image as in Fig. 8, the image processing unit (150) can use the following mathematical expression 3.

As another example, the image processing unit (150) can generate an intensity image, which is a type of ToF IR image, using the following mathematical expression 4. The intensity image can be used interchangeably with a confidence image.

ToF IR images, such as amplitude images or intensity images, can be grayscale images.

Fig. 9 is a drawing for explaining an optical signal of a light emitting unit according to an embodiment of the present invention.

The light emitting unit (110) can output an optical signal having a predetermined duty ratio. The light emitting unit according to an embodiment of the present invention can output an optical signal having a duty ratio in a range greater than 0% and less than 25%.

Figure 9 illustrates the light power of an optical signal at duty ratios of 25% and 10%. As illustrated in Figure 9, the lower the duty ratio, the shorter the time for which light is output during one exposure cycle. A lower duty ratio has the advantage of reducing power consumption because the time for which light is output is shorter. However, the shorter time for which light is output may result in sections where distance measurement is impossible.

FIG. 10 is a diagram showing a measurable phase region according to an optical signal of a light emitting unit according to an embodiment of the present invention.

Fig. 10 (a) is a graph showing a normalized signal value corresponding to the actual phase, and Fig. 10 (b) shows a measured phase value corresponding to the actual phase by calculating the arc tangent (tan ^-1 ) of Fig. 10 (a). As shown in Fig. 10 (a) and (b), when the duty ratio of the optical signal is 25% or more (e.g., in the ideal case, when the duty ratio is 50%, when the duty ratio is 25%), the actual phase and the measured phase have a 1:1 correspondence, so it can be seen that distance measurement to an object is possible in the entire phase region of the exposure cycle.

However, when the duty ratio is less than 25% (e.g., when the duty ratio is 20%, when the duty ratio is 10%), there may be a period where the actual phase and the measured phase do not have a 1:1 correspondence. In other words, it may be impossible to measure the distance to an object in some phase regions among the entire phase region of the exposure cycle.

Therefore, in order to reduce the duty cycle of the optical signal and at the same time improve the accuracy of measuring the distance to the object, the camera module (100) according to the embodiment of the present invention shifts the output phase of the optical signal output by the light emitting unit (110) for each frame according to a plurality of sequences or shifts the reception phase of the sensor unit (130) so that the distance to the object can be measured for the entire phase region of the exposure cycle.

Fig. 11 is a drawing for explaining the output phase shift of a light emitting unit according to an embodiment of the present invention.

Referring to FIG. 11, the control unit (140) can control the light emitting unit (110) to shift the output phase of the optical signal according to a plurality of sequences. For example, the control unit (140) can control the light emitting unit (110) to generate an optical signal of a reference phase (a portion illustrated by a solid line in FIG. 11) according to a first sequence in a first exposure cycle. Then, the control unit (140) can control the light emitting unit (110) to generate an optical signal of an output phase (a portion illustrated by a dotted line in FIG. 11) whose phase is shifted by a predetermined phase from the reference phase according to a second sequence in a second exposure cycle. At this time, the second exposure cycle may mean an exposure cycle following the first exposure cycle. When the plurality of sequences include the first sequence and the second sequence, the control unit (140) can control the light emitting unit (110) according to the second sequence and then control the light emitting unit (110) according to the first sequence in a third exposure cycle. That is, multiple sequences can be repeated sequentially.

Table 1 is a table showing the phase of an optical signal when multiple sequences include the first to second-2 sequences.

Phase of optical signal First sequence 0 Sequence 2-1 0.16π Sequence 2-2 0.32π

According to Table 1, the control unit (140) can control the light emitting unit (110) to generate an optical signal of phase 0 (reference phase) according to the first sequence, and then control the light emitting unit (110) to generate an optical signal whose phase is shifted by 0.16π according to the 2-1 sequence. In addition, the control unit (140) can control the light emitting unit (110) to generate an optical signal whose phase is shifted by 0.16π compared to the 2-1 sequence and whose phase is shifted by 0.32π compared to the 1st sequence according to the 2-2 sequence.

Fig. 12 is a drawing for explaining the reception phase shift of the sensor unit according to an embodiment of the present invention.

Referring to FIG. 12, the control unit (140) can control the sensor unit (130) to shift the reception phase of the optical signal, i.e., the reception phase of the optical signal received by the sensor unit (130), according to a plurality of sequences. As described above, the sensor unit (130) can generate four image signals through four reference signals, i.e., demodulation signals, during one exposure period. The four reference signals can have a phase difference of 90 degrees. For example, the control unit (140) can control the sensor unit (130) to generate a reference signal of a reference phase according to a first sequence in a first exposure period. That is, the control unit (140) can control the sensor unit (130) according to the reception phase corresponding to the reference phase according to the first sequence in the first exposure period. In addition, the control unit (140) can control the sensor unit (130) to generate a reference signal whose phase is shifted by a predetermined phase from the reference phase according to a second sequence in a second exposure period. That is, the control unit (140) can control the sensor unit (130) according to the reception phase corresponding to the phase shifted by a predetermined phase from the reference phase according to the second sequence in the second exposure cycle. At this time, the second exposure cycle may mean the exposure cycle following the first exposure cycle. When the plurality of sequences include the first sequence and the second sequence, the control unit (140) can control the sensor unit (130) according to the second sequence and then control the sensor unit (130) according to the first sequence in the third exposure cycle. That is, the plurality of sequences can be sequentially repeated.

Table 2 is a table showing the phase of a demodulated signal when multiple sequences include the first to third sequences.

1st receiver
phase 2nd receiver
phase Third Reception
phase 4th receiver
phase First sequence 0 π/2 π 3π/2 Sequence 2-1 0.16π π/2+0.16π π+0.16π 3π/2+0.16π Sequence 2-2 0.32π π/2+0.32π π+0.32π 3π/2+0.32π

According to Table 2, the control unit (140) can control the sensor unit (130) to receive an optical signal according to the reception phases of 0, π/2, π, and 3π/2 according to the first sequence, and then control the sensor unit (130) to receive an optical signal according to the reception phases of 0.16π, π/2+0.16π, π+0.16π, and 3π/2+0.16π according to the 2-1 sequence. It can be seen that in the 2-1 sequence, the first to fourth reception phases are each shifted by 0.16π in phase compared to the first to fourth reception phases in the 1st sequence. In addition, the control unit (140) can control the sensor unit (130) according to the reception phases of 0.32π, π/2+0.32π, π+0.32π, and 3π/2+0.32π according to the 2-2 sequence. It can be seen that the first to fourth receiving phases in the 2-2 sequence are shifted by 0.32π in phase compared to the first to fourth receiving phases in the 1st sequence, and are shifted by 0.16π in phase compared to the first to fourth receiving phases in the 2-1 sequence.

Fig. 13 is a drawing for explaining a phase-specific measurable area according to an embodiment of the present invention.

When the duty ratio of the optical signal is 10%, when measuring the distance at 0 degree phase (reference phase), the distance is measured only in the shaded area of (a) of Fig. 15, and the distance may not be measured in other areas. However, when the output phase of the light emitting unit (110) or the reception phase of the sensor unit (130) is shifted by a predetermined phase, the distance for the area not measured at 0 degree phase can be measured, as in (b) to (d) of Fig. 15. Therefore, when generating one depth image through the image signals generated corresponding to each of a plurality of sequences, distance information corresponding to the entire phase can be obtained.

FIG. 14 is a drawing for explaining one embodiment of shift phase setting of a control unit according to an embodiment of the present invention.

Fig. 14 is a diagram showing a case where the duty ratio is 10%. Fig. 14 (a) is a graph showing the values of normalized signals for actual phases for each of multiple sequences, and Fig. 14 (b) is a graph showing measured phases for actual phases for each of multiple sequences. Each shade shown in Fig. 14 (b) represents a phase region where depth information can be acquired for each sequence.

As illustrated in Fig. 14, when the duty ratio is 10%, by performing a phase shift of two times from the reference phase, distance information to the object can be obtained for the entire phase region of the exposure cycle. Therefore, the optimal shift phase at a duty ratio of 10% is 0.16π.

In order to detect the optimal shift phase and measure the distance to the object, the control unit (140) according to the embodiment of the present invention can set the shift phase based on the image signal generated corresponding to the first sequence. Specifically, the control unit (140) can calculate a phase region in which the distance to the object can be measured through the image signal generated corresponding to the first sequence, and set the shift phase according to the calculated phase region. The control unit (140) can determine the number of second sequences according to the shift phase. At this time, the exposure period corresponding to the first sequence may be smaller than the exposure period corresponding to the second sequence. For example, the exposure period corresponding to the first sequence may be 50% of the exposure period corresponding to the second sequence.

Meanwhile, in the case of the ToF camera module, the distance may not be measured for objects beyond a certain distance. For example, when measuring distance information for a specific object, the background of the object may exceed the measured distance, so distance information cannot be obtained. In such cases where the distance is too far to obtain distance information, even if the area not measured during the exposure cycle is measured through phase shift due to the short duty cycle, distance information cannot be obtained. In another example, a part of the object may be very close to the camera module (100), so distance information may be obtained only with the image signal generated in response to the first sequence. In this case, even if the area not measured during the exposure cycle is measured through phase shift, the acquired distance information does not change. In other words, since different distance information is generated for each pixel depending on the situation, there may be cases where it is not necessary to measure the area not measured during the exposure cycle through phase shift.

In order to operate optimally in this situation, the control unit (140) according to the embodiment of the present invention can control the sensor unit (130) to shift the phase of the demodulation signal for the optical signal for each pixel of the sensor unit (130). For example, in the second sequence, the sensor unit (130) can be controlled to shift the phase of the demodulation signal for pixel 1 by 0.16π, while the sensor unit (130) can be controlled to shift the phase of the demodulation signal for pixel 2 by 0.24π. At this time, the shift phase of the demodulation signal for each pixel can be set based on the image signal generated corresponding to the first sequence.

Although the above description focuses on examples, these are merely examples and do not limit the present invention. Those skilled in the art will appreciate that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present invention. For example, each component specifically shown in the examples can be modified and implemented. In addition, differences related to such modifications and applications should be construed as being included within the scope of the present invention defined in the appended claims.

100: Camera module
110: Light-emitting part
120: Light receiving unit
130: Sensor section
140: Control unit
150: Image processing unit
160: Tilt section

Claims (9)

A light emitting unit that outputs a light signal to an object;
A light receiving unit that collects the light signal output from the light emitting unit and reflected on the object;
A sensor unit that receives an optical signal collected by the light receiving unit; and
A control unit that controls at least one of the light emitting unit and the sensor unit to shift at least one of the output phase of the light emitting unit and the reception phase of the sensor unit; and
A tilt unit is included that tilts at least one of the filter and the lens so that the optical path of the optical signal passing through at least one of the filter and the lens shifts.
The above sensor unit includes a plurality of cells spaced at a certain interval,
A camera module in which the tilt unit tilts at least one of the filter and the lens so that the optical path of the optical signal shifts by a unit smaller than one of the cells of the sensor unit. In the first paragraph,
The duty ratio of the above optical signal is
Camera module set within a range greater than 0% and less than 25%. In the first paragraph,
The above control unit,
Controlling the above light emitting unit or the sensor unit according to a plurality of sequences,
The above multiple sequences are,
A camera module comprising a first sequence corresponding to a reference phase and a second sequence shifted by a predetermined phase from the reference phase. In the third paragraph,
Each of the above plurality of sequences,
Camera modules corresponding to different phases. In the third paragraph,
The exposure period of the optical signal corresponding to the above first sequence is,
A camera module having an exposure period smaller than that of the optical signal corresponding to the second sequence. In the third paragraph,
The above control unit,
A camera module that sets the phase of the second sequence based on an image signal generated corresponding to the first sequence. In the third paragraph,
The above control unit,
A camera module in which each of the above plurality of sequences is performed during an exposure cycle corresponding to one image frame. In the third paragraph,
The above control unit,
A camera module that controls the sensor unit to shift the phase of the demodulation signal for the optical signal for each pixel of the sensor unit based on the image signal generated in response to the first sequence. In the first paragraph,
A camera module further comprising an image processing unit that generates a depth image using a plurality of image signals corresponding to the plurality of sequences.