KR102274033B1 - Depth image acquisition apparatus reducing motion blur using multi frequency - Google Patents

Abstract

The present invention relates to a depth image acquisition apparatus for reducing motion blur using multiple frequencies.
The present invention provides a transmitter that transmits two or more multiple signals with different modulation frequencies, a receiver that receives two or more reflected signals in which the multiple signals are reflected on the subject, and generates a luminance image by synthesizing the two or more reflected signals for each pixel. A luminance image generating unit, a motion blur determining unit for determining a pixel region in which the difference in intensity of each pixel of the two or more reflected signals in the luminance image exceeds a threshold as a motion blur region, and temporally setting the motion blur region among the two or more reflected signals and a motion blur compensator for compensating the motion blur region by replacing it with the strength of the latest signal received at a late point in time, and a depth image generator for generating a depth image from the luminance image in which the motion blur region is corrected.

Description

A depth image acquisition device that uses multiple frequencies to reduce motion blur {DEPTH IMAGE ACQUISITION APPARATUS REDUCING MOTION BLUR USING MULTI FREQUENCY}

The present invention relates to a depth image acquisition device. More specifically, the present invention improves the performance of the depth data acquired by the TOF camera using multiple frequencies and simultaneously determines and compensates for the motion blur area that may occur in the process of increasing the detection distance, thereby simultaneously improving the quality of the depth image and the detection distance. It relates to a depth image acquisition apparatus for reducing motion blur by using multiple frequencies that can be increased.

In general, TOF (Time Of Flight) is a method of calculating the distance to the subject by measuring the flight time, that is, the time of light reflected back by the subject. The TOF camera outputs a depth image using the TOF method. refers to a camera that

According to such a conventional TOF camera, there is a problem that motion blur occurs due to a measurement time difference when a subject moves, and there is a problem that the detection distance and detection accuracy cannot be improved at the same time.

This will be described with reference to FIG. 1 for explaining a problem in which motion blur occurs in a depth image photographed by a depth camera of a time of flight (TOF) method using a conventional single frequency.

Referring to FIG. 1 , an image 1-1 in which motion blur is not generated and an image 1-2 in which motion blur is generated due to motion or the like are exemplified.

Since the conventional TOF camera generates one image by irradiating four lights with different phases to the subject, motion blur occurs due to the measurement time difference on the moving object.

In addition, in TOF, the higher the modulation frequency of the light, the better the performance, that is, the depth data of the accuracy can be obtained, but there is a problem in that the maximum sensing distance is reduced.

Republic of Korea Patent Publication No. 10-2015-0133082 (published date: November 27, 2015, title: Depth image generating apparatus and method) Republic of Korea Patent Publication No. 10-2017-0088259 (published date: August 01, 2017, title: Method and apparatus for acquiring a depth image using a ToF sensor)

The technical task of the present invention is to improve the performance of depth data acquired by a TOF camera using multiple frequencies, and at the same time improve the quality of the depth image and the detection distance by accurately determining and compensating the motion blur region that may occur in the process of increasing the detection distance. An object of the present invention is to provide a depth image acquisition device for reducing motion blur by using multiple frequencies that can be increased.

Depth image acquisition apparatus for reducing motion blur by using a multi-frequency according to the present invention for solving the above technical problem, a modulation frequency (modulation frequency) of transmitting two or more different multi-signals A transmitter, a receiver for receiving two or more reflected signals in which the multiple signals are reflected on the subject, a luminance image generator for generating a luminance image by synthesizing the two or more reflected signals for each pixel, and the two or more reflected signals from the luminance image A motion blur determination unit that determines a pixel area in which the difference in intensity for each pixel exceeds a threshold as a motion blur area, and replaces the motion blur area with the intensity of the latest signal received at a later time among the two or more reflected signals. and a motion blur compensator for correcting the motion blur region and a depth image generator for generating a depth image from the luminance image in which the motion blur region is corrected.

In the depth image acquisition apparatus for reducing motion blur using multiple frequencies according to the present invention, the transmitter includes a first transmitter that transmits a first signal having a first modulation frequency and a second transmitter that is different from the first modulation frequency. and a second transmitter for transmitting a second signal having two modulation frequencies.

In the apparatus for obtaining a depth image for reducing motion blur using multiple frequencies according to the present invention, the motion blur determining unit determines a difference in intensity between the first reflected signal and the second reflected signal for all pixels constituting the luminance image. It is characterized by calculating for each pixel, and determining an area in which the difference in intensity calculated for each pixel exceeds the threshold as the motion blur area, and determining an area not exceeding the threshold as a normal area.

In the depth image acquisition apparatus for reducing motion blur using multiple frequencies according to the present invention, the motion blur determining unit generates a motion blur determination map by synthesizing the motion blur region and the normal region. do it with

In the depth image acquisition apparatus for reducing motion blur using multiple frequencies according to the present invention, the threshold is a weight value of the latest signal received at a later time among the first reflected signal and the second reflected signal. It is characterized in that it is a multiplied value.

In the depth image acquisition apparatus for reducing motion blur by using multiple frequencies according to the present invention, the motion blur compensator may include a normal region constituting the motion blur determination map of the first reflected signal and the second reflected signal. It is characterized in that it is replaced with the average value of the intensity.

According to the present invention, a multi-frequency that can improve the quality of a depth image by accurately determining and compensating for a motion blur region that may occur in the process of increasing the sensing distance while improving the performance of depth data acquired by a TOF camera using multiple frequencies There is an effect that a depth image acquisition apparatus for reducing motion blur by using is provided.

In addition, it has the effect of providing users with high-performance TOF depth images in which motion blur is removed.

In addition, by remarkably improving the motion blur phenomenon, which is a fatal weakness in applications such as driving robots and gesture recognition in which a target object moves, it has the effect of greatly expanding the market area to which the ToF application is applied.

1 is a diagram for explaining a problem in which motion blur occurs in a depth image photographed by a depth camera of a time of flight (TOF) method using a conventional single frequency;
2 is a diagram for explaining a problem in which motion blur occurs in a depth image captured by a TOF-type depth camera using multi-frequency;
3 is a view showing a depth image acquisition apparatus for reducing motion blur using multiple frequencies according to an embodiment of the present invention;
4 is a flowchart for explaining an exemplary operation of an apparatus for obtaining a depth image for reducing motion blur using multiple frequencies according to an embodiment of the present invention;
5 is a diagram for illustratively explaining a configuration in which a motion blur determination unit determines a motion blur region according to an embodiment of the present invention;
6 is a diagram for explaining a configuration in which a motion blur compensator corrects a motion blur region according to an embodiment of the present invention
7 is a diagram for explaining a configuration in which a depth image generator corrects a depth image according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention may take various forms. It can be implemented with the above and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from another, for example, without departing from the scope of the inventive concept, a first element may be termed a second element and similarly a second element. A component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that other components may exist in between. will be. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. As used herein, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described herein is present, but one or more other features It is to be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as commonly used dictionary definitions should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in the present specification, they are not to be interpreted in an ideal or excessively formal meaning. .

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 2 is a diagram for explaining a problem in which motion blur occurs in a depth image photographed by a depth camera of a TOF method using a multi frequency.

Referring to FIG. 2 , since the TOF camera generates one image by irradiating four lights with different phases to the subject, when multi-frequency is applied to the TOF camera, one frequency, that is, a single frequency Compared to the case of applying (single frequency), motion blur due to the measurement time difference may occur more severely for a relatively moving object.

Reference numeral 2-1 of FIG. 2 denotes an image 2-11 obtained using a first signal having a first modulation frequency and an image 2-12 obtained using a second signal having a second modulation frequency. is a luminance image synthesized, reference numeral 2-2 denotes a depth image generated using the synthesized luminance image, and reference numeral 2-21 denotes a motion blur component included in the depth image.

The present invention improves the performance of the depth data acquired by the TOF camera using multiple frequencies and at the same time accurately determines and compensates for the motion blur area that may occur in the process of increasing the detection distance, thereby increasing the quality of the depth image and the detection distance at the same time. Provided is an apparatus for acquiring a depth image that reduces motion blur by using multiple frequencies.

3 is a diagram illustrating a depth image acquisition apparatus for reducing motion blur using multiple frequencies according to an embodiment of the present invention.

Referring to FIG. 3 , an apparatus for obtaining a depth image for reducing motion blur using multiple frequencies according to an embodiment of the present invention includes a transmitter 5 , a receiver 30 , a luminance image generator 40 , and a motion blur It includes a determination unit 50 , a motion blur correcting unit 60 , and a depth image generation unit 70 . For example, an embodiment of the present invention may be applied to a TOF (Time Of Flight) type camera that calculates the distance to the subject by measuring the time of light reflected by the subject and returned, but one embodiment of the present invention is applicable The range of products available is not limited thereto. However, hereinafter, for convenience of understanding, an embodiment of the present invention will be described based on a Time Of Flight (TOF) method.

The transmitter 5 is a component that transmits two or more multiple signals having different modulation frequencies. In the following description, the embodiment is described based on the case where the transmitter 5 uses two modulation frequencies, but this is only an example, and the transmitter 5 is It may be configured to transmit a plurality of signals to a subject using three or more modulation frequencies within a satisfactory range.

For example, the transmitter 5 may be configured to include a first transmitter 10 and a second transmitter 20 .

The first transmitter 10 is a component that transmits a first signal having a first modulation frequency to a subject.

The second transmitter 20 is a component that transmits a second signal having a second modulation frequency different from the first modulation frequency to the wave body.

For example, the first signal transmitted by the first transmitting unit 10 and the second signal transmitted by the second transmitting unit 20 may be four lights having different phase differences, respectively. The phase difference of light may be 90 degrees, but this is only an example, and the first signal and the second signal are not limited thereto.

The receiver 30 is a component that receives a first reflected signal from which the first signal is reflected on the subject and a second reflected signal from which the second signal is reflected from the subject. For example, the receiving unit 30 may be a CMOS or CCD type image sensor, and the number of receptions is information about the intensity, reception time, phase difference, etc. of the first and second reflected signals that are reflected back to the subject. can be obtained Of course, such information may be obtained for all pixels constituting one image according to the resolution provided by the image sensor, which is the receiver 30 .

The luminance image generator 40 is a component that generates a luminance image by synthesizing the first reflected signal and the second reflected signal for each pixel. Reference numeral 5-1 of FIG. 5 is a first luminance image generated by using the first reflected signal that is reflected by the first signal transmitted by the first transmitter 10 to the subject and returned to the receiver 30, Reference numeral 5-2 of FIG. 5 is a second luminance image generated by using a second reflected signal that is reflected by a second signal transmitted by the second transmitter 20 to the subject and returned to the receiver 30, 6-1 of FIG. 6 is a luminance image obtained by synthesizing a first luminance image generated using the first reflected signal and a second luminance image generated using the second reflected signal.

The motion blur determining unit 50 determines a pixel area in which the difference in intensity of each pixel of the first reflected signal and the second reflected signal in the luminance image exceeds a threshold as the motion blur area. As described above, in the luminance image, the first luminance image and the second signal generated by using the first reflected signal returned to the receiver 30 after the first signal is reflected by the subject are reflected by the subject and sent to the receiver 30 . It is an image obtained by synthesizing a second luminance image generated using the returned second reflected signal.

For example, the motion blur determination unit 50 calculates the difference in intensity between the first reflected signal and the second reflected signal for every pixel constituting the luminance image, and the intensity difference calculated for each pixel is preset and stored. It may be configured to determine an area exceeding a threshold as a motion blur area, and determine an area not exceeding the threshold as a normal area.

For example, as illustrated in FIG. 5 , the motion blur determination unit 50 may be configured to generate a motion blur determination map by synthesizing the motion blur area and the normal area. Reference numeral 5-3 of FIG. 5 denotes a motion blur determination map, reference numeral 5-31 denotes a motion blur region in which a first signal having a first modulation frequency is caused by a first reflected signal reflected by a subject, Reference numeral 5-32 denotes a motion blur region caused by a second reflected signal reflected by a second signal having a second modulation frequency to a subject, and reference numeral 5-33 denotes a normal region in which motion blur does not occur.

For example, the threshold may be a value obtained by multiplying the strength of the latest signal received at a later time among the first reflected signal and the second reflected signal by a weight.

The motion blur compensating unit 60 replaces the motion blur region determined by the motion blur determining unit 50 with the strength of the latest signal received at a later time among the first reflected signal and the second reflected signal to replace the motion blur region. correct the

An operation of the motion blur correcting unit 60 will be described with reference to FIG. 6 as follows.

Reference numeral 6-1 of FIG. 6 denotes a luminance image generated by the luminance image generator 40 . As described above, the luminance image generating unit 40 generates a first signal generated by using the first reflected signal transmitted by the first transmitting unit 10 is reflected by the subject and returned to the receiving unit 30 . A second luminance image 5 - 11 generated by using the luminance image 5-11 and the second reflected signal returned to the receiver 30 after the second signal transmitted by the second transmitter 20 is reflected by the subject 12) can be synthesized for each pixel to generate a luminance image. In addition, reference numeral 5-2 of FIG. 6 denotes a motion blur determination map generated by the motion blur determination unit 50, and reference numeral 5-31 denotes a second signal in which a first signal having a first modulation frequency is reflected on a subject. 1 denotes a motion blur region due to a reflected signal, reference numeral 5-32 denotes a motion blur region caused by a second reflected signal in which a second signal having a second modulation frequency is reflected on a subject, and reference numeral 5-33 denotes a normal region where motion blur does not occur.

For example, the motion blur compensator 60 converts the motion blur region determined by the motion blur determination unit 50 to the strength of the latest signal received at a later time among the first reflected signal and the second reflected signal. Alternatively, the motion blur region may be corrected, and the normal region constituting the motion blur determination map may be replaced with an average value of the intensities of the first reflected signal and the second reflected signal. Reference numeral 6-3 of FIG. 6 denotes a luminance image in which motion blur is corrected by the motion blur compensator 60 .

The depth image generator 70 generates a depth image from the luminance image in which the motion blur region is corrected. For example, the depth image generating unit 70 generates a motion blur-corrected depth image using information about a luminance image in which the motion blur region is corrected, a reception time of the first reflected signal and the second reflected signal, a phase difference, and the like. can do. Reference numeral 7-1 of FIG. 6 denotes a motion blur determination map generated by the motion blur determination unit 50, and reference numeral 7-2 denotes a luminance image in which motion blur is corrected by the motion blur correcting unit 60. Reference numeral 7-3 denotes a depth image in which the motion blur region generated by the depth image generator 70 is corrected.

Hereinafter, an operation of the apparatus for obtaining a depth image for reducing motion blur using multiple frequencies according to an embodiment of the present invention will be exemplarily described with additional reference to FIGS. 4 to 7 .

4 is a flowchart for explaining an exemplary operation of an apparatus for obtaining a depth image for reducing motion blur using multiple frequencies according to an embodiment of the present invention, and FIG. 5 is a flowchart illustrating motion blur according to an embodiment of the present invention. It is a diagram for explaining a configuration in which the determination unit 50 determines the motion blur area by way of example, and FIG. 6 is a configuration in which the motion blur correcting unit 60 corrects the motion blur area according to an embodiment of the present invention. 7 is a diagram to exemplarily describe a configuration in which the depth image generator 70 corrects a depth image according to an embodiment of the present invention.

4 to 7 , in step S10 , a process of transmitting a first signal having a first modulation frequency to a subject by the first transmitter 10 is performed.

In step S20, the second transmitter 20 transmits a second signal having a second modulation wave number different from the first modulation frequency to the subject.

For example, the modulation frequency of the first signal may be 20 MHz and the modulation frequency of the second signal may be 100 MHz, but the number of modulation waves is not limited thereto. In addition, an embodiment of the present invention is described based on a case in which a first signal having a low frequency is transmitted before a second signal having a high frequency, but is not limited thereto, and the second signal having a high frequency is transmitted before the first signal having a low frequency It may be configured to be

In step S30 , a process of receiving the first reflected signal in which the first signal is reflected by the subject and the second reflected signal in which the second signal is reflected by the subject is performed by the receiver 30 .

In step S40, the process of generating a luminance image by synthesizing the first reflected signal and the second reflected signal for each pixel by the luminance image generator 40 is performed. Reference numeral 6-1 of FIG. 6 denotes a luminance image generated by the luminance image generator 40 .

In step S50, a process of determining, by the motion blur determination unit 50, a pixel area in which the difference in intensity of each pixel of the first reflected signal and the second reflected signal in the luminance image exceeds a threshold value, as the motion blur area is performed. Reference numeral 5-3 of FIG. 5 denotes a motion blur determination map including motion blur areas 5-31 and 5-32 and normal areas 5-33 determined by the motion blur determination unit 50 .

In step S60, the motion blur compensating unit 60 corrects the motion blur region by replacing the motion blur region with the strength of the latest signal received at a later time among the first reflected signal and the second reflected signal. . Reference numeral 6-3 of FIG. 6 denotes a luminance image in which the motion blur region is corrected by the motion blur compensator 60 .

In step S70 , a process of generating the depth image by the depth image generator 70 from the luminance image in which the motion blur region is corrected is performed. Reference numeral 7-3 of FIG. 7 denotes a depth image in which a motion blur region generated by the depth image generator 70 is corrected.

As described in detail above, according to the present invention, the quality of the depth image is improved by accurately determining and compensating for the motion blur region that may occur in the process of increasing the sensing distance while increasing the performance of the depth data acquired by the TOF camera using multiple frequencies. There is an effect of providing a depth image acquisition device for reducing motion blur by using multiple frequencies capable of simultaneously increasing and detecting distance.

In addition, it has the effect of providing users with high-performance TOF depth images in which motion blur is removed.

In addition, by remarkably improving the motion blur phenomenon, which is a fatal weakness in applications such as driving robots and gesture recognition in which a target object moves, it has the effect of greatly expanding the market area to which the ToF application is applied.

5: Sender
10: first transmitter
20: second transmitter
30: receiver
40: luminance image generating unit
50: motion blur determination unit
60: motion blur compensation unit
70: depth image generator

Claims (6)

As a depth image acquisition apparatus for reducing motion blur by using multi frequency,
a transmitter for transmitting two or more multiple signals having different modulation frequencies;
a receiver receiving two or more reflected signals in which the multi-signal is reflected on a subject;
a luminance image generator generating a luminance image by synthesizing the two or more reflected signals for each pixel;
a motion blur determination unit for determining a pixel area in which the difference in intensity of each pixel of the two or more reflected signals in the luminance image exceeds a threshold as a motion blur area, and determining an area not exceeding the threshold as a normal area;
A motion of correcting the motion blur region by replacing the motion blur region with the intensity of the latest signal received at a later time among the two or more reflected signals, and replacing the normal region with the average value of the intensity of the two or more reflected signals blur correction unit; and
and a depth image generator configured to generate a depth image from the luminance image in which the motion blur region is corrected.
According to claim 1,
The sender is
a first transmitter for transmitting a first signal having a first modulation frequency; and
Depth image acquisition apparatus for reducing motion blur using multiple frequencies, characterized in that it comprises a second transmitter for transmitting a second signal having a second modulation frequency different from the first modulation frequency.
delete 3. The method of claim 2,
The motion blur determination unit,
Depth image acquisition apparatus for reducing motion blur using multiple frequencies, characterized in that by synthesizing the motion blur region and the normal region to generate a motion blur determination map.
5. The method of claim 4,
the threshold is
A depth image acquisition apparatus for reducing motion blur using multiple frequencies, characterized in that it is a value obtained by multiplying the intensity of the latest signal received at a later time among the first reflected signal and the second reflected signal.
delete

Images