KR102472767B1 - Method and apparatus of calculating depth map based on reliability - Google Patents

Abstract

A method and apparatus for calculating a depth map according to an embodiment divides an input image into segments, selects one or more of the segments based on reliability calculated for the segments, and uses the selected segments in the input image. Camera pose information is estimated for , and a depth map of the input image is calculated using the camera pose information.

Description

Method and apparatus for calculating depth map based on reliability

The embodiments below relate to a method and apparatus for calculating a depth map based on reliability.

A 2D input image may be reconstructed into a 3D image through camera position estimation and depth estimation. Camera position estimation and depth estimation include, for example, the Structure From Motion (SFM) technique that identifies the structure of an object through information generated from the movement of the object, and the position of the moving camera while simultaneously measuring the surrounding environment. A Simultaneous Localization and Mapping (SLAM) technique for creating a map, a Visual Odometry (VO) measurement technique for determining a position and attitude by analyzing a camera image, and the like may be used.

According to the techniques described above, unnecessary computational resources are required due to error generation through repetitive selection of an area of an object other than a target object to be tracked in an image and/or tracking of a moving object. losses may occur.

According to an embodiment, a method of calculating a depth map includes dividing an input image into segments; calculating reliability of the segments; selecting one or more of the segments based on the confidence level; estimating camera pose information for the input image using the selected segment; and calculating a depth map of the input image using pose information of the camera.

The dividing may include dividing the input image into semantic segments by classifying objects included in the input image into semantic units; and dividing the input image into depth segments based on a depth value of the input image, or a combination thereof.

The calculating of the reliability may include calculating a first reliability of the semantic segments; and calculating the second reliability of the depth segments, or a combination thereof.

The calculating of the first reliability may include calculating the first reliability of the semantic segments based on whether an object included in the input image is a moving object.

The calculating of the first reliability may include determining a first reliability for a semantic segment corresponding to the moving object as a first value when the object is a moving object; and determining a first reliability of a semantic segment corresponding to the fixed object as a second value when the object is a fixed object.

The calculating of the reliability may include fusion of the first reliability and the second reliability; and determining the fused reliability as the reliability of the segments.

The method of calculating the depth map further comprises selecting pixels from the selected segment based on the fused reliability, wherein the estimating the pose information of the camera comprises the pose information of the camera from the selected pixels. It may include estimating.

Selecting pixels from the selected segment may include selecting pixels from the selected segment to be proportional to the fused reliability.

The input image may include frames, and calculating the reliability may include calculating reliability of the segments for each key frame among the frames.

Estimating the pose information of the camera may include estimating the pose information of the camera by applying a cost function to the selected segment.

According to an embodiment, an apparatus for calculating a depth map includes a camera acquiring an input image; and dividing the input image into segments, selecting one or more of the segments based on reliability calculated for the segments, and estimating camera pose information for the input image using the selected segments. and a processor for calculating a depth map of the input image using pose information of the camera.

The processor classifies objects included in the input image into semantic units and divides the input image into semantic segments, or divides the input image into depth segments based on a depth value of the input image, or An input image may be divided into the semantic segments and the depth segments.

The processor may calculate a first reliability level for the semantic segments, a second reliability level for the depth segments, or calculate the first reliability level and the second reliability level.

The processor may calculate a first reliability of the semantic segments based on whether an object included in the input image is a moving object.

The processor may fuse the first reliability level and the second reliability level, and determine the fused reliability level as a reliability level for the segments.

The processor may select pixels from the selected segment based on the fused reliability, and estimate pose information of the camera from the selected pixels.

The processor may select pixels from the selected segment proportional to the fused reliability.

The input image may include frames, and the processor may calculate reliability of the segments for each key frame among the frames.

The processor may estimate pose information of the camera by applying a cost function to the selected segment.

1 is a flowchart illustrating a method of calculating a depth map according to an exemplary embodiment;
2 is a flowchart illustrating a method of calculating reliability according to an exemplary embodiment;
3 is a diagram for explaining a method of selecting one or more segments according to an exemplary embodiment;
4 is a flowchart illustrating a method of estimating pose information of a camera according to an exemplary embodiment;
5 is a diagram for explaining an operation of an apparatus for calculating a depth map according to an exemplary embodiment.
6 is a block diagram of an apparatus for calculating a depth map according to an exemplary embodiment;

Specific structural or functional descriptions disclosed in this specification are only illustrated for the purpose of describing embodiments according to technical concepts. The disclosed embodiments may be modified and implemented in various other forms, and the scope of the present specification is not limited to the disclosed embodiments.

Terms such as first or second may be used to describe various components, but such terms should only be interpreted for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Embodiments to be described below may be used to estimate a depth value for reconstructing a 3D scene of an input image in various augmented reality (AR) application fields. In the embodiments, a dense depth map can be generated within a short period of time using images obtained from a single camera without additional hardware configuration such as a depth camera. Embodiments include, for example, augmented reality head-up displays (HUDs), augmented reality/virtual reality (VR) glasses, autonomous vehicles, intelligent vehicles, smart phones, and mobile devices, etc. It can be applied to implement augmented reality applications in real time. The embodiments may be applied to, for example, camera pose tracking and depth reconstruction for accurate matching between driving images and virtual objects in a head-up display. Embodiments can be applied to matching and 3D image reconstruction of a smart phone, augmented reality/virtual reality device on a mobile platform. Alternatively, the embodiments may be applied to posture control using vision technology in drones, robots, and self-driving cars. Embodiments can be implemented in the form of a chip and mounted on in-vehicle infotainment (IVI), advanced driver assistance systems (ADAS), smart phones, augmented reality/virtual reality devices, etc. have. Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

1 is a flowchart illustrating a method of calculating a depth map according to an exemplary embodiment. Referring to FIG. 1 , an apparatus for calculating a depth map according to an exemplary embodiment (hereinafter referred to as a 'calculating apparatus') divides an input image into segments (110). The input image is an image input to the calculation device, and may be, for example, a live image or a moving picture. Alternatively, the input image may be a mono image or a stereo image. An input image may include a plurality of frames. The input image may be captured through a camera included in the calculation device (eg, the camera 610 shown in FIG. 6 ) or may be acquired from an external source of the calculation device.

Segments may correspond to partial areas in which the input image is classified or divided according to a predetermined criterion.

The calculation device classifies objects included in the input image into semantic units of 20 classes, such as roads, vehicles, sidewalks, people, animals, sky, buildings, etc., and divides them into semantic segments. can do. Classes of semantic units may include, for example, moving objects such as moving people, moving animals, and moving vehicles in addition to fixed objects such as roads, sky, and buildings. The calculation device divides an object from an input image into semantic units, identifies the meaning of the segmented area in units of pixels, and labels each class to obtain a segmentation image including semantic segments. can create

The calculation device uses, for example, a Convolution Neural Network (CNN), a Deep Neural Network (DNN), a Support Vector Machine, etc. that have been previously trained to recognize a plurality of classes. An image can be segmented into semantic segments. The convolutional neural network is obtained by pre-learning various objects, and may be a region-based convolutional neural network. In addition, the calculation device may divide objects included in the input image into semantic segments using various machine learning methods.

Also, the calculation device may divide the input image into depth segments based on a depth value obtained by using a depth map or a normal map inferred from the input image. The regions of the semantic segments and the depth segments may coincide with each other or may be different from each other.

The calculator calculates the reliability of the segments divided in step 110 (120). Reliability here may correspond to, for example, reliability of depth information (eg, depth values) and location information (eg, location coordinates) of segments. The calculator may calculate first reliability for semantic segments, for example. Also, the calculator may calculate the second reliability of the depth segments.

The calculator may calculate reliability of segments for each key frame among frames. A key frame is a frame having all information of an image progressing on a time line, and may correspond to the most central frame, such as a time frame and an end frame of a single motion, for example.

For example, the calculation device may set the reliability of the segment including the moving object low so that the corresponding segment is excluded from the process of estimating the pose information of the camera and the process of calculating the depth map. A method of calculating reliability of segments by the calculator will be described in detail with reference to FIG. 2 below.

The calculation device selects one or more segments from among the segments based on the reliability calculated in step 120 (130). The calculation device may select a pixel to be a feature point used in a process of estimating pose information of a camera and a process of calculating a depth map, which will be described later, from the selected segment based on reliability. A feature point is a feature point within a frame, and may include information (u, v) corresponding to a 2D location within a corresponding frame. Each frame may include a plurality of feature points, and since a general feature point detection algorithm may be applied to an operation of selecting feature points from the frame, a detailed description thereof is omitted. According to an embodiment, at least some of the feature points may additionally include information corresponding to a depth value. For example, information corresponding to 3D positions of at least some of feature points may be obtained in a process of estimating pose information of a camera that captures an input image. A 3D position includes a depth value.

For example, False Negative Selection (False Negative Selection ), a moving object in the image, or a piece of glass on the road surface, causing tracking loss due to high frequency noise caused by a part in which the gradient of the part appears high. can

In an embodiment, segments that may cause such tracking loss according to reliability, that is, segment(s) with low reliability may be excluded, and segment(s) with high reliability may be selected. The calculation device may improve calculation speed and accuracy by estimating pose information of the camera using information extracted from the highly reliable segment(s) and calculating a depth map of the input image. A method of selecting one or more segments by the calculation device will be described in detail with reference to FIG. 3 below.

The calculation device estimates camera pose information for the input image using the selected segment (140). The pose information of the camera may include, for example, rotation information R and translation information T of the camera. Alternatively, the pose information of the camera may be, for example, X (horizontal), Y (vertical), Z (depth) corresponding to the position of the camera, and/or pitch, yaw corresponding to the orientation of the camera. ), and a 6 DoF camera pose including roll.

The calculation device uses, for example, homography representing a correlation between pixels in a series of images (frames) to determine the location of a camera capturing an input image and the location (depth) of a captured object. ) may be estimated. The calculation device is, for example, a feature based simultaneous localization and mapping (SLAM) technique, a direct SLAM technique, an extended Kalman filter (EKF) SLAM technique, a fast SLAM technique, and a large -Scale Direct Monocular) The pose information of the camera can be obtained using various SLAM techniques such as the SLAM technique. A method of estimating pose information of a camera by a calculation device will be described in detail with reference to FIG. 4 below.

The calculator calculates a depth map of the input image using pose information of the camera (150). The calculation device calculates a depth value using camera position coordinates (u.v) obtained in the process of estimating camera pose information, rotation information (R) and translation information (T) of the camera, etc. By doing so, a depth map can be calculated.

2 is a flowchart illustrating a method of calculating reliability according to an exemplary embodiment. Referring to FIG. 2 , the calculation device according to an exemplary embodiment may calculate first reliability for semantic segments based on whether an object included in an input image is a moving object.

The calculation device may determine whether an object included in the input image is a moving object (210). In step 210, if it is determined that the object is not a moving object (ie, if it is determined that the object is a fixed object), the calculation device may determine the first reliability of the semantic segment corresponding to the fixed object as a second value ( 220). The second value may be, for example, '1'.

If it is determined that the object is a moving object in step 210, the calculation device may determine the first reliability of the semantic segment corresponding to the moving object as a first value (230). The first value may be, for example, '0'. According to an embodiment, the calculation device estimates camera pose information or calculates a depth map by setting a low reliability value for a low reliability segment(s) that cause tracking loss or have noise, such as a moving object. In doing so, the use of the corresponding segment can be excluded.

The calculator may calculate the second reliability of the depth segments (240). The calculator may calculate the second reliability (R _Si ) of the depth segments using, for example, Equation 1 below.

는 현재(current)의 키 프레임 i를 나타내고,

는 키 프레임 i에 가장 인접한 다음(next) 키 프레임 j를 나타낸다.

는 깊이 맵을 나타내고,

는

에서

로의 변환 행렬(transformation matrix)을 나타낸다.

는 세그먼트 i에 속한 이미지 영역을 나타내고,

는 현재의 키 프레임에서의 이미지 영역을 나타낸다. In Equation 1,

represents the current key frame i,

represents the next key frame j closest to key frame i.

denotes a depth map,

Is

at

Represents a transformation matrix to .

denotes an image area belonging to segment i,

represents the image area in the current key frame.

로 나타낼 수 있다.

는 고유 행렬(intrinsic matrix)를 나타내고,

는 픽셀 좌표(pixel coordinate)를 나타내고,

는

의 동종 표현(homogeneous representation)을 나타낸다.

는 현재의 키 프레임 i에서의 픽셀 좌표 u의 깊이 맵을 나타낸다.

이다.

can be expressed as

denotes an intrinsic matrix,

represents a pixel coordinate,

Is

represents a homogeneous representation of

denotes the depth map of pixel coordinates u at the current keyframe i.

to be.

는 타겟(target)이 되는 다음 키 프레임 j을 나타내고,

는 변형된 호스트(warped host), 다시 말해 변형된 현재의 키 프레임 i을 나타낸다.in Equation 1

represents the next key frame j to be the target,

denotes a warped host, that is, a warped current key frame i.

The calculation device may fuse the first reliability level and the second reliability level (250). The calculation device may fuse the first reliability for the semantic segment and the second reliability for the depth segment using Equation 2 below, for example.

는 수학식 1에서 산출한 픽셀 좌표

에서의 깊이 세그먼트의 신뢰도를 나타내고,

는 픽셀 좌표

에서의 의미 세그먼트의 신뢰도를 나타낸다.

is the pixel coordinate calculated in Equation 1

Represents the reliability of the depth segment in ,

is the pixel coordinate

Indicates the reliability of the semantic segment in .

The calculation device may determine the fused reliability as the reliability of the segments (260).

3 is a diagram for explaining a method of selecting one or more segments according to an exemplary embodiment. Referring to (a) of FIG. 3 , an input image divided into semantic segments 310 , 320 , 330 , and 340 is shown. As described above, the calculation device may divide the input image into semantic segments by classifying objects included in the input image into semantic units. Hereinafter, a method of selecting one or more segments from semantic segments will be described for convenience of explanation, but the same method may be applied to depth segments.

For example, the road may be divided into segments 310, the building into segments 320, the sky into segments 330, and the car into segments 340 according to the meaning of the objects included in the input image. Among the divided segments 310, 320, 330, and 340, for example, as shown in FIG. Assume that a high frequency noise is generated and the car of the segment 340 divided into cars is a moving object. The calculation device may set the reliability of the segment 330 with high-frequency noise lower than that of segments without noise. Also, the reliability of the segment 340 corresponding to the moving object may be set to '0', for example.

The calculation device may exclude segments that may cause tracking loss or segments with low reliability, and select segment(s) with high reliability. The calculation device selects a segment 310 divided into roads and a segment 320 divided into buildings, for example, as shown in (c) of FIG. 3 , and information extracted from the selected segments 310 and 320 Camera pose information estimation and depth map calculation may be performed using (eg, information on the pixel(s) 350 ).

4 is a flowchart illustrating a method of estimating pose information of a camera according to an exemplary embodiment. Referring to FIG. 4 , the calculation device according to an exemplary embodiment may select pixels from a selected segment based on reliability (410). At this time, the reliability may be, for example, the reliability fused in step 250 of FIG. 2, or may be the first reliability or the second reliability. The calculation device may select pixels from the selected segment in proportion to the reliability. The calculation device may, for example, select pixels from the most reliable segment. Alternatively, the calculation device may evenly select pixels from the segments in order of high reliability. For example, the calculation device may select a large number of pixels from segments having the highest reliability, and may select a smaller number of pixels as the reliability gradually decreases.

)를 적용하여 카메라의 포즈 정보를 추정할 수 있다.The calculation device may estimate camera pose information from the selected pixels (420). The calculation device may estimate camera pose information from 3D points corresponding to pixels including depth values. Alternatively, the calculation device calculates, for example, a cost function such as Equation 3 below for the selected segment (

) can be applied to estimate the pose information of the camera.

는 기준 프레임(reference frame)을 나타내고,

는 타겟 프레임(target frame)을 나타낸다.

는 기준 프레임

에서의 포인트(point), 다시 말해 픽셀을 나타내고,

와 같이 나타낼 수 있다.

는 SSD(sum of squared differences)에 포함된 픽셀들의 집합을 나타낸다.

는 기준 프레임

의 노출 시간(exposure time)을 나타내고,

는 타겟 프레임

의 노출 시간을 나타낸다.

는 손실 함수(loss function)인 후버 놈(Huber norm)을 나타낸다.

는 융합된 신뢰도에 기초한 가중치(weight)를 나타내고,

는 아핀 밝기 전달 함수(affine brightness transfer function)를 나타낸다.

는 기준 프레임의 밝기(brightness)를 나타내고,

는 타겟 프레임의 밝기를 나타낸다.

는 기준 프레임에 대한 밝기 전달 함수(brightness transfer function)의 파라미터(parameter)를 나타내고,

는 타겟 프레임에 대한 밝기 전달 함수의 파라미터를 나타낸다. In Equation 3,

denotes a reference frame,

represents a target frame.

is the reference frame

Represents a point, that is, a pixel, in

can be expressed as

represents a set of pixels included in the sum of squared differences (SSD).

is the reference frame

represents the exposure time of

is the target frame

indicates the exposure time of

denotes the Huber norm, which is a loss function.

represents a weight based on the fused reliability,

denotes an affine brightness transfer function.

represents the brightness of the reference frame,

represents the brightness of the target frame.

denotes a parameter of the brightness transfer function for the reference frame,

represents parameters of the brightness transfer function for the target frame.

는 역 깊이(inverse depth)

를 갖는 포인트

의 투영된 포인트 위치를 나타내고, 아래의 수학식 4를 통해 구할 수 있다.

is the inverse depth

point with

Represents the projected point position of , and can be obtained through Equation 4 below.

이고,

와 같다.

는 카메라 포즈들이 변환 행렬로 표현됨을 나타낸다. here,

ego,

Same as

Indicates that camera poses are expressed as a transformation matrix.

The full photometric error can be expressed as Equation 5 below.

에서 실행되고,

는 프레임 i에서의 모든 포인트들

대하여 실행되며, j는

가 보이는 모든 프레임들

에서 실행된다. In Equation 5, i is every frame

runs on

is all points in frame i

is executed for, and j is

all visible frames

runs on

The aforementioned Equations 3 to 5 are for matching the brightness between frames. Since the brightness between frames can affect the depth value, in one embodiment, the brightness difference is calculated using the above-described Equations. By matching, a more accurate depth map can be calculated.

5 is a diagram for explaining a configuration according to an operation of an apparatus for calculating a depth map according to an exemplary embodiment. Referring to FIG. 5 , a calculation device 500 according to an embodiment includes a camera 510, a segmentation unit 520, a selector 530, a tracking unit 540, and A mapping unit 550 may be included. The segmentation unit 520, the selector 530, the tracking unit 540, and the mapping unit 550 are, for example, the processor 620 shown in FIG. can be implemented by

The camera 510 may photograph or capture a series of input images.

The division unit 520 may divide the input image into segments. The segmentation unit 520 may include a depth segmentation unit 523 that divides the input image into depth segments based on depth values and a semantic segmentation unit 526 that divides the input image into semantic segments of semantic units.

The selection unit 530 may select one or more segments to be used for tracking a camera pose and calculating a depth map from among segments based on reliability of the segments divided by the segmentation unit 520 . For example, the selection unit 530 may select segments in proportion to reliability of the segments.

The selection unit 530 may include a depth reliability evaluation unit 532 , a semantic reliability evaluation unit 534 , a reliability fusion unit 536 , and a pixel selection unit 538 .

The depth reliability evaluation unit 532 may evaluate (or calculate) reliability of depth segments. The semantic reliability evaluation unit 534 may evaluate (or calculate) reliability of semantic segments.

The confidence fusion unit 536 may fuse the confidence levels of the depth segments and the confidence levels of the semantic segments.

The pixel selection unit 538 may select a segment using the reliability fused in the reliability fusion unit 536 and select pixels from the selected segment.

The tracking unit 540 may calculate 6 DOF pose information of the camera including the position and posture of the camera. For example, the tracking unit 540 may continuously track new input images and calculate camera pose information in a current frame based on camera pose information in a previous frame. In this case, the tracking unit 540 may estimate camera pose information from the pixels of the selected segment of the previous frame in the pixel selection unit 538 . The tracking unit 540 may estimate camera pose information (eg, camera rotation information and translation information) by solving the above-described cost function for the selected segment.

The mapping unit 550 may calculate a depth map by calculating the depth of the photographed object. The mapping unit 550 may calculate a depth map of an input image using camera pose information estimated from pixels of a segment selected by the pixel selection unit 538 . The mapping unit 550 calculates a depth map using, for example, a depth value calculated using the position coordinates (u.v) of the camera, rotation information (R) and translation information (T) of the camera. can do.

The mapping unit 550 may generate a new key frame or refine a current key frame using the tracked frames. For example, when a camera capturing an input image moves too far to include objects captured in previous frames, the calculation device 500 may generate a new key frame from the most recently tracked frames. . If a new key frame is generated, the depth map of the corresponding key frame may be initialized by projecting points from the previous key frame to the new key frame. Also, frames that do not become new key frames among the tracked frames may be used to redefine the current key frames.

A depth map newly calculated by the mapping unit 550 may be added to the newly created key frame or the redefined key frame.

6 is a block diagram of an apparatus for calculating a depth map according to an exemplary embodiment. Referring to FIG. 6 , an apparatus 600 for calculating a depth map according to an exemplary embodiment (hereinafter referred to as 'calculating apparatus') 600 includes a camera 610 , a processor 620 , and a memory 630 . The calculation device 600 may further include a communication interface 640 and/or a display device 650 . The camera 610 , processor 620 , memory 630 , communication interface 640 and display device 650 may communicate with each other through a communication bus 605 .

The computing device 600 is an electronic device that implements various augmented reality applications in real time, such as, for example, an augmented reality head-up display, augmented reality/virtual reality glasses, autonomous vehicles, intelligent vehicles, smart phones, and mobile devices. can be

The camera 610 acquires an input image. The camera 610 may be, for example, an RGB camera or an RGB-D (Depth) camera. The input image is an image input to the calculation device 600, and may be, for example, a real-time image or a video. Alternatively, the input image may be a mono image or a stereo image. An input image may include a plurality of frames. The input image may be photographed or captured through the camera 610 or obtained from the outside of the calculation device 600 .

The processor 620 divides the input image into segments and selects one or more of the segments based on the reliability calculated for the segments. The processor 620 estimates camera pose information for the input image using the selected segment. The processor 620 calculates a depth map of the input image using pose information of the camera.

The processor 620 may classify objects included in the input image into semantic units and divide them into semantic segments. The processor 620 may divide the input image into depth segments based on the depth value of the input image. Alternatively, the processor 620 may divide the input image into semantic segments and depth segments.

The processor 620 may calculate a first confidence level for semantic segments or a second confidence level for depth segments. Alternatively, the processor 620 may calculate the first reliability for semantic segments and the second reliability for depth segments. For example, the processor 620 may calculate the first reliability of the semantic segments based on whether an object included in the input image is a moving object.

The processor 620 may fuse the first reliability level and the second reliability level, and determine the fused reliability level as the level of reliability for the segments. The processor 620 may calculate reliability of segments for each key frame among the frames.

The processor 620 may select pixels from the selected segment based on the fused reliability and estimate camera pose information from the selected pixels. Processor 620 may select pixels from the selected segment proportional to the fused confidence.

The processor 620 may estimate pose information of the camera by applying a cost function to the selected segment, for example.

In addition, the processor 620 may perform the method or an algorithm corresponding to the method described above with reference to FIGS. 1 to 5 . The processor 620 may execute a program and control the calculation device 600 . Program codes executed by the processor 620 may be stored in the memory 630 .

The memory 630 may store an input image and/or a plurality of frames. The memory 630 stores the pose information of the camera for the input image estimated by the processor 620, the depth map of the input image calculated by the processor 620, and/or the three images reconstructed using the depth map by the processor 620. dimensional images can be stored.

In addition, the memory 630 may store various pieces of information generated in the process of processing in the processor 620 described above. In addition, the memory 630 may store various data and programs. The memory 630 may include volatile memory or non-volatile memory. The memory 630 may include a mass storage medium such as a hard disk to store various types of data.

According to embodiments, the calculation device 600 may receive an input image captured outside the calculation device 600 through the communication interface 640 . In this case, the communication interface 640 may receive, in addition to the input image, pose information such as rotation information and movement information of the photographing device that captured the input image, location information of the photographing device, and/or calibration information of the photographing device. .

The display device 650 may display a 3D image reconstructed by the depth map calculated by the processor 620.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The estimation device may perform an operating system (OS) and one or more software applications running on the operating system. In addition, the estimation device may access, store, manipulate, process and generate data in response to execution of software. For convenience of understanding, there are cases in which one estimation device is used, but those skilled in the art will understand that the estimation device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, the estimation device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures the estimation device to operate as desired or which, independently or collectively, estimates You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or to provide instructions or data to an estimating device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved. Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

Dividing an input image into segments;
calculating reliability of the segments;
selecting one or more of the segments based on the confidence level;
estimating camera pose information for the input image using the selected segment; and
Calculating a depth map of the input image using pose information of the camera
including,
Calculating the reliability of the segments
calculating a first reliability level for semantic segments obtained by dividing objects included in the input image into semantic units;
calculating second reliability for depth segments obtained by dividing the input image based on a depth value of the input image; and
Determining reliability obtained by fusion of the first reliability and the second reliability as reliability for the segments
Including, a method for calculating a depth map. According to claim 1,
The dividing step is
dividing the input image into semantic segments by classifying objects included in the input image into semantic units; and
Dividing the input image into depth segments based on a depth value of the input image
A method for calculating a depth map, including any one or a combination thereof. delete According to claim 1,
Calculating the first reliability
Calculating first reliability for the semantic segments based on whether an object included in the input image is a moving object
Including, a method for calculating a depth map. According to claim 4,
Calculating the first reliability
determining a first reliability of a semantic segment corresponding to the moving object as a first value when the object is a moving object; and
If the object is a fixed object, determining a first reliability for a semantic segment corresponding to the fixed object as a second value.
Including, a method for calculating a depth map. delete According to claim 1,
selecting pixels from the selected segment based on the fused confidence
Including more,
Estimating the pose information of the camera
Estimating pose information of the camera from the selected pixels
Including, a method for calculating a depth map. According to claim 7,
Selecting pixels from the selected segment comprises
selecting pixels from the selected segment to be proportional to the fused confidence.
Including, a method for calculating a depth map. According to claim 1,
The input image includes frames,
The step of calculating the reliability is
calculating reliability of the segments for each key frame among the frames;
Including, a method for calculating a depth map. According to claim 1,
Estimating the pose information of the camera
Estimating pose information of the camera by applying a cost function to the selected segment
Including, a method for calculating a depth map. A computer program stored in a computer-readable recording medium in order to execute the method of any one of claims 1, 2, 4, 5, 7 to 10 in combination with hardware. a camera that acquires an input image; and
Dividing the input image into segments, selecting one or more of the segments based on reliability calculated for the segments, and estimating camera pose information for the input image using the selected segments; , A processor for calculating a depth map of the input image using the pose information of the camera
including,
The processor
Calculating first reliability for semantic segments obtained by classifying objects included in the input image into semantic units and dividing them;
Calculating a second reliability of depth segments obtained by dividing the input image based on a depth value of the input image;
and determining a reliability obtained by combining the first reliability and the second reliability as reliability for the segments. According to claim 12,
The processor
Objects included in the input image are divided into semantic units and the input image is divided into semantic segments, or the input image is divided into depth segments based on a depth value of the input image, or the input image is divided into semantic segments. Apparatus for calculating a depth map, dividing into the semantic segments and the depth segments. delete According to claim 12,
The processor
An apparatus for calculating a depth map, wherein a first reliability of the semantic segments is calculated based on whether an object included in the input image is a moving object. delete According to claim 12,
The processor
and selecting pixels from the selected segment based on the fused reliability, and estimating pose information of the camera from the selected pixels. According to claim 17,
The processor
and selecting pixels from the selected segment to be proportional to the fused confidence. According to claim 12,
The input image includes frames,
The processor
wherein reliability of the segments is calculated for each key frame among the frames. According to claim 12,
The processor
Apparatus for calculating a depth map, estimating pose information of the camera by applying a cost function to the selected segment.

Images