KR102488813B1 - Method and apparatus for generating disparity map using edge image - Google Patents

Abstract

To generate the disparity map, an initial disparity map is generated based on the first image and the second image, an edge image is generated based on the first image, and pixels in the edge image are generated based on the initial disparity map and the edge image. determine a target pixel of the target pixel, determine a first representative value of a first target region in the set edge image based on the target pixel, and determine a second representative value of a second target region of the initial disparity map corresponding to the first target region A target disparity value for a target pixel is determined by inputting the first representative value and the second representative value into a pre-learned algorithm, and a target disparity map is generated by updating an initial disparity map based on the target disparity value.

Description

Method and apparatus for generating parallax map using edge image {METHOD AND APPARATUS FOR GENERATING DISPARITY MAP USING EDGE IMAGE}

The following embodiments relate to a technique for generating a disparity map, and specifically to a technique for generating a disparity map using an edge image.

A method using a disparity map is used to detect an object in an image. For example, a disparity map may be generated by matching stereoscopic images taken at different viewpoints. When an object is detected using a disparity map, one of the most important information is a clear boundary of the object's outer perimeter. However, the disparity map generated by the above method may have errors due to the inaccuracy of the matching algorithm.

An embodiment may provide a method and apparatus for generating a disparity map using an edge image.

An embodiment may provide a method and apparatus for detecting an object using a disparity map.

According to one aspect,

1 is a schematic diagram of a method of detecting an object using stereoscopic images according to an example.
2 is a configuration diagram of an electronic device generating a disparity map according to an exemplary embodiment.
3 is a flowchart of a method of generating a disparity map according to an exemplary embodiment.
4 is a flowchart of a method of determining a target pixel among pixels in an edge image according to an example.
5 is a schematic diagram of a method for determining a target pixel in an edge image according to an example.
6 is a flowchart of a method of determining a first representative value of a first target area in an edge image according to an example;
7 illustrates a representative matrix generated as a first representative value of a first target region according to an example.
8 is a flowchart of a method of determining a target disparity value by further using a gray image according to an example.
9 is a schematic diagram of an algorithm for determining a target parallax value according to an example.
10 is a flowchart of a method of detecting an object according to an example.
11 is a flowchart of a method of determining an outer pixel based on a matrix in a horizontal direction according to an example.
12 illustrates an object detected based on a disparity map according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these examples. Like reference numerals in each figure indicate like elements.

Various changes may be made to the embodiments described below. The embodiments described below are not intended to be limiting on the embodiments, and should be understood to include all modifications, equivalents or substitutes thereto.

Terms used in the examples are used only to describe specific examples, and are not intended to limit the examples. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

1 is a schematic diagram of a method of detecting an object using stereoscopic images according to an example.

A disparity map may be generated based on the left image and the right image. The left image and the right image may be stereoscopic images generated at different viewpoints.

A disparity map may be generated based on a difference between the matched pixels by matching the pixels of the left image and the pixels of the right image with each other.

An object may be detected based on a difference between disparity values generated between boundaries of the objects. In order to accurately detect an object, the accuracy of the disparity map is required. Accordingly, the initial disparity map may be updated by detecting a pixel with an error among pixels in the initial disparity map generated based on the stereoscopic images and accurately calculating a disparity value of the corresponding pixel.

Below, a method of generating a disparity map will be described in detail with reference to FIGS. 2 to 9 , and a method of detecting an object will be described in detail with reference to FIGS. 10 to 12 .

2 is a configuration diagram of an electronic device generating a disparity map according to an exemplary embodiment.

The electronic device 200 includes a communication unit 210, a processor 220, and a memory 230.

The communication unit 210 is connected to the processor 220 and the memory 230 to transmit and receive data. The communication unit 210 may transmit/receive data by being connected to another external device. Hereinafter, the expression “transmitting and receiving “A” may indicate transmitting and receiving “information or data indicating A”.

The communication unit 210 may be implemented as circuitry within the electronic device 200 . For example, the communication unit 210 may include an internal bus and an external bus. As another example, the communication unit 210 may be an element that connects the electronic device 200 and an external device. The communication unit 210 may be an interface. The communication unit 210 may receive data from an external device and transmit the data to the processor 220 and the memory 230 .

The processor 220 processes data received by the communication unit 210 and data stored in the memory 230 . A “processor” may be a data processing device implemented in hardware having circuitry having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program. For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

Processor 220 executes computer readable code (eg, software) stored in memory (eg, memory 230 ) and instructions invoked by processor 220 .

The memory 230 stores data received by the communication unit 210 and data processed by the processor 220 . For example, the memory 230 may store a program (or application or software). The stored program may be a set of syntaxes coded to generate a disparity map and executed by the processor 220 .

According to one aspect, the memory 230 may include one or more of volatile memory, non-volatile memory and random access memory (RAM), flash memory, a hard disk drive, and an optical disk drive.

The memory 230 stores a command set (eg, software) for operating the electronic device 200 . A set of instructions for operating the electronic device 200 is executed by the processor 220 .

The communication unit 210, the processor 220, and the memory 230 will be described in detail with reference to FIGS. 3 to 12 below.

3 is a flowchart of a method of generating a disparity map according to an exemplary embodiment.

Steps 310 to 370 below are performed by the electronic device 200 described above with reference to FIG. 2 .

In operation 310, the electronic device 200 may generate an initial disparity map based on the first image and the second image. The first image and the second image may be stereoscopic images. A method of generating an initial disparity map based on stereoscopic images is not limited, and a description of a method of generating an initial disparity map will be omitted for a clear and simple description of the invention.

In step 320, the electronic device 200 generates an edge image based on the first image. For example, the electronic device 200 may generate an edge image by detecting contour stones in the first image.

In step 330, the electronic device 200 determines a target pixel among pixels in the edge image. The target pixel may be a pixel corresponding to a corresponding pixel that needs to be improved or updated in the initial disparity map.

A method for determining the target pixel is explained in detail with reference to FIGS. 4 and 5 below.

In step 340, the electronic device 200 determines a first representative value of a first target region in the set edge image based on the target pixel. The reason for setting the first target area may be to determine a parallax value for the target pixel in consideration of neighboring pixels of the target pixel. The reason why the first representative value of the first target area is determined is that it is difficult for an algorithm for determining a parallax value to consider all values in the first target area, so a representative value may be used.

A method of setting the first target region and a method of determining the first representative value will be described in detail with reference to FIGS. 6 and 7 below.

In operation 350, the electronic device 200 determines a second representative value of a second target area of the initial disparity map corresponding to the first target area. Since a method of determining the second representative value of the second target area is similar to the method of determining the first representative value of the first target area, a description thereof will be omitted.

In step 360, the electronic device 200 determines a target parallax value for the target pixel by inputting the first representative value and the second representative value to a pretrained algorithm. For example, the algorithm may be an algorithm that can be trained or learned, such as an algorithm based on a neural network or a support vector machine (SVM), and is not limited to the described embodiments. For example, the algorithm is an algorithm learned based on ground truth, may receive a first representative value and a second representative value as inputs, and output a disparity value of a corresponding pixel of a disparity map as a result. The output parallax value is the target parallax value.

The algorithm is described in detail with reference to FIG. 9 below.

In step 370, the electronic device 200 creates a target disparity map by updating the initial disparity map based on the target disparity value. For example, the initial disparity map may be updated by changing an existing disparity value of a corresponding pixel to a target disparity value.

When there are a plurality of corresponding pixels to be updated, the final disparity map may be generated by repeatedly updating the target disparity map.

The final disparity map may be used to detect objects in the first image or the second image. A method of detecting an object in the first image using the disparity map will be described in detail with reference to FIGS. 10 to 12 .

4 is a flowchart of a method of determining a target pixel among pixels in an edge image according to an example.

According to one aspect, the step 330 described above with reference to FIG. 3 may include steps 410 to 430 below.

In step 410, the electronic device 200 determines a candidate target pixel in which an edge component in the edge image exists.

In step 420, the electronic device 200 determines a corresponding pixel of the initial disparity map corresponding to the candidate target pixel.

In operation 430, the electronic device 200 determines a candidate target pixel as a target pixel when parallax values change within a preset range based on the corresponding pixel. Changes in parallax values may be changes greater than or equal to a preset range.

5 is a schematic diagram of a method for determining a target pixel in an edge image according to an example.

A candidate target pixel having an edge component among pixels in the edge image 510 may be determined, and corresponding pixels of the initial disparity map 520 corresponding to the candidate target pixel may be determined.

When parallax values change within a preset range 521 based on the corresponding pixel, the candidate target pixel is determined as the target pixel 512 .

Also, a first target region 520 may be set based on the target pixel 512 for further processing. The first target area 520 will be described in detail with reference to FIGS. 6 and 7 below.

6 is a flowchart of a method of determining a first representative value of a first target area in an edge image according to an example;

According to one aspect, the step 340 described above with reference to FIG. 3 may include the following steps 610 to 630.

In step 610, the electronic device 200 sets a first target area based on the target pixels. For example, the first target area may be centered on the target pixel.

In step 620, the electronic device 200 generates a plurality of matrices corresponding to the sliding position of the window by sliding a window having a preset size within the first target area. When the window is located at a specific location within the first target area, values of pixels included in the window at that location may be generated in a matrix form.

In step 630, the electronic device 200 generates a representative matrix as a first representative value based on the plurality of matrices. For example, the size of the representative matrix may be the same as the size of the window. An average value, a median value, or a mode value of co-located values of a plurality of matrices may be determined as the value of the representative matrix.

A method of generating a representative matrix based on a plurality of matrices will be described in detail with reference to FIG. 7 below.

7 illustrates a representative matrix generated as a first representative value of a first target region according to an example.

The size of the preset window 710 may be Px×Py, and the size of the preset first target area 720 may be Sx×Sy.

By sliding the window 710 within the first target area 720 , first matrices 730 to (Sx-Px+1)×(Sy-Py+1) 740 matrices are generated.

A representative value of a corresponding position may be determined based on values of the same position of the first matrix 730 to the (Sx-Px+1)×(Sy-Py+1)th matrix 740 . For example, an average value, a median value, or a mode value of the values 731 and 741 at the top left of the matrices may be determined as the representative value 751 at the top left of the representative matrix 750 . As another example, the average value, the median value, or the most frequent value of the values 732 and 742 at the upper rightmost position of the matrices may be determined as the representative value 752 at the uppermost rightmost position of the representative matrix 750 .

8 is a flowchart of a method of determining a target disparity value by further using a gray image according to an example.

The disparity map update method described with reference to FIGS. 3 to 7 uses an edge image and an initial disparity map, but a gray image may be further used to update the disparity map. Step 810 below may be performed independently of steps 310 and 320 and in parallel.

In step 810, the electronic device 200 generates a gray image based on the first image.

In operation 820, the electronic device 200 determines a third representative value of a third target area of the gray image corresponding to the first target area. Since the method of determining the third representative value of the third target region is similar to the method of determining the first representative value of the first target region, a description thereof is omitted.

According to one aspect, step 360 may include step 830 .

In operation 830, the electronic device 200 determines a target parallax value for the target pixel by inputting the first representative value, the second representative value, and the third representative value into an algorithm.

9 is a schematic diagram of an algorithm for determining a target parallax value according to an example.

According to one aspect, algorithm 920 may be composed of a plurality of layers. In the algorithm 920, connection parameters between nodes of layers may be adjusted through training. As the connection parameter is preferably adjusted, a parallax value closer to ground truth may be output.

The input layer 910 may include a first representative value, a second representative value, and a third representative value. For example, when the representative value is a matrix, Px×Py edge representative values 911 to 912, Px×Py disparity representative values 913 to 914, and Px×Py gray representative values 915 to 916 It can be input to the algorithm 920.

A target disparity value 930 may be output as a result of an input by the operation of the algorithm 920 .

10 is a flowchart of a method of detecting an object according to an example.

According to one aspect, step 1000 below may include steps 1010 to 1040, and step 1000 may be performed after step 370 described above with reference to FIG. 3 is performed. Step 1000 is a method of detecting an object.

In step 1010, the electronic device 200 generates a first pixel matrix in a horizontal direction including target pixels of the disparity map. The target pixel is a pixel to be inspected. The first pixel matrix in the horizontal direction will be described in detail with reference to FIG. 11 below.

In step 1020, the electronic device 200 generates a second pixel matrix in the vertical direction including the target pixel.

In operation 1030, the electronic device 200 determines whether the target pixel is a pixel outside the object based on the first pixel matrix and the second pixel matrix.

In step 1040, the electronic device 200 detects the object in the first image based on the target pixel when the target pixel is a pixel outside the object.

11 is a flowchart of a method of determining an outer pixel based on a matrix in a horizontal direction according to an example.

The first pixel matrix 1110 in the horizontal direction includes a plurality of pixels. Parallax values of a plurality of pixels may be represented as a graph 1120 . A position 1121 representing a change in the graph 1120 may be determined. The pixel 1111 corresponding to the location 1121 or pixels around the pixel 1111 may be determined as an outer pixel.

12 illustrates an object detected based on a disparity map according to an example.

Outer pixels 1210 may be detected in the disparity map, and an area including the outer pixels 1210 may be detected as the detection area 1220 .

An area in the first image corresponding to the detection area 1220 may be determined as an object area 1230 .

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

200: electronic device
210: communication department
220: processor
230: memory

Claims (14)

The method for generating a disparity map, performed by an electronic device,
generating an initial disparity map based on the first image and the second image;
generating an edge image based on the first image;
determining a target pixel among pixels in the edge image based on the initial disparity map and the edge image;
determining a first representative value of a first target area in the edge image set based on the target pixel;
determining a second representative value of a second target area of the initial disparity map corresponding to the first target area;
determining a target parallax value for the target pixel by inputting the first representative value and the second representative value to a previously learned algorithm; and
generating a target disparity map by updating the initial disparity map based on the target disparity value;
including,
How to create a parallax map.
According to claim 1,
generating a gray image based on the first image;
Determining a third representative value of a third target area of the gray image corresponding to the first target area
Including more,
Determining a target parallax value for the target pixel,
determining the target parallax value for the target pixel by inputting the first representative value, the second representative value, and the third representative value to the pre-learned algorithm;
including,
How to create a parallax map.
According to claim 1,
The first image and the second image are stereo images,
How to create a parallax map.
According to claim 1,
Determining a target pixel among pixels in the edge image based on the initial disparity map and the edge image,
determining a candidate target pixel in which an edge component exists;
determining a corresponding pixel of the initial disparity map corresponding to the candidate target pixel; and
determining the candidate target pixel as the target pixel when parallax values change within a preset range based on the corresponding pixel;
including,
How to create a parallax map.
According to claim 1,
Determining a first representative value of a first target region in the edge image set based on the target pixel,
setting the first target area based on the target pixel;
generating a plurality of matrices corresponding to positions where the window is slid by sliding a window having a preset size within the first target area; and
generating a representative matrix as the first representative value based on the plurality of matrices;
including,
How to create a parallax map.

According to claim 5,
Generating a representative matrix as the first representative value based on the plurality of matrices,
Determining an average value, a median value, or a mode value of co-located values of the plurality of matrices as a value of the representative matrix
including,
How to create a parallax map.
According to claim 1,
Detecting an object in the first image based on a final disparity map generated by updating the target disparity map
Including more,
How to create a parallax map.
According to claim 7,
The step of detecting an object in the first image based on the final disparity map,
generating a first pixel matrix in a horizontal direction including target pixels of the final disparity map;
generating a second pixel matrix in a vertical direction including the target pixel;
determining whether the target pixel is an outer pixel of the object based on the first pixel matrix and the second pixel matrix; and
Detecting the object in the first image based on the target pixel when the target pixel is an outer pixel of the object
including,
How to create a parallax map.
A computer-readable recording medium containing a program for performing the method of any one of claims 1 to 8.
An electronic device for generating a parallax map,
a memory in which a program for generating a parallax map is recorded; and
Processor that executes the above program
including,
said program,
generating an initial disparity map based on the first image and the second image;
generating an edge image based on the first image;
determining a target pixel among pixels in the edge image based on the initial disparity map and the edge image;
determining a first representative value of a first target area in the edge image set based on the target pixel;
determining a second representative value of a second target area of the initial disparity map corresponding to the first target area;
determining a target parallax value for the target pixel by inputting the first representative value and the second representative value to a previously learned algorithm; and
generating a target disparity map by updating the initial disparity map based on the target disparity value;
to do,
electronic device.
According to claim 10,
said program,
generating a gray image based on the first image;
Determining a third representative value of a third target area of the gray image corresponding to the first target area
do more,
Determining a target parallax value for the target pixel,
determining the target parallax value for the target pixel by inputting the first representative value, the second representative value, and the third representative value to the pre-learned algorithm;
including,
electronic device.
According to claim 10,
Determining a target pixel among pixels in the edge image based on the initial disparity map and the edge image,
determining a candidate target pixel in which an edge component exists;
determining a corresponding pixel of the initial disparity map corresponding to the candidate target pixel; and
determining the candidate target pixel as the target pixel when parallax values change within a preset range based on the corresponding pixel;
including,
electronic device.
According to claim 10,
Determining a first representative value of a first target region in the edge image set based on the target pixel,
setting the first target area based on the target pixel;
generating a plurality of matrices corresponding to positions where the window is slid by sliding a window having a preset size within the first target area; and
generating a representative matrix as the first representative value based on the plurality of matrices;
including,
electronic device.
According to claim 10,
said program,
Detecting an object in the first image based on a final disparity map generated by updating the target disparity map
to do more,
electronic device.

Images