CN104616304A

CN104616304A - Self-adapting support weight stereo matching method based on field programmable gate array (FPGA)

Info

Publication number: CN104616304A
Application number: CN201510072013.4A
Authority: CN
Inventors: 顾国华; 龚文彪; 吕芳; 任建乐; 钱惟贤; 路东明; 任侃; 于雪莲
Original assignee: Nanjing University of Science and Technology
Current assignee: Nanjing University of Science and Technology
Priority date: 2015-02-11
Filing date: 2015-02-11
Publication date: 2015-05-13

Abstract

The invention proposes a stereo matching method based on FPGA-based self-adaptive support weights. Establish a local stereo matching window for the left and right images in the FPGA; then according to the gray similarity between the matching point in the local window and the center point of the local window and the similarity of the Manhattan distance, the gray similarity function value and the Manhattan distance of the points to be matched are obtained. distance similarity function value, so as to obtain the weight cost relationship value of the local window matching point; finally, according to the matching cost weight relationship value of the local matching window, calculate the cost aggregation relationship of each matching point, and then use the winner is king criterion , to obtain the disparity result of each pixel. The present invention improves the overall matching effect, can obtain real-time dense disparity results, and has strong robustness.

Description

A Stereo Matching Method Based on FPGA Based Adaptive Support Weight

技术领域technical field

本发明属于双目立体视觉领域，具体涉及一种基于FPGA的自适应支撑权重的立体匹配方法。The invention belongs to the field of binocular stereo vision, and in particular relates to a stereo matching method based on FPGA-based self-adaptive support weights.

背景技术Background technique

双目立体视觉是直接模拟人类双眼视觉的生理结构，它是三维场景下深度信息提取的重要技术手段，在机器人导航、无人机定位以及三维测量中具有广泛的运用，立体视觉中最关键和难点在于双目图像的立体匹配，匹配过程中由于需要进行大量数据的重复运算，因此在CPU上对双目图像进行立体匹配需要花费大量的运算时间，难以满足实时性的要求。例如，对于主频为1GHZ的CPU，利用区域立体匹配算法对两幅中型大小的图片进行稠密视差图计算，需要花费几秒的时间，这样低速率的处理结果极大的限制了立体视觉的发展，特别是一些需要得到实时匹配视差结果的实际运用场合。因此，近年来如何在保证匹配精度的条件下提高立体匹配算法的实时性，成为了人们研究的热点。Binocular stereo vision is a physiological structure that directly simulates human binocular vision. It is an important technical means for depth information extraction in 3D scenes. It is widely used in robot navigation, UAV positioning and 3D measurement. The most critical and The difficulty lies in the stereo matching of binocular images. Due to the need for repeated calculations of a large amount of data during the matching process, it takes a lot of computing time to perform stereo matching of binocular images on the CPU, which is difficult to meet the real-time requirements. For example, for a CPU with a main frequency of 1GHZ, it takes a few seconds to calculate the dense disparity map of two medium-sized pictures using the regional stereo matching algorithm. Such low-speed processing results greatly limit the development of stereo vision. , especially in some practical applications that require real-time matching disparity results. Therefore, in recent years, how to improve the real-time performance of the stereo matching algorithm under the condition of ensuring the matching accuracy has become a research hotspot.

区域立体匹配算法包含全局区域立体匹配算法和局部区域立体匹配算法，全局区域立体匹配算法匹配准确度高，但计算复杂、运算量大，在提高实时性方面可以采用GPU来实现，但是GPU处理过程中功耗很大，因此限制器实际的运用场合。对应局部区域立体匹配算法，该方法是通过左右图像上的局部窗口的相似性关系来获得每个待匹配点的视差值，计算量小，且匹配过程中如果能够找到一个合适的匹配代价关系，也能获得一个较好的匹配结果，而FPGA作为一种可编程的逻辑门阵列，灵活性强，利用其内部的并行流水线技术可以实时的实现局部区域立体匹配算法。The regional stereo matching algorithm includes a global stereo matching algorithm and a local stereo matching algorithm. The global stereo matching algorithm has high matching accuracy, but the calculation is complex and the amount of calculation is large. GPU can be used to improve real-time performance, but the GPU processing process The power consumption in the middle is very large, so the actual application occasion of the limiter. Corresponding to the local area stereo matching algorithm, this method is to obtain the disparity value of each point to be matched through the similarity relationship between the local windows on the left and right images, the calculation amount is small, and if a suitable matching cost relationship can be found during the matching process , can also obtain a better matching result, and FPGA, as a programmable logic gate array, has strong flexibility. Using its internal parallel pipeline technology, it can realize the local area stereo matching algorithm in real time.

目前，利用FPGA实现实时的局部区域立体匹配方法有Ambrosch K等人提出的基于FPGA实现的SAD立体匹配方法(详见文献Hardware implementation of an SAD basedstereo vision algorithm)、Chen L等人提出的基于FPGA实现的实时立体匹配方法(详见文献A Parallel reconfigurable architecture for real-time stereo vision)。该类方法能够实时的获得到实时的稠密视差结果，但匹配过程中的局部窗口为固定值，且对窗口内的匹配点在匹配过程中没有分配合适的权重信息，因此在深度不连续点处、低纹理处以及场景重复区域处容易匹配出错，匹配准确率不高。At present, the real-time local area stereo matching method using FPGA includes FPGA-based SAD stereo matching method proposed by Ambrosch K et al. Real-time stereo matching method (see literature A Parallel reconfigurable architecture for real-time stereo vision for details). This type of method can obtain real-time dense disparity results in real time, but the local window in the matching process is a fixed value, and no appropriate weight information is assigned to the matching points in the window during the matching process, so the depth discontinuity point It is easy to make matching errors in places with low texture and repeated scenes, and the matching accuracy is not high.

发明内容Contents of the invention

本发明的目的针对现有立体匹配方法在匹配准确率和实时性方面存在的不足，提出了一种基于FPGA的自适应支撑权重的立体匹配方法，该方法可以解决深度不连续点处、低纹理处以及场景重复区域处容易匹配出错的问题，从而可以提高整体的匹配效果，同时，该方法利用FPGA内部的并行流水线技术，可以在FPGA内进行并行的局部区域立体匹配，获得实时的稠密视差结果，具有较强的鲁棒性。The purpose of the present invention aims at the deficiencies of existing stereo matching methods in terms of matching accuracy and real-time performance, and proposes a stereo matching method based on FPGA-based adaptive support weights. It is easy to make matching errors in places and repeated areas of the scene, so as to improve the overall matching effect. At the same time, this method uses the parallel pipeline technology inside the FPGA to perform parallel local area stereo matching in the FPGA to obtain real-time dense disparity results. , has strong robustness.

为了解决上述技术问题，本发明提供一种基于FPGA的自适应支撑权重的立体匹配方法，在FPGA中对左右图像建立一个m×n大小的局部立体匹配窗口，视频输入口的输入数据为预处理后的左右图像，像素点的时钟为左右图像每个像素点进入FPGA内部的系统同步时钟，用先进先出缓存器对匹配点进行行缓存，用FPGA内部的D触发器对匹配点进行列缓存；然后根据局部窗口内匹配点与局部窗口中心点间的灰度相似性和曼哈顿距离相似性，求取待匹配点的灰度相似性函数值和曼哈顿距离相似性函数值，从而得到局部窗口匹配点的权重代价关系值w(p,q)，权重代价关系值w(p,q)如式(1)所示，In order to solve the above-mentioned technical problems, the present invention provides a stereo matching method based on FPGA-based self-adaptive support weights, in which a local stereo matching window of m×n size is established for the left and right images in the FPGA, and the input data of the video input port is preprocessed After the left and right images, the clock of the pixel point is the system synchronous clock that each pixel point of the left and right images enters into the FPGA, and uses the first-in-first-out buffer to cache the matching points, and uses the D flip-flop inside the FPGA to cache the matching points ; Then according to the gray similarity and Manhattan distance similarity between the matching point in the local window and the local window center point, calculate the gray similarity function value and the Manhattan distance similarity function value of the point to be matched, so as to obtain the local window matching The weight-cost relationship value w(p,q) of the point, and the weight-cost relationship value w(p,q) is shown in formula (1),

w(p,q)＝wd_k·wR_l (1)w(p,q)=wd _k ·wR _l (1)

式(1)中，wd_k表示在FPGA内部通过查找表获得的像素点p和q的曼哈顿距离相似性函数值的对应值，R_l表示像素点p和q的灰度值在经过rank变换后的灰度相似性关系，wR_l表示在FPGA内部通过查找表获得的像素点p和q的灰度相似性函数值的对应值；最后根据所述局部匹配窗口的匹配代价权重关系值，计算出每个匹配点的代价聚合关系，然后利用胜者为王准则，求取每个像素点的视差结果；所述每个匹配点的代价聚合关系如式(2)所示，In formula (1), wd _k represents the corresponding value of the Manhattan distance similarity function value of the pixel point p and q obtained through the lookup table inside the FPGA, and R _l represents the gray value of the pixel point p and q after rank transformation The gray-scale similarity relationship of wR ₁ represents the corresponding value of the gray-scale similarity function value of the pixel point p and q obtained by the look-up table inside the FPGA; finally, according to the matching cost weight relationship value of the local matching window, calculate The cost aggregation relationship of each matching point, and then use the winner is king criterion to obtain the disparity result of each pixel point; the cost aggregation relationship of each matching point As shown in formula (2),

$E E. ((p p,, \overset{&OverBar; &OverBar;}{{p p}_{d d}})) = = \frac{{Σ Σ}_{q q &Element; &Element; {N N}_{P P},, \overset{&OverBar; &OverBar;}{{q q}_{d d}} &Element; &Element; {N N}_{\overset{&OverBar; &OverBar;}{Pd PD}}} w w ((p p,, q q)) w w ((\overset{&OverBar; &OverBar;}{{p p}_{d d}},, \overset{&OverBar; &OverBar;}{{q q}_{d d}})) {e e}_{m m} ((q q,, \overset{&OverBar; &OverBar;}{{q q}_{d d}}))}{{Σ Σ}_{q q &Element; &Element; {N N}_{P P},, \overset{&OverBar; &OverBar;}{{q q}_{d d}} &Element; &Element; {N N}_{\overset{&OverBar; &OverBar;}{Pd PD}}} w w ((p p,, q q)) w w ((\overset{&OverBar; &OverBar;}{{p p}_{d d}},, \overset{&OverBar; &OverBar;}{{q q}_{d d}}))} - - - - - - ((22))$

式(2)中，N_p表示左图像的局部匹配窗口，表示右图像的局部匹配窗口，w(p,q)为左图像中匹配点q的权重关系值，为右图像中的匹配点的权重关系值，对于左右图像的局部窗口，p点和点对应，q点和点相对应，为匹配点经rank变换后的比较值，表达式如式(3)所示：In formula (2), N _p represents the local matching window of the left image, Represents the local matching window of the right image, w(p,q) is the weight relationship value of the matching point q in the left image, is the matching point in the right image The weight relationship value of , for the local window of the left and right images, p points and points correspond to points q and corresponding to the point, is the comparison value of the matching point after rank transformation, The expression is shown in formula (3):

${e e}_{m m} ((q q,, \overset{&OverBar; &OverBar;}{{q q}_{d d}})) = = \{\begin{matrix} 00 & {R R}_{pq pq} = = {R R}_{\overset{&OverBar; &OverBar;}{{p p}_{d d}} \overset{&OverBar; &OverBar;}{{q q}_{d d}}} \\ 11 & otherwise otherwise \end{matrix} - - - - - - ((33))$

式(3)中，R_pq表示q点的rank变换值，表示点的rank变换值。In formula (3), R _pq represents the rank transformation value of point q, express The rank transformation value of the point.

较佳地，选用高斯函数对视频图像进行预处理，滤除噪声。Preferably, a Gaussian function is selected to preprocess the video image to filter out noise.

本发明与现有技术相比，其显著优点在于，(1)本发明在硬件FPGA上对双目图像进行局部区域立体匹配，从而可以获得实时的稠密视差结果；(2)本发明利用局部窗口内匹配点与窗口中心点间的灰度相似性、曼哈顿距离相似性建立自适应权重关系，从而可以提高匹配过程中的准确率；(3)本发明在局部窗口的匹配过程中，用rank变换后的值替代原始的灰度相似性函数关系值，从而可以在FPGA内部建立一个固定的灰度查找表，有利于FPGA的实时性计算；(4)本发明在代价聚合过程中，将匹配点先经过rank变换后再进行视差值计算，从而消除光照和噪声对左右图像匹配结果的影响。Compared with the prior art, the present invention has significant advantages in that (1) the present invention performs local area stereo matching on binocular images on the hardware FPGA, thereby obtaining real-time dense parallax results; (2) the present invention utilizes local windows The gray-scale similarity between the inner matching point and the window center point, the Manhattan distance similarity establish an adaptive weight relationship, thereby the accuracy rate in the matching process can be improved; (3) the present invention uses rank transformation in the matching process of the local window The final value replaces the original gray-scale similarity function relationship value, thereby a fixed gray-scale lookup table can be established inside the FPGA, which is conducive to the real-time calculation of the FPGA; (4) the present invention will match the matching point After the rank transformation, the disparity value is calculated, so as to eliminate the influence of illumination and noise on the matching results of the left and right images.

附图说明Description of drawings

图1是本发明方法流程图。Fig. 1 is a flow chart of the method of the present invention.

图2是本发明在FPGA中对左右图像建立m×n大小的局部区域立体匹配窗口。Fig. 2 is the local area stereo matching window of m×n size established for the left and right images in the FPGA of the present invention.

图3是本发明在FPGA中对像素点进行rank变换的过程示意图。Fig. 3 is a schematic diagram of the process of performing rank transformation on pixels in FPGA according to the present invention.

图4是本发明在FPGA中建立的相似性函数查找表，其中(a)为曼哈顿距离相似性查找表，(b)为灰度相似性查找表。Fig. 4 is the similarity function lookup table that the present invention establishes in FPGA, and wherein (a) is the Manhattan distance similarity lookup table, (b) is the gray scale similarity lookup table.

图5是本发明在左右图像的局部窗口中对相应的匹配点进行代价聚合运算示意图，其中(a)为左图像窗口的示意图，(b)为右图像窗口的示意图。Fig. 5 is a schematic diagram of the present invention performing cost aggregation operation on corresponding matching points in the local windows of the left and right images, wherein (a) is a schematic diagram of the left image window, and (b) is a schematic diagram of the right image window.

图6是本发明代价聚合过程在FPGA中的实现示意图。FIG. 6 is a schematic diagram of the implementation of the cost aggregation process in the FPGA of the present invention.

具体实施方式Detailed ways

结合图1，本发明基于FPGA的自适应支撑权重的立体匹配方法，步骤如下：In conjunction with Fig. 1, the stereo matching method of the adaptive support weight based on FPGA of the present invention, the steps are as follows:

步骤1：利用标定好的左右摄像机获得极线矫正好的左右图像，将采集到的左右图像进行滤波处理去除噪声干扰，预处理后的左图像为I_l(i,j)和右图像为I_r(i,j)；Step 1: Use the calibrated left and right cameras to obtain the left and right images after epipolar correction, and filter the collected left and right images to remove noise interference. The preprocessed left image is I _l (i, j) and the right image is I _r (i,j);

本发明选用高斯函数对图像进行滤波，从而有效的消除传感器所引入的高斯白噪声。滤波模板采用(2k+1)×(2k+1)维(其中k＝1,2,3,...)的离散高斯卷积核，计算方式如公式(1)所示：The invention uses Gaussian function to filter the image, thereby effectively eliminating Gaussian white noise introduced by the sensor. The filter template adopts a (2k+1)×(2k+1) dimensional (where k=1,2,3,...) discrete Gaussian convolution kernel, and the calculation method is shown in formula (1):

$\{\begin{matrix} {I I}_{l l} ((i i,, j j)) = = {I I}_{r r 00} ((i i,, j j)) * * G G ((u u,, v v)) = = {Σ Σ}_{u u = = - - k k}^{k k} {Σ Σ}_{v v = = - - k k}^{k k} {I I}_{r r 00} ((i i + + u u,, j j + + v v)) \cdot &Center Dot; G G ((u u,, v v)) \\ {I I}_{r r} ((i i,, j j)) = = {I I}_{r r 00} ((i i,, j j)) * * G G ((u u,, v v)) = = {Σ Σ}_{u u = = - - k k}^{k k} {Σ Σ}_{v v = = - - k k}^{k k} {I I}_{r r 00} ((i i + + u u,, j j + + v v)) \cdot &Center Dot; G G ((u u,, v v)) \\ G G ((u u,, v v)) = = \frac{11}{{22 πσ πσ}^{22}} {e e}^{- - \frac{{u u}^{22} + + {v v}^{22}}{{22 σ σ}^{22}}} \end{matrix} - - - - - - ((11))$

公式(1)中，(i,j)为图像像素点的坐标值，I_l0(i,j)为输入的原始左图像，I_r0(i,j)为输入的原始右图像，(u,v)为离散高斯点坐标，G(u,v)为离散高斯核函数在(u,v)处的归一化值，σ为高斯函数尺度值。In formula (1), (i, j) is the coordinate value of the image pixel, I _l0 (i, j) is the input original left image, I _r0 (i, j) is the input original right image, (u, v) is the discrete Gaussian point coordinates, G(u,v) is the normalized value of the discrete Gaussian kernel function at (u,v), and σ is the scale value of the Gaussian function.

步骤2：结合图2，在FPGA中对左右图像建立一个m×n大小的局部立体匹配窗口，视频输入口的输入数据为预处理后的左右图像，像素点的时钟为左右图像每个像素点进入FPGA内部的系统同步时钟，FIFO为先进先出缓存器，对匹配点进行行缓存，列缓存通过FPGA内部的D触发器进行缓存，win为建立的局部窗口，由此在FPGA中可以得到一个并行的局部窗口，用于左右图像的局部区域立体匹配。Step 2: Combining with Figure 2, establish a local stereo matching window of m×n size for the left and right images in the FPGA, the input data of the video input port is the preprocessed left and right images, and the pixel clock is each pixel of the left and right images Entering the system synchronous clock inside the FPGA, FIFO is a first-in-first-out buffer, which caches the matching points, and the column cache is cached through the D flip-flop inside the FPGA, and win is the local window established, so one can get a Parallel local windows for local region stereo matching of left and right images.

步骤3：根据局部窗口内匹配点与局部窗口中心点间的灰度相似性和曼哈顿距离相似性，求取待匹配点的灰度相似性函数值f(Δc_pq)和曼哈顿距离相似性函数值f(Δd_pq)，从而得到局部窗口匹配点的权重代价关系值w(p,q)，如下式(2)所示：Step 3: According to the gray similarity and Manhattan distance similarity between the matching point in the local window and the center point of the local window, calculate the gray similarity function value f(Δc _pq ) and the Manhattan distance similarity function value of the point to be matched f(Δd _pq ), so as to obtain the weight cost relationship value w(p,q) of the local window matching point, as shown in the following formula (2):

w(p,q)＝f(Δc_pq)·f(Δd_pq) (2)w(p,q)=f(Δc _pq )·f(Δd _pq ) (2)

式(2)中，p和q分别为图像匹配窗口的中心像素点和窗口区域内的像素点，Δc_pq表示p和q点的灰度相似性关系，Δd_pq表示p和q点的曼哈顿距离相似性关系。其中Δc_pq、Δd_pq的关系如式(3)所示，f(Δc_pq)、f(Δd_pq)函数表达式关系如式(4)所示。In formula (2), p and q are the central pixel of the image matching window and the pixels in the window area, respectively, Δc _pq represents the gray similarity relationship between p and q points, and Δd _pq represents the Manhattan distance between p and q points similarity relationship. Among them, the relationship between Δc _pq and Δd _pq is shown in formula (3), and the relationship between the function expressions of f(Δc _pq ) and f(Δd _pq ) is shown in formula (4).

$\{\begin{matrix} {Δc Δ c}_{pq pq} = = {I I}_{p p} - - {I I}_{q q} \\ {Δd Δd}_{pq pq} = = | | {x x}_{p p} - - {x x}_{q q} | | + + {| | y the y}_{p p} - - {y the y}_{q q} | | \end{matrix} - - - - - - ((33))$

$\{\begin{matrix} f f (({Δc Δ c}_{pq pq})) = = exp exp ((- - \frac{{Δc Δ c}_{pq pq}}{{τ τ}_{c c}})) \\ f f (({Δd Δd}_{pq pq})) = = exp exp ((- - \frac{{Δd Δd}_{pq pq}}{{τ τ}_{d d}})) \end{matrix} - - - - - - ((44))$

式(3)中，I_p表示p点的灰度值，I_q表示q点的灰度值，(x_p,y_p)表示p点的行列坐标值，(x_q,y_q)表示q点的行列坐标值，式(4)中exp表示指数函数，τ_c表示颜色相似性函数下的权重比例常数，τ_d表示曼哈顿距离相似性函数下的权重比例常数。In formula (3), I _p represents the gray value of point p, I _q represents the gray value of point q, (x _p , y _p ) represents the row and column coordinates of point p, (x _q , y _q ) represents q The row and column coordinates of the point, exp in formula (4) represents the exponential function, τ _c represents the weight proportional constant under the color similarity function, and τ _d represents the weight proportional constant under the Manhattan distance similarity function.

进一步，本发明中，为了使灰度相似性函数方便在FPGA中计算，对匹配点的灰度值先进行rank变换，然后利用新的灰度相似性关系R_pq代替原始的Δc_pq，rank变换的表达式如下：Further, in the present invention, in order to facilitate the calculation of the gray-scale similarity function in the FPGA, the gray-scale value of the matching point is first subjected to rank transformation, and then the original _{Δc pq} _is replaced by the new gray-scale similarity relationship Rpq, and the rank transformation The expression of is as follows:

${R R}_{pq pq} = = \{\begin{matrix} - - 22 & {I I}_{p p} - - {I I}_{q q} < < - - {τ τ}_{11} \\ - - 11 & - - {τ τ}_{11} \leq \leq {I I}_{p p} - - {I I}_{q q} \leq \leq - - {τ τ}_{22} \\ 00 & - - {τ τ}_{22} < < {I I}_{p p} - - {I I}_{q q} \leq \leq {τ τ}_{22} \\ 11 & {τ τ}_{22} < < {I I}_{p p} - - {I I}_{q q} \leq \leq {τ τ}_{11} \\ 22 & {I I}_{p p} - - {I I}_{q q} > > {τ τ}_{11} \end{matrix} - - - - - - ((55))$

式(5)中τ₁、τ₂为rank变换的分类条件，计算过程中作为固定常数，I_p表示p点的灰度值，I_q表示q点的灰度值。In formula (5), τ ₁ and τ ₂ are the classification conditions of rank transformation, which are used as fixed constants in the calculation process. I _p represents the gray value of point p, and I _q represents the gray value of point q.

因此根据式(5)和图2中建立的局部窗口，结合图3，利用FPGA内部的减法器对窗口内匹配点与窗口中心点的灰度值相减并判断，从而在FPGA内部确定窗口内每个匹配点rank变换等级，由于局部窗口中每个点都是并行得到的，因此变换后的rank像素点也是以局部窗口的形式并行出现，从而不影响算法的实时性。Therefore, according to formula (5) and the local window established in Figure 2, combined with Figure 3, the subtractor in the FPGA is used to subtract and determine the gray value of the matching point in the window and the gray value of the window center point, so as to determine the gray value in the window within the FPGA. The rank transformation level of each matching point, since each point in the local window is obtained in parallel, the transformed rank pixels also appear in parallel in the form of a local window, which does not affect the real-time performance of the algorithm.

结合图4，根据新的灰度相似性函数值f(R_pq)和曼哈顿距离相似性函数值f(Δd_pq)在FPGA中建立查找表，从而确定每个像素点的匹配代价关系权重值w(p,q)，因此在FPGA中计算的局部窗口匹配点的权重w(p,q)表达式如式(6)所示：Combined with Figure 4, according to the new gray similarity function value f(R _pq ) and the Manhattan distance similarity function value f(Δd _pq ), a lookup table is established in the FPGA to determine the matching cost relationship weight value w of each pixel (p,q), so the weight w(p,q) of the local window matching points calculated in the FPGA is shown in formula (6):

w(p,q)＝wd_k·wR_l (6)w(p,q)=wd _k ·wR _l (6)

图4中，Δd_k表示p和q点的曼哈顿距离，wd_k表示在FPGA内部通过查找表获得的f(Δd_pq)对应值，R_l表示p和q点在rank变换后的灰度相似性关系，wR_l表示在FPGA内部通过查找表获得的f(R_pq)对应值。In Figure 4, Δd _k represents the Manhattan distance between points p and q, wd _k represents the corresponding value of f(Δd _pq ) obtained through a lookup table inside the FPGA, and R _l represents the gray similarity between points p and q after rank transformation relationship, wR _l represents the corresponding value of f(R _pq ) obtained through the lookup table inside the FPGA.

由于经过上述步骤变换后，对于大小一定的局部区域匹配窗口，其曼哈顿距离相似性函数值与rank变换后的灰度相似性函数值都是一定范围内的离散变换值，因此查找表的建立可以在FPGA中通过有限状态机来对数据进行映射，其中映射后的wd_k、wR_l数据都为整数，该数据在归一化后，并不会影响其匹配点权重的变换，如此有利于在FPGA中进行定点整数的计算。After the above steps of transformation, for a local area matching window with a certain size, its Manhattan distance similarity function value and the gray level similarity function value after rank transformation are both discrete transformation values within a certain range, so the establishment of the lookup table can be In the FPGA, the data is mapped through the finite state machine, where the mapped wd _k and wR _l data are all integers, and the normalized data will not affect the transformation of its matching point weights, which is beneficial in Perform fixed-point integer calculations in the FPGA.

步骤4：结合图5，根据局部匹配窗口的匹配代价权重关系值，计算出每个匹配点的代价聚合关系如式(7)所示，然后利用胜者为王(WTA)的准则，求取每个像素点的视差结果d_q，如式(9)所示。Step 4: Combined with Figure 5, calculate the cost aggregation relationship of each matching point according to the matching cost weight relationship value of the local matching window As shown in Equation (7), the disparity result d _q of each pixel is obtained by using the winner-takes-all (WTA) criterion, as shown in Equation (9).

$E E. ((p p,, \overset{&OverBar; &OverBar;}{{p p}_{d d}})) = = \frac{{Σ Σ}_{q q &Element; &Element; {N N}_{P P},, \overset{&OverBar; &OverBar;}{{q q}_{d d}} &Element; &Element; {N N}_{\overset{&OverBar; &OverBar;}{Pd PD}}} w w ((p p,, q q)) w w ((\overset{&OverBar; &OverBar;}{{p p}_{d d}},, \overset{&OverBar; &OverBar;}{{q q}_{d d}})) {e e}_{m m} ((q q,, \overset{&OverBar; &OverBar;}{{q q}_{d d}}))}{{Σ Σ}_{q q &Element; &Element; {N N}_{P P},, \overset{&OverBar; &OverBar;}{{q q}_{d d}} &Element; &Element; {N N}_{\overset{&OverBar; &OverBar;}{Pd PD}}} w w ((p p,, q q)) w w ((\overset{&OverBar; &OverBar;}{{p p}_{d d}},, \overset{&OverBar; &OverBar;}{{q q}_{d d}}))} - - - - - - ((77))$

式(7)中，N_p表示左图像的局部匹配窗口，表示右图像的局部匹配窗口，w(p,q)为左图像中匹配点q的权重关系值，为右图像中的匹配点的权重关系值，对于左右图像的局部窗口，p点和点对应，q点和点相对应，为匹配点rank变换后的比较值，表达式如式(8)所示：In formula (7), N _p represents the local matching window of the left image, Represents the local matching window of the right image, w(p,q) is the weight relationship value of the matching point q in the left image, is the matching point in the right image The weight relationship value of , for the local window of the left and right images, p points and points correspond to points q and corresponding to the point, To match the comparison value after the rank transformation of the matching point, the expression is shown in formula (8):

${e e}_{m m} ((q q,, \overset{&OverBar; &OverBar;}{{q q}_{d d}})) = = \{\begin{matrix} 00 & {R R}_{pq pq} = = {R R}_{\overset{&OverBar; &OverBar;}{{p p}_{d d}} \overset{&OverBar; &OverBar;}{{q q}_{d d}}} \\ 11 & otherwise otherwise \end{matrix} - - - - - - ((88))$

式(8)中，R_pq表示q点的rank变换值，表示点的rank变换值。In formula (8), R _pq represents the rank transformation value of point q, express The rank transformation value of the point.

${d d}_{q q} = = arg arg \underset{d d &Element; &Element; {S S}_{d d}}{min min} E E. ((p p,, \overset{&OverBar; &OverBar;}{{p p}_{d d}})) - - - - - - ((99))$

式(9)中，S_d＝{d_min,d_min+1,...,d_max}，d_min为最小视差值，d_max为最大视差值。In formula (9), S _d ={d _min ,d _min +1,...,d _max }, d _min is the minimum parallax value, and d _max is the maximum parallax value.

结合图6，根据FPGA在左右图像中建立起局部权重窗口和局部像素点窗口，根据式(7)进行运算，在FPGA内部，为了减少乘法器资源的使用，左右图像的权重关系值进行相乘的过程中，利用移位的操作代替乘法器的功能(即将数据拆成多个2的n次方形式进行求和)，图6中的w_l表示左图像的权重w(p,q)，w_r表示右图像中的权重w即为用移位操作计算出的左右图像的权重积，图6中e_m的值由式(8)确定，它的取值只能是‘1’或者‘0’，因此式(7)的代价聚合的过程中，在FPGA中，只需要将e_m的最低位与权重值w中的各位进行逻辑与(&)操作，因此根据式(7)，对权重关系值归一化后即可得到代价聚合关系值 Combined with Figure 6, the local weight window and local pixel point window are established in the left and right images according to the FPGA, and the operation is performed according to formula (7). Inside the FPGA, in order to reduce the use of multiplier resources, the weight relationship values of the left and right images are multiplied In the process, the shift operation is used to replace the function of the multiplier (that is, the data is divided into multiple n-th power forms for summing), w _l in Figure 6 represents the weight w(p,q) of the left image, w _r represents the weight in the right image w is the weight product of the left and right images calculated by the shift operation. The value of _em in Figure 6 is determined by formula (8), and its value can only be '1' or '0', so formula (7) In the process of aggregation of the cost of , in the FPGA, only the lowest bit of _em and the bits in the weight value w need to be logically ANDed (&). Therefore, according to formula (7), after normalizing the weight relationship value, it is Aggregated value of cost can be obtained

根据式(9)，在FPGA中对每个匹配像素点并行的计算出(d_max-d_min+1)个匹配代价聚合值，并利用FPGA内部的比较器计算出代价聚合关系值最小的像素点所对应的偏移量，即为该点的视差值。According to formula (9), (d _max -d _min +1) matching cost aggregation values are calculated in parallel for each matching pixel in the FPGA, and the pixel with the smallest cost aggregation relationship value is calculated using the comparator inside the FPGA The offset corresponding to the point is the disparity value of the point.

因此，本发明循环上述步骤1～步骤6，即可对双目视频流进行稠密视差图的计算。Therefore, the present invention can calculate the dense disparity map for the binocular video stream by repeating the above steps 1 to 6.

为了本发明进一步通过仿真实验验证了本发明的有益效果，仿真实验是在Altera公司提供的CYCLONE III EP3C120F780C8 FPGA芯片上实现的，采用的局部匹配窗口的大小为11×11，采集到的双目视频流的帧频为60fps,图像大小分辨率为640×480，表1是本发明方法利用quartus2编译软件生成的硬件资源使用的报表，表2是本发明方法与现有的立体匹配算法的实时性能对比，帧频按60MHZ时钟归一化。实验表明本发明方法比Chen提出的方法搜索的视差范围更大，帧频也更高，能够获得一个实时、匹配准确率较高的稠密视差结果。In order for the present invention to further verify the beneficial effects of the present invention by simulation experiments, the simulation experiments are realized on the CYCLONE III EP3C120F780C8 FPGA chip provided by Altera Corporation, and the size of the local matching window adopted is 11 * 11, and the binocular video collected The frame rate of stream is 60fps, and the image size resolution is 640 * 480, and table 1 is the report form that the inventive method utilizes the hardware resource that quartus2 compiles software to generate to use, and table 2 is the real-time performance of the inventive method and existing stereo matching algorithm In contrast, the frame rate is normalized to a 60MHZ clock. Experiments show that the method of the present invention searches a larger disparity range and a higher frame rate than the method proposed by Chen, and can obtain a real-time dense disparity result with high matching accuracy.

表1Table 1

表2Table 2

Claims

1. the solid matching method of the support of the self-adaptation based on a FPGA weight, it is characterized in that, left images is set up to the sectional perspective match window of m × n size in FPGA, the input data of video input mouth are pretreated left images, the clock of pixel is the system synchronization clock that each pixel of left images enters FPGA inside, with first in first out buffer, row cache is carried out to match point, with the d type flip flop of FPGA inside, row buffer memory is carried out to match point.

2. as claimed in claim 1 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that, according to match point in local window and the grey similarity between local window central point and manhatton distance similarity, ask for grey similarity functional value and the manhatton distance similarity function value of point to be matched, thus obtain weight cost relation value w (p, q) of local window match point, weight cost relation value w (p, q) such as formula shown in (1)

w(p,q)＝wd _k·wR _l(1)

In formula (1), wd _krepresent the respective value of the manhatton distance similarity function value of pixel p and q obtained by look-up table in FPGA inside, R _lrepresent that the gray-scale value of pixel p and q is in the grey similarity relation after rank conversion, wR _lrepresent the respective value of the grey similarity functional value of pixel p and q obtained by look-up table in FPGA inside.

3. as claimed in claim 2 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that, according to the Matching power flow weight relationship value of described local matching window, calculate the cost paradigmatic relation of each match point, then utilize the victor is a king criterion, ask for the parallax result of each pixel.

4., as claimed in claim 3 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that,

The cost paradigmatic relation of described each match point shown in (2),

E (p, \overset{&OverBar;}{p_{d}}) = \frac{Σ_{q &Element; N_{P}, \overset{&OverBar;}{q_{d}} &Element; N_{\overset{&OverBar;}{P_{d}}}} w (p, q) w (\overset{&OverBar;}{p_{d}}, \overset{&OverBar;}{q_{d}}) e_{m} (q, \overset{&OverBar;}{q_{d}})}{Σ_{q &Element; N_{P}, \overset{&OverBar;}{q_{d}} &Element; N_{\overset{&OverBar;}{P_{d}}}} w (p, q) w (\overset{&OverBar;}{p_{d}}, \overset{&OverBar;}{q_{d}})} - - - (2)

In formula (2), N _prepresent the local matching window of left image, represent the local matching window of right image, w (p, q) is the weight relationship value of match point q in left image, for the match point in right image weight relationship value, for the local window of left images, p point and point is corresponding, q point with point is corresponding, for the fiducial value of match point after rank conversion, expression formula is such as formula shown in (3):

e_{m} (q, \overset{&OverBar;}{q_{d}}) = \{\begin{matrix} 0 & R_{pq} = R_{\overset{&OverBar;}{p_{d}} \overset{&OverBar;}{q_{d}}} \\ 1 & otherwise \end{matrix} - - - (3)

In formula (3), R _pqrepresent the rank transformed value of q point, represent the rank transformed value of point.

5., as claimed in claim 1 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that, select Gaussian function to carry out pre-service to video image, filtering noise.