CN116347055A - FPGA-based real-time binocular stereoscopic vision architecture method and system - Google Patents
FPGA-based real-time binocular stereoscopic vision architecture method and system Download PDFInfo
- Publication number
- CN116347055A CN116347055A CN202310244816.8A CN202310244816A CN116347055A CN 116347055 A CN116347055 A CN 116347055A CN 202310244816 A CN202310244816 A CN 202310244816A CN 116347055 A CN116347055 A CN 116347055A
- Authority
- CN
- China
- Prior art keywords
- image
- visible light
- infrared
- module
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 79
- 238000012937 correction Methods 0.000 claims abstract description 27
- 238000003702 image correction Methods 0.000 claims abstract description 27
- 239000000872 buffer Substances 0.000 claims description 49
- 230000001360 synchronised effect Effects 0.000 claims description 24
- 238000004364 calculation method Methods 0.000 claims description 13
- 230000002776 aggregation Effects 0.000 claims description 8
- 238000004220 aggregation Methods 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 3
- 230000003139 buffering effect Effects 0.000 claims description 2
- 238000013461 design Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/246—Calibration of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/25—Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
本发明公开了一种基于FPGA的实时双目立体视觉架构方法及系统，包括如下步骤：同步获取可见光图像和红外图像并按照可见光图像和红外图像的顺序轮序储存；在原图上发模式下，将所述储存的可见光图像和红外图像以轮询优先级的方式上发；在校正图及视差图上发模式下，通过图像处理模块对可见光图像和红外图像进行图像校正得到可见光校正后图像和红外校正后图像，通过半全局匹配算法获得可见光校正后图像和红外校正后图像的视差图，并将所述可见光校正后图像、所述红外校正后图像以及所述视差图上发，所述视差图包括可见光视差图像和红外视差图像；该架构方法及系统可全天候条件下实时工作且系统延迟低、视差图像精度较高，造价成本较小。
The invention discloses an FPGA-based real-time binocular stereo vision framework method and system, comprising the following steps: synchronously acquiring visible light images and infrared images and storing them sequentially in sequence; in the original image uploading mode, Upload the stored visible light image and infrared image in a polling priority manner; in the correction map and disparity map upload mode, perform image correction on the visible light image and infrared image through the image processing module to obtain the visible light corrected image and For the infrared corrected image, the disparity map of the visible light corrected image and the infrared corrected image is obtained through a semi-global matching algorithm, and the visible light corrected image, the infrared corrected image and the disparity map are uploaded, and the disparity The image includes visible light parallax images and infrared parallax images; the architecture method and system can work in real time under all-weather conditions, with low system delay, high parallax image accuracy, and low cost.
Description
Technical Field
The invention relates to the technical field of image acquisition, in particular to a real-time binocular stereoscopic vision architecture method and system based on an FPGA.
Background
In recent years, binocular stereoscopic vision systems are widely used in the fields of intelligent robots, 3D scene reconstruction, unmanned aerial vehicles, automatic driving and the like. In order to meet all-weather and real-time working requirements, the industry mostly uses laser radar ranging to meet the scene requirements of all-weather working, and the method has good effects in foggy days and rainy days, but has the defect of high cost.
Disclosure of Invention
Based on the technical problems in the background art, the invention provides a real-time binocular stereoscopic vision architecture method and a real-time binocular stereoscopic vision architecture system based on an FPGA, which can work in real time under all-weather conditions, and have the advantages of low system delay, higher parallax image precision, lower manufacturing cost and high economic benefit.
The invention provides a real-time binocular stereoscopic vision architecture method based on an FPGA, which comprises the following steps:
synchronously acquiring visible light images and infrared images and storing the visible light images and the infrared images in turn according to the sequence of the visible light images and the infrared images;
in an original image uploading mode, uploading the stored visible light images and infrared images in a polling priority mode;
in a correction chart and parallax chart uploading mode, performing image correction on a visible light image and an infrared image through an image processing module to obtain a visible light corrected image and an infrared corrected image, obtaining a parallax chart of the visible light corrected image and the infrared corrected image through a semi-global matching algorithm, and uploading the visible light corrected image, the infrared corrected image and the parallax chart, wherein the parallax chart comprises a visible light parallax image and an infrared parallax image;
in the correction chart and parallax chart uploading mode, whether the current data amount in the DRAM meets the consumption of the image processing module is judged in the frame blanking period of the image processing module, so that the starting and stopping of the image processing module are controlled in real time.
Further, in the step of synchronously acquiring the visible light image and the infrared image and sequentially storing the visible light image and the infrared image in turn, the method specifically comprises the following steps:
synchronously receiving visible light images and infrared images uploaded by the visible light binocular camera module and the infrared binocular camera module;
two frame buffer spaces are respectively arranged in the DRAM and used for respectively storing visible light images and infrared images, and the two frame buffer spaces are written into the DRAM through an AXI4 bus interface in turn according to the sequence of the visible light images and the infrared images.
Further, in the step of synchronously receiving the visible light image and the infrared image uploaded by the visible light binocular camera module and the infrared binocular camera module, the method specifically comprises the following steps:
synchronizing a frame synchronizing signal given by the visible light binocular camera module to a clock domain of the infrared binocular camera module;
and acquiring a frame of infrared image which is matched with the synchronized frame synchronizing signal, and synchronously aligning the visible light image and the infrared image to obtain the synchronized visible light image and the synchronized infrared image.
Further, in the data writing process according to the sequence of the visible light image and the infrared image, whether the data is written currently or whether the data of a complete frame is discarded is judged by the number of the data writing packets, whether the data is in the last stage of the frame blanking period or not and the specific address of the current read address bus.
Further, in the correction chart and parallax chart up-transmitting mode, performing image correction on the visible light image and the infrared image to obtain a visible light corrected image and an infrared corrected image, specifically including:
setting a lens correction table of the visible light binocular camera module and the infrared binocular camera module;
the 3-8 decoder divides the input visible light image and infrared image data into 8 groups of line caches according to the low three bits of the line number;
the visible light image and the infrared image data are read out from the 8 groups of line caches by 8 16-to-1 multiplexers;
selecting four visible light images and infrared image data in two required lines through 2 8-by-1 multiplexers;
bilinear difference calculation is carried out on the four visible light images and the infrared image data, so that corrected visible light corrected images and infrared corrected images are obtained;
further, a specific bilinear difference calculation formula is as follows:
I[x，y]＝I[x i ，y i ](1-x f )(1-y f )+I[x i +1，y i ]x f (1-y f )
+I[x i ，y i +1](1-x f )y f +I[x i +1，y i +1]x f y f
wherein (x, y) represents the coordinates of the input pixel, divided into an integer part and a fractional part, (x) i ，y i ) Is an integer part for selecting four nearest pixel points, (x f ，y f ) Is a fractional part used to weight the pixel values.
Further, in obtaining the disparity map of the visible light corrected image and the infrared corrected image by the semi-global matching algorithm, the method specifically includes:
denoising gray images of the input visible light corrected image and the input infrared corrected image through Gaussian filtering;
calculating the initial cost of each pixel under all parallaxes through a Census transformation algorithm;
calculating the aggregation cost of each pixel by adopting a semi-global cost aggregation method;
finally, selecting the parallax corresponding to each pixel by using a WTA principle;
and optimizing parallax accuracy through subpixel level parallax calculation, spot filtering, median filtering and hole filling, and outputting visible light parallax images and infrared parallax images.
Further, in the correction chart and disparity chart uploading mode, whether the current data amount in the DRAM meets the consumption of the image processing unit is judged in the frame blanking period of the image processing module, so that the starting and stopping of the image processing module are controlled in real time, and the specific process is as follows:
judging how many data packets are still stored in the current DRAM, and if the data in the cached data packets reach a set threshold value, sending an enabling signal to the VTC;
judging whether a pfull signal of the level 1 FIFO is pulled up or not and whether a pepty signal of the level two FIFO is pulled down or not, if the pfull signal of the level 1 FIFO is pulled up or the pepty signal of the level two FIFO is pulled down, sending an enabling signal to the VTC to start the image processing module;
judging whether the current frame blanking period exists, if so, sending an enabling signal to the VTC to start the image processing module;
if the image processing modules meet the conditions, the image processing modules are in a starting or a starting maintaining state, continue to carry out image correction on the visible light images and the infrared images, and continue to obtain parallax images of the images after the visible light correction and the images after the infrared correction through a semi-global matching algorithm;
if the image processing module does not fully meet the conditions, the image processing module is in a pause state, the image correction of the visible light image and the infrared image is paused, the parallax images of the visible light corrected image and the infrared corrected image are obtained through a semi-global matching algorithm, and the image processing module is restarted until the number of data packets corresponding to the original visible light image and the infrared image exceeds a set threshold value.
A real-time binocular stereoscopic vision architecture system based on FPGA, a real-time binocular stereoscopic vision architecture method and system based on FPGA, including visible light/infrared image synchronous receiving module, frame buffer write controller module, frame buffer read controller module, image processing unit rate matching module, image correction module, binocular stereoscopic matching module, USB receiving and transmitting controller module;
the visible light/infrared image synchronous receiving module is used for acquiring a visible light image and an infrared image, synchronizing the visible light image and the infrared image and storing the synchronized visible light image and the synchronized infrared image into fifo;
the frame buffer write controller module is used for writing the visible light images and the infrared images into the DRAM in turn according to the sequence of the visible light images and the infrared images for frame buffer;
the frame buffer reading controller module is used for interacting with the USB transceiver controller module or interacting with the image processing unit rate matching module to obtain a later state so as to judge whether the visible light image and the infrared original image stored in the DRAM are read out;
the image processing unit rate matching module is used for judging whether the data amount in the current DRAM meets the consumption of the image processing unit in the frame blanking period of the image processing module so as to control the start and stop of the image processing module in real time;
the image correction module is used for carrying out image correction on the visible light image and the infrared image and outputting the visible light corrected image and the infrared corrected image to the binocular stereo matching module;
the binocular stereo matching module is used for respectively processing the visible light corrected image and the infrared corrected image through a semi-global matching algorithm to obtain a visible light parallax image and an infrared parallax image;
the USB transceiver controller module is used for receiving a starting mode instruction sent by the upper computer, and sending up a visible light image and an infrared image in an original image sending-up mode, and sending up a visible light corrected image, an infrared corrected image, a visible light parallax image and an infrared parallax image in a correction chart and parallax image sending-up mode.
A computer readable storage medium having stored thereon a number of classification procedures for being invoked by a processor and performing a real-time binocular stereoscopic architecture method as described above.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
The real-time binocular stereoscopic vision architecture method and system based on the FPGA provided by the invention have the advantages that: the real-time binocular stereoscopic vision architecture method and the real-time binocular stereoscopic vision architecture system based on the FPGA provided by the structure can work in real time under all-weather conditions, the system delay is low, the parallax image precision is higher, the manufacturing cost is lower, the economic benefit is high, under the synchronous work of the visible light binocular camera module and the infrared binocular camera module, the image processing tasks such as binocular image correction, binocular stereoscopic matching and the like can be completed in real time facing different working scenes under different climatic environments, and the processed visible light and infrared corrected images and visible light and infrared parallax images are uploaded to an upper computer through USB 3.0; finally, an image processing frame rate of 25fps and a system delay of less than 4 milliseconds are realized on an Xilinx XZU15EG type FPGA core board.
Drawings
FIG. 1 is a schematic flow chart of the present invention;
FIG. 2 is a flow chart of a real-time binocular stereoscopic architecture;
FIG. 3 is a schematic diagram of a real-time binocular stereoscopic architecture hardware architecture;
FIG. 4 is a schematic diagram of a frame buffer write controller based on an AXI4 bus interface;
FIG. 5 is a schematic diagram of a frame buffer read controller based on an AXI4 bus interface;
fig. 6 is a schematic diagram of the structure of the rate matching module of the image processing unit.
Detailed Description
In the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The invention may be embodied in many other forms than described herein and similarly modified by those skilled in the art without departing from the spirit or scope of the invention, which is therefore not limited to the specific embodiments disclosed below.
As shown in fig. 1 to 6, the method and the system for real-time binocular stereoscopic vision architecture based on FPGA provided by the invention comprise the following steps:
s1: synchronously acquiring visible light images and infrared images and storing the visible light images and the infrared images in turn according to the sequence of the visible light images and the infrared images;
s2: in an original image uploading mode, uploading the stored visible light images and infrared images in a polling priority mode;
s3: in a correction chart and parallax chart uploading mode, performing image correction on a visible light image and an infrared image to obtain a visible light corrected image and an infrared corrected image, obtaining a parallax chart of the visible light corrected image and the infrared corrected image through a semi-global matching algorithm, and uploading the visible light corrected image, the infrared corrected image and the parallax chart, wherein the parallax chart comprises a visible light parallax image and an infrared parallax image; meanwhile, whether the current data amount in the DRAM meets the consumption of the image processing unit is judged in the frame blanking period of the image processing module, so that the start and stop of the image processing module are controlled in real time.
Through the steps S1 to S3, the visible light image and the infrared image are synchronized, and the image processing of the visible light image and the infrared image is completed through reasonably designed image cache storage in a limited time window, so that the overall delay of the system is controlled below 4 milliseconds at the frame rate of 25fps, and the hardware acceleration of the all-weather real-time binocular stereoscopic vision system is realized.
In addition, the architecture method can finish image processing tasks such as binocular image correction, binocular stereo matching and the like in real time in different climatic environments under the synchronous working of the visible light binocular camera module and the infrared binocular camera module, and upload processed visible light and infrared corrected images and visible light and infrared parallax images to an upper computer through USB 3.0; finally, an image processing frame rate of 25fps and a system delay of less than 4 milliseconds are realized on an Xilinx XZU15EG type FPGA core board.
For convenience of description, a construction system is used for implementing the construction method.
As shown in fig. 2, after synchronizing the image data of the visible light binocular camera module and the infrared binocular camera module, the system obtains paired visible light images and infrared images, creates a frame buffer with two frames as a ping-pong buffer of the image data, designs an efficient frame buffer read-write controller for matching the image data speed of the visible light binocular camera module and the infrared binocular camera module with the speed of the USB transceiver controller, designs a low-delay image processing unit rate matching module for starting and stopping the image processing module in real time to match the tiny change of the original image data speed when the visible light and infrared binocular cameras work, wherein the image processing module is used for realizing image correction and image binocular stereo matching in the step S3 so as to output a visible light corrected image, an infrared corrected image, a visible light parallax image and an infrared parallax image.
Architecture system specification:
1. visible light/infrared image synchronous receiving module
The visible light/infrared image synchronous receiving module is used for receiving the visible light images and the infrared images uploaded by the visible light binocular camera module and the infrared binocular camera module, synchronously aligning the visible light images and the infrared images, and facilitating the synchronization of the two paths of image processing flows and the visible light/infrared data packet sent by the USB; for achieving the synchronous acquisition of the visible light image and the infrared image in step S1.
The visible light/infrared image synchronous receiving module synchronizes the frame synchronization signal given by the visible light binocular camera module (25 fps) to the clock domain of the infrared binocular camera module (50 fps), and uses the synchronized frame synchronization signal to match the latest frame of infrared image, so that the visible light and the infrared image are synchronized (the time difference between the visible light and the infrared image ranges from 0s to 0.02 s).
2. Frame buffer write controller module
The frame buffer write controller module is used for safely writing the received visible light image and infrared image into the DRAM through the AXI4 bus interface, controlling the writing of new data by opening up two frame buffer spaces for the visible light image and the infrared image in the DRAM and monitoring the data reading condition in the DRAM in real time, avoiding that the data in the DRAM is covered by the new data when not used, and realizing the sequential round-robin storage of the visible light image and the infrared image in the step S1.
As shown in fig. 4, the frame buffer write controller module is composed of a visible light left and right image data buffer, an infrared image left and right data buffer, a two-channel polling read-write controller, a write data counter, a write data controller, a frame blanking period counter, and the like. The visible light left and right image data cache and the infrared image left and right data cache are respectively used for caching visible light images and infrared images; the two-channel polling read-write controller is used for arbitrating and distributing the use right of the write-in bus, and writing data according to the use bus of the sequence of the visible light images and the infrared images; the frame blanking period counter is used for recording the frame blanking period of the visible light image and infrared image data which are originally uploaded and providing the data for the write data controller to judge whether the write data or the data losing operation is carried out at the current moment; the write data counter is used for recording how many data packets are written, the write data controller is used for carrying out communication interaction with the AXI4 bus interface bus, writing data into the DRAM, judging whether the data are written currently or discarding the data of a complete frame according to the number of the data packets to be written, whether the data are in the last stage of a frame blanking period and the specific address of the current read address bus, and specifically comprises the following steps: when the size of the current writing data packet reaches the size of one frame of image and is at the end of the frame blanking period, judging the space position of the read address bus at the moment: 1. if the read address bus is still in the address space of the unread image data, the upcoming complete frame data is discarded completely; 2. at this time, if the read address bus is in the address space of the read image data, the upcoming image data is normally written.
3. Frame buffer read controller module
The frame buffer memory read controller module is used for reading out the image data and the correction table data in the DRAM in a polling priority mode through an AXI4 bus interface, respectively writing the visible light data into the intermediate stage fifo according to a Y, U, V format, reading out the stored visible light image and infrared image in the step S2 in a polling priority mode in an original image uploading mode, and uploading the read-out image through the USB read-write controller module; in the correction chart and parallax chart uploading mode, the method is used for reading out the visible light image and the infrared image before performing image correction on the visible light image and the infrared image in the step S3 so as to perform subsequent processing of the image, and then uploading the read-out image through the USB read-write controller module.
As shown in fig. 5, the frame buffer read controller module is composed of a read data counter, a read data controller, a four-way polling read controller, a visible light data buffer (FIFO 0), an infrared data buffer (FIFO 1), a visible light correction table data buffer (FIFO 2), an infrared correction table data buffer (FIFO 3), and the like. The read data counter is used for counting the number of the read data packets; the read data controller is used for carrying out communication interaction with the AXI4 bus interface bus, reading data from the DRAM, and judging whether the data are read currently or not according to the number of read data packets and read data requests of the FIFO; the four-channel polling read-out controller is used for reading out data according to the use bus of the sequence of the visible light image data, the infrared image data, the visible light correction table data and the infrared correction table data. The visible light image data buffer (FIFO 0), the infrared image data buffer (FIFO 1), the visible light corrected image data buffer (FIFO 2) and the infrared corrected image data buffer (FIFO 3) are respectively used for buffering four paths of data read out from the DRAM.
4. Image processing unit rate matching module
The image processing unit rate matching module is used for adjusting the starting and the stopping of the image processing module between frames in real time according to the front-stage data rate so that the image processing unit matches with the tiny speed difference in the operation of the visible light binocular camera module and the infrared binocular camera module, and the image processing unit is used for controlling the starting and the stopping of the image processing module in the step S3 in the image processing process, wherein the front-stage data specifically refer to data packets corresponding to the original visible light image and the infrared image.
As shown in fig. 6, the rate matching module of the image processing unit is composed of a buffered data packet counter, a VTC start-stop judgment, a FIFO status judgment, and other functional modules. The cached data packet counter judges how many data packets are still stored in the current DRAM according to the values of the read-write data packet counter in the front-stage read-write cache controller module; the FIFO state judging module is used for judging whether a pfull signal of the 1-level FIFO of the read cache is pulled up or not and whether a pepty signal of the second-level FIFO is pulled down or not, and if the conditions are met, an enabling signal is sent to the VTC start-stop judging module; the VTC start-stop judging module is configured to judge whether the current frame blanking period is present, whether the value of the buffered data packet counter reaches a set threshold value, and whether an enable signal of the FIFO state judging module is received, if the current frame blanking period is satisfied, the value of the buffered data packet counter reaches the set threshold value, and the enable signal of the FIFO state judging module is received, the VTC starts or maintains the start state, otherwise, the VTC is suspended (i.e. the operation of the image processing module is suspended), the VTC is restarted when the number of preceding image data packets exceeds the set threshold value, and the image processing module continues to operate, where the preceding image data packets are specifically data packets corresponding to the original visible light image and the infrared image.
5. Image correction module
The image correction module is used for correcting the original visible light image and infrared image, is convenient for improving the precision of the post-binocular stereo matching, and is used for realizing the image correction of the visible light image and the infrared image in the step S3 to obtain a visible light corrected image and an infrared corrected image.
The calculation principle of the image correction module is based on a bilinear interpolation algorithm, i.e. image correction is performed on the original image by using bilinear interpolation according to camera correction table data prepared in advance. The image correction module mainly comprises three parts of a line buffer and line buffer read-write controller and bilinear interpolation calculation, wherein the line buffer and line buffer read-write controller consists of 128 double-port RAMs. The line buffer is composed of 128 (8 groups of 16 each) 8-bit dual-port RAM with depth of 1280, the design size of the line buffer is determined by searching the maximum line difference between the upper line and the lower line in the correction table, and the maximum line difference of the correction table is 64 in the embodiment, so that the image data with the line difference of 64 can be maximally searched upwards and downwards by designing the line buffer to 128 lines, and the accuracy of algorithm data is ensured. The line buffer read-write controller is mainly composed of 1 3-8 decoders, 16 1-8 multiplexers and 2 1-8 multiplexers. In the write control path, the 3-8 decoder divides the input image data into 8 groups of line caches according to the low three bits of the line number, in the read control path, the data are read out from 8 groups by 8 16-1-8 multiplexers, and finally four data in two required lines are selected by 2 8-1-8 multiplexers. The bilinear interpolation calculation module mainly comprises 6 multipliers and 6 adders, and the bilinear interpolation algorithm is as follows:
I[x，y]＝I[x i ，y i ](1-x f )(1-y f )+I[x i +1，y i ]x f (1-y f )
+I[x i ，y i +1](1-x f )y f +I[x i +1，y i +1]x f y f
wherein (x, y) represents the coordinates of the input pixel, divided into an integer part and a fractional part, (x) i ，y i ) Is an integer part for selecting four nearest pixel points, (x f ，y f ) Is a fractional part used to weight the pixel values.
6. Binocular stereo matching module
The binocular stereo matching module is used for obtaining a visible light parallax image and an infrared parallax image through an SGM algorithm according to Y channel data of the corrected visible light image and the infrared image, and obtaining parallax images of the visible light corrected image and the infrared corrected image through a semi-global matching algorithm in the implementation step S3.
The hardware architecture of the binocular stereo matching module is designed according to an improved algorithm based on an SGM algorithm, and the binocular stereo matching algorithm mainly comprises a preprocessing module, an initial cost calculation module, a semi-global cost aggregation module, a parallax selection module and a parallax optimization module. The preprocessing module performs denoising on an input gray image through Gaussian filtering, then calculates initial cost of each pixel under all parallaxes through a Census transformation algorithm with good robustness, and calculates aggregation cost of each pixel by adopting a semi-global cost aggregation method in a cost aggregation stage. Finally, the WTA (winner general eating) principle is used for selecting the parallax corresponding to each pixel, and the parallax accuracy is further improved through sub-pixel level parallax calculation, spot filtering, median filtering and hole filling in the parallax optimization stage. The entire hardware design requires two clock frequencies, the input of the initial left and right gray images and the output of the final disparity map are performed at a 52MHz clock frequency, one data per cycle is input/output. The middle stereo matching calculation module is used for processing data under the control of 104MHz clock frequency, and processes one data every four cycles, and the hardware architecture design of the whole stereo matching calculation module is a pipeline architecture and has higher throughput rate.
7. USB read-write controller module
The USB read-write controller module is used for sending four paths of images, namely the visible light corrected image, the infrared corrected image, the visible light parallax image and the infrared parallax image, generated by the image processing module to the upper computer in an arbitration strategy with fixed priority, and is used for realizing the final image sending stage in the step S2 and the step S3.
The USB read-write controller module is used for receiving the instruction of the upper computer or sending the corresponding instruction head and image data to the upper computer. The USB read-write controller receives and analyzes the working mode instruction from the upper computer through the self-defined instruction, sends and receives the instruction to the visible light binocular camera module and the infrared binocular camera module and sends the working instruction to the image processing module, and the whole system starts working. The USB read-write controller respectively defines data packet instruction heads of the visible light corrected image, the infrared corrected image, the visible light parallax image and the infrared parallax image through the user-defined instruction heads and polls the uplink image data according to the working mode.
In this embodiment, the main processing flow of the hardware architecture includes: the USB read-write controller module receives and analyzes the mode starting instruction sent by the upper computer, and starts working under the condition that the initialization of the visible light binocular camera module and the infrared binocular camera module is completed. In an original image uploading mode, the visible light/infrared image synchronous receiving module receives left and right images from the visible light binocular camera module and the infrared binocular camera module, synchronizes the left and right images and stores the synchronized images into fifo; the frame buffer write controller module takes out original visible light image and infrared image data of the original image data stored in fifo in turn according to the order of fixed priority and safely writes the data into the DRAM (without covering the data which is not read out in the DRAM); the frame buffer memory read controller module reads the original visible light image and the infrared image stored in the DRAM in turn according to the sequence of fixed priority in a state that the USB read/write controller module can send data, and writes the original visible light image and the infrared image data into fifo interacted with the USB read/write controller module to wait for the USB read/write controller module to send data; when the USB read-write controller module detects that the data exists in the front stage fifo, the original left and right images of the visible light image and the infrared image are alternately sent out in the sequence of the visible light image and the infrared image by taking the frame as a unit. In the correction chart and parallax chart uplink mode, the visible light/infrared image synchronous receiving module receives left and right charts from the visible light binocular camera module and the infrared binocular camera module, synchronizes the left and right charts and stores the charts in fifo; the frame buffer write controller module takes out original visible light image and infrared image data of the original image data stored in fifo in turn according to the order of fixed priority and safely writes the data into the DRAM (without covering the data which is not read out in the DRAM); the frame buffer memory read controller module starts to read the original visible light image and infrared image data in turn according to the sequence of the fixed priority when the number of the data packets in the DRAM is larger than 1, and writes the read data into fifo for temporary storage according to the left image and the right image respectively (for visible light, corresponding fifo is written into according to Y, U, V three channels respectively); the image processing unit rate matching module detects the number of data packets in the current DRAM and a data threshold signal in the former fifo in a frame blanking period of the image data processed by the image processing module, judges whether the image processing module can be started currently, pauses the operation of the image processing module if the number of data packets in the current DRAM and the data amount in the former fifo do not reach a set threshold (at the moment, the processing speed of the image processing module is slightly faster than the maximum rate of an image generated by an actual camera), and starts the image processing module to start working after the data is enough for the image processing module to complete processing of one frame. The image correction module corrects the original visible light image and the infrared image by utilizing camera lens correction table data obtained in advance based on a bilinear interpolation algorithm to obtain a visible light corrected image and an infrared corrected image. The binocular stereo matching module obtains a visible light parallax image and an infrared parallax image (parallax is used for calculating distance) according to a semi-global stereo matching algorithm by using the visible light corrected image and the infrared corrected image; when the USB read-write controller module detects that the data exists in the front stage fifo, the data are alternately sent up in the order of the visible light corrected image, the infrared corrected image, the visible light parallax image and the infrared parallax image by taking the frame as a unit. Finally, the upper computer can obtain the distance information from the object on the image to the camera according to the parallax map.
The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310244816.8A CN116347055B (en) | 2023-03-09 | 2023-03-09 | A real-time binocular stereo vision architecture method and system based on FPGA |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310244816.8A CN116347055B (en) | 2023-03-09 | 2023-03-09 | A real-time binocular stereo vision architecture method and system based on FPGA |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116347055A true CN116347055A (en) | 2023-06-27 |
| CN116347055B CN116347055B (en) | 2024-11-22 |
Family
ID=86875642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310244816.8A Active CN116347055B (en) | 2023-03-09 | 2023-03-09 | A real-time binocular stereo vision architecture method and system based on FPGA |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116347055B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5416510A (en) * | 1991-08-28 | 1995-05-16 | Stereographics Corporation | Camera controller for stereoscopic video system |
| CN1798957A (en) * | 2003-05-29 | 2006-07-05 | 奥林巴斯株式会社 | Stereo camera system and stereo optical module |
| CN107917701A (en) * | 2017-12-28 | 2018-04-17 | 人加智能机器人技术（北京）有限公司 | Measuring method and RGBD camera systems based on active binocular stereo vision |
| US20200008899A1 (en) * | 2017-04-24 | 2020-01-09 | Truevision Systems, Inc. | Stereoscopic camera with fluorescence visualization |
| US20200084353A1 (en) * | 2018-08-23 | 2020-03-12 | Ethersec Industries Ltd | Stereo camera and imaging system |
| US20200099912A1 (en) * | 2018-09-26 | 2020-03-26 | Snap Inc. | Depth sculpturing of three-dimensional depth images utilizing two-dimensional input selection |
| US20200396407A1 (en) * | 2018-03-01 | 2020-12-17 | Canon Kabushiki Kaisha | Image processing device and control method therefor, and storage medium |
-
2023
- 2023-03-09 CN CN202310244816.8A patent/CN116347055B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5416510A (en) * | 1991-08-28 | 1995-05-16 | Stereographics Corporation | Camera controller for stereoscopic video system |
| CN1798957A (en) * | 2003-05-29 | 2006-07-05 | 奥林巴斯株式会社 | Stereo camera system and stereo optical module |
| US20200008899A1 (en) * | 2017-04-24 | 2020-01-09 | Truevision Systems, Inc. | Stereoscopic camera with fluorescence visualization |
| CN107917701A (en) * | 2017-12-28 | 2018-04-17 | 人加智能机器人技术（北京）有限公司 | Measuring method and RGBD camera systems based on active binocular stereo vision |
| US20200396407A1 (en) * | 2018-03-01 | 2020-12-17 | Canon Kabushiki Kaisha | Image processing device and control method therefor, and storage medium |
| US20200084353A1 (en) * | 2018-08-23 | 2020-03-12 | Ethersec Industries Ltd | Stereo camera and imaging system |
| US20200099912A1 (en) * | 2018-09-26 | 2020-03-26 | Snap Inc. | Depth sculpturing of three-dimensional depth images utilizing two-dimensional input selection |
Non-Patent Citations (1)
| Title |
|---|
| 陈松: "双目图像校正VLSI 硬件电路结构设计", 计算机硬件, 31 December 2019 (2019-12-31) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116347055B (en) | 2024-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11263767B2 (en) | Method for processing image in virtual reality display device and related virtual reality display device | |
| US12204869B2 (en) | Natural language understanding for visual tagging | |
| US10650574B2 (en) | Generating stereoscopic pairs of images from a single lens camera | |
| US10719939B2 (en) | Real-time mobile device capture and generation of AR/VR content | |
| CN105872432B (en) | Device and method for fast adaptive frame rate conversion | |
| CN110570457B (en) | A 3D Object Detection and Tracking Method Based on Streaming Data | |
| KR20180002607A (en) | Pass-through display for captured images | |
| WO2020093850A1 (en) | Dual-light image integration method and apparatus, and unmanned aerial vehicle | |
| US12182930B2 (en) | Enclosed multi-view visual media representation | |
| US8866887B2 (en) | Computer graphics video synthesizing device and method, and display device | |
| US12293474B2 (en) | Video pass-through computing system | |
| KR20210044506A (en) | Apparatus of displaying augmented reality object and operating methode thereof | |
| TW202332263A (en) | Stereoscopic image playback apparatus and method of generating stereoscopic images thereof | |
| WO2017003680A1 (en) | Video frame processing | |
| WO2023093792A1 (en) | Image frame rendering method and related apparatus | |
| CN115862081A (en) | Image acquisition method, device, equipment and computer readable storage medium | |
| US20250168315A1 (en) | Multiview interactive digital media representation inventory verification | |
| CN116347055B (en) | A real-time binocular stereo vision architecture method and system based on FPGA | |
| CN113556366A (en) | Multi-sensor data synchronization method and system and electronic equipment | |
| CN111445380A (en) | Method and device for realizing real-time binocular stereo matching based on ZYNQ | |
| US20220406003A1 (en) | Viewpoint path stabilization | |
| US20220408019A1 (en) | Viewpoint path modeling | |
| CN110443887A (en) | Characteristic point positioning method, device, method for reconstructing, system, equipment and medium | |
| US20130315554A1 (en) | Method for determining the movements of an object from a stream of images | |
| CN117392518B (en) | Low-power-consumption visual positioning and mapping chip and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |