CN114554155A - Remote image acquisition device - Google Patents

Abstract

The invention discloses a remote image acquisition device, which first collects images through an image acquisition module, and then in the image processing module, the FPGA of the ZYNQ chip is used to mainly complete system control, data storage and transmission and post-processing, and the ARM processor uses It is used for preprocessing ADC input data and outputting data to DAC; data exchange between PS terminal and PL terminal through AXI bus, including AXI4 and AXI4_Lite bus, in which the data bit width of AXI4 bus is 64bit, which is used to transmit data volume Large image data with high transmission delay requirements; the data bit width of the AXI4_Lite bus is 32bit, which is used to control the transmission of data.

Description

A remote image acquisition device

technical field

The invention belongs to the technical field of image acquisition, and more particularly, relates to a remote image acquisition device.

Background technique

Image acquisition and storage equipment has been widely used in industrial production, medical and health care, aerospace and other fields. It is of great significance to improve image data processing speed and reduce image transmission delay for image acquisition terminal equipment. In specific scenarios, such as remote surgery, space probes, UAV power inspection, etc., there are high requirements for the real-time and stability of remote image transmission, and a large number of images are required in the process of image acquisition, transmission and processing. time. For example, the 8-bit digital image data size of 1728*625 converted by the analog-to-digital converter is 1.03MB. When the sensor has high frame rate and continuous image acquisition, it needs to have a high transmission bandwidth to ensure the integrity of the image data. sex. In addition, in order to effectively utilize the image information, the data also needs to be processed. The traditional DSP or ARM processor has excellent control ability, but the sampling rate is low, the instructions are executed serially and the system uses floating point, it is difficult to directly process the image data with large data volume, high pixel correlation and wide frequency band .

At present, the characteristics of data concurrent processing, pipeline technology, receiving-while-processing, high-speed interface and other characteristics of FPGA devices are very compatible with the needs of image data transmission and processing, but they are insufficient in peripheral control capabilities. In order to build an image acquisition system with low image transmission delay, remote network image transmission, and programmable control, this paper adopts the ZYNQ high-performance chip launched by Xilinx. The ZYNQ chip integrates ARM processor and FPGA, and the processor and FPGA pass through High-speed AXI bus interconnection, these features can maximize the processing speed, control capability, and transmission rate. At the same time, the ARM processor can also use the Linux system to carry the Gstreamer streaming media application to push the processed image data to the network for remote image transmission.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide a remote image acquisition device, which performs real-time data processing on the data collected by the image sensor through embedded software and hardware and an expansion port, and pushes the data to the network port to realize the image acquisition. Remote collection of data.

In order to achieve the above purpose of the invention, a remote image acquisition device of the present invention is characterized by comprising: a dual-channel image acquisition module, a high-speed real-time image processing module and a human-computer interaction module;

The dual-channel image acquisition module includes two image acquisition sensors, the image acquisition sensors are used to acquire images, and then the acquired images are converted into electrical signals in the form of optical signals, and the converted electrical signals are input into high-speed real-time through a coaxial cable. The image processing module;

The high-speed real-time image processing module includes a video decoding chip, a Zynq chip, a network port and a memory; the Zynq chip also includes an FPGA and an ARM processor;

Among them, the ARM processor runs the embedded control program and is responsible for receiving the instructions of the remote PC. After parsing the instructions, it transmits the control signal to the FPGA side through the AXI-Lite bus to realize the video decoding chip configuration module, channel selection module, AXI_DMA IP The configuration of the core, after the configuration is completed, the function scheduling of the acquisition device is performed, specifically: the video decoding chip decodes the electrical signal converted by the image acquisition sensor, and the output format is the digital video signal of the ITU-R BT.656 standard, which is processed by ARM. The controller controls the FPGA to reorganize the image data of the digital video signal, and then transmits the reorganized image data to the memory for storage by DMA, and transmits it to the human-computer interaction module through the network port at the same time;

The human-computer interaction module is used to configure the IP address of the image acquisition device, realize the remote transmission of the signal, and at the same time display the acquired image in real time through the display.

The purpose of the invention of the present invention is achieved in this way:

A remote image acquisition device of the present invention first collects images through an image acquisition module, and then in the image processing module, the FPGA of the ZYNQ chip is used to mainly complete system control, data storage and transmission, and post-processing, and the ARM processor is used to Pre-processing of ADC input data and output data to DAC; data exchange between PS terminal and PL terminal through AXI bus, including AXI4 and AXI4_Lite bus, where AXI4 bus has a data bit width of 64 bits, which is used to transmit large amounts of data, Image data with high transmission delay requirements; the data bit width of the AXI4_Lite bus is 32bit, which is used to control the transmission of data.

Meanwhile, a remote image acquisition device of the present invention also has the following beneficial effects:

(1), the image processing part of the device utilizes the logic resources of the FPGA to realize high-speed parallel data processing, which can realize the processing while receiving the image data stream;

(2) Using the format of the input image data, keep the gray-scale value Y unchanged, first expand the transmission bit width through the serial-parallel conversion module, and then further compress Cr and Cb, so as to retain the valid data and reduce the amount of data transmission, Effectively improve the efficiency of data transmission;

(3) Using AXI-DMA to transfer the processed data, the image data stream can be directly converted into data with memory mapping and stored in the DDR memory. Compared with other interfaces, the DMA transfer through the AXI interconnect bus inside the Zynq chip DMA transmission, the transmission efficiency is higher;

(4) The function selection and partial circuit power-down processing are realized by software programming, so as to get rid of the phenomenon that some functions of the traditional hardware platform are not used but still have a large amount of power consumption.

Description of drawings

1 is a structural block diagram of a remote image acquisition device of the present invention;

Fig. 2 is the structural block diagram of the image processing module;

Figure 3 is a schematic diagram of the ITU-R BT.656 standard 625-column data format;

Figure 4 is a schematic diagram of row data in the ITU-R BT.656 standard 625-column data format;

5 is a schematic diagram of an auxiliary signal encoding method;

Fig. 6 is the schematic diagram of valid video signal data arrangement;

Fig. 7 is the schematic diagram of the human-computer interaction interface of the image acquisition system;

Figure 8 is an image acquisition remote display effect diagram.

Detailed ways

The specific embodiments of the present invention are described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that, in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

Example

For the convenience of description, the relevant technical terms appearing in the specific implementation manner are explained first:

FPGA (Field Programmable Gate Array): Field Programmable Gate Array;

ARM (Advanced RISC Machine): advanced reduced instruction set processor;

LVDS (Low-Voltage Differential Signaling): Low-voltage differential signal;

DMA (Direct Memory Access): direct memory access;

FIFO (First Input First Output): first in first out;

PC (Personal Computer): personal computer;

FIG. 1 is an architectural diagram of a specific implementation manner of a remote image acquisition device according to the present invention.

In this embodiment, as shown in FIG. 1 , a remote image acquisition device of the present invention includes: a dual-channel image acquisition module, a high-speed real-time image processing module, and a human-computer interaction module;

The dual-channel image acquisition module includes two image acquisition sensors. The image acquisition sensor is used to acquire images, and then converts the acquired images into electrical signals in the form of optical signals. The converted electrical signals are input to high-speed real-time image processing module;

The high-speed real-time image processing module includes video decoding chip, Zynq chip, network port and memory; among them, Zynq chip also includes FPGA and ARM processor;

Among them, the video decoding chip is used to convert the electrical signal output by the image sensor into a digital video signal whose format is the ITU-R BT.656 standard; as shown in Figure 2, the working state of the video decoding chip is determined by the FPGA part of the Zynq chip. The video decoding chip configuration module is configured through the I2C bus. When the video decoding chip configuration is completed, the chip continuously converts the analog signal of the image sensor under the clock of 27Mhz, and connects the converted digital signal through the LVDS interface. The FPGA side of the Zynq chip;

As shown in Figure 2, the FPGA includes a video chip configuration module, a channel selection module, a data analysis module, an image preprocessing module, and an image data reorganization module;

The FPGA first selects a channel of digital video signal and outputs it to the data analysis module through the channel selection module. The data analysis module analyzes the digital video signal's sign information according to the composition characteristics of the ITU-R BT.656 format, including: valid data, blanking signal sign, The data error flag, the head and tail signals of the image frame, the horizontal synchronization signal, the vertical synchronization signal, and the composite synchronization signal; then firstly, according to the data error flag, it is judged whether the currently parsed image data contains complete information. When the information is incomplete, the current The image frame is discarded, and the data parsing module waits for the input of the next frame of image data; otherwise, the data parsing module inputs the complete and correct image data along with the parsed flag information to the image preprocessing module;

In this embodiment, the ITU-R BT.656 standard divides a video sequence into N frames, the BT.656 standard after PAL format conversion has 625 lines, the bottom field valid data is also 288 lines, and the remaining lines are for marking and A vertical blanking signal that distinguishes the two fields. Interlaced scanning is used when capturing images. Each frame generally has two fields, one is called the top field and the other is called the bottom field. Due to the interlaced scanning, the top and bottom fields can also be called even and odd fields. Among them, the ITU-R BT.656 standard 625-column data format is shown in Figure 3;

Each line of the BT.656 standard is mainly composed of the following four parts: line = end code (EAV) + horizontal blanking (Horizontal Vertical Blanking) + start code (SAV) + valid data (Active Video), as shown in Figure 4. Show.

Each line of data signal is encoded in the form of 8 bits, including auxiliary signals (SAV, EAV), line blanking signals, and effective video signals. The auxiliary signal includes SAV and EAV, which respectively represent the start and end of the data line, and 4byte data is composed of hexadecimal FF00 00XY. Among them, FF 00 00 is the data flag bit of SAV and EAV, XY is the information bit of the auxiliary signal, its coding format is shown in Figure 5, the highest bit (bit7) of XY is fixed data 1; F=0 means even field F= 1 means odd field; V=0 means valid video data for this row; V=1 means there is no valid video data for this row; H=0 means SAV signal H=1 means EAV signal; P3~P0 are protection signals, which are represented by F, V and H signals are calculated and generated; P3=V XOR H; P2=F XOR H; P1=F XOR V; P0=F XOR V XOR H. When V=0 of the time-base signal, it indicates the behavior of video data; when V=1, it indicates the behavior of auxiliary data (when there is no auxiliary data, it is usually 10 and 80 alternately appearing for blanking). Each row corresponds to different EAV and SAV as shown in Table 1.

Table 1 is the row auxiliary signal corresponding to 656 columns of data

Rows F V EAV SAV 1-22 0 1 0xb6 0xab 23-310 0 0 0x9d 0x80 311-312 0 1 0xb6 0xab 313-335 1 1 0xf1 0xec 336-623 1 0 0xda 0xc7 624-625 1 1 0xf1 0xec

In the image preprocessing module, the data separation module firstly uses the data flag information to extract the lines containing valid data in the even field and odd field from the image data in ITU-R BT.656 format, and strips the invalid field blanking at the same time. Signal;

Among them, the preliminary extracted lines still contain auxiliary signals, line blanking signals, and valid image signals; the auxiliary signals include SAV and EAV, which respectively indicate the start and end of the data line, and the auxiliary signals are used to further eliminate the line blanking data. , the removed image data only contains valid grayscale values Y and chrominance values Cr and Cb;

In this embodiment, the blanking line data is composed of 80 and 10, with a total of 280 bytes. In this design, the useless data of the blanking line can be eliminated at the PL end according to the change of the auxiliary signal, so as to reduce the transmission of a frame of useful signals. time and the time the software processes the useless data. For the valid video signal (Valid data), its arrangement order is shown in Figure 6, where Y represents the brightness (Luminance or Luma), that is, the grayscale value; while Cr and Cb are used to represent the chrominance, where Cr reflects The difference between the red part of the RGB input signal and the luminance value of the RGB signal is calculated. Cb reflects the difference between the blue part of the RGB input signal and the luminance value of the RGB signal.

In this embodiment, the combination of valid image data is such that one pixel contains a single Y value, and two adjacent pixels share a pair of Cr and Cb, and the ratio of Y, Cr, and Cb is 4:2: 2; In order to further reduce the amount of data transmission, the latter stage needs to process the Cr and Cb data separately, so the data separation module outputs the Y data and the Cr and Cb data in the valid data separately. The serial-parallel conversion module in the image preprocessing module is used to expand the bandwidth of data transmission and improve the data transmission capability. In the image data, the Y data has a greater impact on the image quality, while the Cr and Cb components have less impact on the imaging of the image data. Therefore, the amount of data is reduced by further compressing the effective data of the Cr and Cb parts. The specific processing process is: Retain the gray scale The value Y remains unchanged. The serial-parallel conversion module first expands the transmission bit width, and then further compresses Cr and Cb: add Cb and Cr at the corresponding positions in the progressive image data and divide by 2 to obtain new Cb and Cr , and finally, the processed Cb, Cr and Y are output to the image data reorganization module synchronously after using the FIFO buffer;

The image data recombination module is used to recombine the valid image data of Y, Cr and Cb; the specific recombination method is: using a Y value corresponding to a group of Cr and Cb to recombine the image data into YUV 4:2:0 data Format, the combined image data is output to the AXI_DMA IP core through the interface conforming to the AXI data stream, and is transmitted to the network port of the acquisition device in the form of network data packets and sent to the human-computer interaction module;

In this embodiment, the external PAL standard TV broadcast signal is an analog video signal. In order to realize ZYNQ data processing, A/D conversion is also required. After the analog video signal is converted by TVP5150AM1, the analog video signal is converted into ITU-R BT. 656 standard digital video signal, but the video signal at this time contains a large number of column blanking signals, row blanking signals, auxiliary signals and other useless signals, which cannot be directly transmitted to the host computer for data processing and image display. The final display still requires us to rearrange the line and field data in the ITU-R BT.656 video standard, extract the grayscale and chromaticity information in the effective line, that is, accurately separate Y, Cr, and Cb, and press the odd field. Even field rearrangement can be used for software playback. Therefore, the video decoding work is particularly important in this system. At present, there are mainly two ways to realize the video decoding: software decoding and hardware decoding.

The design uses software decoding at first, and the hardware is only responsible for the data transmission function. Under this architecture, a large number of software cycles are repeated and the amount of data to be processed is huge, which not only increases the system power consumption and CPU occupancy rate, but also frequently accesses DDR. It also increases the time for data transmission and parsing. In the end, only 3 to 5 frames per second video playback can be achieved, which is far lower than the 25 frames per second speed of the image acquisition front end. Image preprocessing through FPGA needs to analyze the BT.656 standard digital video signal, and analyze the field synchronization signal (F), vertical synchronization signal (V), and horizontal synchronization signal (H) according to the XY control word of each line. By extracting the effective data in each frame of image, a progressive image is assembled.

Every two adjacent rows are grouped together, and all the Y data, Cb data, and Cr data of each two rows are extracted from the original data in the order of "CbYCrY...". According to the first line of 720 bytes of Y data (dma_tvalid for 180 consecutive 100MHz clock cycles), the second line of 720 bytes of Y data, 360 bytes of preprocessed Cb data, and 360 bytes of preprocessed Cr data (No. The two lines of Y, Cb, and Cr have 360 consecutive dma_tvalid) arrangements of 100MHz clock cycles. After that, they can be grouped as a group. A total of 288 groups of such data are sent to the DMA after being buffered by the fifo.

In order to further improve the data transmission efficiency and ensure the real-time performance of the system, it is necessary to further compress the digital video data, and convert YCrCb=4:2:2 to YCrCb=4:2:0.

The ratio between the luminance and chrominance values delivered when a line is scanned is often used to describe various sampling methods. This ratio is usually based on luma values and then described as 4:X:Y, where X and Y are the relative numbers of values in each of the two chroma channels. 4:2:2 means that each horizontal scan line corresponds to two chrominance values for every 4 luminance values. Simply, 4:1:1 means that every 4 luminance values corresponds to 1 chrominance value, 4:4 :4 means that the chroma values are not subsampled. However, this is not completely continuous. 4:2:0 will use one chroma value to correspond to four luma values. For the first chroma element, there are two sampling values, and the second chroma element is not sampled. A full color image cannot be produced. In practice, 4:2:0 means that there are two chroma samples per scan line and only interlace is sampled.

While converting 4:2:2 to 4:2:0 may reduce color saturation in fine detail, it usually does not reduce color saturation in larger objects. The specific processing process is: retain the Y data in the original image, add Cb and Cr at the corresponding position in the progressive image data and divide by 2 to obtain new Cb and Cr, and then the data can be grouped by two rows, A total of 288 groups of such data are sent to the DMA after passing through the FIFO buffer for each frame of image. This processing method further compresses the data amount of one frame of image, which effectively improves the data transmission efficiency.

Finally, a frame of data of a 625-line BT.656 standard digital video signal is removed at the PL end according to the change of the auxiliary signal, and the line blanking signal is removed. After the useful data is rearranged and processed, it is transferred to the memory through DMA, and the software passes the data through the parity. After interspersed, the player can display the image correctly.

The ARM processor runs the embedded control program and is responsible for receiving the instructions of the remote PC. After parsing the instructions, it transmits the control signal to the FPGA side through the AXI-Lite bus to realize the video decoding chip configuration module, channel selection module, and AXI_DMA IP core. Configuration, after the configuration is completed, the function scheduling of the acquisition device is performed, specifically: the video decoding chip decodes the electrical signal converted by the image acquisition sensor, and the output format is the digital video signal of the ITU-R BT.656 standard, which is controlled by the ARM processor. The FPGA reorganizes the image data of the digital video signal, and then transmits the reorganized image data to the memory for storage by DMA, and transmits it to the human interaction module through the network port at the same time;

The human-computer interaction module is used to configure the IP address of the image acquisition device, realize the remote transmission of the signal, and display the acquired image in real time through the display.

Experimental results and analysis

Use the Vivado software of Xilinx Company to complete the development on the FPGA side and export the hardware platform file. Use Petalinux software to build an embedded Linux system adapted to the hardware platform, and import the driver software and applications into the embedded Linux system after the system is transplanted. After completing the above operations, power on the device again, load the driver program and run the application program after entering the embedded system, first initialize each module of the device, enable network monitoring, and then wait for remote client access.

Functional test: Connect the remote monitoring equipment and image acquisition equipment to the same subnet for network transmission and video display tests. The test steps are as follows:

a) Connect the designed image acquisition device to the computer through a network cable, connect the DC power supply to the acquisition device through a 12V power cable, set the DC power supply current to 3A and output;

b) Start the host computer software in the PC, set the IP address of the image acquisition device, and click "Open Connection";

c) Connect the video source to the input channel of the image acquisition device and output image data. Configure the acquisition channel to be viewed through the host computer software, and enable the transmission of video data.

The host computer software comes with a video display window. When the data transmission is closed, the display window defaults to a black screen state, and the configuration menu bar is in an unconfigurable state. The human-computer interaction interface of the image acquisition system is shown in Figure 7.

When the host computer software establishes a network connection with the image acquisition device and the acquisition device completes the self-test and sends back a self-test success signal to the host computer software, other control windows of the human-computer interaction interface switch to a configurable state. Configure the acquisition device to open channel 1 and enable remote acquisition. The experimental results are shown in Figure 8.

The upper left of the display window displays the display duration of the collected data and the accumulation of the number of image display frames. The entire video collection and display process is very smooth, and the refresh rate is 25 frames per second, which is completely consistent with the front-end ADC collection rate.

Although the illustrative specific embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, As long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the inventive concept are included in the protection list.

Claims (3)

1. A remote image acquisition device, comprising: the system comprises a dual-channel image acquisition module, a high-speed real-time image processing module and a human-computer interaction module;

the dual-channel image acquisition module comprises two image acquisition sensors, the image acquisition sensors are used for acquiring images, converting the acquired images into electric signals in an optical signal mode, and inputting the converted electric signals into the high-speed real-time image processing module through a coaxial cable;

the high-speed real-time image processing module comprises a video decoding chip, a Zynq chip, a network port and a memory; the Zynq chip also comprises an FPGA and an ARM processor;

the ARM processor runs an embedded control program, is responsible for receiving instructions of a remote PC, transmits control signals to the FPGA end through an AXI-Lite bus after the instructions are analyzed, realizes the configuration of a video decoding chip configuration module, a channel selection module and an AXI _ DMA IP core, and performs function scheduling on an acquisition device after the configuration is completed, and specifically comprises the following steps: the video decoding chip decodes the electric signal converted by the image acquisition sensor, outputs a digital video signal with an ITU-R BT.656 standard format, and the ARM processor controls the FPGA to carry out image data recombination on the digital video signal, transmits the recombined image data to a memory in a DMA mode for storage, and simultaneously transmits the recombined image data to the human-computer interaction module through a network port;

the man-machine interaction module is used for configuring the IP address of the image acquisition device, realizing remote transmission of signals and displaying acquired images in real time through the display.

2. The remote image capturing device as claimed in claim 1, wherein the working state of the video decoding chip is configured by the video chip configuration module through an I2C bus, and after the configuration of the video decoding chip is completed, the chip continuously decodes the electrical signal converted by the image capturing sensor under a 27Mhz clock, and the converted digital video signal is connected to the FPGA side of the Zynq chip through the LVDS interface.

3. The remote image capturing device as claimed in claim 1, wherein the FPGA comprises a video chip configuration module, a channel selection module, a data parsing module, an image preprocessing module, and an image data reorganizing module

FPGA selects a path of digital video signal to be output to a data analysis module through a channel selection module, and the data analysis module analyzes the mark information of the digital video signal, and the method comprises the following steps: valid data, blanking signal flag, data error flag, head and tail signals of image frame, line synchronizing signal, field synchronizing signal, composite synchronizing signal; then judging whether the currently analyzed image data contains complete information according to the data error mark, discarding the current image frame when the information is incomplete, and waiting for the input of the next frame of image data by the data analysis module; otherwise, the data analysis module inputs the complete and error-free image data to the image preprocessing module along with the analyzed mark information;

in the image preprocessing module, a data separation module firstly utilizes data mark information to respectively extract lines containing valid data in an even field and an odd field from image data in an ITU-R BT.656 format, and simultaneously strips an invalid field blanking signal; the method comprises the steps that lines extracted preliminarily still comprise auxiliary signals, line blanking signals and effective image signals, line blanking data are further removed through the auxiliary signals, at the moment, the removed image data only comprise effective gray-scale values Y and chromatic values Cr and Cb, wherein Cr reflects the difference between the red part of an RGB input signal and the brightness value of the RGB signal, and Cb reflects the difference between the blue part of the RGB input signal and the brightness value of the RGB signal; keeping the gray-scale value Y unchanged, firstly expanding the transmission bit width through a serial-parallel conversion module, then further compressing Cr and Cb to new Cr and Cb, and finally synchronously outputting the processed Cr, Cb and Y to an image data recombination module after FIFO (first in first out) caching;

the image data recombination module is used for recombining the effective image data of Y, Cr and Cb; the specific recombination mode is as follows: recombining the image data into YUV 4by using a Y value corresponding to a group of Cr and Cb: 2: and the data format of 0, the combined image data is output to an AXI _ DMA IP core through an interface conforming to an AXI data stream, and is transmitted to a network port of the acquisition device in the form of a network data packet and is sent to the man-machine interaction module.

Images