CN116342394B - A FPGA-based real-time image demosaicing method, device and medium - Google Patents

Abstract

The invention discloses an FPGA-based real-time image demosaicing method, device and medium. The method realizes real-time conversion from the RAW domain to the RGB domain based on the FPGA. Firstly, the image arrangement mode, image resolution, and quantization bit width of the RAW domain are determined. , the number of pixels input by each clock and the number of pixels output by each clock; then determine the interpolation templates of different components; secondly, perform row-column pipeline buffering on the image according to the interpolation template to obtain the matrix to be interpolated; then calculate the target pixels according to the interpolation template and the matrix to be interpolated The missing channel component; finally, the interpolation result is converted and output. The present invention adopts a pipeline design, has low processing delay, can improve interpolation quality by using inter-pixel related information, has low computational complexity, flexible calculation, and can be flexibly adapted to different resolutions, different quantization bit widths, different RAW domain arrangement modes, and different The number of pixels input by each clock and the number of pixels output by each clock.

Description

A FPGA-based real-time image demosaicing method, device and medium

technical field

The present invention relates to the technical field of digital image processing, in particular to an FPGA-based real-time image demosaicing method, device and medium.

Background technique

Digital color image is one of the main channels for people to obtain media information. Each pixel of a digital color image is composed of quantized K-bit width red (R), green (G), and blue (B). The bit width K is usually 8, 10, 12, 16 bits. However, due to technical and cost reasons, the common color digital cameras currently on the market are single-sensor digital cameras, that is, an image sensor is used to capture the color information of each pixel in a color image. The image sensor in a single-sensor color digital camera is usually a photosensitive coupling device (Charge-coupled Device, CCD) or a complementary metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS) chip, while CCD and CMOS can only perceive the intensity of light instead of Color, and color information is obtained through a color filter array (Color Filter Array, CFA), usually with the suffix .raw, so it is also called a RAW domain map. The process of recovering an RGB color image from a RAW domain image is called CFA Interpolation or Demosaicing. Demosaicing is the most front-end and one of the most critical links in the entire camera image processing.

Although the current demosaic methods based on FPGA (Field Programmable Gate Arrays, programmable array logic) have their own advantages, most of them are only designed for a certain resolution and a certain quantization bit width, which has poor flexibility and cannot adapt to different front-end systems. Enter an image.

Contents of the invention

The object of the present invention is to provide an FPGA-based real-time image demosaicing method, device and medium for the deficiencies of the prior art.

The object of the present invention is achieved through the following technical solutions: the first aspect of the embodiment of the present invention provides a real-time image demosaicing method based on FPGA, comprising the following steps:

(1) Determine the arrangement mode of the RAW domain image, image resolution, quantization bit width, number of input pixels per clock and output pixels per clock;

(2) Determine the interpolation templates for different components;

(3) performing row-column pipeline buffering on the RAW domain image in the step (1) according to the interpolation template obtained in the step (2), so as to obtain the row count and column count of the matrix to be interpolated and the central pixel;

(4) Calculate the missing channel component of the central pixel according to the interpolation template obtained in step (2) and the matrix to be interpolated obtained in step (3);

(5) Perform conversion processing on the components obtained in step (4), and stitch R, G, and B of the central pixel in order.

Further, the arrangement mode includes RGGB, GRBG, BGGR and GBRG.

Further, the number of pixel points for each clock output is greater than or equal to the number of pixel points for each clock input.

Further, the step (2) includes the following sub-steps:

(2.1) Determine the G component interpolation template on the R channel;

(2.2) Determine the G component interpolation template on the B channel;

(2.3) Determine the R component interpolation template on the Gr channel;

(2.4) Determine the B component interpolation template on the Gr channel;

(2.5) Determine the R component interpolation template on the Gb channel;

(2.6) Determine the B component interpolation template on the Gb channel;

(2.7) Determine the B component interpolation template on the R channel;

(2.8) Determine the R component interpolation template on the B channel.

Further, the step (3) includes the following sub-steps:

(3.1) Multiple row buffers are connected in series behind the original data; the number of row buffers is determined according to the size of the interpolation template obtained in step (2);

(3.2) Connect multiple D flip-flops in series after each row buffer and original data to obtain a matrix to be interpolated with a size of M×(NPPC2+N-1), and count rows and columns according to the resolution of the RAW domain image , to obtain the row count and column count of the central pixel; where the number of D flip-flops is determined according to the number of input pixels per clock and the number of output pixels per clock of the RAW domain image determined in the step (1) and the step (2) The size of the obtained interpolation template is determined.

Further, the step (4) includes the following sub-steps:

(4.1) Split the matrix to be interpolated with a size of M×(NPPC2+N-1) obtained in step (3) to obtain 2 matrices to be interpolated with a size of M×N in NPPC, where NPPC2 represents the RAW domain The number of pixels output per clock of the image;

(4.2) According to the row count and column count obtained in the step (3), determine in parallel whether the central pixel channel of the NPPC2 matrices to be interpolated is the R channel, the B channel, the Gr channel, or the Gb channel;

(4.3) Determine whether the matrix to be interpolated includes pixels at boundary positions, and if the matrix to be interpolated includes pixels at boundary positions, perform mirror mapping on the missing elements of the matrix to be interpolated; otherwise, go directly to step (4.4);

(4.4) Shift the template to be interpolated to the left by 8 bits to expand the matrix to be interpolated by 256 times;

(4.5) According to the central pixel point channel of each matrix to be interpolated and the corresponding matrix to be interpolated and the interpolation template, two parallel NPPC interpolation calculation modules are used to calculate the two missing components of the central pixel point.

Further, the step (4.5) specifically includes: if the central pixel point channel of the matrix to be interpolated is the Gr channel, select the R component interpolation template on the Gr channel and perform point multiplication with the matrix to be interpolated to obtain the missing R component, and select Gr Do point multiplication of the B component interpolation template on the channel and the matrix to be interpolated to obtain the missing B component; if the center pixel channel of the matrix to be interpolated is the Gb channel, select the R component interpolation template on the Gb channel to perform point multiplication with the matrix to be interpolated To obtain the missing R component, select the B component interpolation template on the Gb channel and perform point multiplication with the matrix to be interpolated to obtain the missing B component; if the center pixel channel of the matrix to be interpolated is the R channel, select the G component on the R channel Perform point multiplication between the interpolation template and the matrix to be interpolated to obtain the missing G component, and select the B component interpolation template on the R channel to perform point multiplication with the matrix to be interpolated to obtain the missing B component; if the center pixel point channel of the matrix to be interpolated is B Channel, select the R component interpolation template on the B channel and perform point multiplication with the matrix to be interpolated to obtain the missing R component, select the G component interpolation template on the B channel and perform point multiplication with the matrix to be interpolated to obtain the missing G component.

Further, the step (5) includes the following sub-steps:

(5.1) The components obtained in the step (4) are reduced by 256 times;

(5.2) Determine the reduced component: if the reduced component is less than 0, convert it to 0; if the reduced component is greater than , it is converted to /> ; If the reduced component is less than or equal to /> And greater than or equal to 0, no conversion will be performed and the original value will be kept; where K represents the quantization bit width;

(5.3) The R, G, and B of the 2 central pixels of NPPC are spliced in order and output.

The second aspect of the embodiment of the present invention provides an FPGA-based real-time image demosaicing device, including one or more processors, configured to implement the above-mentioned FPGA-based real-time image demosaicing method.

The third aspect of the embodiment of the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it is used to implement the FPGA-based real-time image demosaicing method described above.

The beneficial effect of the present invention is that the present invention adopts the pipeline design method, and the delay of image processing is low; the present invention can effectively improve the interpolation quality by using the relevant information between pixels, the calculation complexity is low, the calculation is flexible, and different resolutions can be flexibly adapted , Different quantization bit widths, different RAW domain arrangement modes, different number of input pixels per clock and output pixels per clock, can adapt to different input images at the front end.

Description of drawings

Fig. 1 is the flowchart of the FPGA-based real-time image demosaicing method of the present invention;

Fig. 2 is a schematic diagram of the RGGB arrangement mode of the RAW domain image of the present invention;

Fig. 3 is a schematic diagram of the GRBG arrangement mode of the RAW domain image of the present invention;

Fig. 4 is a schematic diagram of the BGGR arrangement mode of the RAW domain image of the present invention;

Fig. 5 is a schematic diagram of the GBRG arrangement mode of the RAW domain image of the present invention;

Fig. 6 is a schematic diagram of the B component on the Gr channel and the R component interpolation template on the Gb channel determined by the bilinear interpolation method under the GRBG arrangement mode of the present invention;

Fig. 7 is a schematic diagram of the R component on the Gr channel and the B component interpolation template on the Gb channel determined by the bilinear interpolation method under the GRBG arrangement mode of the present invention;

Fig. 8 is a schematic diagram of the B component on the R channel and the R component interpolation template on the B channel determined by the bilinear interpolation method under the GRBG arrangement mode of the present invention;

Fig. 9 is a schematic diagram of the G component on the R channel and the G component interpolation template on the B channel determined by bilinear interpolation in the GRBG arrangement mode of the present invention;

Fig. 10 is a schematic diagram of the B component on the Gr channel and the R component interpolation template on the Gb channel determined by the HQLI method in the GRBG arrangement mode of the present invention;

11 is a schematic diagram of the R component on the Gr channel and the B component interpolation template on the Gb channel determined by the HQLI method in the GRBG arrangement mode of the present invention;

Fig. 12 is a schematic diagram of the G component on the R channel and the G component interpolation template on the B channel determined by the HQLI method under the GRBG arrangement mode of the present invention;

Fig. 13 is a schematic diagram of the B component on the R channel and the R component interpolation template on the B channel determined by the HQLI method under the GRBG arrangement mode of the present invention;

Fig. 14 is a flowchart of a method for obtaining a matrix to be interpolated in the present invention;

Fig. 15 is a matrix map to be interpolated for pixels in the lower right corner of the image under the GRBG arrangement mode of the present invention;

Fig. 16 is a matrix map to be interpolated for pixels in the upper left corner of the image under the GRBG arrangement mode of the present invention;

Fig. 17 is a schematic diagram of decomposing the matrix to be interpolated into NPPC2 M×N matrices to be interpolated with a size of M×(N+NPPC2-1) according to the present invention;

FIG. 18 is a schematic structural diagram of an FPGA-based real-time image demosaicing device of the present invention.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with aspects of the invention as recited in the appended claims.

The terminology used in the present invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein and in the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present invention, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

The present invention will be described in detail below in conjunction with the accompanying drawings. If there is no conflict, the features in the following embodiments and implementations can be combined with each other.

Referring to Fig. 1, the FPGA-based real-time image demosaicing method of the present invention specifically comprises the following steps:

(1) Determine the arrangement mode of the RAW domain image, the image resolution size (rows × cols), the quantization bit width K, the number of input pixels per clock (NPPC1) and the number of output pixels per clock (NPPC2).

In this embodiment, the arrangement modes of the RAW domain images include four types: RGGB, GRBG, BGGR, and GBRG. Therefore, it is necessary to first determine which arrangement mode among the above four arrangement modes is used for the RAW domain images. Among them, the arrangement pattern of RGGB is shown in FIG. 2 , the arrangement pattern of GRBG is shown in FIG. 3 , the arrangement pattern of BGGR is shown in FIG. 4 , and the arrangement pattern of GBRG is shown in FIG. 5 . In addition, for the convenience of subsequent calculations, the G that goes with R is denoted as Gr, and the G that goes with B is denoted as Gb.

Specifically, at the beginning of receiving a picture, determine and latch the arrangement of the RAW domain image input by the front end, the image resolution size (rows × cols), the quantization bit width K of the image, and the number of pixels per clock input (NPPC1) And the number of pixels output per clock (NPPC2), and these parameters are latched, which can effectively prevent the settings in the image processing from changing and causing errors.

In this embodiment, the number of pixels output by each clock is greater than or equal to the number of pixels input by each clock, that is, NPPC2≥NPPC1. For example, you can configure each clock to input 1 pixel, and each clock to output 1, 2, 4, and 8 pixels; you can configure each clock to input 2 pixels, and each clock to output 2, 4, and 8 pixels points, but the number of pixels per clock output cannot be less than 2.

(2) Determine the interpolation templates for different components.

In this embodiment, the component interpolation template is a matrix with a size of M×N, representing a matrix of M rows×N columns. It should be understood that the size of the matrix is determined according to actual conditions.

(2.1) Determine the G component interpolation template on the R channel.

(2.2) Determine the G component interpolation template on the B channel.

(2.3) Determine the R component interpolation template on the Gr channel.

(2.4) Determine the B component interpolation template on the Gr channel.

(2.5) Determine the R component interpolation template on the Gb channel.

(2.6) Determine the B component interpolation template on the Gb channel.

(2.7) Determine the B component interpolation template on the R channel.

(2.8) Determine the R component interpolation template on the B channel.

In this embodiment, input of multiple interpolation templates may be supported, and interpolation templates of different components may be determined by using a bilinear interpolation method, a high quality linear interpolation (High quality linear interpolation, HQLI) method, or other methods.

Exemplarily, taking the GRBG arrangement pattern shown in Figure 3 as an example, the interpolation template determined by the bilinear interpolation method is shown in Figure 6-Figure 9, and the interpolation template is expressed as a matrix with a size of 3×3, namely M=3, N=3. Among them, the B component on the Gr channel and the R component interpolation template on the Gb channel are shown in Figure 6, the R component on the Gr channel and the B component interpolation template on the Gb channel are shown in Figure 7, and the B component on the R channel The R component interpolation templates on the R and B channels are shown in FIG. 8 , and the G component interpolation templates on the R channel and the G component interpolation templates on the B channel are shown in FIG. 9 .

Exemplarily, taking the GRBG arrangement pattern shown in Figure 3 as an example, the interpolation template determined by the High quality linear interpolation (HQLI) method is shown in Figure 10-Figure 13, the interpolation template is expressed as a matrix, its The size is 5×5, that is, M=5, N=5. Among them, the B component of the Gr channel and the R component interpolation template of the Gb channel are shown in Figure 10, the R component of the Gr channel and the B component interpolation template of the Gb channel are shown in Figure 11, the G component of the R channel and the G component of the B channel The component interpolation template is shown in Figure 12, and the B component of the R channel and the R component interpolation template of the B channel are shown in Figure 13.

It should be understood that the bilinear interpolation method is simpler to calculate, occupies less resources, and has lower delay; while the HQLI method takes into account the relevant information of the pixels in the interpolation window, and the interpolation effect is better; therefore, the appropriate one can be selected according to the actual situation. Interpolation template. Of course, templates in other methods can also be selected, for example, a gradient calculation module can also be added to integrate gradient and boundary information, and the most suitable interpolation template can be calculated in real time and input to the demosaic module for interpolation.

(3) According to the interpolation template obtained in step (2), the RAW domain image in step (1) is cached in a row-column pipeline to obtain the row count and column count of the matrix to be interpolated and the central pixel.

(3.1) Multiple row buffers are connected in series behind the original data; the number of row buffers is determined according to the size of the interpolation template obtained in step (2).

Specifically, the size of the interpolation template obtained according to step (2) is M×N, and M−1 line buffers are connected in series after the original data.

(3.2) Connect multiple D flip-flops in series after each row buffer and original data to obtain a matrix to be interpolated with a size of M×(NPPC2+N-1), and count rows and columns according to the resolution of the RAW domain image , to obtain the row count and column count of the central pixel; where the number of D flip-flops is obtained according to the number of input pixels per clock and the number of output pixels per clock of the RAW domain image determined in step (1) and step (2) The size of the interpolation template is determined.

Specifically, the number of D flip-flops connected in series is determined according to the number of input pixels per clock and the number of output pixels per clock of the RAW domain image determined in step (1) and the size of the interpolation template obtained in step (2), that is, respectively concatenated after each row buffer and raw data D flip-flops, /> Represents rounding up, and a matrix to be interpolated with a size of M×(NPPC2+N-1) can be obtained, as shown in Figure 14.

It should be noted that the series D flip-flop can delay the original data and other related data by one cycle. For example, D0, D1, D2, and D3 are sent sequentially on each rising edge of the clock. If the D flip-flop is not added, at a certain moment only One data can be obtained. If three D flip-flops are connected in series, at the moment of sending D3, the data after the first D flip-flop is D2, the data after the second D flip-flop is D1, and the data after the third D flip-flop The data is D0, and 4 data can be obtained at the same time. It should be understood that in digital logic, D flip-flops are generally used for delay, and of course, other flip-flops can also be used for delay.

On the other hand, count the rows and columns according to the resolution of the RAW domain image, and output the row count and column count of the central pixel of the matrix to be interpolated while outputting the matrix to be interpolated. Specifically, M×(NPPC2+N-1) is the size of the matrix to be interpolated, which can be regarded as the size of the window function. This window function slides on the original input image, and the selected data is multiplied and added with the interpolation template as in step (2.8). , calculate the missing channel information of the center point of the window function. For example, the size of the original image is 500×500 pixels, and the size of the window function is 5×5 pixels. By multiplying and adding according to the interpolation template, the row count and column count of the interpolation center can be obtained, that is, the row count and column count of the center pixel.

(4) According to the interpolation template obtained in step (2) and the matrix to be interpolated obtained in step (3), calculate the missing channel component of the central pixel.

(4.1) Split the matrix to be interpolated with a size of M×(NPPC2+N-1) obtained in step (3) to obtain NPPC2 matrixes to be interpolated with a size of M×N, where NPPC2 represents the RAW domain image Output pixels per clock.

In this embodiment, the size of the matrix to be interpolated obtained in step (3) is M×(NPPC2+N-1), the size of the interpolation template is M×N, and the matrix to be interpolated can be split to obtain NPPC2 with a size of The M×N matrix to be interpolated is shown in Figure 17.

(4.2) According to the row count and column count obtained in step (3), determine in parallel whether the central pixel channel of the NPPC2 matrices to be interpolated is the R channel, the B channel, the Gr channel, or the Gb channel.

Specifically, taking the GRBG arrangement mode as an example, the row count obtained in step (3) is 0~rows-1, and the column count is 0~cols-1. When the row count and column count are both even numbers, the current matrix to be interpolated The central pixel is the Gr component; when the row count is even and the column count is odd, the central pixel of the matrix to be interpolated is the R component; when the row count is odd and the column count is even, the central pixel of the current matrix to be interpolated is B component; when the row count and column count are both odd, the center pixel of the current matrix to be interpolated is the Gb component.

It should be understood that the value of the central pixel of the matrix to be interpolated in the Gr channel is called the Gr component of the central pixel, so the central pixel of the matrix to be interpolated is the Gr component, indicating that the channel of the central pixel of the matrix to be interpolated is Gr channel.

(4.3) Determine whether the matrix to be interpolated includes boundary position pixels, and if the matrix to be interpolated includes boundary position pixels, perform mirror mapping on the missing elements of the matrix to be interpolated; otherwise, go directly to step (4.4).

Specifically, take the GRBG arrangement mode, whose size is 5×5, NPPC1=NPPC2=1 as an example, as shown in Figure 15, the remaining missing elements can be obtained by mirroring the elements of the known part, the first row 5 rows of mapping, the second row is the mapping of the fourth row, the first column is the mapping of the fifth column, and the second column is the mapping of the fourth column. After mapping, the arrangement order of the four types of pixels can be maintained.

Exemplarily, as shown in Figure 16, only the upper left part of the matrix to be interpolated at the last point is in the image, and the rest is obtained through mirror mapping. The fourth row is mapped to the second row, the fifth row is mapped to the first row, and the Column 4 is mapped to column 2, and column 5 is mapped to column 1.

In summary, the upper half of the matrix to be interpolated at the upper boundary of the image, the lower half of the matrix to be interpolated at the lower boundary of the image, the left half of the matrix to be interpolated at the left boundary of the image, and the matrix to be interpolated at the right boundary of the image The right half is obtained through mirror mapping by the above method.

(4.4) Shift the template to be interpolated to the left by 8 bits to expand the matrix to be interpolated by 256 times, which can ensure the precision of the decimal place of the calculation result.

(4.5) According to the central pixel point channel of each matrix to be interpolated and the corresponding matrix to be interpolated and the interpolation template, two parallel NPPC interpolation calculation modules are used to calculate the two missing components of the central pixel point.

In this embodiment, the NPPC2 interpolation calculation modules are parallelized according to the corresponding matrix to be interpolated and the interpolation template, and the missing two components are calculated according to the type of the central pixel channel of each matrix to be interpolated in the 2 matrices to be interpolated. If the center pixel point channel of the matrix to be interpolated is the Gr channel, select the R component interpolation template on the Gr channel and the matrix to be interpolated to perform point multiplication to obtain the missing R component, and select the B component interpolation template on the Gr channel to perform multiplication with the matrix to be interpolated. Point multiplication to obtain the missing B component; if the central pixel point channel of the matrix to be interpolated is the Gb channel, select the R component interpolation template on the Gb channel and the matrix to be interpolated to perform point multiplication to obtain the missing R component, and select the Gb channel on the Do point multiplication between the B component interpolation template and the matrix to be interpolated to obtain the missing B component; if the center pixel channel of the matrix to be interpolated is the R channel, select the G component interpolation template on the R channel and perform point multiplication with the matrix to be interpolated to obtain the missing G component, select the B component interpolation template on the R channel and perform point multiplication with the matrix to be interpolated to obtain the missing B component; if the center pixel point channel of the matrix to be interpolated is the B channel, select the R component interpolation template on the B channel and Perform point multiplication of the matrix to be interpolated to obtain the missing R component, and select the G component interpolation template on the B channel to perform point multiplication with the matrix to be interpolated to obtain the missing G component.

(5) Convert the components obtained in step (4), and stitch the R, G, and B of the center pixel in order.

(5.1) Reduce the components obtained in step (4) by 256 times.

It should be understood that since the matrix to be interpolated is enlarged by 256 times in step (4.4), the components calculated in step (4.5) need to be reduced by 256 times according to the rounding principle.

(5.2) Determine the reduced component: if the reduced component is less than 0, convert it to 0; if the reduced component is greater than , it is converted to /> ; If the reduced component is less than or equal to /> And greater than or equal to 0, no conversion will be performed and the original value will be kept; where K represents the quantization bit width.

Specifically, the conversion can be performed according to the following formula:

Among them, K is the quantization bit width.

(5.3) The R, G, and B of the 2 central pixels of NPPC are spliced in order and output.

It should be understood that splicing can be performed according to actual needs, such as [R, G, B] or [B, G, R] and so on.

The present invention adopts the design method of the pipeline, and the delay of image processing is low; the present invention can effectively improve the interpolation quality by using the relevant information between pixels, the calculation complexity is low, the calculation is flexible, and it can be flexibly adapted to different resolutions, different quantization bit widths, different The RAW domain arrangement mode, the number of pixels per clock input and the number of pixels per clock output can be adapted to different input images at the front end.

Corresponding to the foregoing embodiments of the FPGA-based real-time image demosaicing method, the present invention also provides embodiments of an FPGA-based real-time image demosaicing device.

Referring to FIG. 18 , an FPGA-based real-time image demosaicing device provided by an embodiment of the present invention includes one or more processors for implementing the FPGA-based real-time image demosaicing method in the foregoing embodiments.

The embodiment of the FPGA-based real-time image demosaicing device of the present invention can be applied to any device with data processing capability, and any device with data processing capability can be a device or device such as a computer. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for operation by the processor of any device capable of data processing. From the hardware level, as shown in Figure 18, it is a hardware structure diagram of any device with data processing capabilities where the FPGA-based real-time image demosaicing device of the present invention is located, except for the processor, memory, and network shown in Figure 18 In addition to the interface and the non-volatile memory, any device with data processing capability where the device in the embodiment is usually located may also include other hardware according to the actual function of any device with data processing capability, which will not be repeated here.

For the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method for details, and will not be repeated here.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. It can be understood and implemented by those skilled in the art without creative effort.

An embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the FPGA-based real-time image demosaicing method in the foregoing embodiments is implemented.

The computer-readable storage medium may be an internal storage unit of any device capable of data processing described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device capable of data processing, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, or a flash memory card (Flash Card) equipped on the device. wait. Further, the computer-readable storage medium may also include both an internal storage unit of any device capable of data processing and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device capable of data processing, and may also be used to temporarily store data that has been output or will be output.

The above embodiments are only used to illustrate the design concept and characteristics of the present invention, and its purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims (8)

1. a real-time image demosaicing method based on FPGA, is characterized in that, comprises the following steps: (1) Determine the arrangement mode of the RAW domain image, the image resolution size, the quantization bit width, the number of input pixels per clock and the number of output pixels per clock; (2) determine the interpolation template of different components; (3) performing row-column pipeline buffering on the RAW domain image in the step (1) according to the interpolation template obtained in the step (2), to obtain the row count and the column count of the matrix to be interpolated and the center pixel; (4) according to the interpolation template that described step (2) obtains and the matrix to be interpolated that described step (3) obtains, calculate the missing channel component of central pixel point; Described step (4) comprises following substep: (4.1) split the matrix to be interpolated that is M×(NPPC2+N-1) obtained in step (3) to obtain NPPC2 matrixes to be interpolated that are M×N in size, where NPPC2 represents the RAW domain The number of pixels output per clock of the image; (4.2) according to the row count and column count that described step (3) obtains, determine in parallel the central pixel point channel of NPPC2 interpolation matrices, be R channel, B channel, Gr channel or Gb channel; (4.3) Determine whether the matrix to be interpolated includes boundary position pixels, if the matrix to be interpolated comprises boundary position pixels, then the missing elements of the matrix to be interpolated are mirrored; otherwise, directly enter step (4.4); (4.4) Shift the template to be interpolated by 8 bits to the left to enlarge the matrix to be interpolated by 256 times; (4.5) According to the central pixel point channel of each matrix to be interpolated and the corresponding matrix to be interpolated and the interpolation template parallel NPPC2 interpolation calculation modules to calculate the two missing components of the central pixel; The step (4.5) specifically includes: if the central pixel point channel of the matrix to be interpolated is the Gr channel, select the R component interpolation template on the Gr channel and perform point multiplication with the matrix to be interpolated to obtain the missing R component, and select the R component on the Gr channel Perform point multiplication between the B component interpolation template and the matrix to be interpolated to obtain the missing B component; if the center pixel channel of the matrix to be interpolated is the Gb channel, select the R component interpolation template on the Gb channel and perform point multiplication with the matrix to be interpolated to obtain the missing R component, select the B component interpolation template on the Gb channel and perform point multiplication with the matrix to be interpolated to obtain the missing B component; if the center pixel point channel of the matrix to be interpolated is the R channel, select the G component interpolation template on the R channel and Perform point multiplication of the matrix to be interpolated to obtain the missing G component, select the B component interpolation template on the R channel and perform point multiplication with the matrix to be interpolated to obtain the missing B component; if the center pixel channel of the matrix to be interpolated is the B channel, select The R component interpolation template on the B channel is multiplied by the matrix to be interpolated to obtain the missing R component, and the G component interpolation template on the B channel is selected to be multiplied by the matrix to be interpolated to obtain the missing G component; (5) Perform conversion processing on the components obtained in the step (4), and stitch R, G, and B of the central pixel in order. 2. FPGA-based real-time image demosaic method according to claim 1, is characterized in that, described arrangement pattern comprises RGGB, GRBG, BGGR and GBRG. 3. FPGA-based real-time image demosaic method according to claim 1, is characterized in that, described each clock output pixel number is greater than or equal to each clock input pixel number. 4. the real-time image demosaicing method based on FPGA according to claim 1, is characterized in that, described step (2) comprises the following substeps: (2.1) determine the G component interpolation template on the R channel; (2.2) determine the G component interpolation template on the B channel; (2.3) determine the R component interpolation template on the Gr channel; (2.4) determine the B component interpolation template on the Gr channel; (2.5) determine the R component interpolation template on the Gb channel; (2.6) determine the B component interpolation template on the Gb channel; (2.7) determine the B component interpolation template on the R channel; (2.8) Determine the R component interpolation template on the B channel. 5. the FPGA-based real-time image demosaicing method according to claim 1, is characterized in that, described step (3) comprises the following substeps: (3.1) a plurality of line buffers are connected in series after the original data; wherein the quantity of the line buffers is determined according to the size of the interpolation template obtained in the step (2); (3.2) Multiple D flip-flops are connected in series after each row buffer and original data to obtain a matrix to be interpolated with a size of M×(NPPC2+N-1), and count rows and columns according to the resolution of the RAW domain image , to obtain the row count and column count of the central pixel; wherein the number of D flip-flops is based on the number of input pixels per clock and the number of pixels output by each clock of the RAW domain image determined in the step (1) and the steps (2) The size of the obtained interpolation template is determined. 6. the real-time image demosaic method based on FPGA according to claim 1, is characterized in that, described step (5) comprises following substep: (5.1) reducing the component obtained by the step (4) by 256 times; (5.2) Determine the reduced component: if the reduced component is less than 0, convert it to 0; if the reduced component is greater than 2 ^K -1, convert it to 2 ^K -1; if the reduced component If the component is less than or equal to 2 ^K -1 and greater than or equal to 0, no conversion will be performed and the original value will be kept; where K represents the quantization bit width; (5.3) The R, G, and B of the 2 central pixels of NPPC are spliced in order and output. 7. An FPGA-based real-time image demosaicing device, characterized in that it comprises one or more processors for implementing the FPGA-based real-time image demosaicing method according to any one of claims 1-6. 8. A computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, it is used to realize the real-time image demosaicing based on FPGA according to any one of claims 1-6 method.