RU2688616C1 - Method of compensation of geometric noise of infrared images - Google Patents

Abstract

FIELD: data processing.SUBSTANCE: invention relates to the field of digital image processing and concerns a method for compensating the geometric noise of infrared images from sensors with a vertical arrangement of photosensitive elements. Method consists in receiving the radiation flux and subtracting from the brightness array of pixels of the input image of the array compensated constant components of the signals from the photosensitive elements. Array of constant component signals from photosensitive elements is obtained as a result of recurrent averaging of the brightness of pixels in a set of frames. In each frame, the rows are randomly rearranged, the variance of the brightness gradient is estimated in the recurrently averaged frame in the row direction, and compared with its previous maximum value. If this value is exceeded, the recurrent averaged frame is recorded to the memory, divided into low-frequency and high-frequency components, the high-frequency component is stored in memory and subtracted from the current frame. If the value is not exceeded, the previously stored high-frequency component is subtracted from the current frame.EFFECT: technical result consists in forming a calibration frame with uniform brightness, regardless of the brightness distribution of the observed scene.1 cl, 10 dwg

Description

The invention relates to the field of digital image processing, in particular - filtering noise, and can be used to improve the visual quality of infrared images, for which the actual task of compensating for the heterogeneity of the constant component of the signal from a matrix photodetector device (MFP) - geometric noise.

The prior art method and system for the correction of geometric noise (patent US 8503821, published 06.08.2013, IPC: G06K 9/40). In this method, compensation is based on the comparison of two frames of a video sequence and contains the following steps:

- assessment of the pixel offset between the first and second frames with mfpu, and this assessment is performed on the characteristic features (special points) of the scene in the first and second frames;

- determination of differences in brightness of pixels from the first and second frames corresponding to the same objects of the scene;

- compensation of brightness inhomogeneity based on differences in the average brightness of frames and information on the direction of displacement from frame to frame.

The disadvantage of this method is the limitation of its use for low-contrast images, in which the interframe offset at specific points is estimated with a large error.

Known methods for correcting the heterogeneity of scanning multi-element photodetectors by scene signals (patent RU 2347324, published 05/28/2007, IPC: H04N 5/33 and patent RU 2411684, published on 10.02.2011, IPC: H04N 5/33, H04N 1/409) in which:

- sequential registration of the scene elements by adjacent photosensitive elements of the sensor is performed during scanning, which is performed in the direction perpendicular to the lines of photosensitive elements;

- the dependence of the signals of each element on the signals of the neighboring element is determined and the parameters of the corrective functions are estimated by these dependencies: coefficients of first-order polynomials (patent RU 2347324, published on 28.05.2007, IPC: H04N 5/33) or second order (patent RU 2411684, published 10.02 .2011, IPC: H04N 5/33, H04N 1/409);

- after evaluating the parameters of the functions, the signals of each element of the photoreceiver are corrected sequentially relative to the previous one.

The disadvantages of the considered methods are:

- the need for the light fluxes of the same elements of the scene to hit the adjacent elements of the mfpu and the flows of the neighboring elements of the scene to the same element of the device;

- large processing time associated with the calculation of the linear regression coefficients for each pair of adjacent frame pixels.

From these drawbacks, there is a free method for leveling the uneven sensitivity of photodetectors of scanning lines of thermal imagers (patent RU 2113065, published 10.06.1998, H04N 5/33), which contains the following steps:

- line by line decompose the video signal from the outputs of photodetectors,

- video signals from the output of each photodetector summarize along each line,

- smooth the resulting sequence of total signals in the direction of the vertical scan,

- for each line produce a correction signal by dividing the smoothed sum signal for a given line by the sum signal corresponding to this line,

- form the resulting aligned signal on each line by multiplying the video line from the output of the photodetector to the corresponding correction signal.

The smoothing of the sequence of total signals in the direction of the frame sweep can also be carried out by replacing the values of the total signals received in the time interval of forming the rows with the value of the sum of the neighboring values of the total signals.

The method, according to its description, implies the direction of reading charge packets with mfpu in rows, however, when changing the direction of summing video signals (from row to column) and smoothing the sum signals (in the direction of line scanning), it can also be applied to mfpu with vertical reading direction of charge packages.

The disadvantages of the method include high quality geometric noise correction only when shooting scenes with a uniform background, in which the luminous flux incident on all elements of the mfp is approximately the same. If there are objects in the frame (in the reading direction of the charge from the photosensitive elements of the mfpu) objects with brightness different from the brightness of the background (more or less), the alignment of the average brightness in the mfp lines / columns leads to artifacts: the appearance of bands in the background background (dark or light ) with a width corresponding to the width of an extended object. This effect is illustrated in FIG. 1 and FIG. 2, where a fragment of an image from an Xenics Bobcat 640 short-range infrared camera is shown before and after the geometric noise correction according to the method under consideration, respectively. As seen in FIG. 2, a band appears above the low-brightness cell tower image, the brightness of which is higher than the background brightness.

A uniformity of the brightness of the calibration scene allows for averaging the sequence of frames in time, which gives a positive effect, but only on a large set of different scenes with uncorrelated scenes. However, for most practical tasks related to shooting outdoors, the frame scene usually consists of two areas of different brightness: less contrast in the upper part of the frame (sky) and more contrast - in the lower part (objects on the ground). Also, often the angles of movement of the camera in azimuth exceed the angles of movement in the plane of the elevation angle. In such cases, averaging over a sequence of frames leads to the formation of a calibration image with two regions with different brightness.

A calibration frame that is homogeneous in brightness can be selected as a prototype to select a method for compensating the heterogeneity of amplification of photosensitive elements of a multi-element photodetector (patent RU 2449491 published 27.04.2012, IPC: H04N 5/33, G02B 23/12). In this method, the constant components of signals from photosensitive elements are obtained as a result of defocusing, by introducing a thermal imaging device of a defocusing element into the optical system path, which forms a uniform radiation flux or a stream with a given uniformity at the input of the photoreceiver. The compensation consists in receiving the radiation flux to be registered and processing the signal produced by the photo-receiver in accordance with the formula

S is an array of image pixels as a result of compensation,

T ⁽ⁱⁿ⁾ is the input array of pixels of the image coming from the multi-element photodetector,

T ^(fon) is an array of constant components of signals from photosensitive elements obtained as a result of optical defocusing,

K - an array of coefficients for the correction of the sensitivity of the elements,

i and j - the number of pixels in a row and the row number of the image, respectively. When forming compensated constant components of signals from photosensitive elements by defocusing, the required field of view of the thermal imaging device is retained.

The disadvantage of the prototype method is the complexity of the design of the lens of a thermal imaging device associated with the need to introduce an additional defocusing lens into it.

The technical problem solved by the creation of the claimed invention is the need to form a uniform brightness gauge frame for estimating the geometric noise of the matrix photoreceiver (MFP), regardless of the plot of the observed scene, without using a defocusing lens in the optical path of the infrared camera.

The technical result consists in forming a calibration frame with uniform brightness, regardless of the brightness distribution of the observed scene for mfpu with the vertical direction of reading the charge packets.

The technical result is achieved due to the fact that the method of compensating the geometric noise of infrared images from sensors with vertical lines of photosensitive elements consists in receiving the radiation flow to be recorded and subtracting the input image of the array of compensated constant components from the photosensitive elements from the brightness array of pixels. (geometric noise estimates). At the same time, it differs from the prototype in that the array of constant components of the signals from the photosensitive elements is obtained as a result of recurrent averaging of the brightness of the pixels of the set of frames. When forming a frame with averaged brightness, the permutation of the rows is carried out in such a way that the brightness of the pixels corresponding to the photosensitive elements of the matrix photoreceiver (MFP) in the rows with even numbers is averaged with the brightness of the pixels that are only in the even lines, and the brightness of the pixels corresponding to the photosensitive elements of the mfpU in rows with odd numbers, averaged with pixel brightness, located only in odd lines. Moreover, in each frame, the rows are randomly rearranged, the variance of the brightness gradient is estimated in the recurrently averaged frame in the row direction and compared with its previous maximum value. If this value is exceeded, they write to the memory of the recurrent averaged frame, divide it into low-frequency and high-frequency components, retain the high-frequency component in memory and subtract it from the current frame. If the value is not exceeded, the previously stored high-frequency component is subtracted from the current frame.

The presence of geometric noise due to the principle of signal generation with mfpu by sequentially polling the signals of all photosensitive elements built-in electronic switch, because of which the output signal contains stationary noise caused by the heterogeneity of the dark current. Geometric noise appears on the image as horizontal or vertical stripes of different brightness depending on the direction of reading the charge packets with mfpu: in rows or in columns.

The achievement of the technical result is based on the adoption of two hypotheses: the hypothesis of the additive nature of the geometric noise and the hypothesis of the same constant component of all photosensitive elements of each column of the mfpu. The specified offset is due to the technology of manufacturing mfpu for infrared cameras (Tarasov, VV, Yakushenkov, Yu.G. Infrared systems of the "looking" type. - M .: Logos, 2004, p. 452).

Geometric noise (article Perry D.L., Dereniak E.L. Linear theory of nonuniformity correction in infrared staring sensors // Optical Engineering. - 1993. Vol. 32. - P. 1854-1859), described by a linear model:

Y _ij - the brightness of the image pixels with geometric noise,

i and j are the row and column of the sensor,

g _j - the gain of the elements of the j-th line,

X _ij - the brightness of the pixels of the image without geometric noise,

b _j - constant brightness offset at the output of the elements of the j-th ruler.

In a number of literary sources, for example (article Hardie RC at al. Scene-based nonuniformity correction and registration sequences // Appl. Opt. - 2000. - Vol. 39, No. 8. - P. 1241-1250; article Zuo C. at al., Improved interframe registration for the focal plane arrays // Infrared Phys. Technol. - 2012. - Vol. 55, No. 4. - P. 263-269; article by Cao Y. at al. Correction in Infrared Imaging System using ID Horizontal Differential Statistics // IEEE Photonics Journal. - 2017. - Vol. 9, No. 5), it is shown that the model (2) can be simplified to one parameter - a constant offset:

and the correction of additive geometric noise is simply to subtract its estimates for each j-th column:

which excludes from the processing according to (1) the division by the gain factor _{j j} .

Thus, the acceptance of the two hypotheses given above is fair.

The method of compensation of the geometric noise of infrared images as follows.

1. Receive video signals from mfpu and convert them to digital signals.

2. Averaging the frames received from the mfpU in time according to the following algorithm:

в результате перестановки строк пиксели j-го столбца кадра соответствующие одной линейке фоточувствительных элементов с вертикальным направлением считывания зарядовых пакетов, в кадре

по-прежнему остаются в j-м столбце;2.1 - in a frame received at the k-th instant of time, randomly stop all its lines and get a frame

as a result of row permutation, pixels of the j-th column of the frame corresponding to one line of photosensitive elements with a vertical direction of reading charge packets, in the frame

still remain in the jth column;

2.2 - according to a recurrent rule, an auxiliary frame N _{k is} estimated:

where n is the number of previously received frames;

2.3 - frame N _k estimate the variance of the brightness gradient when moving along the line (in the horizontal direction):

означает вычисление математического ожидания;where is the operator

means calculating the expectation;

2.4 - When forming N _k with a uniform background brightness, the dispersion estimate Dh _k will be maximum and equal to the noise power of the sensor.

if, when a new frame is received, the dispersion estimate Dh _{k is} greater than its previous maximum value Dh _max , this means that the “geometric noise-background” ratio in the auxiliary frame has increased and the quality of geometric noise estimation, and, consequently, its correction, will increase therefore write N _k to the calibration frame K:

3. The estimated calibration frame K is divided into two components: a low-frequency K _LF containing a background and a high-frequency K _RF containing the geometric noise of the sensor:

4. Geometric noise compensation is performed according to the mathematical model (4):

followed by linear contrasting frame X _k .

The random permutation of rows in Y _k frame in step 2.1, followed by averaging of frames in step 2.2 can align the background brightness in the N _k frame even in the presence on the scene areas of different brightness (for shooting outdoors - regions "sky" and "earth's surface") .

When forming a frame with averaged brightness, the permutation of the rows is carried out in such a way that the brightness of the pixels corresponding to the photosensitive elements of the photodetector array device (MFP) in the even-numbered rows is averaged with the brightness of the pixels that are only in the even rows, and the brightness of the pixels corresponding to the photosensitive elements The mfpu in the rows with odd numbers are averaged with the brightness of the pixels that are only in the odd lines.

FIG. 3 shows a block diagram of the geometric noise compensation algorithm discussed above.

The principle of operation of the algorithm is explained in FIG. 4-9.

FIG. Figure 4 shows a frame of a video sequence from a short-range infrared camera.

FIG. 5 shows an auxiliary frame N ₁₀₀₀ , obtained by averaging 1000 frames after randomly swapping rows.

FIG. 6 shows the result of the geometric noise compensation in FIG. 4 according to (5) with the subsequent expansion of the dynamic range of brightness to 255 by linear contrasting.

FIG. 6 that the result of geometric noise compensation by the present method for a static camera position when shooting contains correction artifacts similar to compensation according to the method of patent RU 2113065, published June 10, 1998, IPC: H04N 5/33). Elimination of artifacts is achieved by the movement of the camera during the shooting.

FIG. 7 shows a frame from an infrared camera.

FIG. 8 shows the auxiliary frame N ₁₀₀₀ , formed during the execution of steps 2.1 - 2.4 in a video sequence of 1000 frames with a changing plot: during the shooting, the camera is slowly moved in the clockwise and counterclockwise directions.

FIG. 9 shows the result of the geometric noise compensation in FIG. 7 with the subsequent expansion of the dynamic range of brightness to 255.

It should be noted that if the camera moves during the evaluation of the calibration frame K, and scenes with different scenes fall in its field of view, the quality of compensation of the geometric noise increases. This is explained by the fact that, due to the central limit theorem, the law of brightness distribution of the background of the calibration frame tends to be normal, and the accumulation of geometric noise statistics is effective. Indeed, in FIG. 9, unlike FIG. 6, there are no correction artifacts in the form of stripes with a brightness different from the brightness of the background. Compensation artifacts occur when there are extended vertical objects in the frame.

A feature of some infrared cameras with digital output is the formation of frames with a pronounced interlace structure. This effect for the Xenics Bobcat 640 camera is illustrated in FIG. 10 (scale - 1600%). For such cameras in step 2.1, the permutation of even and odd lines is performed separately. In this case, even lines after permutation fall only on even positions, and odd lines - only on odd positions.

Thus, the proposed method of compensating the geometric noise of infrared images allows the formation of a calibration frame with uniform brightness, regardless of the brightness distribution of the observed scene for mfpu with vertical reading direction of charge packets.

Claims (1)

The method of compensating the geometric noise of infrared images from sensors with a vertical arrangement of photosensitive element lines, which consists in receiving the radiation flux to be recorded and subtracting the input image of the compensated constant components of the photosensitive elements from the brightness array of pixels from the photosensitive elements, characterized in that the array is constant component signals from photosensitive elements are obtained as a result of recurrent averaging the brightness of the pixels of the set of frames, and when forming a frame with averaged brightness, the permutation of the rows is carried out in such a way that the brightness of the pixels corresponding to the photosensitive elements of the photodetector array device (mfpu) in even-numbered rows is averaged with pixel brightness that are only in even rows, and the brightness the pixels corresponding to the photosensitive elements of the mfpu in the rows with odd numbers are averaged with the brightness of the pixels that are only in the odd lines, with each m frame randomly rearranged the line, estimate the variance of the brightness gradient in a recurrently averaged frame in the direction of the lines and compare it with its previous maximum value; if this value is exceeded, write a recurrent averaged frame into memory, divide it into low-frequency and high-frequency components, store the high-frequency component in memory and subtract it from the current frame; if the value is not exceeded, the previously stored high-frequency component is subtracted from the current frame.

Images