DE19524577C2 - Sensor for CCD camera - Google Patents

Description

The invention relates to a sensor for a CCD camera several arranged in parallel existing vertical rows of light-sensitive sensor cells according to the preamble of claim 1 (additional application to Acc. P 44 36 426.1-31).

The basic principle of combining sensor charges in the CCD sensor is also used in DE 39 42 615 A1 in order to enable electronic zooming in in the vertical direction. Fig. 1a shows a typical CCD sensor for this. In addition to two vertically adjacent sensor cells ( 1 , 2 ; 4 , 5 ;...; 28 , 29 ; 10 , 11 ; 13 , 14 ;...) There is one memory cell ( 3 ; 6 ; .. ; 30 ; 12 ; 15 ;.... In normal operation, the charges from two vertically adjacent sensor cells ( 1 , 2 ; 4 , 5 ;...) Are transferred together into the respectively adjacent memory cell ( 3 ; 6 ;...) And read from there. When zooming in, only one sensor charge ( 1 ; 4 ;...) Is transferred to the corresponding memory cell ( 3 ; 6 ;...) And read out. The respective underlying sensor cells ( 2 ; 5; ... ) are transferred to the same memory cells ( 3 ; 6 ; ... ) in the next field transfer and read out. This document describes further details, e.g. B. how to avoid the so-called line offset.

However, zooming or summarizing sensor charges in the horizontal direction is unsolved, e.g. (....... 1. 4; 10, 13;.; 19, 22) as the transfer of sensor charges into the memory cells (3;...; 12;...; 21;. ..). To do this, adjacent charges on a line would have to be moved in opposite directions. This is not possible with the normal CCD architecture.

Furthermore, DE 39 42 615 A1 did not recognize how and to what extent the combination of sensor cells can be used to increase sensitivity to light. It is also in principle not possible with the described method to charge the charges from a large number of sensor cells, for. B. 1 , 4 , 7 , 2 , 5 , 8 , 10 , 13 , 16 , 11 , 14 , 17 in the horizontal and vertical directions. The merging would then have to take place across the intervening memory cells 3 , 6 , 12 , 15 .

For the goal of an increase and free adjustability this would be desirable in light sensitivity.

The basic possibility of adjacent sensor charges add is known and is z. B. by W. Yang and A. Chiang in the article: "A Full Fill-FActor CCD Imager with Integrated Siganl Processors ", / 1990 IEEE International Solid-State Circuits Conference, FAM 13.4, pages 218-219, 1990 / described. The solution is complex, however, and it is required in addition the sensor area additional chip area for the Processing unit. This chip area also goes light-integrating sensor area is lost, which is the objective counteracts the increase in light sensitivity.

Finally, DE 39 42 615 A1 also speaks the possibility of Use of color CCD sensors based on the mosaic principle. In the mosaic sensor are optical filters with four mixed colors laid like a checkerboard over the sensor cells. In the well-known  Normal operation will be four vertically adjacent ones Sensor cells combined in mixed colors. In zoom mode are only two vertically adjacent sensor cells summarized. Outside of the sensor, the Mixed colors calculated. The procedure described is however principally on zooming in and on the vertical direction limited. Would the principle described be based on the horizontal direction results inevitably Mixed color white and the color information is lost. In the It is important to suppress the shot noise Sensor charges in both vertical and horizontal directions summarize.

When combining sensor charges, another occurs undesirable effect. With the same read cycle, fewer pixels are read out and the images appear as a result downsized. In DE 39 42 615 A1 this zoom effect is the Task. In the case of shot noise reduction, the Zoom effect just not wanted.

The invention is based on the object

• (1) Shot noise from vertical and horizontal grouping of sensor cell charges Suppress (macro cell integration) and a accompanying image reduction (zoom) whole and one Avoid resolution resolution as much as possible
• (2) enable the use of color CCD sensors and Color macro cells form from which identical to Normal operation, so without additional Processing unit, the color signal can be derived.

This object is achieved by the Characteristic features of claim 1 solved. The advantage is that by combining the sensor charges in the sensor the shot noise is suppressed more than one Summarize behind the readout amplifier of the sensor. By the shift of the macro cells according to the invention and the  simultaneous use of an image memory will continue to be Image reduction avoided. The one for preserving the image size necessary intermediate values arise in real time and without one elaborate computer or special module, which for a Compute the intermediate values necessary in real time ought to. In addition, the one with the grouping of the charges associated loss of resolution largely avoided.

Another advantage is according to claim 1b that Color sensors can be used and despite horizontal and vertically extended macro cells the color information surprisingly not lost. Is particularly advantageous thereby that according to claim 2 and 3 for color image display necessary further processing unchanged despite macro cell formation remains, so no complex computer or additional special module is necessary.

Claim 4 gives the advantage of moving objects the shot noise due to multiple exposures suppress without causing motion blur. As a result of that the image in the memory cells synchronized with the movement of the Object is moved, active compensation arises the motion blur. This can be especially true when observing of production processes can be an advantage. With claim 5 the task solved, the camera on the speed of the synchronize moving object. It can continue after Claim 7 be advantageous, the shift of To carry out macro cells mechanically. This is beneficial when out A CCD sensor is used for easier control that allows macro cell integration, but one Do not allow displacement of the macro cells. Through the configuration according to the invention will nevertheless Image reduction entirely and the reduction in resolution largely avoided.

In the following, exemplary embodiments are described with reference to drawings of the invention explained. It shows  

FIG. 1a shows the basic construction and the reading of an imaging sensor of a CCD camera,

FIG. 1b shows the basic structure of a sensor with only three times six sensor cells and three times three memory cells,

Fig. 1c shows the basic structure of a sensor with only four times eight sensor cells and four times four memory cells,

Fig. 2 shows the improvement in the signal / noise ratio when combining of charges in the sensor as compared to the averaging of adjacent image intensities outside the sensor,

Fig. 3 is a block diagram of a camera with a CCD sensor

Fig. 4 shows the amount of the local transfer function at four, once two and once one macro cells

Fig. 5a the combining of only three color macrocells

Fig. 5b the grouping of three times one color macrocells

Fig. 5c summarizing once five color macro cells.

Summarizing the charges from sensor cells in the CCD sensor brings an improvement in photosensitivity, as experiments ( FIG. 2) show. It is surprising that the averaging 47 of the charges in the CCD sensor is superior to the averaging 48 outside the sensor. In the first case, the signal-to-noise ratio improves in proportion to the number N of averaged sensor cells, in the second case only in proportion to the root of N. The reason is the occurrence of shot noise (photon noise) at low light intensities, as also more complicated theoretical considerations show / JW Goodmann and JF Belscher, "Photon limited images and their restoration," ARPA Order No. 2646, Technical Rep. RADC-TR- / 6-50 (Rome Air Develop Center, New York, 1976; B. Wirnitzer: "Bispectral analysis at low light levels and astronomical speckle masking", J.Opt.Soc.Am.A , Vol.2. No. 1, 1985 /.

The task is now the shot noise (photon noise) too avoid by grouping loads as follows:  

In Fig. 1b it is shown how the charges of the sensor cells 1 ', 2 ', 4 ', 5 ', 7 ', 8 ', 10 ', 11 ', 13 ', 14 ', 16 ', 17 ', 19 ', 20 ', 22 ', 23 ', 25 ', 26 can be combined into a three times three macro cell. Of course, in another exemplary embodiment, fewer or more units can also be combined to form a macro cell without departing from the scope of the invention. It is also possible that the sensor is made up of many identical or different macro cells.

The functional sequence here is that, after exposure, the charges of the sensor cells 1 ', 2 '; 4 ′, 5 ; 7 ′, 8 ′; 10 ′, 11 ′; 13 ′, 14 ′; 16 ′, 17 ′; 19 ′, 20 ′; 22 ′, 23 ′; 25 ', 26 ' in a field transfer to the adjacent memory cells 3 '; 6 ′; 9 ′; 12 ′; 15 ′; 18 ′; 21 ′; 24 ′; 27 'are transferred. In a second step, the charges 3 ′ + 12 ′ + 21 ′, 6 ′ + 15 ′ + 24 ′ and 9 ′ + 18 ′ + 27 ′ are created by two subsequent vertical transfers in the memory cells 38 ′, 39 ′, 40 ′ . The further merging into larger macro-cells with horizontal expansion takes place in a third step many cycles later at the output 37 'of the horizontal shift register 37 ', 38 '; 39 ′; 40 'by repeated horizontal transfers without reading the cell 37 '.

It is surprising that steps two and three also result in an increase in sensitivity which is proportional to the number N of sensor cells combined, and this despite the fact that the actual process of combining takes place many cycles later in the horizontal shift register! In the case of sensors with a large number of macro cells in particular, the combination can take place several hundred cycles after exposure. Measurements show that outside the sensor, directly behind the readout amplifier, the gain is now only a root of N, as shown in FIG. 2.

The macro cell integration described before Readout amplifier causes a reduction (zoom) and Reduction of the resolution of the image. In contrast to DE 39 42 615 A1 zoom is not wanted here and should be avoided  will. The following describes how both are preferably avoided or minimized.

Fig. 3 shows an arrangement preferably used. The CCD sensor 42 is controlled by a special unit 43 . The read image data either go directly to the video output unit 44 or into an image memory 46 . The data in the image memory can also be processed by a computer 45 . However, the computer is not necessary for normal motion picture operation.

For the case of two macro cells twice, FIG. 1 c describes how an image reduction and a resolution reduction are avoided without the computer 45 being required. For this purpose, the image intensity of the macro cell 3 ′ ′ + 6 ′ ′ + 12 ′ ′ + 15 ′ ′ at the image storage position b (0.0) and the neighboring macro cell 9 ′ ′ + 30 ′ ′ + 18 ′ ′ + 33 ′ ′ on the image memory position b (0.2) is written and the image memory positions b (0.1), b (1.0), b (1.1) and b (0.3), b (1.2), b ( 1,3) remain blank. In the image below, the macro cells are gradually shifted according to their horizontal and vertical dimensions and the intermediate image storage positions are described as follows. In the first picture below, the macro cell 6 ′ ′ + 9 ′ ′ + 15 ′ ′ + 18 ′ ′ is formed and written to the picture storage position b (0,1). In the second picture below the macro cell 12 ′ ′ + 15 ′ ′ + 21 ′ ′ + 24 ′ ′ is formed and written to the image storage position b (1.0) and in the third picture below the macro cell 15 ′ ′ + 18 ′ + 24 ′ + 27 ′ 'Formed and written to the storage position b (1,1). All other macro cells are treated accordingly. The complete image is then stored in the image memory without reduction. The image in the image memory is continuously read out sequentially, ie b (0.0), b (0.1), b (0.2), b (0.3), independently of the write processes by the video output unit 44 . . ., b (1.0), b (1.1). . .

Of course, other known interpolation methods, such as. B. the interpolation with a function of the form sin (x) / x or a linear interpolation can be performed by the computer 45 .

The disadvantage is that these calculations can only be carried out in real time using expensive special modules. The above-described interpolation with shifting of the macro cells has the additional advantage that a loss of resolution is avoided. A calculation of the theoretical resolution carried out using standard methods shows that the maximum possible spatial resolution of 1 / (2 _* scanning distance) is achieved for macro cells the size of two. Fig. 4 shows the results are the calculated transfer function for one-dimensionally extended macro cells and the interpolation according to the invention. The calculation assumed that the sensor cells directly adjoin one another. This assumption is correct, since there are normally lenticular grids over the sensor cells, which bring the light that would fall on the memory cells to the sensor cells. In the case of two macro cells, the transfer function (ÜTF) has no zeros below the theoretical resolution limit. The slight loss of contrast at high frequencies can be compensated for. The spatial frequencies up to the theoretical limit are also available for macro cells of four. The zero at the frequency 1 / (2 _* sampling distance) shows that a completely defined frequency component is lost.

The interpolation procedure described shows for still images interference from moving objects. There is a moving car when two macro cells are two times shifted four times in the picture. However, the shift can be calculated using known methods and be compensated /J.,R. Ohm: "Digital image coding," Chap. 16, Springer, Berlin, 1995) /. As long as the car doesn't stop only the spatial resolution behind the car reduced.

The task of forming color macro cells is as follows solved.

To describe color images, 3 Color components e.g. B. Red R, Green G and Blue B (RGB) required. In video technology often does not become RGB directly, but that  Luminance signal Y = R + G + B and the color difference signals Y-R and Y-B transmitted.

Modern color CCD sensors have color filters above the sensor cells, which have such mixed colors that when reading a sensor line as standard, alternate here with a, c, a, c ,. . . designated color mixing signals and on the next line the color mixing signals b, d, b, d,. . . arise. The relationship is shown in FIG. 5. It shows how the color mixing signals are stored in an image memory 46 after the sensor has been read out. Of course, as with the monochrome sensor, sensor and memory cells are again arranged in the sensor. To simplify matters, these details are not drawn.

The mixed colors a, b, c, d now have the property that from this very simply the luminance signal Y and the color difference signal R-Y and B-Y can be calculated. It applies approximately:

Y = a + c; R = ac; Eq. (1)
Y = b + d; B = bd;

If two macro cells are formed twice as described above, see above one receives signals with the mixed color (a + c + c + d) = 2Y. So that is the color information disappeared. The same applies to larger ones Macro cells.

To obtain the color information, one would have to skip one color cell in the horizontal and one vertical direction. So add the a to the a, the b to the b and so on. This is certainly possible outside the CCD sensor in the image memory 46 . The disadvantage is that outside the sensor the increase in light sensitivity is less, as explained above.

It would also be in the sense of a simple, uniform Further processing of the sensor signals important that despite Shot noise reduction through macrocell formation Color difference signals continue according to Eq. (1) from the Macrocell charges are formed.  

The object is achieved according to the invention by combining diagonally adjacent sensor cells, as shown in FIG. 5a. Two adjacent and three macro cells have the mixed colors a + d + c and c + b + a. The grouping of diagonally adjacent sensor cells takes place again many cycles after the field transfer in the horizontal shift register. According to the above explanation, the vertically mixed colors d + c and b + a should be in registers 39 and 40 and the colors a and c in registers 21 and 24, respectively. A horizontal transfer brings 39 → 38 and 40 → 39 . A subsequent vertical transfer vertical adds 21 + 38 = a + d + c or 24 + 39 = c + b + a as requested above. Summation of these mixed colors gives (a + d + c) + (c + b + a) = 3Y. Difference formation gives (c + b + a) - (a + d + c) = bd = BY.

The three macro cells underneath result in the Mixed colors b + c + d and d + a + b with the sum (b + c + d) + (d + a + b) = 3Y and the difference (d + a + b) - (b + c + d) = a-c = R-Y.

This is exactly the result you get with the standard Readout also gets d. H. Eq. (1) also applies to the Macro cells.

Fig. 5b shows how three times one macrocells are summarized. Again the sum and the difference of the macro cells give the desired color mixing signals.

Fig. 5c shows a possibility as five times one macrocell can be summarized. The sum of adjacent macro cells gives the luminance SY, the difference between which gives the color mixing signals 3 (RY) and 3 (BY). This means an improvement in light sensitivity by a factor of 5 in luminance and a factor of 3 in color.

Of course, color macro cells can also be used at the same time horizontal and vertical expansion can be realized and it is possible the macro cells in successive  Moving images to the inventive To get interpolation.

A reduction in shot noise is also possible through longer ones Integration times possible. However, this leads to a increased motion blur when looking at camera or a observed object relatively during the exposure time move towards each other. In case of movement in one The camera is now oriented in such a way that the Direction of movement in column direction.

In order to avoid the motion blur, the procedure according to claim 7 is as follows. After the field transfer from the sensor cells into the memory cells, there is a vertical transfer, that is from 3 → 12 , 12 →. . . 21 , 21 → 38 ; 6 → 15 , 15 . . . → 24 , 24 . . . → 39 ; . . . for reading out the image. Before the next vertical transfer, a new field transfer takes place after a defined waiting time, that is to say from the sensor cells into the memory cells. The waiting time is chosen so that the movement of the object image on the sensor takes place synchronously with the movement of the charge image in the memory cells. If the relative speed between the camera and the observed object is fixed, the waiting time between the field transfers can be fixed. If the relative speed changes, it can be readjusted. The rule criterion is that maximum image sharpness is created. A measure of the sharpness can be z. B. calculate by storing the recorded image in a memory 44 and calculating the difference between vertically adjacent image intensities. The mean of the squares of the 10 largest difference values turned out to be a good measure of sharpness.

The above principle can be applied directly to color sensors according to the Apply the strip principle. With stripe color sensors stripe-shaped filter masks in the vertical direction side by side over the sensor cells. For color sensors after So much mosaic principle between 2 field transfers Vertical transfers lie like an integer multiple corresponds to the mosaic period.  

The shot noise can also be prolonged as mentioned above Integration times can be reduced. Longer integration times of the sensor cells means that the reading of the Memory cells can take place more slowly. It turns out that too this reduces the so-called transfer noise. It is advantageous (according to claim 9) the readout of the To slow down memory cells to the extent that Exposure time of the sensor cells is increased.

In a further embodiment of the sensor, the corresponding Claim 10 moving the macro cells through a mechanical movement of the sensor generated relative to the lens. Either the sensor directly, or the lens, or parts of the lens can also be moved. The only thing that matters is that the image at the sensor location moves relative to the macro cells becomes. Such a shift is also possible, for example, in that a single lens of the lens is eccentric is rotated. The lens then acts like a centrally mounted one Lens with a downstream prism that is rotated.

Claims (7)

1.Sensor for a CCD camera with a plurality of vertical lines arranged parallel to one another and consisting of light-sensitive sensor cells, with memory cells associated therebetween, each with two vertically adjacent sensor cells, into which the camera transfers the two assigned sensor charges simultaneously or sequentially by means of a field transfer in the case of two successive field recordings, from which the charges are read out in a row arranged at one end of the vertical charge flow by horizontal transfers at the CCD output, whereby charges of individual cells can be adjusted by software to horizontally and vertically extended macro cells by means of vertical transfers directly connected in series without a horizontal reading process and direct transfers in series without reading process of the sensor output be summarized (additional application to Akz. P 44 36 426.1-31), characterized in that

• a) the charge of a macro cell after an analog / digital conversion is written in an image memory to the position which corresponds to the macro cell position on the sensor, that is to say the macro cell 3 ′ ′ + 6 ′ ′ + 12 ′ ′ + 15 ′ ′ to the Image storage position b (0.0) and the neighboring macro cell 9 ′ ′ + 30 ′ ′ + 18 ′ ′ + 33 ′ ′ to the image storage position b (0.2) and the image storage positions b (0.1) b (1.0) , b (1,1) as well as b (0,3), b (1,2), b (1,3) initially remain blank, but the macro cells in subsequent image recordings are gradually shifted according to their horizontal and vertical dimensions and the intermediate image storage positions are described, i.e. in the first picture the macro cell 6 ′ ′ + 9 ′ ′ + 15 ′ ′ + 18 ′ ′ is formed and at the image storage position b (0.1), in the second picture the macro cell 12 ′ ′ + 15 ′ ′ + 21 '' + 24 '' formed and at the image storage position b (1.0) and in the third image below the macro lle 15 ′ ′ + 18 ′ ′ + 24 ′ + 27 ′ ′ is formed and written to the storage position b (1,1) and the image memory is read out sequentially independently of the writing processes, i.e. b (0,0), b (0 , 1), b (0.2), b (0.3). . ., b (1.0), b (1.1). . .
• b) a color processing mode is provided in which, after a vertical transfer, at least one horizontal transfer with subsequent vertical transfer without a horizontal readout is provided, which results in diagonally extended macro cells 15 ′ ′ + 21 ′ ′ or 18 ′ ′ + 24 ′ ′.

2. Sensor according to claim 1, which is characterized through the mosaic principle. 3. Sensor according to claim 1 and 2, characterized records that the macro cells are chosen geometrically be that the sum of horizontally adjacent macro cells the Luminance signal Y and their difference the color difference signals R-Y or B-Y in the line below result in the case of macro cells of one by three dimension the macro cells a + d + c and c + b + a. 4. Sensor according to any one of claims 1-2, characterized ge  indicates that between each Vertical transfers with at least one additional field transfer previous defined exposure takes place. 5. Sensor according to claim 4, characterized in that the Quotient from the time and the number of vertical transfers between the field transfers take place gradually increased until the image sharpness is at its maximum. 6. Sensor according to any one of claims 1-5, characterized characterized in that the beat with which the Memory cells are read out proportional to Integration time of the sensor cells can be reduced. 7. Sensor according to any one of claims 1-6, characterized in that the macro cells shifted relative to the image at the sensor location by a mechanical displacement of the sensor relative to the lens or by shifting or rotating Lens parts.