JPH09327027A - Image processing unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To process signals from a full picture element imaging device in response to a color filter arrangement and to allow the processing unit to cope with picture element deviation. SOLUTION: Signals by 6 lines obtained by extracting signals from a full picture element read CCD 2 by line memories 11-14 are given to a color separate circuit 16 consisting of each color separate section of a primary color system and a complementary color system corresponding to the color filter arrangement of the CCD 2, and broad band signals of G and Y signals and color signals R, G, B are separated by the color separate section selected depending on the color filter arrangement. In the case that no picture element deviation is conducted, the signals are processed through LPFs 25-28 and in the case that picture element deviation is conducted, the signals are processed with high resolution while including high frequency components not through the LPFS 25-28.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus suitable for use in semiconductor integrated circuits for digital cameras and the like.

[0002]

2. Description of the Related Art In recent years, with the progress of semiconductor technology, a solid-state image sensor (all-pixel readout solid-state image sensor, hereinafter referred to as all-pixel image sensor) capable of reading out signals by progressive scanning has been developed. Such an all-pixel image pickup device is more advantageous than a conventional interlaced scan (interlaced scan) type image pickup device.
An image with high resolution can be captured with less blur even for a moving subject. That is, when the image is picked up by the interlaced image sensor, the image of one frame is 1/6.
Since it is composed of two fields with a time difference of 0 seconds, when a moving subject is imaged, jaggedness occurs due to the time difference between the fields. Further, when the image of one field is taken out, the jaggedness is eliminated, but the vertical resolution is halved. On the other hand, in the all-pixel image pickup device, the state at the same time can be picked up for one frame, so the above-mentioned problem does not occur. Taking advantage of such features, the all-pixel image sensor is applied to still image capturing cameras, computer input devices, and the like.

Generally, in a solid-state image pickup device, a mosaic-like color filter array is arranged on the image pickup surface in front of the device,
Color imaging with a single plate is possible, and a low-cost and compact color camera can be realized. Color imaging with a single plate is possible in the same way with an all-pixel image sensor, but since the two vertically adjacent pixels are not added when signals are read out, unlike the interlace scanning image sensor, the color filter array is interlaced. It is different from the color filter array of the image sensor.
FIG. 8 shows an example of a color filter applied to the all-pixel image sensor.

On the other hand, in order to meet the demands for miniaturization and high performance of video cameras, an integrated circuit for signal processing is disclosed in, for example, 12 of the 1994 Annual Conference of the Television Society.
It is disclosed on pages 5-126. Although this example is a signal processing integrated circuit of an interlaced image pickup device, it can be easily applied to an all-pixel image pickup device.

As a signal processing integrated circuit of a single-plate all-pixel image pickup device, for example, a configuration as shown in FIG. 9 can be considered. Hereinafter, FIG. 9 will be described as a conventional example. In the figure, incident light from a subject is imaged by the imaging optical system 101 on the CCD 102, which is an all-pixel image sensor, and is converted into an electrical signal. A color filter array for color imaging is attached on the CCD 102. Here, the color filter array is a primary color filter array as shown in FIG. The electric signal converted by the CCD 102 is C
Signals for two scanning lines are output in parallel from the two output terminals of the CD 102. That is, the signal of the odd-numbered scanning line is output from one, and the signal of the even-numbered scanning line is output from the other. By adopting such an output method, it is possible to output the signals of all pixels at the same reading speed as the CCD for interlaced scanning.

Each of the signals is a noise reduction circuit 103,
After being processed by 104 and the AGC amplifier circuits 105 and 106, they are converted into digital signals by the AD conversion circuits 107 and 108, and are input from the input terminals 136 and 137 to the integrated circuit 10.
Input to 0. These signals are stored in the 1H memory 109, 1
By 10, the time-axis conversion is performed into a line-sequential signal. The timing in this operation is shown in FIG. After that, the signals of three consecutive lines are further synchronized by the 1H memories 111 and 112, and the signals are input to the color separation circuit 113 and the contour emphasis circuit 117, respectively.

The RGB primary colors demodulated by the color separation circuit 113 are limited to a required band by low-pass filters 114, 115 and 116 to remove interference components such as moire. The band-limited RGB signals are adjusted in level by the white balance circuit 118, and then the contour signal extracted by the contour emphasizing circuit 117 and the adder 119.
Is added by Further, the high-pass component of the G signal whose contour has been corrected by the high-pass filter 120 is extracted and added to the RB signal by the adders 121 and 122. After that, γ correction is performed by the correction circuits 123, 124, and 125, and then the luminance signal Y and the color difference signals RY and BY are converted by the matrix circuit 126. Converted RY, B
The -Y signal is band-limited by the low-pass filters 127 and 128, and then decimated to 1/2 by the dot-sequencing circuit 129 to be a dot-sequential signal. That is, R-Y, B-Y, R-Y, and B have the same frequency as the luminance sampling frequency.
-Y, ... are signals in a row.

After that, in the scan conversion circuits 130 and 131, the signals are converted into two scanning line parallel signals similar to the CCD output. Further, the multiplexing circuits 132 and 133 multiplex the luminance signal and the color difference signal. The arrangement of signals at this time is, for example, Y, RY, Y, BY, .... The multiplexed signal is output from the output terminals 134 and 135.

[0009]

However, the above-mentioned conventional example has the following drawbacks. Unlike the interlaced scanning CCD, the CCD for all-pixel reading does not add charges between pixels at the time of reading, and therefore has essentially poor sensitivity as compared with the interlaced CCD. In addition, when a primary color filter as shown in FIG. 8A is used, the light transmittance of the primary color filter is as shown in FIG.
Since the transmittance is lower than that of the complementary color filter of (b), the sensitivity is further deteriorated. Considering the sensitivity, it is preferable to use the complementary color filter. On the other hand, from the viewpoint of color reproducibility and color S / N, the primary color filter is superior to the complementary color filter. That is, it is necessary to properly use the primary color filter and the complementary color filter according to the system specifications. Therefore, as a signal processing integrated circuit,
It is desirable that both primary colors and complementary colors can be supported, but the conventional example described above has a drawback that only primary color filters can be supported.

Further, the pixel shift imaging method has been well known as one of the methods for imaging a high-definition image using a CCD having a small number of pixels. An example of this pixel shift imaging method will be described below with reference to FIG. 11 by taking the case where the color filter of the CCD is as shown in FIG. In the case of the color filter array as shown in FIG. 8A, the RB sampling frequency is 1/2 in the horizontal and vertical directions with respect to the G sampling frequency, so that RB moire occurs. The band of the low-pass filter applied to the RB signal in order to suppress this is set to about 1/2 of that of the low-pass filter of the G signal. For this reason, the resolution is deteriorated when a highly saturated red or blue subject is imaged.

Therefore, in order to obtain the signals of the three primary colors in each pixel in order to equalize the band of each color, an optical system such as a parallel plate that shifts the optical path is used, as shown in FIGS. (A) Reference image forming position, (b) Image forming position is shifted by one pixel in the horizontal direction with respect to the reference position, (c) Image forming position is shifted by one pixel in the vertical direction with respect to the reference position, (d) Reference position On the other hand, the image formation position is divided into four pixels, that is, the horizontal position is shifted by one pixel and the vertical position is shifted by one pixel.
It is possible to capture a single high-definition image in which the bands of the three primary colors are equal.

That is, in the pixel shift imaging, the band is widened by performing band limitation according to the synthesizing method upon synthesizing a plurality of images in which high frequency components are stored without band limiting in the signal processing at the time of image capturing. Images can be obtained. However, in the signal processing circuit of the conventional example, the band of RB is limited to about 1/2 of G at the time of image pickup as described above, and therefore the effect of pixel shifting cannot be obtained in the image processed by this circuit. There was a flaw.

A first object of the present invention is to obtain an image processing apparatus capable of processing the signals from the all-pixel image pickup device according to the color filter array. Further, a second object is to obtain an image processing device that can cope with a case where pixel shift imaging is performed using an all-pixel imaging element.

[0014]

According to the present invention, a color filter is arranged corresponding to each pixel on the image pickup surface, and signals of all pixels are sequentially read out in a predetermined period. A plurality of color separation means for respectively separating a wideband signal and a color signal from a signal obtained from the image pickup means according to the arrangement of the color filters, a wideband signal processing means for processing the wideband signal, and a processing for the color signal. A color signal processing means and a selecting means for selecting one of the plurality of color separating means are provided.

According to a second aspect of the present invention, the filter means for band-limiting the image signal obtained from the image pickup means, which is arranged to sequentially read out the signals of all pixels on the image pickup surface in a predetermined period, and the output of the filter means. And a selecting means for selecting one of the image signals.

[0016]

According to the invention of claim 1, one of the plurality of color separation means is selected by the selection means according to the color filter arrangement,
By sending the wideband signal and the color signal obtained from the selected color separation means to the wideband signal processing means and the color signal processing means, respectively, one semiconductor integrated image processing device can handle a plurality of color filter patterns. can do.

According to the second aspect of the present invention, the high-frequency component is stored without band limitation in the signal processing when the image is picked up by the all-pixel image pickup element, and a plurality of images picked up by pixel shifting are combined in the post-processing. As a result, a high resolution image with a wide band can be obtained.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a video camera according to the present invention will be described below with reference to FIG. In the figure, 1 is an imaging optical system for forming a subject image on an image pickup device, 2 is a CCD as a solid-state image pickup device for reading out signals by sequential scanning, 3 and 4 are noise reductions for reducing noise of an output signal of the CCD 2. Circuits 5, 6 are amplification circuits for amplifying CCD signals to proper levels, 7, 8 are AD conversion circuits for converting CCD signals into digital signals, 9 and 10 are input terminals of the semiconductor integrated circuit 47, 11, 12, 13, 14 are CCDs
A line memory for delaying the signal by one scanning line, 15 is a vertical contour extraction circuit for extracting a vertical contour signal from the signals of 6 scanning lines synchronized by the line memories 11 to 14, 16
Is the luminance signal and GB from the signals synchronized in the line memory.
It is a color separation circuit that generates an R primary color signal.

Reference numeral 17 is a base clip circuit for slicing a small amplitude portion of a vertical contour signal, 18 is a low-pass filter for limiting the band of the vertical contour signal, 19 is a horizontal contour extraction circuit for extracting a horizontal contour signal from a wide band signal, 20, 21,
22 and 23 are low-pass filters that limit the band of the signal,
Reference numeral 24 is a base clip circuit for slicing a small amplitude portion of the horizontal contour signal, 25, 26, 27 and 28 are selectors for selecting the presence or absence of low-pass filter processing, 29 is a gain adjusting circuit for adjusting the gain of the horizontal contour signal, 30 Is an adder for adding the horizontal contour signal and the vertical contour signal, 31 is a gain adjusting circuit for adjusting the gain of the contour signal, 32 is a white balance circuit for adjusting the white balance, 33 is an adder for adding the contour signal to the wideband signal Is.

Reference numeral 34 is a high-pass filter for extracting the high frequency band of the wide band signal, and 35 and 36 are adders for adding the high frequency band of the wide band signal extracted by the high pass filter 34 to the R and B signals, 37, 38, 39, 40 is a γ correction circuit, 41
Is a broadband signal and a primary color signal, and a luminance signal Y and a color difference signal R-
Y and B-Y matrix circuits, 42 and 43 low pass filters that limit the band of color difference signals, 44 scan conversion circuits, 45 and 46 output terminals of semiconductor integrated circuit 47, and 47 signal processing of CCD signals Integrated semiconductor circuit.
A selector 48 selects a signal to be added to 35 and 36.

Next, the operation of the above configuration will be described. The incident light from the subject is imaged by the imaging optical system 1 on the CCD 2 which is an all-pixel image pickup element and converted into an electric signal. A color filter array for color imaging is attached on the CCD 2. The color filter array has, for example, an array as shown in FIGS. The electric signal converted by the CCD 2 is output from two output terminals of the CCD 2 in parallel for two scanning lines. That is, the signal of the odd-numbered scanning line is output from one, and the signal of the even-numbered scanning line is output from the other. Output signals from the CCD 2 are noise reduction circuits 3 and 4 and amplification circuits 5 and 6.
After being processed by, the signals are converted into digital signals by the AD conversion circuits 7 and 8. The signals of the two scanning lines converted into digital signals are input from the input terminals 9 and 10 to the semiconductor integrated circuit 4
The signals of 6 scanning lines are simultaneously input to the line memories 11, 12, 13 and 14 by being input to the line 7.

The signals for the six scanning lines are input to the vertical contour extraction circuit 15 and the vertical contour signals are extracted. The vertical contour extraction circuit 15 is composed of a vertical high-pass filter, and has a transfer characteristic as shown in FIG. 2, for example, when the number of pixels of the CCD 2 is 640 × 480. Further, the vertical contour signals for two scanning lines extracted by the vertical contour extraction circuit 15 are supplied to the CCD 2 in order to reduce the circuit scale of the subsequent stage.
Is converted into a dot-sequential signal having a frequency twice as high as that of the signal read from. That is, as shown in FIG. 3, the signals of the odd-numbered scanning lines and the even-numbered scanning lines are converted into time-sequentially multiplexed dot-sequential signals. The vertical contour signal extracted by the vertical contour extraction circuit 15 is sliced into a small amplitude portion by a pace clip circuit 17 for noise reduction, and then an unnecessary band in the horizontal direction is removed by a low pass filter 18.

On the other hand, the signals synchronized by the line memory are separated in the color separation circuit 16 into a wide band signal and a GBR primary color signal according to a timing signal from a timing generator (not shown). These signals also have a form in which signals for two scanning lines are time-division multiplexed. In addition, the wideband signal referred to here corresponds to a G signal in the case of the color filter array shown in FIG. 8A, and corresponds to a luminance signal in the case of the color filter array shown in FIG. 8B. Each signal is limited to an appropriate band by the low pass filters 20, 21, 22, 23. For example, in the case of the color filter array as shown in FIG. 8A, since the G sampling points are present in each column, the horizontal band is ½ of the horizontal sampling frequency of the CCD 2. Therefore, the low pass filter 2
The pass band of 0 is about 1 / s of the sampling frequency of CCD2.
Set to 2. On the other hand, since there is only one R and B sampling point in two columns, the band is 1/2 of the G signal,
The pass bands of the low pass filters 22 and 23 are set to about 1/4 of the sampling frequency of the CCD 2. The band of the low-pass filter 21 is set to the same pass band as the low-pass filters 22 and 23 for the purpose of aligning the band with RB.

The outputs of these low pass filters 20 to 23 are input to one of the selectors 25, 26, 27 and 28. The other inputs of the selectors 25 to 28 are connected to the input terminals of the low pass filters 20 to 23, so that the low pass filter processing can be bypassed. With such a configuration, it is possible to bypass the low-pass filter and preserve the high-frequency component when a plurality of images captured by shifting the phases are combined to obtain a high-definition image. .

At the same time, the separated luminance signal is input to the horizontal contour extraction circuit 19 and the horizontal contour signal is extracted.
The extracted horizontal contour signal is supplied to the base clip circuit 24,
After being processed by the gain adjusting circuit 29, it is added to the above-mentioned vertical contour signal by the adder 30. After the added contour signal is adjusted to an appropriate level by the gain adjustment circuit 31,
The luminance signal is added by the adder 33.

For each band-limited signal, the white balance circuit 32 changes the gain of the RB signal to match the level of the white portion. Next, the high-pass filter 34 extracts the high frequency band of the wideband signal, and this is added to the adders 35 and 36.
To add to the R and B signals. This high pass filter 3
The characteristic No. 4 is a characteristic complementary to the low-pass filters 22 and 23, and is intended to align the bands of the wideband signal and the primary color signal. When this processing is unnecessary, such as when pixel-shifted imaging is performed, the addition can be disabled by the selector 48.

Then, after being .gamma.-corrected by .gamma.-correction circuits 31, 32 and 33, it is converted into a luminance signal Y and color difference signals RY and BY by a matrix circuit 41, and RY and B.
-Y is a low-pass filter 42, 43 which is approximately 1 / Y of the Y band.
Band-limited to 2. Finally, the Y, RY, and BY signals are input to the scan conversion circuit 44. In the scan conversion circuit 44, the input Y, RY, BY is switched by the switching pulse generated by the timing generator,
Two systems of video signals for each scanning line are generated. For example, this signal is in a state in which RY and BY are selected for every two pixels of Y and time-divided in the order of Y, RY, Y, and BY. At this time, Y, RY, and BY are all signals on the same scanning line. The above timing explanatory diagram is shown in FIG. At this time, one clock of the output signal is Y
It is twice the sampling rate of the signal. If you take out one of the two systems, it is the interlaced scanning signal,
If both are used, it becomes a signal for progressive scanning. 2 generated
The system video signal is output from the output terminals 45 and 46.

As described above, by performing the dot-sequential processing of two scanning lines inside the signal processing, the number of storage elements used in the entire signal processing is smaller than that of the conventional example, the circuit scale is small, and the consumption is small. It is possible to realize a configuration with low power consumption. In addition, there is an advantage that the circuit scale related to scan conversion is also reduced.

FIG. 5 shows the structure of the color separation circuit 16 in detail. The color separation circuit 16 includes a primary color separation unit 1
6A and a complementary color system color separation unit 16B. First, the operation and configuration of the primary color system color separation unit 16A will be described. The primary color system color separation unit 16A operates when the color filter array of the CCD 2 is as shown in FIG. 8A, and color-separates signals for one scanning line based on signals for three scanning lines. Since the G signal is obtained with the sampling phase deviated by π for each scanning line,
By alternately sampling the signal of the central scanning line and the signal obtained by adding the upper and lower scanning lines, a G signal having a cycle equal to the sampling frequency of the CCD signal can be obtained. FIG.
(A) shows a signal of the central scanning line, and (b) shows a signal obtained by adding the upper and lower scanning lines. By selecting these signals with the switch 63 at the timing shown in FIG.
The G signal shown in (d) is obtained. Although the 3H signal in FIG. 5 has been described above, the G signal is separated by the same processing for the 4H signal.

Since the RB signal is obtained in the form of being sampled every other scanning line at a frequency half the sampling frequency of the CCD signal, the signal of the central scanning line and the signal of the upper and lower scanning lines are added. It can be obtained by sampling and holding each of them at a cycle of twice the sampling of the CCD signal and alternately selecting at a cycle of twice the horizontal scanning frequency.
That is, the signals shown in FIGS. 6E and 6F are obtained by sample-holding the signals of FIGS. 6A and 6B by the D flip-flops 62 and 64 and the switches 65 and 66. Since RB appears in this signal by being switched every horizontal scanning period, RB is selected by the switches 67 and 68 with a switching pulse having a cycle twice the horizontal frequency.
The signal is obtained. When the center scanning line is a signal represented by 4H, the RB signal is similarly separated. After that, the signals for two scanning lines obtained as described above are time-division multiplexed at a frequency twice as high as the sampling frequency by the CCD signal switches 69, 70, and 71 to be combined into one system of signals.

On the other hand, the complementary color separation unit 16B operates when the color filter array of the CCD 2 is as shown in FIG. 8 (b).
First, the separation of the broadband luminance signal will be described. When signals of two vertically adjacent lines are added by the adder 80, a dot-sequential signal of Wr and Gb is obtained. Here, Wr = Ye + Mg = 2R + G + B Gb = Cy + G = 2G + B. Nu this signal to 1/2 the sampling frequency
A luminance signal is obtained by smoothing with a low-pass filter having 11 points. That is, the average of Wr and Gb is (2R + G + B + 2G + B) /2=R+1.5G+B, which is a signal almost equal to the composition ratio of the luminance signal.

Next, separation of RGB signals will be described. Ye at each pixel position for the signals of two lines
D flip-flop 8 so that Cy and MgG exist
4, 85, 88, 89 and adders 86, 90 interpolate linearly from the preceding and following pixels. The CCD 2 is shown in FIGS.
A signal obtained by delaying the signal read from the D flip-flop is shown, and (f) and (h) show signals interpolated from the preceding and succeeding pixels. Further, (e) + (h) is added by the adders 87 and 91.
And (f) + (g) are added, (i)
A dot-sequential signal of Wb and Gr as shown in (j) is obtained. Here, Wb = Cy + Mg = R + G + 2B Gr = Ye + G = R + 2G. Wb and Gr are respectively obtained by rearranging these two signals by the switches 94 and 95 for each clock. Further, the previously obtained point sequential signal of WrGb is similarly linearly interpolated from the preceding and succeeding pixels by the D flip-flops 81 and 82 and the adder 83 so that Wr and Gb are synchronized, and the switches 92 and 93 are used. Wr and Gb are obtained by rearranging for each clock.

For the 4H signal, the signals are separated by the same process, and then the switches 109, 110 and 11 are used.
The signals for two scanning lines are transferred to CCD by 1, 112, 123
The dot-sequentialization is performed at a frequency twice the read frequency of The color separation circuit 114 separates primary color signals from the four signals Wr, Gb, Wb, and Gr thus obtained. As a method of separating the primary color signals, a method similar to the conventional color difference line sequential method may be used, or a method of generating from WrGbWbGr by matrix calculation may be used. Finally selector 1
15, 116, 117 and 118 select and output one of the outputs from the primary color system color separation unit and the complementary color system color separation unit.

As described above, the color separation unit 16 is provided with the primary color system color separation unit 16B, and the signal processing is configured by the wide band signal processing unit and the primary color signal processing unit. Instead, the same signal processing semiconductor can be used.

[0035]

As described above, according to the first aspect of the present invention, a plurality of color separating means such as a primary color separating section and a complementary color separating section are provided and any one of the outputs is selected. hand,
By inputting signals to the wideband signal processing means and the color signal processing means, the same image processing semiconductor integrated circuit can be used regardless of the type of color filter of the all-pixel image pickup device.

According to the second aspect of the present invention, by providing means for bypassing the filter for band limiting the video signal, a high-definition original image can be obtained when the pixel shift imaging is performed using the all-pixel imaging device. There is an effect that can be.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a video camera according to the present invention.

FIG. 2 is a characteristic diagram showing a transfer characteristic of a vertical contour emphasis circuit.

FIG. 3 is a timing chart showing an operation of time division multiplexing.

FIG. 4 is a timing chart showing an output signal format from the semiconductor integrated circuit in FIG.

FIG. 5 is a configuration diagram showing a configuration of a color separation circuit.

FIG. 6 is a timing chart showing the operation of the primary color system color separation unit.

FIG. 7 is a timing chart showing an operation of a complementary color system color separation unit.

FIG. 8 is a configuration diagram of a color filter array.

FIG. 9 is a block diagram showing a conventional image processing apparatus.

FIG. 10 is a timing chart showing the operation of the line memory in the conventional example.

FIG. 11 is a configuration diagram illustrating pixel shift imaging.

[Explanation of symbols]

 2 CCD 16 color separation circuit 16A primary color system color separation unit 16B complementary color system color separation unit 19 horizontal contour extraction circuit 20-23 low pass filter 25-28 switch 32 white balance circuit 37-40 gamma correction circuit 41 matrix circuit

Claims (3)

[Claims] 1. A color filter is arranged corresponding to each pixel on an image pickup surface, and the color filter is arranged from a signal obtained from an image pickup means configured to sequentially read out signals of all pixels in a predetermined period. A plurality of color separation means for respectively separating a wideband signal and a color signal, a wideband signal processing means for processing the wideband signal, a color signal processing means for processing the color signal, and a plurality of color separation means. An image processing apparatus comprising: a selection unit that selects one of the above. 2. A filter means for band-limiting an image signal obtained from an image pickup means, which is configured to sequentially read out signals of all pixels on an image pickup surface in a predetermined period, and one of an output of the filter means and the image signal. An image processing apparatus comprising: a selecting unit for selecting. 3. The image processing apparatus according to claim 2, wherein the image signal comprises a wide band signal and a color signal separated from the signal output from the image pickup means.