JP2010226514A - Imaging device - Google Patents

Abstract

Generation of a false signal due to a decrease in the degree of color modulation, which is a problem when different color pixels are mixed in an image sensor in order to achieve both a moving image and a still image, is prevented.
A pixel mixing unit 701 for mixing adjacent pixels is provided in an image sensor 102, and a mixture combination changing unit 702 for changing a combination of pixel mixing for each frame is provided in an image sensor driving unit 103. In the signal output from the image sensor 102, the sign of the difference between signals adjacent in the oblique direction is inverted for each frame at the same position. A luminance signal processing unit 703 and a color signal processing unit 704 are provided in the digital signal processing unit 106 that receives the signal of the image sensor 102, and the color signal calculation unit 705, the color signal frame memory 706, and the frame interval are provided in the color signal processing unit 704. A color signal subtraction unit 707 and other processing unit 708 are provided. As a result, the color signal is amplified by a factor of 2 to compensate for the decrease in the color modulation degree, and the influence of the light / dark pattern of the subject having a high temporal correlation is eliminated.
[Selection] Figure 7

Description

  The present invention relates to an image pickup apparatus used in a digital still camera (DSC), a video movie, a mobile phone, and the like, and in particular, in a device that handles both still images and moving images, pixel mixing is performed on the image pickup device for moving image shooting. It is related with the imaging technique which performs.

  In recent years, imaging devices that handle both still images and moving images are increasing in DSCs, video movies, mobile phones, and the like. Since the number of pixels required for still images and moving images is different, these image capture devices reduce the number of pixels by processing such as thinning out or mixing pixels of the image sensor during moving images, and at the same time increase the high frame rate required for moving images. Realized.

  Patent Document 1 discloses a technique for mixing pixels of the same color and reducing the number of pixels to 1/9, for example. It is interlaced pixel mixing to mix pixels of the same color, but it is realized by devising the electrodes of the CCD image sensor, and at the same time, by making the horizontal and vertical odd pixel mixing, the uniformity of the center of gravity after mixing is achieved. Secured.

  The same color pixel mixing can obtain the same color as before mixing, but only the color filter pixels of the same color are mixed. Therefore, there is a problem that free mixing is difficult due to restrictions of the color filter pattern.

Patent Document 2 discloses a different color pixel mixing technique. This makes it possible to colorize even a mixture of different color pixels by devising a color filter in a CCD type image pickup device. In particular, since a conventional RGB Bayer color filter has a 2 × 2 pattern, only one color can be obtained by mixing four adjacent pixels, but a method of colorizing with a 3 × 1 or 3 × 2 pattern is disclosed. ing.
Japanese Patent Laid-Open No. 2004-312140 Japanese Patent Laid-Open No. 2003-116061

  However, different color pixel mixing has the disadvantage that the color becomes light because different colors are mixed, that is, the so-called color modulation degree is lowered, specifically, that each color signal obtained from the mixed signal becomes smaller. Present in principle. Therefore, it is necessary to amplify the color signal by image processing. Amplification amplifies not only the signal but also noise, resulting in a problem that the color S / N deteriorates. In order to improve the color S / N, a noise suppression technique for suppressing noise has been conventionally devised, and the problem of the color S / N is not considered here. However, a decrease in the color modulation degree also causes a problem of an increase in false colors. In other words, in the colorization of a single-panel camera, since signals of pixels provided with different color filters arranged at spatially different positions are used, the light / dark pattern of the subject is misjudged as a color and does not originally exist. The problem of generating colors will occur. The amplification of the color signal also has the disadvantage of further increasing this false signal.

  An object of the present invention is to prevent the generation of a false signal due to a decrease in the degree of color modulation, which is a serious problem when different-color pixel mixing is performed with an image pickup device in order to achieve compatibility between a moving image and a still image. .

  In order to achieve the above object, a solid-state imaging device according to the present invention has a plurality of photoelectric conversion units arranged as pixels in the horizontal direction and the vertical direction in order to photoelectrically convert a subject image, and a color A combination of an image sensor provided with a color filter that passes a specific color for each photoelectric conversion unit to obtain an image, a pixel mixing unit that mixes and outputs charges of the plurality of pixels, and a pixel mixture A mixing combination changing means for changing, and by changing the combination of the mixing, in the signal after mixing, the sign of the difference value of signals adjacent in the horizontal and vertical or diagonal directions is inverted at the same position. To do.

  Specifically, the mixed combination changing unit changes the combination for each frame. Furthermore, the color signal calculating means for obtaining the color signal by taking the difference between the signals adjacent in the horizontal and vertical or diagonal directions in the mixed signal, and the color for storing the output of the color signal calculating means for one frame The signal frame memory further includes an inter-frame color signal subtracting unit that performs subtraction between the stored previous frame color signal and the input current frame color signal.

  In the imaging apparatus according to the present invention, by changing the combination of pixel mixing for each frame, in the signal after mixing, the sign of the difference value of signals adjacent in the horizontal and vertical or diagonal directions is inverted at the same position. Color signal calculation means for generating a difference value as a color signal, a color signal frame memory for storing a color signal for one frame, and subtraction between the stored previous frame color signal and the input current frame color signal By providing the inter-frame color signal subtracting means to be executed, the color signals are in an inversion relationship at the same position, and the color signals are amplified twice by subtraction of the color signals, and at the same time, correlated. By canceling the influence of the light / dark pattern of the subject with high brightness, it is possible to effectively suppress the occurrence of false colors due to the light / dark pattern of the subject. In particular, the utility is very high in different color pixel mixture in which the degree of color modulation decreases.

  FIG. 1 shows the overall configuration of a digital still camera (DSC) that is an imaging apparatus according to an embodiment of the present invention. The DSC includes an optical lens 101, an image sensor 102 such as a CCD, an image sensor driver 103, an analog signal processor 104, an analog / digital converter 105, a digital signal processor 106, and an image compression unit. The decompression unit 107, the image recording unit 108, and the image display unit 109 are configured.

  According to the DSC in FIG. 1, the image of the subject that has passed through the lens 101 is formed on the image sensor 102. The imaging element 102 is driven by the imaging element driving unit 103 to perform photoelectric conversion and output an imaging signal. Next, the analog signal processing unit 104 performs processing such as noise removal and amplification, and the analog / digital conversion unit 105 converts the imaging signal into a digital signal. The digital signal processing unit 106 receives the digitized imaging signal and generates an image signal composed of a luminance signal (Y) and a color signal (C). Upon receiving this image signal, the image display unit 109 displays the image. In parallel with this image display, the image compression / decompression unit 107 compresses the image signal received from the digital signal processing unit 106, and the compressed image data is recorded in the image recording unit 108. The image data recorded in the image recording unit 108 can be expanded by the image compression / decompression unit 107, and the image can be reproduced on the image display unit 109 via the digital signal processing unit 106.

  Hereinafter, the operation in the moving image shooting mode in the DSC of FIG. 1 will be described. Since the present invention relates to pixel mixing, the description of the still image shooting mode in which all pixels are read independently will be omitted.

  The moving image shooting mode of the present embodiment is a mode in which adjacent four pixels are mixed and read. A specific method for performing the adjacent four-pixel mixing in the image sensor 102 and the image sensor drive unit 103 is known, and a detailed method for the CCD type image sensor is described in Patent Document 2 and will not be described.

  The adjacent four-pixel mixing has a feature that the structure of the image sensor 102 can be simplified because interlaced mixing is unnecessary. Moreover, since the low-pass filter effect by mixing is weak, it has the feature that there is little degradation in resolution. However, when the color filter array is an RGB Bayer array that is generally used in the past, since it is a 2 × 2 repetitive pattern, only one type of signal can be obtained after pixel mixing, and a color image cannot be generated. . Patent Document 2 discloses a method for realizing colorization when four adjacent pixels are mixed by making a 3 × 1 or 3 × 2 repetitive pattern. In the present invention, the color filter array shown in FIG. 2 is adopted. To do. In this pattern, the transmittance is modulated with a predetermined pattern in the RGB Bayer arrangement so that the RGB balance after mixing varies. As a result, in the still image mode in which all the pixels are read independently, the image processing corresponding to the conventional RGB Bayer arrangement can be applied only by adding a gain correction process for canceling the variation in transmittance. Can do.

  FIG. 3 schematically shows a color signal generation method when adjacent four pixels are mixed in the present invention from the viewpoint of how color signals are generated on a color filter. There are four kinds of combinations of filters A to D in the adjacent four-pixel mixture, and four kinds of signals having different RGB balances are obtained. As shown in FIG. 3, there are lines that are mixed by pattern A (301) and pattern B (302), and lines that are mixed by pattern C (303) and pattern D (304) shifted by one pixel in the horizontal direction. Is switched line by line. The signal arrangement after pixel mixing becomes a so-called offset sampling pattern, and the horizontal resolution can be improved.

Color signal generation is performed by taking the difference between the upper and lower lines, and RG and BG signals are obtained. These can be said to be signals corresponding to so-called color difference signals. As shown in FIG. 3, when taking the difference in the diagonal direction,
(1) Pattern A-Pattern C = (2G + 0.5R + 0.5B)-(1.5G + 0.5R + B) = -0.5 (BG)
(2) Pattern B-Pattern C = (G + R + B)-(1.5G + 0.5R + B) = 0.5 (RG)
(3) Pattern B-Pattern D = (G + R + B)-(1.5G + R + 0.5B) = 0.5 (BG)
(4) Pattern A-Pattern D = (2G + 0.5R + 0.5B)-(1.5G + R + 0.5B) = -0.5 (RG)
In this manner, four types of color difference signals are obtained in order. Here, R and B color difference signals are obtained for each pixel, and they are inverted every two pixels. As a result, colorization is possible even during moving image shooting in which adjacent four pixels are mixed.

  In FIG. 3, generation of color difference signals is schematically shown using arrows. The mixed pixels are connected by arrows, and a color difference signal is generated by calculation between the pixels. In addition, the four pixels on the side indicated by the arrow are positive elements of the obtained signal.

In the next frame, the combination of pixel mixture is changed as shown in FIG. That is, the patterns A and B are shifted by one pixel in the horizontal direction so as to become patterns C and D. Similarly, the lines of patterns C and D are shifted by one pixel so as to become patterns A and B. As a result, the pattern A and B lines and the pattern C and D lines are interchanged up and down. In this state, as in the case of FIG.
(1) 'Pattern C-Pattern A = (1.5G + 0.5R + B)-(2G + 0.5R + 0.5B) = 0.5 (BG)
(2) 'Pattern C-Pattern B = (1.5G + 0.5R + B)-(G + R + B) = -0.5 (RG)
(3) 'Pattern D-Pattern B = (1.5G + R + 0.5B)-(G + R + B) = -0.5 (BG)
(4) 'Pattern D-Pattern A = (1.5G + R + 0.5B)-(2G + 0.5R + 0.5B) = 0.5 (RG)
In this manner, color difference signals are obtained in order. That is, it can be seen that the color signal at the same position is inverted with respect to the previous frame.

When the color signal shown in FIG. 4 is subtracted from the color signal of FIG. 3 which is the previous frame,
(1)-(1) '= BG
(2)-(2) '=-(RG)
(3)-(3) '=-(BG)
(4)-(4) '= RG
In this way, color signals are obtained in order. That is, the color signal is amplified twice. In addition, the influence of the contrast difference of the subject in the vertical direction is canceled out. FIG. 5 and FIG. 6 show this situation as a specific example.

5 and 6 schematically illustrate a case where there is a difference in brightness in the vertical direction, assuming a red subject from which only an R signal can be obtained. That is, the upper line of the subject pattern is R, and the lower line is 0.5R. In this case, when the same combination as in FIG. 3 is performed, as shown in FIG.
(1) Pattern A-Pattern C = 0.25 (= -0.5 (BG))
(2) Pattern B-Pattern C = 0.75 (= 0.5 (RG))
(3) Pattern B-Pattern D = 0.5 (= 0.5 (BG))
(4) Pattern A-Pattern D = 0 (= -0.5 (RG))
In this manner, color difference signals are obtained in order. Originally, there is no B component and no G component, so (1) and (3) should be 0, but there are values. This is a false color.

Next, when the same combination as in FIG. 4 is performed, the result shown in FIG. 6 is obtained.
(1) 'Pattern C-Pattern A = 0.25 (= 0.5 (BG))
(2) 'Pattern C-Pattern B = 0 (= -0.5 (RG))
(3) 'Pattern D-Pattern B = 0.5 (= -0.5 (BG))
(4) 'Pattern D-Pattern A = 0.75 (= 0.5 (RG))
In this manner, color difference signals are obtained in order.

Taking the difference between the results of FIG. 5 and FIG.
(1)-(1) '= 0 (= BG)
(2)-(2) '= -0.75 (=-(RG))
(3)-(3) '= 0 (=-(BG))
(4)-(4) '= 0.75 (= RG)
As a result. That is, the false BG component is 0, and only the correct RG component is obtained.

  In this embodiment, since the color signal is obtained by the difference in the vertical direction, the color signal is inverted by changing the combination combination in the vertical direction, and the influence of the contrast difference of the subject in the vertical direction is negated. However, strictly speaking, since the color signal is obtained by the difference in the diagonal direction shifted by one pixel in the horizontal direction, the color signal is also affected by the contrast of one pixel in the horizontal direction. As shown by the directions of the arrows in FIGS. 3 and 4, the pixel components that work positively and negatively are reversed in the vertical direction, but not reversed in the horizontal direction. That is, it is affected by the difference in brightness of one horizontal pixel.

  However, since the difference in brightness for one horizontal pixel also affects the still image mode in which all pixels are read independently, it is usually suppressed by an optical low-pass filter. In other words, when mixing pixels, the distance between pixels after mixing is an important cause of false color generation. Here, the calculation object is two pixels away in the vertical direction, and the false color that is generated due to the effect of the optical low-pass filter is reduced effectively in this embodiment. On the other hand, since only one pixel is separated in the horizontal direction and the false color is originally small, this embodiment does not cause a big problem even if the measure in the horizontal direction is not taken.

  FIG. 7 is a diagram illustrating a configuration of an imaging apparatus that performs the above processing. The same parts as those in the overall configuration example of the DSC shown in FIG. A pixel mixing unit 701 for mixing adjacent pixels is provided in the image sensor 102. In the image sensor driving unit 103, a mixed combination changing unit 702 for changing a combination of pixel mixing is provided. The mixing combination changing means 702 changes the mixing combination for each frame as shown in FIGS. Actual pixel mixing is performed by the pixel mixing means 701 in the image sensor 102. As shown in FIGS. 3 and 4, the signal output from the image sensor 102 is inverted for each frame at the same position in the difference between the signals adjacent in the oblique direction. The signal of the image sensor 102 is digitized through an analog signal processing unit 104 and an analog / digital conversion unit 105 and input to the digital signal processing unit 106. The digital signal processing unit 106 is provided with a luminance signal processing unit 703 and a color signal processing unit 704, and in the color signal processing unit 704, a color signal calculation unit 705, a color signal frame memory 706, and an inter-frame color signal subtraction unit 707. And other processing units 708 are provided.

  The color signal frame memory 706 may be configured with one semiconductor chip together with other processing units, or may be configured as another semiconductor chip.

  A luminance signal (Y signal) is generated by the luminance signal processing unit 703 and a color signal is generated by the color signal processing unit 704 from the signal of the image sensor 102 input to the digital signal processing unit 106. As the color signal, a color difference signal corresponding to the difference from the luminance signal is usually used. There are two types of color difference signals, RY and BY. In the color signal processing unit 704, first, as described above, the difference in the diagonal direction of the mixed signal is obtained. This is performed by the color signal calculation means 705, and RG and BG are obtained as described above. Since the G signal is spectrally close to the luminance signal, it may be considered that these correspond to color difference signals. The generated color difference signal is stored in the color signal frame memory 706. Next, the inter-frame color signal subtracting unit 707 performs subtraction between the current color difference signal and the color difference signal of the previous frame, and obtains a color difference signal that eliminates the influence of the light / dark pattern of the subject as described above. Thereafter, the color difference signal is processed in the other processing unit 708 as in the conventional case. This includes γ processing, matrix processing for bringing the spectrum closer to RY and BY, and the like. The generated luminance signal and color difference signal are sent to the image compression / decompression unit 107 and the image display unit 109 as described in the overall configuration described above.

  The combination of the color filter pattern and the mixture is not limited to the present embodiment, and various embodiments can be taken without departing from the gist of the present invention.

  As described above, the imaging apparatus according to the present invention generates false signals due to a decrease in the degree of color modulation, which is a serious problem when different color pixel mixing is performed with an imaging device in order to achieve compatibility between moving images and still images. Can be prevented very effectively. According to the present invention, a major problem in different color pixel mixing is solved, and an unprecedented free pixel mixing pattern can be selected. As a result, various combinations of still images and moving images can be realized, and the utility is extremely high.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to an embodiment of the present invention. FIG. 2 is a color filter array diagram of the image sensor in FIG. 1. It is explanatory drawing of the color signal generation method at the time of the video recording in the imaging device of FIG. FIG. 3 is a diagram in which the combination of pixel mixture is changed. It is a figure which shows the example of a color signal when there exists a light / dark pattern of a to-be-photographed object. FIG. 5 is a diagram in which the combination of pixel mixture is changed. It is a detailed block diagram of the principal part in FIG.

DESCRIPTION OF SYMBOLS 101 Lens 102 Image pick-up element 103 Image pick-up element drive part 104 Analog signal processing part 105 Analog / digital conversion part 106 Digital signal processing part 107 Image compression / expansion part 108 Image recording part 109 Image display part 301 Pixel mixing combination pattern A
302 Pixel combination pattern B
303 Mixed pixel combination pattern C
304 Pixel combination pattern D
701 Pixel mixing unit 702 Mixing combination changing unit 703 Luminance signal processing unit 704 Color signal processing unit 705 Color signal calculation unit 706 Color signal frame memory 707 Inter-frame color signal subtraction unit 708 (in the color signal processing unit) Other processing units

Claims (6)

Each photoelectric conversion unit has a plurality of photoelectric conversion units arranged as pixels in the horizontal direction and the vertical direction in order to photoelectrically convert an image of a subject, and a specific color filter that passes a specific color to obtain a color image. An image sensor provided in
Pixel mixing means for mixing and outputting the charges of the plurality of pixels;
A mixing combination changing means for changing a combination of pixel mixing,
An image pickup apparatus characterized in that, by changing a combination of mixing, in a signal after mixing, the positive / negative of a difference value of signals adjacent in the horizontal and vertical or diagonal directions is inverted at the same position. The imaging device according to claim 1,
The image pickup apparatus, wherein the mixed combination changing unit changes the combination for each frame. The imaging device according to claim 1,
Color signal calculation means for obtaining a color signal by taking a difference between signals adjacent in the horizontal and vertical or diagonal directions in the mixed signal;
A color signal frame memory for storing the output of the color signal calculation means for one frame;
An imaging apparatus, further comprising: an inter-frame color signal subtracting unit that performs subtraction between the stored previous frame color signal and the input current frame color signal. The imaging device according to claim 1,
The combination of a plurality of mixed pixels includes a pixel having two or more kinds of color filters. The imaging device according to claim 1,
The image pickup apparatus, wherein the color filter is an RGB primary color filter. The imaging device according to claim 1,
2. The image pickup apparatus according to claim 1, wherein the pixel mixture is performed by four pixels adjacent in the horizontal and vertical directions.

Images