JP2009105925A - Image signal processor and method therefor - Google Patents

Abstract

An object of the present invention is to enable an auto white balance process that is not affected by the spectral characteristics of a color filter and to obtain good color reproducibility.
An extraction means for extracting each color component from an output of an image sensor provided with a color filter, and spectral characteristics of the color filter by a first linear process for each color component extracted by the extraction means. And a white balance processing means 11 for adjusting white balance by a second linear process for each color component corrected by the color adjustment means 10.
[Selection] Figure 1

Description

The present invention relates to an image signal processing apparatus suitable for white balance adjustment of an imaging apparatus using an image sensor and a method thereof.

In recent years, an electronic camera that obtains an image signal by converting a subject image into an electric signal by a photoelectric conversion element such as an image sensor has become widespread. In such an electronic camera, an optical image of a subject that has passed through an optical system is converted into an electrical signal by an image sensor that is an image sensor. The electrical signal from the image sensor is subjected to predetermined image signal processing as an image signal. The image signal processing includes processing such as gamma correction processing, color separation processing, and white balance processing for the image signal, and an image signal in a predetermined format is obtained through these processing.

The gamma correction process is a process for correcting non-linearity in the relationship between the image signal level and the transmittance (luminance) in the display system. The white balance process is a process for correctly reproducing white.

In an electronic camera, a color image is obtained by arranging a color filter in an image sensor. For example, R (red), G (green), and B (blue) light is incident on the image sensor by the color filter, thereby obtaining R, G, and B image signals.
However, even when the same subject (same object color) is imaged, the levels of the R, G, and B signals obtained from the image sensor vary individually according to the color temperature of the light source (environment color). Therefore, white balance processing is performed in order to correctly reproduce the white color and appropriately reproduce the color balance.

White balance processing is performed by multiplying R, G, and B image signals by predetermined coefficients (color coefficients). As a result, the levels of the R, G, and B signals are individually adjusted to achieve an appropriate color balance.

By the way, in the display system, each color is reproduced by mixing R, G, B signals. For example, in a color filter of a liquid crystal panel, the spectral characteristics of the R, G, B filter are made relatively steep. R, G, B light is reliably obtained from the G, B filters. However, in the imaging system, it is necessary to allow transmission of colors other than R, G, and B of the optical image of the subject at the time of imaging, and the spectral characteristics of R, G, and B of the color filter are as follows. The band characteristics are set to be relatively wide and have overlapping portions. Therefore, each R, G,
The R, G, and B signals based on the light transmitted through the B filter also include signal components based on other color lights.

White balance processing, which is linear processing, is performed on such R, G, and B signals.
Even in this case, white reproducibility can be improved by appropriately setting the R, G, and B signals. However, other color components included in the R, G, and B signals are changed at the same time by adjusting the levels of the R, G, and B signals, and the color reproducibility of the other color components is deteriorated. there were.

The present invention has been made in view of such problems, and by performing auto white balance processing after matrix processing for color adjustment, not only white reproducibility but also reproducibility of each color balance is improved. An object of the present invention is to provide an image signal processing apparatus and method that can be used.

An image signal processing apparatus according to the present invention includes an extracting unit that extracts each color component from an output of an image sensor provided with a color filter, and the color by a first linear process on each color component extracted by the extracting unit. A color adjustment unit that corrects spectral characteristics of the filter, and a white balance processing unit that adjusts a white balance by a second linear process for each color component corrected by the color adjustment unit are provided.

According to such a configuration, the extraction unit extracts each color component from the output of the image sensor. Each color component is a color based on the object color of the subject, the environmental color that illuminates the subject, and the spectral characteristics of the color filter. The extracted color component is subjected to a first linear process by the color adjusting means. As a result, the spectral characteristics of the color filter are corrected, and the color component is based on the object color and the environmental color of the subject. Further, each color component is subjected to a second balance by white balance processing means.
The linear processing is applied. As a result, the influence of the environmental color is eliminated, and each color component is based on the physical color of the subject, and good color reproducibility can be obtained.

The color adjusting unit and the white balance processing unit may perform the first and second linear processes by matrix calculation, respectively.

According to such a configuration, it is possible to easily correct the spectral characteristics of the color filter and adjust the white balance by appropriately setting the matrix coefficients used for the matrix calculation.

The color component extracted by the extraction unit is a primary color signal, and the white balance processing unit converts the primary color signal extracted by the extraction unit into a luminance-color difference color space, and converts the converted luminance signal and A pixel used for a color coefficient for white balance processing is extracted from a color difference signal, and the color coefficient for white balance processing is calculated from each color component after color adjustment by the color adjustment unit corresponding to the extracted pixel. And

According to such a configuration, the primary color signal extracted by the extraction unit is converted into a luminance-color difference color space. Thereby, extraction of pixels in a predetermined color range using color difference signals and extraction of pixels in a predetermined luminance range using luminance signals can be performed. As pixels used for calculation of color coefficients, for example, achromatic pixels Thus, it is possible to improve the accuracy of the color coefficient and enable a good white balance adjustment.

  The luminance-color difference color space is a YUV color space.

According to such a configuration, conversion from the primary color signal to the YUV color space can be easily performed, and a simple device can be configured.

Further, the color component extracted by the extraction unit is a primary color signal, the primary color signal extracted by the extraction unit is converted into a luminance-color difference color space, and color suppression is performed on the color difference signal after conversion. The image processing apparatus is further characterized by further including a color suppression unit provided to the color adjustment unit.

According to such a configuration, the color difference signal is color-suppressed by the color suppression means. For example, by performing color suppression for dark pixels, it is possible to prevent the dark pixel portion from being conspicuous and improve the overall image quality.

Further, the color adjustment unit performs color adjustment after converting a luminance signal in a luminance-color difference color space and a color difference signal from the color suppression unit into a primary color signal.

According to such a configuration, it is possible to perform reliable color adjustment by performing adjustment in a color space according to the spectral characteristics of the color filter.

The image processing apparatus may further include a gamma correction unit that performs gamma correction on each of the color components subjected to white balance processing by the white balance processing unit.

  According to such a configuration, the image quality can be improved by gamma correction.

The image signal processing apparatus according to the present invention includes an extraction unit that extracts R, G, and B signals from an output of an image sensor provided with a color filter, and R, G, and B extracted by the extraction unit.
A first matrix means for converting a signal into a signal in a YUV color space; and an extracted pixel determination means for determining a pixel to be used for a color coefficient for white balance processing from the luminance signal Y and the color difference signal UV from the first matrix means. An image quality correction unit for band limiting the luminance signal Y and the color difference signal UV from the first matrix unit, a color suppression unit for color suppressing the color difference signal UV from the image quality correction unit, and the image quality correction unit. The luminance signal Y and the color difference signal UV from the color suppression means are converted into R, G, B signals, and the first matrix for the R, G, B signals from the second matrix means. The third matrix means for correcting the spectral characteristics of the color filter by the linear processing of R, G, from the third matrix means
A color coefficient calculating means for calculating a color coefficient for white balance processing using a signal of a pixel selected by the extracted pixel determining means from among the B signals, and R,
Fourth matrix means for adjusting white balance by second linear processing using the color coefficients for the G and B signals, and gamma for performing gamma correction processing for the R, G and B signals from the fourth matrix means. And a correcting means.

According to such a configuration, the R, G, B signals extracted by the extracting means are converted into signals in the YUV color space by the first matrix means, and then band-limited by the image quality correcting means to correct the image quality. Is done. The extracted pixel determining means determines a pixel to be used for the color coefficient for white balance processing from the luminance signal Y and the color difference signal UV from the first matrix means.
Thereby, only pixels in a predetermined color range and luminance range can be used for calculating the color coefficient,
The accuracy of the color coefficient can be improved. The color suppression means suppresses the color difference signal UV and prevents, for example, dark pixel defects from being noticeable. The second matrix means converts the luminance-color-difference color space signal into R, G, B signals, and the third matrix means corrects the spectral characteristics of the color filter by the first linear processing. As a result, the R, G, and B signals are based on the environmental color and the object color of the subject. The color coefficient calculation means calculates a color coefficient for white balance processing using a pixel signal selected by the extraction pixel determination means from among the R, G, and B signals from the third matrix means. Then, the fourth matrix means adjusts the white balance by the second linear processing using the color coefficients for the R, G, B signals from the third matrix means. Thereby, the influence of the environmental color is eliminated, and the object color of the subject is reproduced. The R, G, B signals from the fourth matrix means are gamma corrected by the gamma correction means. Thus, an image signal having excellent color reproducibility can be obtained.

The image signal processing method according to the present invention includes an extraction procedure for extracting each color component from the output of the image sensor provided with the color filter, and a first linear process for the extracted color component. A color adjustment procedure for correcting spectral characteristics; and a white balance processing procedure for adjusting white balance by a second linear process for each color component corrected by the color adjustment procedure.

According to such a configuration, each color component is extracted from the output of the image sensor. Each extracted color component is a color based on the object color of the subject, the environmental color that illuminates the subject, and the spectral characteristics of the color filter. In the color adjustment procedure, a first linear process is performed on the extracted color component. As a result, the spectral characteristics of the color filter are corrected, and the color component is based on the object color and the environmental color of the subject. Furthermore, the second linear processing is performed on each color component in the white balance processing procedure. As a result, the influence of the environmental color is eliminated, and each color component is based on the physical color of the subject, and good color reproducibility can be obtained.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing an image signal processing apparatus according to the first embodiment of the present invention.

The optical image of the subject captured by the image sensor is colored light based on the object color of the subject, the illumination light (environment color) of the subject, and the spectral characteristics of the filter. In this embodiment, after correcting the spectral characteristics of the filter by performing matrix processing for color adjustment in the previous stage of auto white balance processing, the environmental color is removed by auto white balance processing by linear processing, and the subject The correct object color is reproduced.

An optical image from a subject is incident on an image sensor (not shown) through an optical system (not shown) including a color filter. The image sensor photoelectrically converts incident light and generates an image signal based on the optical image of the subject. This image signal is supplied to an AFE (analog front end) circuit 1 in FIG. The AFE circuit 1 includes an amplifier and an analog / digital converter (not shown), converts the input image signal into a digital signal, and outputs the digital signal to the OB clamp circuit 2.

The OB clamp circuit 2 is a circuit for adjusting the input image signal to an appropriate black reference level. In the image sensor, several predetermined pixels are shielded by a light shielding plate or the like and set in an OB (optical black) region. The OB clamp circuit 2 is the O
The black level of the image signal is adjusted based on the pixel signal level of the pixels in the B area. The output of the OB clamp circuit 2 is given to the data range correction circuit 3. The data range correction circuit 3 corrects the range of the input digital image data and corrects the CFA color interpolation circuit 4.
Output to.

The color filter provided in the image sensor is, for example, one in which R, G, B filters are cyclically arranged according to an appropriate arrangement rule. Thereby, each pixel of the image sensor individually constitutes an R pixel, a G pixel, or a B pixel. The CFA color interpolation circuit 4 has a line buffer (not shown) that holds image signals for a plurality of lines (for example, 3 lines).
Interpolation processing for obtaining R, G, and B signals at each pixel position is performed by arithmetic processing using R, G, and B pixels at adjacent positions. The R, G, B signals from the CFA color interpolation circuit 4 are supplied to the matrix calculator 5.

The matrix calculator 5 is for converting the inputted R, G, B signals into YUV ′ signals. The matrix calculator 5 converts the R, G, B signals into YUV ′ signals by, for example, 3 × 3 matrix calculation, gives the luminance signal Y to the high-pass filter (HPF) 6, and passes the color difference signals U, V ′ to the low-pass filter. (LPF) 7 is provided. The HPF 6 passes the high frequency component of the luminance signal Y, performs image quality correction such as contour compensation by high frequency emphasis, and outputs it to the matrix calculator 9. In addition, the LPF 7 limits the input color difference signals U and V ′ to only a low frequency and outputs them to the color suppression circuit 8.

In this embodiment, the color suppression circuit 8 suppresses the color component of the color difference signal UV ′ and outputs it to the matrix calculator 9. When the gain for the pixel signal is relatively large, the defect of the pixel signal becomes conspicuous due to the influence of quantization noise or the like. Therefore, by using the fact that the sensitivity of the human eye to the color is relatively low for dark portions, the color components of pixels with relatively low luminance (dark portions) are suppressed, or the monochrome mode is set. The defect is made inconspicuous.

In the present embodiment, the image signal converted into the YUV ′ signal is returned to the R, G, B signal again. The matrix calculator 9 performs matrix processing on the input YUV ′ signal and converts it into R ₁ , G ₁ , B ₁ signals, and then outputs them to the matrix calculator 10. The matrix calculator 10 performs matrix processing on the input R ₁ , G ₁ , and B ₁ signals to perform color adjustment, and outputs the R ₂ , G ₂ , and B ₂ signals after color adjustment to the auto white balance circuit 11. It is supposed to be. In the present embodiment, the spectral characteristics of the color filter are corrected by this color adjustment.

In the present embodiment, the color coefficient used in the auto white balance circuit 11 is generated using only a specific pixel signal in the input image signal. The AWB (auto white balance) extraction pixel determination circuit 12 determines a pixel signal for obtaining a color coefficient for auto white balance processing from the output YUV ′ signal of the matrix calculator 5. For example, the AWB extraction pixel determination circuit 12 is a pixel signal for obtaining a color coefficient for pixels in a predetermined color range among the outputs of the matrix calculator 5 and having a luminance level within the predetermined range. The position information indicating the pixel position is output to the AWB pixel data storage circuit 13.

The AWB pixel data storage circuit 13 is supplied with the outputs R ₂ , G ₂ , and B ₂ signals of the matrix calculator 10, and stores and holds the R ₂ , G ₂ , and B ₂ signals corresponding to the pixel positions given by the position information. At the same time, the color coefficient is output to the AWB coefficient calculation circuit 14 for calculation. The AWB coefficient calculation circuit 14 calculates a color coefficient for auto white balance adjustment based on the input R ₂ , G ₂ , and B ₂ signals and outputs the color coefficient to the auto white balance circuit 11.

The auto white balance circuit 11 adjusts each color level by a matrix process that multiplies the input R ₂ , G ₂ , and B ₂ signals by a color coefficient to make the white balance appropriate. Yes. The R ₃ , G ₃ , and B ₃ signals that have been subjected to auto white balance adjustment by the auto white balance circuit 11 are output to the user color adjustment circuit 15.

The user color adjustment circuit 15 receives an external signal based on a user operation, and adjusts the level of the R ₃ , G ₃ , and B ₃ signals input based on the external signal, thereby adjusting the color balance to the user's preference. It comes to adjust. R ₄ , G ₄ , B after color adjustment from the user color adjustment circuit 15
_{The four} signals are supplied to the user digital gain setting circuit 16. The user digital gain setting circuit 16 receives an external signal based on a user operation, and adjusts the gain for the R ₄ , G ₄ , and B ₄ signals input based on the external signal, so that the luminance is set to the user's preference. It comes to adjust. R ₅ , G ₅ , B ₅ after brightness adjustment from the user digital gain setting circuit 16
The signal is supplied to a gamma (γ) correction circuit 17.

The gamma correction circuit 17 applies predetermined nonlinear processing to the input R ₅ , G ₅ , and B ₅ signals to perform gamma correction, and then outputs the gamma correction circuit 17 to the matrix calculator 18. The matrix calculator 18 is for converting the input R, G, B signals into YUV signals.
The matrix calculator 18 converts the R, G, B signals to Y by, for example, 3 × 3 matrix calculation.
It is converted into a UV signal and output.

  Next, the operation of the embodiment configured as described above will be described.

Assume that a subject of a predetermined object color is illuminated by illumination light of a predetermined environmental color. An optical image of a subject is incident on an image sensor via an optical system including a color filter and is photoelectrically converted to obtain R, G, and B signals. The R, G, and B signals input to the AFE circuit 1 correspond to the color light based on the object color of the subject, the illumination light (environment color) of the subject, and the spectral characteristics of the filter. The AFE circuit 1 converts the input R, G, B signals into digital signals and converts them to OB.
Output to the clamp circuit 1. The OB clamp circuit 1 sets the black level of the input pixel signal to an appropriate black level and outputs it to the data range correction circuit 3. The data range correction circuit 3 corrects the R, G, and B signals to an appropriate range and adjusts the brightness of the image. Further, the R, G, B signals are subjected to color interpolation in the CFA color interpolation circuit 4 to obtain R, G, B image signals at the respective pixel positions.

In the present embodiment, in order to calculate the color coefficient for auto white balance in the YUV ′ color space and perform image quality correction, first, the matrix interpolator 5 outputs the R, G, B signals that have undergone color interpolation. 'Converted to signal. The luminance signal Y is high-frequency emphasized by the HPF 6 and supplied to the matrix calculator 9, and the color difference signal UV ′ is band-limited to the low frequency by the LPF 7 and supplied to the color suppression circuit 8. The color suppression circuit 8 suppresses the color of the dark part of the image and outputs it to the matrix calculator 9 in order to reduce the influence of noise such as quantization noise.

In the present embodiment, in order to adjust the color balance, the YUV ′ signal is once converted into R, G, B.
Return to signal. That is, the matrix calculator 9 obtains R ₁ , G ₁ , B ₁ signals by matrix processing on the input YUV ′ signal. The R ₁ , G ₁ , and B ₁ signals are given to the matrix calculator 10 for color adjustment. The matrix calculator 10 corrects the spectral characteristics of the color filter by matrix processing on the input R ₁ , G ₁ , B ₁ signals, and has the same characteristics as when the characteristics of the R, G, B filters are steep. R ₂ , G ₂ and B ₂ signals are obtained. The R ₂ , G ₂ , and B ₂ signals are given to the auto white balance circuit 11.

On the other hand, the extraction pixel determination circuit 12 for AWB is supplied with the YUV ′ signal from the matrix calculator 5 and determines the pixel for calculating the color coefficient for the auto white balance process. For example, AWB
The extracted pixel determination circuit 12 extracts pixels in a predetermined color range and a predetermined luminance range as pixels for color coefficient calculation, and outputs position information indicating the pixel position to the AWB pixel data storage circuit 13.

The AWB pixel data storage circuit 13 takes in and stores the R ₂ , G ₂ , and B ₂ signals from the matrix calculator 10 at a timing corresponding to the given position information, and outputs it to the AWB coefficient calculation circuit 4. To do. The AWB coefficient calculation circuit 4 calculates color coefficients for auto white balance adjustment from the input R ₂ , G ₂ , and B ₂ signals. The calculated color coefficient is given to the auto white balance circuit 11.

The auto white balance circuit 11 adjusts the white balance using, for example, 3 × 3 linear matrix processing using the input color coefficient. As a result, the inputs R, G, B
The influence of the environmental color included in the signal is removed. The R ₃ , G ₃ , and B ₃ signals that have been subjected to auto white balance adjustment by the auto white balance circuit 11 are output to the user color adjustment circuit 15.

The user color adjustment circuit 15 adjusts the color balance to the user's preference based on the user operation. Further, the user digital gain setting circuit 16 adjusts the luminance to the user's preference based on the user operation. Based on the user operation, the image signal subjected to the color adjustment and the brightness adjustment is supplied to the gamma correction circuit 17 and subjected to gamma correction.
The gamma-corrected image signal is given to the matrix calculator 18 and inputted R, G, B
The signal is converted into a YUV signal.

As described above, in the present embodiment, the spectral processing of the filter is corrected by performing matrix processing that performs color adjustment in the previous stage of auto white balance processing, and further, environmental colors are removed by auto white balance processing, The correct object color of the subject is reproduced.

1 is a block diagram showing an image signal processing apparatus according to a first embodiment of the present invention.

Explanation of symbols

5, 9, 10 ... Matrix calculator, 11 ... Auto white balance circuit, 12 ... AWB
Extraction pixel determination circuit, 13... AWB pixel data storage circuit, 14... AWB coefficient calculation circuit, 17.

Claims (8)

Primary color signal acquisition means for acquiring a primary color signal according to the output of the image sensor;
Color adjusting means for correcting spectral characteristics of the image sensor with respect to the primary color signal;
Pixel determining means for determining a pixel for white balance processing using a signal that has not been corrected by the color adjusting means;
White balance processing means for adjusting the white balance of each color component corrected by the color adjustment means based on the pixels for white balance processing;
An image signal processing apparatus comprising: Primary color signal acquisition means for acquiring a primary color signal according to the output of the image sensor;
Color adjusting means for correcting spectral characteristics of the image sensor with respect to the primary color signal;
The primary color signal is converted into a luminance-color difference color space, pixels for white balance processing are extracted based on the converted luminance signal and color difference signal, and corrected by the color adjustment unit based on the extracted pixels. An image signal processing apparatus comprising: white balance processing means for adjusting white balance of each color component. The pixel determination unit converts the primary color signal not corrected by the color adjustment procedure into a luminance-color difference color space, and determines a pixel for white balance processing based on the converted luminance signal and color difference signal. The image signal processing apparatus according to claim 1. The image signal processing apparatus according to claim 2, wherein the luminance-color difference color space is a YUV color space. 5. The white balance processing unit performs white balance adjustment based on the value of each color component corrected by the color adjustment unit in a pixel for white balance processing. 6. Image signal processing apparatus. 6. The image signal processing apparatus according to claim 1, wherein the primary color signal and the signal of each color component corrected by the color adjusting means are signals for R, G, and B. 7. The apparatus according to claim 1, further comprising color suppression means for converting the primary color signal into a luminance-color difference color space and performing color suppression on the converted color difference signal before adjusting the white balance. An image signal processing apparatus according to any one of the above. A primary color signal acquisition procedure for acquiring a primary color signal according to the output of the image sensor;
A color adjustment procedure for correcting spectral characteristics of the image sensor with respect to the primary color signal;
A pixel determination procedure for determining a pixel for white balance processing using a signal that has not been corrected by the color adjustment procedure;
A white balance processing procedure for adjusting the white balance of each color component corrected by the color adjustment procedure based on the pixels for white balance processing;
An image signal processing method characterized by comprising:

Images