JP5261922B2 - Image processing method, image processing program, and digital camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method with which the effect of illumination due to an illumination means can be appropriately corrected by using one image of a subject illuminated by the illumination means. <P>SOLUTION: An image taken by illuminating a subject with a flash unit 107 and surrounding illuminations is captured, the captured image is divided into at least two or more regions of a main subject region and a background subject region, information about illuminations of the divided regions is determined and based on the determined information about the illuminations of the main subject region and the background subject region, the color of the main subject region is corrected. <P>COPYRIGHT: (C)2008,JPO&amp;INPIT

Description

The present invention relates to an image processing method, an image processing program, and a digital camera that appropriately correct an image of a subject illuminated by illumination means.

  When shooting with flash using artificial lighting such as a strobe, an image illuminated with a light source that is different from the illumination that is actually hitting the subject that the photographer is looking at is taken. Reproduced in different colors. For example, when a flash light is photographed in a dark room, the amount of light by the flash light is stronger than the background, so that the color dominant in the flash light is reproduced.

  For example, when the indoor lighting is an incandescent lamp, the color temperature is about 2800K, while the color temperature of the flash light is as high as 5500K. Therefore, the light source visible to the photographer is a subject illuminated with a slightly reddish incandescent bulb color, but in the actually photographed image, the main subject (person, etc.) that is exposed to the flash light is the photographer. Is reproduced in a paler image than the image actually viewed.

  On the other hand, when shooting without using flash light, the color temperature of the incandescent light bulb is reflected, resulting in a slightly reddish color. The latter warm photo is generally preferred, and it has the disadvantage of being dark and prone to blur, so you can shoot without using flash light, or use flash light at the expense of noise Some users shoot with increased ISO sensitivity. Therefore, even if flash light is used, there is a strong demand for color reproduction that is close to the case where flash light is not used.

  In this way, as a method of reproducing an image under illumination different from the illumination used for shooting, two images, an image using flash light and an image not using it, are taken in succession, and two images with and without flash emission are taken. There has also been proposed a correction method using this method (Patent Document 1). According to this Patent Document 1, a difference between images with and without flash emission is created as an illumination image using only flash light, color conversion correction is performed on the difference image, and then combined with an image without flash to obtain desired illumination light. The image illuminated with is generated.

JP 2003-296720 A

  However, when two consecutive images were taken, no matter how short the continuous shooting was, the image was shaken due to the camera shake of the photographer himself or the movement of the subject. . Also, if there is a flashing light source in the background light, or if the background light flickers at a certain frequency, such as a fluorescent lamp, the difference between the continuous shot images is not necessarily the component due to the flash emission. However, there was a possibility that the correction was wrong. Furthermore, since two images with and without flash emission are necessary, there is a problem that correction cannot be performed for an image that has not been shot.

The invention of claim 1 is an image processing method in a digital camera, wherein an image captured by illuminating a subject with flash illumination means for flashing the subject and ambient illumination means for illuminating the subject from the periphery is acquired, and the acquired image is acquired. Based on the image, at least two of the first image that is the main subject that is mainly illuminated by the flash illumination means and the second image that is not the main subject that is mainly illuminated by the ambient illumination means. Dividing the image into two or more images, obtaining at least a color component by the flash illumination means of the first image and at least a color component by the ambient illumination means of the second image ; by the ratio of white color component of the first image and the color component, child converts the color components of each pixel of the first image to the color component that has been illuminated by the ambient lighting means The features.
The invention of claim 2 is an image processing method in a digital camera, wherein an image captured by flash illumination means for flashing a subject and ambient illumination means for illuminating the subject from the periphery is acquired and the acquired Based on the image, at least two of the first image that is the main subject that is mainly illuminated by the flash illumination means and the second image that is not the main subject that is mainly illuminated by the ambient illumination means. Dividing the image into two or more images, obtaining at least a color component by the flash illumination means of the first image and at least a color component by the ambient illumination means of the second image ; by the ratio of white color component of the color components first image, the color components of each pixel of the first image, the ambient lighting hand and illumination by the flash illumination means And converting the color component which is illuminated with a mixed illumination with illumination by.
According to a third aspect of the present invention, there is provided an image processing method in a digital camera, wherein an image captured by illuminating a subject with flash illumination means for illuminating the subject and ambient illumination means for illuminating the subject from the periphery is acquired, and the acquired image is acquired. Based on the image, at least two of the first image that is the main subject that is mainly illuminated by the flash illumination means and the second image that is not the main subject that is mainly illuminated by the ambient illumination means. An image pickup device that divides the image into two or more images, obtains at least a color component by the flash illumination unit of the first image and a color component by at least the ambient illumination unit of the second image, and captures the image A color space corresponding to the spectral characteristic is converted into a predetermined color space, and a white color component under illumination of the second image and a white color component under illumination of the first image By, and converting the color component of each pixel of the first image to the color component that has been illuminated by the ambient lighting means.
According to a fourth aspect of the present invention, there is provided an image processing method for a digital camera, wherein an image captured by illuminating a subject with flash illumination means for flashing the subject and ambient illumination means for illuminating the subject from the periphery is acquired, and the acquired image is acquired. Based on the image, at least two of the first image that is the main subject that is mainly illuminated by the flash illumination means and the second image that is not the main subject that is mainly illuminated by the ambient illumination means. An image pickup device that divides the image into two or more images, obtains at least a color component by the flash illumination unit of the first image and a color component by at least the ambient illumination unit of the second image, and captures the image A color space corresponding to the spectral characteristic is converted into a predetermined color space, and a white color component under illumination of the second image and a white color component under illumination of the first image Accordingly, the color components of each pixel of the first image, and converting the color component which is illuminated with a mixed illumination with illumination by the ambient lighting means and illumination by the flash illumination means.

  Since the present invention is configured as described above, it is possible to appropriately correct the influence of illumination of the illumination means using one image of the subject illuminated by the illumination means.

-First embodiment-
FIG. 1 is a diagram showing a configuration of a digital camera 100 according to an embodiment of the present invention. The digital camera 100 includes a photographing lens 102, an image pickup device 103 including a CCD, a control device 104 including a CPU and peripheral circuits, a memory 105, a lens driving unit 106, a flash unit (strobe) 107, a photometry unit 108, and the like. .

  The image sensor 103 captures (captures) the subject 101 via the photographing lens 102 and outputs the captured image data to the control device 104. The control device 104 performs image processing described below on the image data captured by the image sensor 103 and stores the image data after the image processing in the memory 105 as appropriate. Further, the control device 104 drives the photographing lens 102 via the lens driving unit 106 and acquires distance measurement data up to the subject. Further, the control device 104 acquires photometric data of subject brightness via the photometric unit 108. Furthermore, the control device 104 drives the flash unit 107 to perform flash (flash illumination, strobe) photography. Note that the image processing program executed by the control device 104 is stored in a ROM (not shown).

  In the digital camera 100 of the present embodiment, even when flash photography is performed on an image (specifically, image data) photographed (captured) by the image sensor 103, the flash is not used. The image is appropriately corrected in a certain state.

  Specifically, the image is divided into a main subject area illuminated by a flash and a background area where the flash is difficult to reach, and information regarding illumination, that is, an illumination light source is estimated in each area. Based on the illumination light source of each area estimated in this way, the color conversion processing is performed so that the main subject area where the flash illumination is dominant is illuminated by the illumination light source of the background area.

  FIG. 2 is a diagram illustrating a flowchart of an image processing program executed by the control device 104 of the digital camera 100 according to the present embodiment.

  In step S1, an image photographed using the flash unit 107 is read as a correction target image, and the process proceeds to step S2.

  In step S2, the captured image is divided into a plurality of regions. In step S3, the distance to the representative point (for example, the center) of each divided area is calculated using the distance measuring function of the camera. The area division may be simply divided into blocks as shown in FIG. 3, but may be divided into arbitrary shapes using the result of edge extraction. For example, in the composition as shown in FIG. 4, it is desirable to divide each subject, such as subject A, subject B and other backgrounds.

  Next, in step S4, the main subject is recognized. As the main subject, the region is recognized as the main subject region based on conditions such as the focused region, the distance calculated in the previous step S3 is short, or the average luminance in each region is high. In the image example of FIG. 4, the person A closest to the distance (the areas 13, 18, and 23 in the example of FIG. 3) is in focus, and the person A is determined as the main subject.

Next, in step S5, the proper exposure conditions computed by the camera when not using the flash unit 107 (ISO sensitivity S _1, F-number F _1, the shutter speed T ₁₎ and the camera when using the flash unit 107 From the appropriate exposure conditions (ISO sensitivity S ₂ , F number F ₂ , shutter speed T ₂ ) calculated in, the brightness L ₂ of the main subject (under mixed lighting of strobe light and ambient lighting) when using the flash unit 107 And the brightness L ₁ of the main subject (under ambient lighting) when the strobe is not used is calculated from the following equation. K is a constant. The appropriate exposure condition is an appropriate exposure condition for photographing the main subject. The appropriate exposure condition is calculated based on the photometric data measured by the photometric unit 108.

  FIG. 5 is a table showing the relationship between the exposure conditions when the flash unit 107 is not used and when it is used, and the brightness of each subject.

In step S6, the brightness of the subject A using only flash light is calculated from the following equation.

  In step S7, a region (subject B) that is estimated to be the least affected by the flash light is selected. When the area is divided into the main subject and the other two parts, the area other than the object A is selected. However, when the area is divided into three or more areas, the distance to each area is long, and the image area is long. In this case, the distance from the optical axis to the subject (image height) is long, the average luminance is low, etc., and the process proceeds to step S8.

Next, in step S8, the brightness L _b (per unit area) of the subject B at the distance b on the optical axis from the flash unit 107 to the subject B at the distance b on the optical axis from the flash unit 107 is measured. the light beam) incident on, is calculated by the following procedure. FIG. 6 is a diagram for explaining this calculation. When the digital camera 100 and the flash unit 107 are photographed toward the main subject A, the angle formed by the flash unit 107 and the main subject is considered to be almost vertical, but the subject B in the background is at an angle with respect to the flash unit 107. Consider the case of θ. When the photographing lens is focused on a subject at a distance a and the focal length is f, an angle θ ′ formed by the subject at the image height y on the imaging surface and the optical axis can be obtained by the following equation.

  To be precise, the optical axis and the position of the flash unit 107 are misaligned, so the distance may be taken into account. However, if the distance to the subject is longer than the misalignment between the optical axis and the flash unit 107, it is considered that θ≈θ ′. It is done. When the influence of distortion is large, the calculation is performed in consideration of distortion.

で与えられる。一方、立体角Ωが距離aの位置に作る面積は、入射方向に対して垂直の場合をSとすると、角度θの方向に作る面積S'は、

となる。また立体角Ωの作る面積は、距離の２乗に比例する為、距離bの位置に作る面積S"は、

と表される。式(4)〜(6)より、角度θの方向に、光軸上の距離b（θの方向の距離はb/cosθ）の位置にある被写体Bに照射される単位面積辺りの光束は、

と表すことができる。一方、フラッシュユニット１０７と垂直方向に距離aの位置にある被写体Aに照射される単位面積辺りの光束は次式で表される。

式(7),(8)より、被写体Bに於けるフラッシュ光による明るさは、

と算出できる。 Next, referring to FIG. 6, the brightness of the subject B by the flash light is calculated. A luminous flux dφ ′ radiated at a solid angle Ω from the unit area dA of the flash unit 107 in the direction of the angle θ is expressed as luminance B.

Given in. On the other hand, the area S ′ created in the direction of the angle θ, where S is a case where the solid angle Ω is created at the position of the distance a and S is perpendicular to the incident direction,

It becomes. Also, since the area created by the solid angle Ω is proportional to the square of the distance, the area S "created at the position of the distance b is

It is expressed. From the equations (4) to (6), the luminous flux per unit area irradiated to the subject B at the position of the distance b on the optical axis in the direction of the angle θ (the distance in the direction of θ is b / cos θ) is

It can be expressed as. On the other hand, the luminous flux per unit area irradiated to the subject A at a distance a in the direction perpendicular to the flash unit 107 is expressed by the following equation.

From Equations (7) and (8), the brightness of subject B with flash light is

And can be calculated.

On the other hand, brightness of ambient lighting is substantially equal to L _1. Accordingly, the ambient illumination and the brightness by the flash in the subjects A and B in the image taken with the flash are expressed as shown in FIG.

Next, in step S9, the illumination light sources of the main subject A and the background subject B in the image taken with the flash are estimated by a normal auto white balance (AWB) method. The response of the camera RGB before white balance to white under the light source illuminating the main subject A is (R _wa , G _wa , B _wa ). The response of the camera RGB before white balance to white under the light source illuminating the background subject B is (R _wb , G _wb , B _wb ).

In step S10, the white response to the illumination light source in each region calculated in step S9 and the contribution of the ambient illumination and the illumination by the flash light in each region calculated in steps S5 and S6, The response (R _ws , G _ws , B _ws ) is calculated as follows.

(R _wa , G _wa , B _wa ), (R _wb , G _wb , B _wb ) are (R _ws , G _ws , B _ws ) and the white response (R _wf , G _wf , B _w ) only with flash light illumination it can be expressed as follows by using the _wf).

(12)式同様、G_ws,B_wsも算出する。 If R _wf is _calculated from these two equations and R _ws is obtained, the following equation is obtained.

(12) Similarly, G _ws, B _ws also calculated.

In step S11, a matrix for converting the color components (R _a , G _a , B _a ) of each pixel of the subject A at the distance a into a color when illuminated by only ambient illumination is calculated (see the following equation). . In step S12, color conversion is performed for each pixel of subject A using the matrix obtained in step S11 according to the following equation.

  In step S13, it is determined whether the color conversion has been performed for all the pixels in the area. If not, the process returns to step S12 and the process is repeated. If it is performed, the process proceeds to step S14. In step S14, it is determined whether or not conversion processing has been completed for all regions. If it is determined that it has not been completed, the process proceeds to step S15. If it is determined that it has been completed, the process ends. At the end, output the image after color conversion.

In step S15, the white response (R _wk , R) to the illumination light source in the region k is similarly applied to the region k where color conversion is not performed (N = 0, where N = 0 is the number of image region divisions). G _wk , B _wk ) is calculated. Next, it returns to step S11 and repeats a process. The color conversion of the region k is performed by the following matrix having a ratio with (R _ws , G _ws , B _ws ) as a diagonal component, as in the above equation.

The digital camera 100 according to the first embodiment configured as described above has the following operational effects.
(1) In one image captured by flash shooting of a main subject, the main subject region is divided into a main subject region in which flash light is dominant and a background subject region in which ambient illumination is uncontrollable. Since the illumination light source (white response to illumination) is estimated for each area, the color of the main subject area can be easily changed to an image of a shade that does not use flash light illumination, that is, an image illuminated by ambient illumination. Can be corrected (converted).

(2) Since the area is divided using the distance measuring function of the camera, it is not necessary to add a new function for dividing the image.

(3) Since the color component of each pixel in the main subject region is converted by the ratio of the white color component under illumination of the background subject region and the white color component under illumination of the main subject region, simple processing is performed. Can be appropriately converted to a color under illumination of the background subject area.

(Modification 1 of the first embodiment)
When the lens used is the focal length f of the lens, the shortest shooting distance a _min of the lens is f << a _min (≦ a), and in the case of an image that is not a wide-angle lens, that is, in Equation (3) maximum image height y _max at that, can be regarded as (1 / f-1 / a ) If the _{^{-1 »y max, θ ≒ θ '}} ≒ 0. In this case, it can be approximated as cos θ≈1, and the calculation of equation (12) can be simplified as follows.

と簡略化できる。この場合、第１の実施の形態の図２のステップＳ５、６、８、１０((1)〜(12)式の計算)や領域Bの色変換を省略することができ、図８のようなフローとなる。 (Modification 2 of the first embodiment)
FIG. 8 is a diagram illustrating a flowchart of an image processing program executed by the control device 104 of the digital camera 100 according to the second modification of the first embodiment. When the distance to the subject B is far from the distance to the subject A (b >> a) or the background illumination brightness is sufficiently lower than the brightness at the time of flash light irradiation (L ₁ >> L ₂ ), the formula (12) Is

And can be simplified. In this case, steps S5, S6, S8, S10 (calculation of equations (1) to (12)) in FIG. 2 of the first embodiment and color conversion of the region B can be omitted, as shown in FIG. Flow.

(Modification 3 of the first embodiment)
In the above, the example which converts into the color at the time of illuminating only with ambient illumination was shown. However, it is also possible to perform color conversion assuming that illumination is performed by appropriately mixing ambient illumination and flash illumination. In this case, if the white response due to ambient lighting is (R _ws , G _ws , B _ws ) and the white response due only to the flash is (R _wf , G _wf , B _wf ), it can be obtained as follows: it can. m can be arbitrarily set within the range of 0-1.
R _{w, mix} = m * R _wf + (1-m) R _ws
G _{w, mix} = m * G _wf + (1-m) G _ws
B _{w, mix} = m * B _wf + (1-m) B _ws

  The same applies to the subject B. By doing in this way, arbitrary styles can be made possible. In the above, when m = 0, only ambient illumination is provided, when m = 1, only flash illumination is provided, and when 0 <m <1, mixed illumination of flash illumination and ambient illumination is provided.

-Second Embodiment-
In the first embodiment, color conversion is performed by characterizing the divided areas according to the distance, but in the second embodiment, the area is estimated from the average luminance in each area. Taking a histogram of brightness, the average brightness is high overall in the area hit by the flash, and the average brightness is low in the background area not hit by the flash due to insufficient light. The difference in illumination light source is estimated from the average brightness in the area. Is possible.

If only the luminance of each pixel is determined, it is difficult to determine when the reflectance of the subject is originally low or when the illumination luminance is low. Therefore, when determining by luminance, the area is divided into large areas, and the average luminance within that area is determined. It is preferable to make a judgment with In addition, a luminance histogram is taken as the luminance threshold of the region division, and the luminance value of the valley is set to the threshold Y ₀ (FIG. 10). A region where the average luminance of each region is> Y ₀ is a subject A, and the other region is a background subject B. Subsequent processing processes as in the first embodiment.

  Since the digital camera of the second embodiment is the same as the digital camera 100 of the first embodiment, reference is made to FIG. 1 of the first embodiment, and description thereof is omitted. FIG. 9 is a diagram illustrating a flowchart of an image processing program executed by the control device 104 of the digital camera 100 according to the present embodiment. Steps similar to those in the flow of FIG. 2 of the first embodiment are denoted by the same step numbers.

In step S1, an image photographed using the flash unit 107 is read as a correction target image. In step S31, it creates the luminance distribution histogram of each pixel. FIG. 10 is a diagram illustrating an example of a luminance distribution histogram of an acquired image. In step S32, it determines the brightness threshold value Y _0. Luminance threshold Y ₀ is set as described above, the luminance value of the valley of the intensity distribution in the luminance threshold Y _0.

In step S33, the image is divided into a plurality of regions. In this division, the image is mechanically divided into predetermined pixel ranges. In step S34, it calculates the average luminance Y _m of each region divided. Next, in steps S35 to S38, the areas divided in step S33 are divided into a subject A area (area A) and a subject B area (area B).

First, in step S35, the average luminance Y _m of the divided one certain regions are, if greater than the brightness threshold value Y ₀ is determined. If it is determined that the area is large, the process proceeds to step S36, and the area is classified as subject A. If it is determined that the area is not large, the process proceeds to step S37, and the area is classified as subject B. In step S38, it is determined whether or not the determination has been completed for all the areas. If the determination has not been completed, the process returns to step S35, and the next area is sequentially selected and the process is repeated.

  When the determination of all the areas is completed, the division of the subjects A and B is completed and the process proceeds to step S9. Since the process after step S9 is the same as that of FIG. 8 of the modification 2 of 1st Embodiment, description is abbreviate | omitted.

  As described above, even when the captured image is divided by luminance, the color of the main subject region is changed by the image of the hue that does not use the flash, that is, the ambient illumination, as in the first embodiment. It can be easily corrected (converted) to an illuminated color image.

-Third embodiment-
Since the digital camera of the third embodiment is the same as the digital camera 100 of the first embodiment, reference is made to FIG. 1 of the first embodiment, and description thereof is omitted. FIG. 11 is a diagram illustrating a flowchart of an image processing program executed by the control device 104 of the digital camera 100 according to the present embodiment. Steps similar to those in the flow of FIG. 2 of the first embodiment are denoted by the same step numbers.

  In the first and second embodiments, the color conversion is performed with the ratio of white response in the camera RGB. In the third embodiment, the white color is converted from the camera RGB to the optimum color space for changing the illumination color. Create a matrix that takes the response ratio of.

  Specifically, an optimal color space is obtained in advance by color conversion when the light source is changed from the spectral sensitivity of the camera. Let M be the 3 × 3 conversion matrix to this color space. In FIG. 11, steps S1 to S10 are the same as those in the first embodiment. In step S42, using the conversion matrix M, the color conversion matrix M is calculated as follows.

但し、

とする。 From the color components (R _a , G _a , B _a ) of each pixel under illumination with an illumination light source of (R _wa , G _wa , B _wa ), the white response to (R _ws , G a The color conversion to the color components (R _s , G _s , B _s ) of each pixel under ambient illumination represented by _ws , B _ws ) is expressed as in equation (19).

However,

And

となる。 Therefore, the matrix M ′ obtained in step S42 is

It becomes.

に従い、各画素の色変換を行う。その後は、第１の実施の形態と同様に、他の領域に付いても色変換を行う（ステップＳ１３、Ｓ１４、Ｓ１５、Ｓ４２〜Ｓ４３）。 In step S43,

In accordance with the above, color conversion of each pixel is performed. After that, as in the first embodiment, color conversion is performed for other areas (steps S13, S14, S15, and S42 to S43).

但し、照明X下の白に対する色成分データ(R_wa,G_wa,B_wa)、照明Y下の白に対する色成分データ(R_ws',G_ws',B_ws')であり、

It will now be described with reference to FIG 12 a method for determining the transformation matrix M. Color component data (R _ai , G _ai , B _ai ) (i = 0,1, ..., N-1) obtained by photographing a large number of color charts (N colors) under illumination X Prepared (step S101), color component data (R _si ', G _si ', B _si ') obtained by photographing the same color chart under illumination Y different from illumination X under the same photographing conditions other than illumination i = 0,1, ..., N-1) are prepared (step S102). (R _ai , G _ai , B _ai ) (i = 0,1, ..., N-1) color component data (R _si , G _si , B _si ) ( i = 0,1, ..., N-1) (step S103) and (R _si ', G _si ', B _si ') (i = 0,1, ..., N-1) N components of M are optimized so that (average of N colors) ΔE is minimized.

However, the color component data (R _wa , G _wa , B _wa ) for white under illumination X, and the color component data (R _ws ', G _ws ', B _ws ') for white under illumination Y,

Note that ΔE is evaluated (step S108) by converting (R _si , G _si , B _si ) and (R _si ′, G _si ′, B _si ′) to XYZ space (steps S104 and S105), and further CIE This is performed after conversion to the LAB space (steps S106 and S107). M _{RGB → XYZ} is a conversion matrix from a color space depending on the spectral sensitivity of the camera to a colorimetric value XYZ, which is calculated in advance from the spectral sensitivity of the camera. Also, (X _ws ', Y _ws ', Z _ws ') is the white point of the illumination Y.

The digital camera 100 according to the third embodiment configured as described above has the following operational effects.
(1) Since processing is performed after conversion from a color space depending on the spectral sensitivity of the camera to a color space optimal for changing the illumination color, colors other than white are also appropriately color-converted with high accuracy. In other words, the color of the main subject area can be corrected (converted) easily and with high accuracy into an image having a hue that does not use a flash, that is, an image illuminated by ambient illumination.

-Fourth embodiment-
Since the digital camera of the fourth embodiment is the same as the digital camera 100 of the first embodiment, reference is made to FIG. 1 of the first embodiment, and description thereof is omitted. FIG. 13 is a diagram illustrating a flowchart of an image processing program executed by the control device 104 of the digital camera 100 according to the present embodiment. Steps similar to those in the flow of FIG. 2 of the first embodiment are denoted by the same step numbers.

In the first to third embodiments, processing for obtaining a white response ratio is performed in a color space depending on the spectral sensitivity of the camera or a color space optimal for illumination color conversion. In the fourth embodiment, instead of taking the white response ratio, the color component data RGB1 of the camera under color temperature 1 illumination → the color component of the camera under color temperature 2 illumination for each combination of different color temperatures. A plurality of optimum color conversion matrices M _opt for data RGB2 are stored for possible combinations, the illumination of the main subject (mixed illumination of flash and background illumination), and the color temperature of the background illumination are estimated, and then the illumination performing illumination color converting reads the appropriate matrix M _opt to conditions.

In order to convert the illumination color of each region, it is only necessary to assume a case where the color under the illumination that is strongly influenced by the flash is converted into the color under the illumination with a color temperature of 2500K to 10000K as the background illumination. Therefore, nine coefficients of a 3 × 3 conversion matrix M _opt are calculated in advance every 500 K from 2500 K to 10000 K, and stored in an image processing apparatus such as the digital camera 100.

In FIG. 13, steps S1 to S10 are the same as those in the first embodiment. In step S61, read out by selecting an appropriate transformation matrix M _opt from a plurality of conversion matrices M _opt for storing. When an appropriate conversion matrix M _opt is selected, the information on the illumination light source in the main subject A in the image photographed with the flash estimated in step S9 and the information on the ambient illumination light source calculated in step S10 are used. That is, an appropriate conversion matrix M _opt corresponding to the combination of the color temperatures of these illumination light sources is selected from the memory and read out.

If there is no condition in which the color temperature of the background illumination coincides with the color temperature in which the conversion data is stored in advance, a 3 × 3 conversion matrix M _opt suitable for the shooting conditions is determined by interpolation from the data of the close condition. .

In step S62, the color conversion of each pixel is performed according to the following equation using the read matrix M _opt or the matrix M _opt obtained by interpolation.

When the color conversion is completed for all the pixels in the target area (step S13), the illumination color is similarly estimated for the other areas (step S15), and the optimum conversion matrix _Mopt is determined (step S61). ), Color conversion is performed (step S62). When the color conversion is completed for all the areas in the image (step S14), the image of the conversion result is output and the process ends.

Next, a method for determining a conversion matrix M _opt created in advance will be described with reference to FIG. Color component data (R _ai , G _ai , B _ai ) (i = 0,1, ..., N-1) obtained by photographing a large number of color charts (N colors) under illumination X Prepared (step S201), and color component data (R _si ', G _si ', B _si ') obtained by photographing the same color chart under the same illumination conditions other than illumination under illumination Y different from illumination X i = 0,1, ..., N-1) are prepared (step S202). (R _ai , G _ai , B _ai ) (i = 0,1, ..., N-1) color component data (R _si , G _si , B _si ) ( i = 0,1, ..., N-1) (Step S203) and (R _si ', G _si ', B _si ') (i = 0,1, ..., N-1) color difference ( average of N color) such that ΔE is minimized, to optimize the nine components of M _opt.

Note that ΔE is evaluated (step S208) by converting (R _si , G _si , B _si ) and (R _si ′, G _si ′, B _si ′) into XYZ space, as in the third embodiment. (Steps S204 and S205), and further converted into the CIE LAB space (Steps S206 and S207). M _{RGB → XYZ} is a conversion matrix from the color space depending on the spectral sensitivity of the camera to the colorimetric values XYZ, and is calculated in advance from the spectral sensitivity of the camera. Also, (X _ws ', Y _ws ', Z _ws ') is the white point of the illumination Y.

The digital camera 100 according to the fourth embodiment configured as described above has the following operational effects.
(1) Since a plurality of transformation matrices M _opt are created in advance, the processing for creating the transformation matrix is not required, and the processing becomes lighter and quicker.
-Other variations-
In the above example, the area division is divided for each subject. However, when the block division is performed as shown in FIG. 3, block distortion occurs at the block boundary. Therefore, at the boundary between regions having different color conversions, for example, about 100 pixels are used as boundary regions, and interpolation is performed using color components calculated by color conversion of adjacent regions and color components calculated by color conversion of the regions. Accordingly, it corrected so that the color conversion changes smoothly.

    Also, instead of interpolating the difference between multiple color conversions only at the boundary, the white point at the representative point is determined for each region, and the color conversion is performed by approximating the two-dimensional white point distribution function within the image plane. Also good.

  Further, in the above embodiment, an example in which processing is performed by the digital camera 100 has been described. However, the above-described program processed by the digital camera 100 may be processed by a personal computer. In this case, the image data file includes information such as whether or not the flash unit emits light, distance measurement data for each area, photometry data, and focal length f.

  FIG. 15 is a diagram illustrating a case where processing is performed by a personal computer. A personal computer 201 is connected to a digital camera 202, a recording medium 203 such as a CD-ROM, another computer 204, and the like, and receives image data to be processed. The personal computer 201 performs the above-described image processing on the provided image data. The computer 204 is connected via the Internet or other telecommunication line 205.

  A program executed by the personal computer 201 for image processing is provided from a recording medium such as a CD-ROM, another computer via the Internet or other electric communication line, and installed in the personal computer 201. The personal computer 201 includes a CPU (not shown) and its peripheral circuits (not shown), and executes a program in which the CPU is installed.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

It is a figure which shows the structure of the digital camera 100 which is one embodiment of this invention. It is a figure which shows the flowchart of the image processing program which the control apparatus 104 of the digital camera 100 of 1st Embodiment performs. It is a figure which shows the example which divides | segments an image into a block. It is a figure which shows the example which divides | segments an image for every to-be-photographed object. It is a table | surface which shows the relationship between the exposure conditions when not using the flash unit 107, and when using it, and the brightness of each subject. FIG. 10 is a diagram for explaining calculation of brightness L _b by subject flash unit 107. It is a table | surface which shows the brightness by the surrounding illumination and flash in subject A and B in the image image | photographed with the flash. It is a figure which shows the flowchart of the image processing program which the control apparatus 104 of the digital camera 100 of the modification 2 of 1st Embodiment performs. It is a figure which shows the flowchart of the image processing program which the control apparatus 104 of the digital camera 100 of 2nd Embodiment performs. It is a figure which shows an example of the luminance distribution histogram of the acquired image. It is a figure which shows the flowchart of the image processing program which the control apparatus 104 of the digital camera 100 of 3rd Embodiment performs. 6 is a diagram illustrating a method for determining a conversion matrix M. FIG. It is a figure which shows the flowchart of the image processing program which the control apparatus 104 of the digital camera 100 of 4th Embodiment performs. It is a figure explaining the determination method of conversion matrix _Mopt . It is a figure which shows the case where it processes with a personal computer.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Digital camera 101 Subject 102 Shooting lens 103 Imaging element 104 Control apparatus 105 Memory 106 Lens drive part 107 Flash unit

Claims (9)

An image processing method in a digital camera,
A flash illumination means for flashing the subject and an ambient illumination means for illuminating the subject from the periphery to obtain an image captured,
Based on the acquired image, a first image which is a main subject mainly illuminated by the flash illumination unit and a second image which is not the main subject mainly illuminated by the ambient illumination unit. Divided into at least two images of
Determining at least a color component by the flash illumination means of the first image and a color component by at least the ambient illumination means of the second image;
Converting said by the ratio of white color component of the second white color component and the first image of the image, the color components of each pixel of the first image to the color component that has been illuminated by the ambient lighting means An image processing method. An image processing method in a digital camera,
A flash illumination means for flashing the subject and an ambient illumination means for illuminating the subject from the periphery to obtain an image captured,
Based on the acquired image, a first image which is a main subject mainly illuminated by the flash illumination unit and a second image which is not the main subject mainly illuminated by the ambient illumination unit. Divided into at least two images of
Determining at least a color component by the flash illumination means of the first image and a color component by at least the ambient illumination means of the second image;
According to the ratio of the white color component of the second image and the white color component of the first image, the color component of each pixel of the first image is illuminated by the flash illumination unit and the ambient illumination unit. An image processing method comprising: converting to a color component illuminated by mixed illumination with illumination by a light source. An image processing method in a digital camera,
A flash illumination means for flashing the subject and an ambient illumination means for illuminating the subject from the periphery to obtain an image captured,
Based on the acquired image, a first image which is a main subject mainly illuminated by the flash illumination unit and a second image which is not the main subject mainly illuminated by the ambient illumination unit. Divided into at least two images of
Determining at least a color component by the flash illumination means of the first image and a color component by at least the ambient illumination means of the second image;
The converted image from the color space corresponding to the spectral characteristics of the image sensor of the captured in a predetermined color space, the second color white under illumination of the first image and the color component of the white under illumination of the image An image processing method, comprising: converting a color component of each pixel of the first image into a color component illuminated by the ambient illumination unit according to a ratio to the component. An image processing method in a digital camera,
A flash illumination means for flashing the subject and an ambient illumination means for illuminating the subject from the periphery to obtain an image captured,
Based on the acquired image, a first image which is a main subject mainly illuminated by the flash illumination unit and a second image which is not the main subject mainly illuminated by the ambient illumination unit. Divided into at least two images of
Determining at least a color component by the flash illumination means of the first image and a color component by at least the ambient illumination means of the second image;
The converted image from the color space corresponding to the spectral characteristics of the image sensor of the captured in a predetermined color space, the second color white under illumination of the first image and the color component of the white under illumination of the image The color component of each pixel of the first image is converted into a color component illuminated by a mixed illumination of the illumination by the flash illumination unit and the illumination by the ambient illumination unit according to the ratio with the component. Image processing method. The image processing method according to any one of claims 1 to 4,
The image processing method according to claim 1, wherein the image is divided based on a luminance distribution of the acquired image. The image processing method according to claim 1 or claim 2,
The image processing method according to claim 1, wherein the conversion is performed in a color space corresponding to a spectral characteristic of an image pickup device that picks up the image. The image processing method according to any one of claims 1 to 6,
In the first image, the lighting ratio of the influence of the illumination by the flash illumination means and the influence of the illumination by the ambient illumination means is calculated,
An image processing method comprising: converting a color of the first image in accordance with the calculated illumination ratio. The image processing program for executing the image processing method according to the computer in any one of claims 1 to 7. Digital camera equipped with an image processing program according to claim 8.

Images