WO2009096232A1

WO2009096232A1 - Image processing device and image processing method

Info

Publication number: WO2009096232A1
Application number: PCT/JP2009/050453
Authority: WO
Inventors: Koichi Sugiyama
Original assignee: Sharp Kabushiki Kaisha
Priority date: 2008-01-31
Filing date: 2009-01-15
Publication date: 2009-08-06

Abstract

An estimate matrix calculation unit (12) calculates a first estimate matrix (W(1)) based on autocorrelation matrix (B) of spectral radiance of a light source candidate that can be used for providing an illumination environment in an object, a spectral transmittance (f(1)) of a dispersion member and a spectral sensitivity (S) of an image picking-up device. A spectral radiance calculation unit (11) calculates spectral radiance (E(1)) of illumination light that is incident on an object (OBJ) from image picking-up data (g(1)RGB) by using the first estimate matrix (W(1)) calculated by the estimate matrix calculation unit (12).

Description

Image processing apparatus and image processing method

The present invention relates to an image processing apparatus and an image processing method capable of calculating the spectral radiance of illumination light applied to a subject when the subject is imaged.

In recent years, techniques for accurately reproducing the color of a subject imaged under various lighting environments in an output device such as a display device or a printing device have been proposed.

As a representative technique, a color management technique based on the spectral reflectance (reflection spectrum) of a subject is known. This technique is realized by handling the color of the subject in the wavelength region, and enables accurate color reproduction regardless of the illumination environment of the subject. An imaging method based on the spectral reflectance of such a subject is disclosed in “Yoichi Miyake,“ Introduction to Spectral Image Processing ”, The University of Tokyo Press, February 24, 2006”. Has been.

By the way, in order to estimate the spectral reflectance of the subject, it is necessary to acquire in advance the spectral radiance (illumination spectrum) under the illumination environment at the time of imaging, which is the premise. This is because the spectral radiance from the subject is determined in accordance with the spectral radiance of the illumination light and the spectral reflectance of the subject, so if the spectral radiance of the illumination light is not known, imaging data obtained by imaging the subject This is because the spectral reflectance of the subject cannot be accurately calculated based on the above.

The spectral radiance of the illumination light is also used for white balance adjustment of the imaging device. This white balance adjustment is an operation for determining a coefficient for mutually adjusting the levels of luminance values output from a plurality of image sensors constituting the imaging apparatus. When white balance is lost, when a white subject is imaged by an imaging device, a color that is different from the original white (for example, a reddish color) is output, and color reproduction is performed accurately. I can't. Note that white balance adjustment is disclosed in Japanese Patent Application Laid-Open Nos. 2001-057680 and 2005-328386.
Japanese Patent Laid-Open No. 2001-056780 JP 2005-328386 A Yoichi Miyake, "Introduction to Spectral Image Processing", The University of Tokyo Press, February 24, 2006

Conventionally, the spectral radiance of illumination light has been measured exclusively using a dedicated measuring device such as a spectroradiometer. A spectroradiometer separates light incident through an optical system with a diffraction grating (grating), and receives the light with an image sensor (CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), etc.) for each wavelength. Get the brightness value of. If it is going to implement | achieve the color management techniques as mentioned above, in addition to an imaging device such as a still camera or a video camera, a measuring device for measuring the spectral radiance is required, and there is a problem that costs increase.

Normally, in order to measure the illumination light irradiated to the subject, it is necessary to install a standard white plate with a known spectral reflectance near the subject and measure the reflected light from the plate surface with a spectroradiometer. There is a problem that it takes time and effort for measurement as well as imaging.

Accordingly, the present invention has been made to solve such a problem, and an object of the present invention is to easily reduce the spectral radiance of illumination light applied to the subject using an imaging device for imaging the subject. It is to provide an image processing apparatus and an image processing method that can be calculated.

According to an aspect of the present invention, there is provided an image processing device capable of performing image processing on imaging data imaged by an imaging device. The image processing apparatus uses an imaging device to receive first imaging data obtained by imaging at least a part of light incident on a subject through a diffusing member in an illumination environment, and an illumination unit Using the first estimation matrix calculated based on the autocorrelation matrix of the spectral radiance of the light source candidates that can be used to provide the environment, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device, A first calculator that calculates the spectral radiance of the illumination light incident on the subject from the first imaging data.

Preferably, the spectral radiance of the light source candidate is a characteristic value acquired in advance for each type of light source.

Preferably, the diffusing member is disposed on the optical axis of the imaging apparatus, and the incident intensity of the diffusing member is indicated by a predetermined function value with respect to an angle with respect to the optical axis.

More preferably, the function value is a cosine function with respect to an angle with respect to the optical axis.
Preferably, the imaging device is configured to output coordinate values defined in the RGB color system as imaging data. The image processing apparatus uses a spectral radiance of the illumination light and a color matching function to calculate a coordinate value in the RGB color system corresponding to the spectral radiance of the illumination light, and a second calculator And a third calculation unit that calculates white balance in the imaging apparatus based on the calculated ratio of coordinate values.

Preferably, the image processing apparatus has a second estimation matrix calculated based on the spectral radiance of the illumination light, the spectral sensitivity of the imaging apparatus, and the autocorrelation matrix of the spectral reflectance of colors that can be included in the subject. And a fourth calculation unit that calculates the spectral reflectance of the subject from the second imaging data obtained by imaging the subject with the imaging device in an illumination environment.

More preferably, the image processing device includes a generation unit that generates image data acquired when the subject is imaged under a predetermined illumination environment based on the spectral reflectance of the subject calculated by the fourth calculation unit. In addition.

According to another aspect of the present invention, there is provided an image processing device capable of image processing with respect to imaging data imaged by an imaging device. The image processing apparatus uses an imaging device to receive first imaging data obtained by imaging at least a part of light incident on a subject through a diffusing member in an illumination environment, and an illumination unit A selection unit that selects one of the predetermined calculation matrices for each type of a plurality of light source candidates that can be used to provide an environment according to an external command, a calculation matrix selected by the selection unit, A first calculation for calculating the spectral radiance of the illumination light incident on the subject from the first imaging data using the first estimation matrix calculated based on the spectral transmittance of the diffusing member and the spectral sensitivity of the imaging device. Part. Each of the calculation matrices is an autocorrelation matrix of a matrix indicating the spectral radiance of the light source candidate.

According to still another aspect of the present invention, there is provided an image processing apparatus capable of performing image processing on imaging data captured by an imaging apparatus. The image processing apparatus uses an imaging device to receive first imaging data obtained by imaging at least a part of light incident on a subject through a diffusing member in an illumination environment, and an illumination unit A selection unit that selects one of a plurality of first estimation matrices according to an external command, which is calculated in advance based on spectral radiances of a plurality of light source candidates that can be used to provide an environment, and a selection unit And a first calculator that calculates the spectral radiance of the illumination light incident on the subject from the first imaging data using the selected first estimation matrix. The first estimation matrix is calculated based on the autocorrelation matrix of the matrix indicating the spectral radiance of the corresponding light source candidate, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device.

According to still another aspect of the present invention, there is provided an image processing apparatus capable of performing image processing on imaging data captured by an imaging apparatus. The image processing apparatus uses an imaging device to receive first imaging data obtained by imaging at least a part of light incident on a subject through a diffusing member in an illumination environment, and an illumination unit The spectral radiance of illumination light incident on the subject from the first imaging data using a plurality of first estimation matrices respectively calculated based on the spectral radiances of a plurality of light source candidates that can be used to provide an environment A first calculation unit for calculating each candidate and each calculated spectral radiance candidate are evaluated by comparison with a predetermined reference pattern, and one of them is output as the spectral radiance of illumination light in an illumination environment. And an evaluation unit. The first estimation matrix is calculated based on the autocorrelation matrix of the matrix indicating the spectral radiance of the corresponding light source candidate, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device.

The image processing method according to still another aspect of the present invention is a step of acquiring first imaging data by imaging at least part of light incident on a subject under a lighting environment through a diffusing member using an imaging device. A first estimation matrix calculated based on the autocorrelation matrix of spectral radiance of light source candidates that can be used to provide an illumination environment, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device And calculating the spectral radiance of the illumination light incident on the subject from the first imaging data.

The image processing method according to still another aspect of the present invention is a step of acquiring first imaging data by imaging at least part of light incident on a subject under a lighting environment through a diffusing member using an imaging device. Selecting one of a plurality of predetermined calculation matrices for each type of light source candidates that can be used to provide an illumination environment, the selected calculation matrix, and spectral transmission of the diffusing member Calculating the spectral radiance of the illumination light incident on the subject from the first imaging data using a first estimation matrix calculated based on the rate and the spectral sensitivity of the imaging device. Each of the calculation matrices is an autocorrelation matrix of a matrix indicating the spectral radiance of the light source candidate.

The image processing method according to still another aspect of the present invention is a step of acquiring first imaging data by imaging at least part of light incident on a subject under a lighting environment through a diffusing member using an imaging device. Selecting one of a plurality of first estimation matrices calculated in advance based on spectral radiances of a plurality of light source candidates that can be used to provide an illumination environment, and the selected first estimation matrix And calculating the spectral radiance of the illumination light incident on the subject from the first imaging data. The first estimation matrix is calculated based on the autocorrelation matrix of the matrix indicating the spectral radiance of the corresponding light source candidate, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device.

The image processing method according to still another aspect of the present invention is a step of acquiring first imaging data by imaging at least part of light incident on a subject under a lighting environment through a diffusing member using an imaging device. And using the plurality of first estimation matrices calculated based on the spectral radiances of the plurality of light source candidates that can be used to provide the illumination environment, the spectrum of the illumination light incident on the subject from the first imaging data Each radiance candidate calculation step is evaluated by comparing each calculated spectral radiance candidate with a predetermined reference pattern, one of which is output as the spectral radiance of the illumination light in the illumination environment Including the step of. The first estimation matrix is calculated based on the autocorrelation matrix of the matrix indicating the spectral radiance of the corresponding light source candidate, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device.

According to the present invention, it is possible to easily calculate the spectral radiance of the illumination light applied to the subject using the imaging device for imaging the subject.

1 is a functional configuration diagram of an image processing device according to a first embodiment of the present invention. It is a figure for demonstrating the acquisition method of the imaging data used as the process target in the image processing apparatus according to Embodiment 1 of this invention. It is a figure for demonstrating the production | generation process of the autocorrelation matrix of the spectral radiance according to Embodiment 1 of this invention. It is a flowchart which shows the whole process sequence in the image processing apparatus according to Embodiment 1 of this invention. It is a function block diagram of the image processing apparatus according to Embodiment 2 of this invention. It is a flowchart which shows the whole process sequence in the image processing apparatus according to Embodiment 2 of this invention. It is a function block diagram of the image processing apparatus according to the modification of Embodiment 2 of this invention. It is a flowchart which shows the whole process sequence in the image processing apparatus according to the modification of Embodiment 2 of this invention. It is a function block diagram of the illumination spectrum estimation part of the image processing apparatus according to Embodiment 3 of this invention. It is a figure for demonstrating the comparison process of the spectral radiance of the illumination light by an evaluation part, and a reference pattern. It is a figure for demonstrating the calculation process of the similarity in FIG. It is a flowchart which shows the whole process sequence in the image processing apparatus according to Embodiment 3 of this invention. It is a flowchart which shows the process sequence of the evaluation subroutine shown to step S108 shown in FIG. It is a schematic block diagram of the computer which implement | achieves the image processing apparatus according to the modification of embodiment of this invention.

Explanation of symbols

Ax1, Ax2 optical axis, OBJ subject, 1, 1A, 1B image processing device, 10, 20 input unit, 11, 11A, 11B, 11C, 11D spectral radiance calculation unit, 12 estimation matrix calculation unit, 13, 13A light source data Storage unit, 14 Tristimulus value conversion unit, 15 Coordinate conversion unit, 16 White balance calculation unit, 17 Estimation matrix storage unit, 18 Selection unit, 19 Evaluation unit, 21 Spectral reflectance calculation unit, 22 Estimation matrix calculation unit, 23 Spectroscopy Reflectance data storage unit, 24 image data generation unit, 25 coordinate conversion unit, 100, 100A, 100B, 100C illumination spectrum estimation unit, 150 computer body, 152 monitor, 154 keyboard, 156 mouse, 162 memory, 164 fixed disk, 166 FD drive, 168 CD-ROM drive 170 communication interface, 200-color reproduction unit, 300 light source, 400 an imaging device, 402 diffusing member.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are denoted by the same reference numerals and description thereof will not be repeated.

[Embodiment 1]
<Overall configuration>
FIG. 1 is a functional configuration diagram of an image processing apparatus 1 according to the first embodiment of the present invention.

Referring to FIG. 1, the image processing apparatus 1 includes first imaging data g ⁽¹⁾ _RGB (m, n) and second imaging data g ⁽²⁾ _RGB (m, n) captured by an imaging apparatus described later. On the other hand, the image processing method according to the present embodiment can be executed.

More specifically, the image processing apparatus 1 includes an illumination spectrum estimation unit 100 and a color reproduction unit 200. The illumination spectrum estimation unit 100 calculates the spectral radiance E ⁽¹⁾ (illumination spectrum) of the illumination light incident on the subject using the first imaging data g ⁽¹⁾ _RGB (m, n). Subsequently, the color reproduction unit 200 calculates the spectral reflectance of the subject from the second imaging data g ⁽²⁾ _RGB (m, n) using the calculated spectral radiance E ⁽¹⁾ . Furthermore, the color reproduction unit 200 outputs image data g ^(OUT) _RGB (m, n) obtained by performing color reproduction of the subject based on the calculated spectral reflectance of the subject. The image data g ^(OUT) _RGB (m, n) output from the color reproduction unit 200 is typically output to an output device (not shown) such as a display device (display) or a printing device (printer). Alternatively, it may be stored in a storage device (not shown).

The image processing apparatus 1 is typically realized by hardware, but a part or all of the image processing apparatus 1 may be realized by software as will be described later.

<Acquisition of imaging data>
FIG. 2 is a diagram for describing a method for acquiring imaging data to be processed in image processing apparatus 1 according to the first embodiment of the present invention. FIG. 2 shows a case where the subject OBJ is imaged under a predetermined illumination environment. 2A shows a procedure for acquiring the first imaging data g ⁽¹⁾ _RGB (m, n), and FIG. 2B shows the second imaging data g ⁽²⁾ _RGB (m, n). The procedure to acquire is shown.

First, the imaging device 400 is used for acquisition (imaging) of imaging data. As an example, the imaging apparatus 400 is a digital still camera or a digital video camera, and includes an imaging element (typically, CCD, CMOS, etc.) having spectral sensitivity characteristics in a specific wavelength band. The imaging element includes a plurality of pixels arranged in a matrix, and outputs luminance corresponding to the intensity of light incident on each pixel as imaging data. At this time, the luminance output from each image sensor has a value corresponding to the spectral sensitivity. A specific wavelength band that can be imaged by the imaging apparatus is referred to as a band. In the present embodiment, typically, three bands mainly having spectral sensitivity characteristics of R (red), G (green), and B (blue). A case of using the imaging device 400 will be described. As the device structure, either a structure in which a plurality of types of image sensors are formed on the same substrate or a structure in which a corresponding type of image sensor is formed on a plurality of substrates can be adopted. Further, as a method for determining the spectral sensitivity characteristics of each band, the spectral sensitivity of the element itself may be made different, or an element having the same spectral sensitivity is used, and R, G are input to the input light side of each element. , B may be provided.

As described above, the imaging data output by the imaging apparatus 400 is three-dimensional color information of R, G, and B luminance values (typically, each of 12 bits: 0 to 4095 gradations). As described above, the imaging data output by the imaging apparatus 400 is defined in the RGB color system. Hereinafter, (m, n) of the imaging data g ⁽¹⁾ _RGB (m, n), g ⁽²⁾ _RGB (m, n) represents the coordinates of the corresponding pixel in the imaging device of the imaging device 400. That is, the imaging data g ⁽¹⁾ _RGB (m, n), g ⁽²⁾ _RGB (m, n) = [(the luminance value detected by the R imaging element at coordinates (m, n)), (coordinate ( (luminance value detected by G image sensor at m, n)), (luminance value detected by B image sensor at coordinates (m, n))].

Referring to FIG. 2A, it is assumed that illumination light emitted from some light source 300 is irradiated on the subject OBJ. In the illumination environment realized by such a light source 300, when first imaging data g ⁽¹⁾ _RGB (m, n) is first acquired, the imaging device 400 is used to enter the subject OBJ from the light source 300. The light obtained through the diffusing member 402 (that is, the light after passing through the diffusing member 402) is imaged at least part of the light to be transmitted.

More specifically, when the first imaging data g ⁽¹⁾ _RGB (m, n) is acquired, imaging is performed so that the optical axis Ax1 is on any path on which the illumination light enters the subject OBJ. A device 400 is arranged. Furthermore, a diffusing member 402 is disposed between the imaging device 400 and the light source 300 on this optical axis Ax1 (preferably in the immediate vicinity of the imaging device 400). The path of illumination light incident on the subject OBJ includes a path directly incident on the subject OBJ from the light source 300 and a path incident on the subject OBJ after being reflected from the light source 300 by a wall material or the like.

The diffusion member 402 is a member for spatially diffusing the light imaged by the imaging device 400, that is, for spatially averaging, and a milky white diffusion plate having a known spectral transmittance is typically used. Alternatively, an integrating sphere or the like may be used. By using such a diffusing member 402, the intensity distribution of the illumination light incident on the imaging device 400 can be made uniform, thereby increasing the estimation accuracy of the spectral radiance of the illumination light described later.

Further, when a milky white diffusion plate is used as the diffusion member 402, it is preferable to use a diffusion plate having a predetermined incident angle characteristic (generally referred to as a cosine collector, a cosine diffuser, a cosine receptor, or the like). . In such a diffusing plate, the incident intensity of light after passing through the diffusing member 402 is indicated by a cosine function (cosine) with respect to an angle (solid angle) with respect to the optical axis Ax1 of the imaging device 400. By using a diffuser with such an incident angle characteristic of a cosine function, imaging data that reflects the amount of illumination light energy (spectral irradiance) incident per unit area without performing any special calculations Can be obtained. In addition, when a diffuser plate having no incident angle characteristic is used, it is necessary to relatively reduce the viewing angle of the imaging device 400 in order to suppress disturbance light. In the case of using the diffusing plate, it is possible to image the illumination light from the light source 300 without considering the viewing angle of the imaging device 400.

The first imaging data g ⁽¹⁾ _RGB (m, n) acquired according to the above procedure includes color information reflecting illumination light incident on the subject OBJ under the illumination environment.

Next, referring to FIG. 2B, the second imaging data g ⁽²⁾ _RGB (m, n) is obtained by imaging the subject OBJ using the same imaging device 400 as in FIG. To be acquired. At this time, unlike the case of FIG. 2A, the diffusing member 402 is not disposed on the optical axis Ax2 of the imaging device 400. Note that the imaging device 400 used for acquiring the first imaging data g ⁽¹⁾ _RGB (m, n) and the imaging device 400 used for acquiring the second imaging data g ⁽²⁾ _RGB (m, n). Are not necessarily the same, and different imaging devices 400 may be used as long as at least the spectral sensitivity of the imaging device is substantially known.

2A, the optical axis Ax1 of the imaging device 400 when imaging the first imaging data g ⁽¹⁾ _RGB (m, n), and the second imaging data g ⁽²⁾ in FIG. 2B. It is preferable to match the optical axis Ax2 of the imaging device 400 when imaging _RGB (m, n). The second imaging data g ⁽²⁾ _RGB (m, n) acquired in FIG. 2B is determined mainly depending on the reflected light from the subject OBJ. This reflected light is reflected by the subject OBJ and propagates in the opposite direction on the optical axis Ax2, and the illumination light that generates this reflected light is mainly directed toward the subject OBJ on the optical axis Ax2 of the imaging device 400. Propagate. Therefore, by capturing the illumination light that generates the reflected light as the first imaging data g ⁽¹⁾ _RGB (m, n), a more appropriate spectral radiance of the illumination light can be calculated.

<Calculation processing of spectral radiance of illumination light>
With reference to FIG. 1 again, the calculation process of the spectral radiance E ⁽¹⁾ (illumination spectrum) of the illumination light incident on the subject executed by the illumination spectrum estimation unit 100 will be described. Note that the spectral radiance E ⁽¹⁾ of the illumination light should originally be a continuous function with respect to the wavelength λ, but in the present embodiment, the spectral radiance E ^{(1) is represented} as a visible light region (380 to For 780 nanometers), discrete values sampled with a predetermined wavelength width (1 nanometer width) shall be used. For the convenience of matrix operation described below, the spectral radiance E ^{(1) according to} the present embodiment is 401 rows × 401 columns including a total of 401 luminance values of wavelengths λ = 380, 381,. A matrix. In the matrix indicating the spectral radiance E ⁽¹⁾ , the luminance value at each wavelength is set in the diagonal element, and zero is set in elements other than the diagonal element.

The illumination spectrum estimation unit 100 includes an input unit 10, a spectral radiance calculation unit 11, an estimation matrix calculation unit 12, and a light source data storage unit 13.

As shown in FIG. 2A, the input unit 10 is obtained by imaging at least a part of light incident on the subject OBJ through the diffusing member 402 using an imaging device 400 in an illumination environment. First imaging data g ⁽¹⁾ _RGB (m, n) is received. Further, the input unit 10 based on the first image data ^{_{g (1) RGB (m,}} n) , the first imaging data ^{_{g (1) RGB (m,}} n) image pickup data ^g ₍₁₎ representing the _RGB Is output. The imaging data g ⁽¹⁾ _RGB is linearized color data composed of three luminance values (representative values) of R, G, and B. As an example, the input unit 10, the first imaging data ^{_{g (1) RGB (m,}} n) includes logic for averaging the luminance value included in the first imaging data ^{_{g (1) RGB (m,}} n ) Is averaged separately for each of R, G, and B, and the averaged value (R, G, B) is output as imaging data g ⁽¹⁾ _RGB .

If the first imaging data g ⁽¹⁾ _RGB (m, n) has an inverse gamma characteristic (non-linearity), the input unit 10 performs processing for canceling the inverse gamma characteristic. The first imaging data g ⁽¹⁾ _RGB (m, n) may be linearized. In general, a display device has a non-linear relationship (gamma characteristic) between an input signal level and an actually displayed luminance level. The imaging device 400 has a non-linearity (inverse gamma characteristic) opposite to the gamma characteristic of the display device so that an image adapted to human vision is displayed by canceling such non-linearity in the display device. Such imaging data is often output. As described above, when the inverse gamma characteristic is given to the imaging data, the subsequent processing cannot be executed accurately. For example, the input unit 10 cancels the inverse gamma characteristic and linearizes the first data. Imaging data g ⁽¹⁾ _RGB (m, n) is generated.

In general, the gamma characteristic and the inverse gamma characteristic can be expressed as a power function. For example, when the inverse gamma value in the imaging apparatus 400 is γc, the first imaging data g ⁽¹⁾ _RGB (m, n) can be linearized according to the following arithmetic expression.

g ′ ⁽¹⁾ _RGB (m, n) = g ⁽¹⁾ _RGB (m, n) ^{1 / γc}
Further, such a linearization process needs to be performed before the execution of the averaging process for calculating the above-described imaging data g ⁽¹⁾ _RGB . On the other hand, if the pixel size of the image sensor that constitutes the image capturing apparatus 400 is relatively large, the above-described linearization process requires a huge amount of calculation. For this reason, when the arithmetic processing capability of the input unit 10 is sufficiently high, the above power calculation may be directly executed. However, when the arithmetic processing capability is limited, a lookup table (LUT: Look) is used. It is effective to use (Up Table). This lookup table is a data table in which the result of the above-described conversion formula is stored in advance in association with each of all the luminance values that can be taken by the input imaging data. Refer to the correspondence between this input and output. Since the converted value can be acquired simply by doing this, the amount of calculation can be greatly reduced.

The spectral radiance calculation unit 11 uses the first estimation matrix W ⁽¹⁾ calculated by the estimation matrix calculation unit 12 to be described later, and the spectral radiance of illumination light incident on the subject OBJ from the imaging data g ⁽¹⁾ _RGB. E ⁽¹⁾ is calculated. More specifically, the spectral radiance calculation unit 11 calculates the spectral radiance E ⁽¹⁾ of the illumination light based on the matrix product of the first estimation matrix W ⁽¹⁾ and the imaging data g ⁽¹⁾ _RGB . As described above, in the present embodiment, the spectral radiance E ⁽¹⁾ of 401 rows × 401 columns sampled with a predetermined wavelength width (typically 1 nanometer width ⁾ is used, so the first estimation matrix W ^{( 1)} is the number of wavelength components × the number of bands of the imaging device 400, that is, a matrix of 401 rows × 3 columns.

The estimation matrix calculation unit 12 includes the autocorrelation matrix B of the spectral radiance of the light source candidates that can be used to provide the illumination environment in the subject OBJ, the spectral transmittance f ^{(1) of the} diffusing member 402, and the spectral of the imaging device 400. Based on the sensitivity S, a first estimation matrix W ⁽¹⁾ is calculated. Hereinafter, it is assumed that the spectral sensitivity S is a matrix of 401 rows × 3 columns, and the spectral transmittance f ⁽¹⁾ is a matrix of 401 rows × 3 columns.

Hereinafter, the principle by which the spectral radiance E ⁽¹⁾ can be calculated from the first imaging data g ⁽¹⁾ _RGB (m, n) will be described.

The light (spectrum) that has passed through the diffusing member 402 incident on the imaging device 400 includes the spectral radiance E ⁽¹⁾ (λ) of the illumination light applied to the diffusing member 402 (or the subject OBJ), and the diffusing member. This corresponds to the product of the spectral transmittance f ⁽¹⁾ (λ) of 402. The spectral transmittance f ⁽¹⁾ (λ) of the diffusing member 402 is assumed to be constant over the entire diffusing member 402. The imaging data g ⁽¹⁾ _RGB component values g ⁽¹⁾ _i (i = R, G, B) representing the first image data output from the imaging device 400 are the spectral sensitivities S of the imaging elements. _This is equivalent to a product obtained by further multiplying _i (λ) (i = R, G, B) and integrating the light energy over the wavelength region. Such a relationship can be expressed as a relational expression shown in Expression (1).

Here, n _i (m, n) is additive noise generated by white noise or the like appearing in each image sensor, and is a value depending on the characteristics of the image sensor and the lens of the imaging apparatus 400, the illumination environment, and the like.

As described above, in the present embodiment, a matrix arithmetic expression sampled with a predetermined wavelength width (typically 1 nanometer width) is used. That is, the integral expression of the first term on the right side of the equation (1) is expressed by the spectral sensitivity S that is a matrix indicating the sensitivity at each wavelength of each image sensor and the spectral radiance E ¹ that is a matrix indicating the radiance at each wavelength. ⁾ And spectral transmittance f ⁽¹⁾ which is a matrix indicating the transmittance at each wavelength. The spectral sensitivity S and the spectral transmittance f ⁽¹⁾ are already known.

Here, the additive noise n _i (m, n) is generally a sufficiently small value, and therefore, if ignored from the expression (1), the following matrix operation expression can be derived from the expression (1).

g ⁽¹⁾ = ^St · E ⁽¹⁾ · f ⁽¹⁾ ... (2)
Consider calculating the spectral radiance E ⁽¹⁾ based on the equation (2). Specifically, the spectral radiance E ⁽¹⁾ of the illumination light is calculated according to the following equation (3).

f ⁽¹⁾ = W ⁽¹⁾ · g ⁽¹⁾ (3)
In the equation (3), W ⁽¹⁾ is a first estimation matrix. The first estimation matrix W ⁽¹⁾ is calculated by the winner estimation method described below. Specifically, the first estimation matrix W ⁽¹⁾ is derived as shown in equation (5) by modifying the equation (2) after defining the system matrix I as the following equation (4). The

I = S ^t × f ^{(1) t} (4)
W ⁽¹⁾ = B · I ^t · (I · B · I ^t ) ⁻¹ (5)
However, “×” means a product of matrix elements, “ ^t ” means a transposed matrix, and “ ⁻¹ ” means an inverse matrix.

In the equation (5), B is an autocorrelation matrix (hereinafter also referred to as “calculation matrix”) of spectral radiances of light source candidates that can be used to provide an illumination environment. In the present embodiment, spectral radiance for a plurality of light source candidates is acquired in advance, and from a statistical viewpoint, the spectral radiance of illumination light is used by utilizing the correlation with the spectral radiance of each light source. E ⁽¹⁾ is estimated. That is, statistical data acquired for each type of light source is prepared in advance, and the spectral radiance E ⁽¹⁾ of illumination light is calculated according to the characteristics of the statistical data.

Since the autocorrelation matrix B of the spectral radiance serves as a reference for estimating the spectral radiance E ⁽¹⁾ of the illumination light, the type of the light source that is likely to be used to provide the illumination environment ( For example, it is preferable to use appropriate statistical data according to the light emission principle of fluorescent lamps, incandescent lamps, xenon lamps, mercury lamps, and the like.

The spectral radiance of such a light source can be obtained experimentally in advance for each light source, or standardized by the International Commission on Illumination (CIE), ISO (International Organization for Standardization), or JIS (Japan Industrial Standards). Statistical data may be used.

FIG. 3 is a diagram for describing processing for generating an autocorrelation matrix B of spectral radiance according to the first embodiment of the present invention. Referring to FIG. 3, first, a light source group matrix _Est is generated with spectral radiance values of at least one type of light source candidates (light source 1 to light source N) as elements. That is, assuming that the component value (radiance) at each sampling wavelength λ _j (1 ≦ j ≦ k) of the light source i (1 ≦ i ≦ N) is e _i (λ _j ), the component value e _i (λ Create a light source group matrix _Est with _j ) arranged in the row direction.

Further, an autocorrelation matrix B is calculated based on the group matrix _Est according to the following arithmetic expression.

B = E _st · E _st ^t (6)
In order to calculate the spectral radiance E ⁽¹⁾ obtained by sampling the visible light region (380 to 780 nm) with a width of 1 nm, a group matrix E _st having the same sampling interval (number of elements ⁾ is calculated. It is necessary to use it. Accordingly, a group matrix _Est in which n matrixes of 401 rows × 1 column indicating the spectral radiance of one light source are combined is a 401 row × n column matrix, and the autocorrelation matrix of the group matrix _Est is , 401 rows × 401 columns matrix.

As the spectral radiance of the light source, for example, spectral radiance emitted from a single light source such as a fluorescent lamp or an incandescent lamp may be used, or spectral radiance generated by combining a plurality of types of light sources may be used. Further, outdoors, spectral radiance such as sunlight may be combined. That is, in this embodiment, in order to estimate the spectral radiance E ⁽¹⁾ of illumination light, the autocorrelation matrix B obtained from the spectral radiance of a light source that is likely to be used to provide an illumination environment. Is preferably used.

For details of the winner estimation, please refer to “Yoichi Miyake,“ Introduction to Spectral Image Processing ”, The University of Tokyo Press, February 24, 2006” above.

Referring to FIG. 2 again, the light source data storage unit 13 stores in advance the autocorrelation matrix B, which is an arithmetic matrix, calculated by the procedure as described above.

The estimation matrix calculation unit 12 calculates the system matrix I based on the spectral transmittance f ^{(1) of the} diffusing member 402 stored in advance and the spectral sensitivity S of the imaging device 400 according to the above-described equation (4). The first estimation matrix W ⁽¹⁾ is calculated based on the system matrix I and the autocorrelation matrix B read from the light source data storage unit 13 in accordance with the above equation (5). Subsequently, the spectral radiance calculation unit 11 converts the first estimation matrix W ⁽¹⁾ from the estimation matrix calculation unit 12 and the imaging data g ⁽¹⁾ _RGB from the input unit 10 according to the above equation (3). Based on this, the spectral radiance E ⁽¹⁾ of the illumination light is calculated.

Thus, the spectral radiance E ⁽¹⁾ of the illumination light calculated by the estimation matrix calculation unit 12 is used for white balance calculation processing and color reproduction processing described later.

<White balance calculation processing>
The illumination spectrum estimation unit 100 further includes a tristimulus value conversion unit 14, a coordinate conversion unit 15, and a white balance calculation unit 16. These parts calculate the white balance of the imaging apparatus 400 based on the calculated spectral radiance E ⁽¹⁾ of the illumination light. Based on this white balance value, it is possible to perform white balance adjustment for mutually adjusting the levels of the R, G, and B luminance values output from the image sensor of the imaging apparatus 400.

The tristimulus value conversion unit 14 calculates tristimulus values X, Y, and Z in the XYZ color system from the spectral radiance E ⁽¹⁾ defined in the wavelength region. The tristimulus values X, Y, and Z indicate characteristic values when it is assumed that the human has observed the spectral radiance E ⁽¹⁾ in the illumination environment where the subject OBJ is imaged. More specifically, the tristimulus values X, Y, and Z of the XYZ color system for the spectral radiance E ⁽¹⁾ of the illumination light are expressed by the following equation (7).

In the equation (7), h _i (λ) (i = R, G, B) is a color matching function, which is a function corresponding to human visual sensitivity characteristics. This color matching function h _i (λ) is defined by the International Commission on Illumination (CIE).

As described above, since the spectral radiance E ⁽¹⁾ is a matrix of 401 rows × 401 columns, the tristimulus value conversion unit 14 performs an operation corresponding to Equation (7) by a matrix operation shown below. Realize.

Tristimulus values [X ⁽¹⁾ , Y ⁽¹⁾ , Z ⁽¹⁾ ] = h ^t · E ⁽¹⁾ (8)
Here, the matrix h is a matrix of 401 rows × 3 columns whose elements are values at the respective sampling wavelengths of the color matching function h _i (λ).

Subsequently, the coordinate conversion unit 15 converts the tristimulus values X ⁽¹⁾ , Y ⁽¹⁾ , Z ⁽¹⁾ into coordinate values R ⁽¹⁾ , G ⁽¹⁾ , B ⁽ defined in the RGB color system. Convert to ¹⁾ . More specifically, the coordinate conversion unit 15 calculates coordinate values R ⁽¹⁾ , G ⁽¹⁾ , and B ⁽¹⁾ defined in the RGB color system according to the arithmetic expression shown below.

R ⁽¹⁾ = a ₁₁ X ⁽¹⁾ + a ₁₂ Y ⁽¹⁾ + a ₁₃ Z ⁽¹⁾
G ⁽¹⁾ = a ₂₁ X ⁽¹⁾ + a ₂₂ Y ⁽¹⁾ + a ₂₃ Z ⁽¹⁾
B ⁽¹⁾ = a ₃₁ X ⁽¹⁾ + a ₃₂ Y ⁽¹⁾ + a ₃₃ Z ⁽¹⁾
Here, a ₁₁ to a ₃₃ are 3 rows × 3 columns representing the correspondence between the colorimetric values of the subject (XYZ color system) and the signal values (RGB color system) actually recorded in the imaging apparatus. Is the transformation matrix. Such a matrix is called an RGB⇔XYZ conversion matrix in the display device. If a ₁₁ to a ₃₃ are _viewed as a 3 × 3 matrix, the following equation (15)
h ^t · E ⁽¹⁾ · W ⁽²⁾ = M
This corresponds to the inverse matrix M− ¹ .

Further, the white balance calculation unit 16 calculates the white balance in the imaging apparatus 400 based on the ratio of the coordinate values R ⁽¹⁾ , G ⁽¹⁾ , B ⁽¹⁾ . In general, completion of white balance adjustment means that R ⁽¹⁾ : G ⁽¹⁾ : B ⁽¹⁾ = 1: 1: 1 is established. It can be said that the white balance adjustment is not sufficient. In such a case, the white balance is adjusted by independently adjusting the output gains of the image pickup elements of the respective colors constituting the image pickup apparatus 400. That is, the adjustment gain to be multiplied by the R, G, and B image sensors is 1 / R ⁽¹⁾ : 1 / G ⁽¹⁾ : 1 / B ⁽¹⁾ .

Therefore, the white balance calculation unit 16 calculates the ratio of the coordinate values R ⁽¹⁾ , G ⁽¹⁾ , B ⁽¹⁾ or the inverse ratio 1 / R ⁽¹⁾ : 1 / G ⁽¹⁾ : 1 / B ⁽¹⁾ is output as white balance. The white balance output from the white balance calculation unit 16 is used for manual gain adjustment by the user. Alternatively, a gain adjustment unit (not shown) of the imaging device 400 may be provided so that the gain adjustment unit automatically adjusts the gain of the imaging device 400.

<Color reproduction processing>
Next, using the spectral radiance E ⁽¹⁾ calculated by the above-described processing, the color of the subject is reproduced from the second imaging data g ⁽²⁾ _RGB (m, n), and the image data g ^(OUT) _RGB A process for generating (m, n) will be described.

The color reproduction unit 200 includes an input unit 20, a spectral reflectance calculation unit 21, an estimation matrix calculation unit 22, a spectral reflectance data storage unit 23, an image data generation unit 24, and a coordinate conversion unit 25.

As illustrated in FIG. 2B, the input unit 20 receives second imaging data g ⁽²⁾ _RGB (m, n) obtained by imaging the subject OBJ using the imaging device 400. Then, the input unit 20 outputs the second imaging data g ⁽²⁾ _RGB (m, n) to the spectral reflectance calculation unit 21 according to the processing.

Note that when the second imaging data g ⁽²⁾ _RGB (m, n) is provided with an inverse gamma characteristic (nonlinearity), the input unit 20 also has the inverse gamma, as with the input unit 10 described above. You may make it perform the process for negating a characteristic. That is, when the inverse gamma value in the imaging apparatus 400 is γc, the second imaging data g ⁽²⁾ _RGB (m, n) can be linearized according to the following arithmetic expression.

g ′ ⁽²⁾ _RGB (m, n) = g ⁽²⁾ _RGB (m, n) ^{1 / γc}
Furthermore, as described above, such a linearization process may be executed using a lookup table.

The spectral reflectance calculator 21 calculates the spectral reflectance of the subject OBJ from the second imaging data g ⁽²⁾ using the second estimation matrix W ⁽²⁾ calculated by the estimation matrix calculator 22 described later. Further, the spectral reflectance calculator 21 outputs image data g ^(OUT) _RGB (m, n) that is color reproduction data of the subject OBJ under an arbitrary illumination environment. This color reproduction data is a reproduction of how the subject OBJ is observed under an arbitrary illumination environment based on the spectral reflectance of the subject OBJ.

The estimation matrix calculation unit 22 includes an autocorrelation matrix A calculated from spectral reflectances of colors that can be included in the subject OBJ, the spectral radiance E ^{(1) of} the illumination light calculated by the illumination spectrum estimation unit 100, and imaging. Based on the spectral sensitivity S of the apparatus 400, a second estimation matrix W ⁽²⁾ is calculated.

Hereinafter, the principle of generating image data g ^(OUT) _RGB (m, n) that is color-reproduced from the second imaging data g ⁽²⁾ _RGB (m, n) will be described.

The light (spectrum) from the subject OBJ that enters the pixel at the coordinates (m, n) of the imaging device 400 is the spectral radiance E ⁽¹⁾ (λ) of the illumination light irradiated on the subject OBJ and the subject OBJ. This corresponds to the product of the spectral reflectance f ⁽²⁾ (m, n; λ) at the position corresponding to the pixel. Then, each component value g ⁽²⁾ _i (m, n) (i = R, G, B) of the second imaging data g ⁽²⁾ _RGB (m, n) output from the imaging device 400 is obtained for each imaging. This corresponds to a product obtained by further multiplying the spectral sensitivity S _i (λ) (i = R, G, B) of the element and integrating the light energy over the wavelength region. Such a relationship can be expressed as a relational expression shown in Expression (9).

As described above, in the present embodiment, a matrix arithmetic expression sampled with a predetermined wavelength width (typically 1 nanometer width) is used. That is, the integral expression of the first term on the right side of the equation (9) is expressed as follows: spectral sensitivity S that is a matrix indicating the spectral sensitivity at each wavelength of each image sensor, and spectral radiance E that is a matrix indicating the spectral radiance at each wavelength. ^This is realized by matrix calculation of ⁽¹⁾ and spectral reflectance f ⁽²⁾ (m, n) which is a matrix indicating the spectral reflectance of the subject OBJ at each wavelength. Typically, when the visible light region (380 to 780 nanometers) is sampled with a width of 1 nanometer, the spectral reflectance f ⁽²⁾ (m, n) is expressed as a matrix of 401 rows × 1 column for each element. Become.

Here, the additive noise n _i (m, n) is generally a sufficiently small value, and therefore, if ignored from the expression (9), the following matrix operation expression can be derived from the expression (9).

g ⁽²⁾ _RGB (m, n) = ^St · E ⁽¹⁾ · f ⁽²⁾ (m, n) (10)
Consider calculating the spectral reflectance f ⁽²⁾ (m, n) based on the equation (10). Specifically, the spectral reflectance f ⁽²⁾ (m, n) of the subject OBJ is calculated according to the following equation (11).

f ⁽²⁾ (m, n) = W ⁽²⁾ · g ⁽²⁾ _RGB (m, n) (11)
In equation (11), W ⁽²⁾ is the second estimation matrix. The second estimation matrix W ⁽²⁾ is calculated by the winner estimation method, similarly to the calculation of the first estimation matrix W ⁽¹⁾ described above. Specifically, the second estimation matrix W ⁽²⁾ is derived as the following equation (13) by modifying the equation (11) after defining the system matrix H as the following equation (12). The

H = S ^t · E ⁽¹⁾ (12)
W ⁽²⁾ = A · H ^t · (H · A · H ^t ) ⁻¹ (13)
However, “ ^t ” means a transposed matrix, and “ ⁻¹ ” means an inverse matrix.

In equation (13), A is an autocorrelation matrix calculated from spectral reflectances of colors that can be included in the subject OBJ, and serves as a reference for estimating the spectral reflectance of the subject OBJ. As an example, the autocorrelation matrix A can be determined by referring to a standard object color sample (SOCS) which is a database of spectral reflectance standardized by ISO. Alternatively, when the material of the subject OBJ is known in advance, the spectral reflectance of the subject OBJ itself may be measured in advance by another method to determine the autocorrelation matrix A.

This autocorrelation matrix A is generated by a process similar to the process of generating the autocorrelation matrix B in FIG. As the group matrix used to generate the autocorrelation matrix A, for example, the spectral reflectance of each color of a color chart composed of a plurality of color samples can be used. In the present embodiment, a spectral radiance E ^{(1) of} 401 rows × 401 columns obtained by sampling the visible light region (380 to 780 nanometers) with a width of 1 nanometer is used. The matrix is 401 rows × 401 columns.

The autocorrelation matrix A is stored in advance in the spectral reflectance data storage unit 23.
Further, a principal component analysis technique may be used instead of the above-described winner estimation technique.

As described above, the estimation matrix calculation unit 22 can be included in the spectral radiance E ^{(1) of} the illumination light, the spectral sensitivity S of the imaging device 400, and the subject OBJ according to the equations (12) and (13). Based on the autocorrelation matrix A obtained from the spectral radiance of the color, a second estimation matrix W ⁽²⁾ is calculated. Then, the spectral reflectance calculation unit 21 uses the second estimation matrix W ⁽²⁾ according to the equation (11), and uses the second imaging data g ⁽²⁾ _RGB (m, n) as the spectral reflectance of the subject OBJ. f ⁽²⁾ Calculate (m, n).

The spectral reflectance f ⁽²⁾ (m, n) calculated in this way is the essence of the color of the subject OBJ. By using the spectral reflectance f ⁽²⁾ (m, n), which subject OBJ is selected Even if it is observed under such an illumination environment, the color reproduction can be performed.

That is, the tristimulus values X, Y, and Z of the XYZ color system when an object having a spectral reflectance f (m, n; λ) is observed under the condition of an arbitrary spectral radiance E (λ) are as follows: (14) shown in FIG.

In the equation (14), h _i (λ) (i = R, G, B) is a color matching function, which is a function corresponding to human visual sensitivity characteristics.

In this equation (14), the spectral radiance E (λ) used for color reproduction can be arbitrarily determined. However, in this embodiment, color reproduction is performed under the same illumination environment as when the subject OBJ is imaged. Illustrate.

That is, the image data generation unit 24 uses the color matching function h, the spectral radiance E ^{(1) of} the illumination light incident on the subject OBJ, and the spectral reflectance f ⁽²⁾ (m, n) (= W ⁽²⁾ · g ⁽²⁾ _RGB (m, n)) and image data g ^(OUT) _XYZ (m ^{) in} which the color reproduction of the subject OBJ is performed in the illumination environment having the spectral radiance E ^(1). , N). That is, the image data generation unit 24 executes the arithmetic expression shown in Expression (15).

g ^(OUT) _XYZ (m, n) = ^ht · E ⁽¹⁾ · W ⁽²⁾ · g ⁽²⁾ _RGB (m, n) (15)
Here, the image data g ^(OUT) _XYZ (m, n) is defined as coordinate values of the XYZ color system.

Subsequently, the coordinate conversion unit 25 converts the image data g ^(OUT) _XYZ (m, n) into image data g ^(OUT) _RGB (m, n) defined in the RGB color system. Since the coordinate conversion process executed by coordinate conversion unit 25 is the same as the process in coordinate conversion unit 15 described above, detailed description will not be repeated.

Through the above processing, image data g ^(OUT) _RGB (m, n), which is color reproduction data of the subject OBJ, is generated from the second imaging data g ⁽²⁾ _RGB (m, n).

When the image data g ^(OUT) _RGB (m, n) is output to a display device having a gamma characteristic, it is preferable to perform a process for canceling the gamma characteristic of the output destination. In this case, the coordinate conversion unit 25 may include a process for giving a gamma characteristic. When the gamma value γd of the display device is used, the process for imparting the gamma characteristic is realized by calculating the power of the gamma value γd for the generated image data g ^(OUT) _RGB (m, n). . Note that, similarly to the

input units

10 and 20 described above, the amount of calculation can be significantly reduced by using a lookup table (LUT).

In the above description, the configuration in which the image data generation unit 24 performs color reproduction under the same illumination environment as when the subject OBJ is imaged is illustrated, but the illumination environment in which color reproduction is performed may be different. That is, the spectral radiance E used by the image data generation unit 24 to generate the image data g ^(OUT) _XYZ (m, n) can be arbitrarily determined.

<Processing procedure>
The processing procedure in the image processing apparatus 1 according to the present embodiment is summarized as follows.

FIG. 4 is a flowchart showing an overall processing procedure in image processing apparatus 1 according to the first embodiment of the present invention.

With reference to FIGS. 1 and 4, first, the input unit 10 captures first imaging data g ⁽¹⁾ _RGB (m, n) obtained by imaging at least part of the light incident on the subject OBJ through the diffusing member 402. Is accepted (step S100). Subsequently, the input unit 10 generates imaging data g ⁽¹⁾ _RGB representing the received first imaging data g ⁽¹⁾ _RGB (m, n) (step S102). Note that the input unit 10 linearizes the first imaging data as necessary.

Next, the estimation matrix calculation unit 12 uses the autocorrelation matrix B of the spectral radiance of the light source candidates that can be used to provide the illumination environment in the subject OBJ, the spectral transmittance f ^{(1) of the} diffusing member 402, and the imaging device 400. The first estimation matrix W ⁽¹⁾ is calculated based on the spectral sensitivity S (step S104). Subsequently, the spectral radiance calculation unit 11 uses the first estimation matrix W ⁽¹⁾ calculated in step S104, and the spectral radiance E ⁽ ^{1) of} the illumination light incident on the subject OBJ from the imaging data g ⁽¹⁾ _RGB. ¹⁾ is calculated (step S106).

Next, the tristimulus value conversion unit 14 calculates tristimulus values X, Y, and Z of the XYZ color system from the spectral radiance E ⁽¹⁾ (step S110). Subsequently, the coordinate conversion unit 15 converts the tristimulus values X, Y, and Z in the XYZ color system to coordinate values R ⁽¹⁾ , G ⁽¹⁾ , and B ⁽¹⁾ defined in the RGB color system. (Step S112). Further, the white balance calculation unit 16 calculates the white balance in the imaging apparatus 400 based on the ratio of the coordinate values R ⁽¹⁾ , G ⁽¹⁾ , B ⁽¹⁾ (step S114).

On the other hand, the input unit 20 receives the second imaging data g ⁽²⁾ _RGB (m, n) obtained by imaging the subject OBJ by the imaging device 400 in an illumination environment (step S120). The input unit 20 linearizes the second imaging data as necessary.

Next, the estimation matrix calculation unit 22 calculates the autocorrelation matrix A calculated from the spectral reflectances of colors that can be included in the subject OBJ, the spectral radiance E ^{(1) of} the illumination light calculated by the illumination spectrum estimation unit 100, Based on the spectral sensitivity S of the imaging device 400, a second estimation matrix W ⁽²⁾ is calculated (step S122). Subsequently, the spectral reflectance calculation unit 21 uses the second estimation matrix W ⁽²⁾ calculated in step S122 to calculate the spectral reflectance f ⁽²⁾ (m ⁾ of the subject OBJ from the second imaging data g ^(2). , N) is calculated (step S124). Further, the image data generation unit 24 uses the color matching function h, the spectral radiance E ^{(1) of} the illumination light incident on the subject OBJ, and the spectral reflectance f ⁽²⁾ (m ^{) of} the subject OBJ calculated in step S124. , N) is used to generate image data g ^(OUT) _XYZ (m, n) obtained by performing color reproduction of the subject OBJ (step S126). Further, the coordinate conversion unit 25 converts the image data g ^(OUT) _XYZ (m, n) generated in step S126 into image data g ^(OUT) _RGB (m, n) defined in the RGB color system. (Step S128), the converted image data g ^(OUT) _RGB (m, n) is output.

<Operational effects of the present embodiment>
According to the first embodiment of the present invention, the spectral radiance of the illumination light applied to the subject OBJ can be calculated using the imaging device for imaging the subject OBJ. Therefore, the spectral radiance can be easily acquired without using a dedicated measuring device for measuring the spectral radiance of the illumination light.

Furthermore, based on the spectral radiance of the illumination light calculated in this way, the spectral reflectance of the subject OBJ is accurately estimated, and then the color that will be imaged (observed) in the illumination environment to be imaged. Can be reproduced appropriately.

Also, since the white balance of the imaging device can be adjusted appropriately based on the spectral radiance of the illumination light, more accurate color reproduction can be realized without being affected by variations in the characteristics of the imaging device.

In the above-described embodiment, the configuration in which the three processes of “spectral radiance calculation processing of illumination light”, “white balance calculation processing”, and “color reproduction processing” are realized by one image processing apparatus is illustrated. However, the problem of the present invention can be solved as long as the apparatus can execute at least “calculation processing of spectral radiance of illumination light”.

[Embodiment 2]
In the first embodiment described above, the configuration in which one type of autocorrelation matrix B is used to estimate the spectral radiance of the illumination light applied to the subject OBJ has been illustrated. On the other hand, it is known that the spectral radiance (spectrum) of illumination light varies greatly depending on the type of light source. This is because the emission spectrum (bright line spectrum) has various unique characteristics depending on the emission principle of the light source. Therefore, it is preferable to selectively use the autocorrelation matrix B generated in advance for each type of light source according to the illumination environment in which the subject OBJ is imaged.

Therefore, the second embodiment exemplifies a configuration in which a plurality of autocorrelation matrices are stored for each type of a plurality of light sources (for each category), and the user can select a suitable one for the illumination environment for imaging the subject OBJ.

<Overall configuration>
FIG. 5 is a functional configuration diagram of an image processing apparatus 1A according to the second embodiment of the present invention.

Referring to FIG. 5, the image processing apparatus 1 A includes an illumination spectrum estimation unit 100 A in place of the illumination spectrum estimation unit 100 in the image processing apparatus 1 shown in FIG. 1. On the other hand, color reproduction unit 200 is similar to color reproduction unit 200 of image processing apparatus 1 shown in FIG. 1, and therefore detailed description thereof will not be repeated.

The illumination spectrum estimation unit 100A is provided with a light source data storage unit 13A in place of the light source data storage unit 13 in the illumination spectrum estimation unit 100 shown in FIG. Since other parts are the same as those in the first embodiment, detailed description will not be repeated.

The light source data storage unit 13A stores in advance an autocorrelation matrix B ₁ , B ₂ ,..., B _M, which is a predetermined calculation matrix for each of M types of light source candidates that can be used to provide an illumination environment. Store. Then, the light source data storage unit 13A selects the selected _{one of} the autocorrelation matrices B ₁ , B ₂ ,..., B _M stored in advance in accordance with an external command from the user or the like. Output to.

Hereinafter, a plurality of autocorrelation matrices stored in the light source data storage unit 13A will be described.

For example, the spectral irradiation luminance (spectrum) of a general fluorescent lamp has a waveform having a peak at a wavelength corresponding to an emission line spectrum of mercury or the like enclosed therein. On the other hand, there is no peak in the spectral illumination luminance (spectrum) of the incandescent lamp due to its emission principle. In this way, the spectral irradiation brightness (spectrum) of the light source candidate is different for each type. Therefore, in order to estimate the spectral radiance E ⁽¹⁾ (illumination spectrum) of the illumination light incident on the subject OBJ, it is necessary to appropriately select the autocorrelation matrix B serving as a reference.

On the other hand, a user having a certain level of prior knowledge can determine what light source is used in the illumination environment when the subject OBJ is imaged using the imaging device 400. For example, whether the subject OBJ is imaged indoors or outdoors, or if the imaged location is indoors, it is possible to determine whether a fluorescent light or an incandescent light is used as the light source. It is. Therefore, if a plurality of autocorrelation matrices are prepared in advance for each type of light source under such a classification that can be determined by the user and the user can arbitrarily select according to the imaging state of the subject OBJ, The estimation accuracy of the spectral radiance E ⁽¹⁾ (illumination spectrum) can be increased.

Therefore, as an example, the light source data storage unit 13A according to the present embodiment includes a plurality of autocorrelation matrices for each type of “fluorescent lamp”, “incandescent lamp”, “xenon lamp”, “mercury lamp”, and “sunlight”. B ₁ , B ₂ ,..., B _M are stored in advance, and in response to a selection command SEL by a user or the like, the corresponding one is output as an autocorrelation matrix B to the estimation matrix calculation unit 12. That is, the autocorrelation matrix B ₁ is generated only from the statistical data of the spectral radiance of “fluorescent lamp”, and the autocorrelation matrix B ₂ is generated only from the statistical data of the spectral radiance of “incandescent lamp”. is there.

The estimation matrix calculation unit 12 estimates the spectral radiance E ⁽¹⁾ of the illumination light based on the autocorrelation matrix B received from the light source data storage unit 13A.

In order to increase the estimation accuracy of the spectral radiance E ⁽¹⁾ of the illumination light, it is preferable to store in advance an autocorrelation matrix that further classifies the type of light source. Furthermore, not only a single light source, but also an autocorrelation matrix may be generated based on the spectral radiance generated when, for example, a “fluorescent lamp” and an “incandescent lamp” are combined. That is, it is preferable to store in advance in the light source data storage unit 13A autocorrelation matrices generated based on various spectral radiances that are assumed as illumination environments when the subject OBJ is imaged.

Other processes are basically the same as those in the first embodiment described above, and thus detailed description will not be repeated.

<Processing procedure>
FIG. 6 is a flowchart showing an overall processing procedure in image processing apparatus 1A according to the second embodiment of the present invention. Of the steps in the flowchart shown in FIG. 6, steps having the same contents as the steps in the flowchart shown in FIG.

With reference to FIGS. 5 and 6, first, the input unit 10 captures at least part of the light incident on the subject OBJ through the diffusion member 402. First imaging data g ⁽¹⁾ _RGB (m, n) Is accepted (step S100). Subsequently, the input unit 10 generates imaging data g ⁽¹⁾ _RGB representing the received first imaging data g ⁽¹⁾ _RGB (m, n) (step S102). Note that the input unit 10 linearizes the first imaging data as necessary.

Next, among the autocorrelation matrices B ₁ , B ₂ ,..., B _M stored in advance by the light source data storage unit 13A, one autocorrelation matrix is estimated as an autocorrelation matrix B according to the selection command SEL. It outputs to the calculation part 12 (step S103). Thereafter, the estimation matrix calculator 12 determines the first estimation matrix W ⁽ based on the autocorrelation matrix B from the light source data storage 13 A, the spectral transmittance f ^{(1) of the} diffusing member 402, and the spectral sensitivity S of the imaging device 400. ¹⁾ is calculated (step S104). Subsequently, the spectral radiance calculation unit 11 uses the first estimation matrix W ⁽¹⁾ calculated in step S104, and the spectral radiance E ⁽ ^{1) of} the illumination light incident on the subject OBJ from the imaging data g ⁽¹⁾ _RGB. ¹⁾ is calculated (step S106).

Next, the estimation matrix calculation unit 22 calculates the autocorrelation matrix A calculated from the spectral reflectances of colors that can be included in the subject OBJ, the spectral radiance E ^{(1) of} the illumination light calculated by the illumination spectrum estimation unit 100, Based on the spectral sensitivity S of the imaging device 400, a second estimation matrix W ⁽²⁾ is calculated (step S122). Subsequently, the spectral reflectance calculation unit 21 uses the second estimation matrix W ⁽²⁾ calculated in step S122 to calculate the spectral reflectance f ⁽²⁾ (m ⁾ of the subject OBJ from the second imaging data g ^(2). , N) is calculated (step S124). Further, the image data generating unit 24 uses the color matching function h, the spectral radiance E ^{(1) of} the illumination light incident on the subject OBJ, and the spectral reflectance f ⁽²⁾ (m, ^{) of} the subject OBJ calculated in step S124. n) is used to generate image data g ^(OUT) _XYZ (m, n) in which color reproduction of the subject OBJ is performed (step S126). Further, the coordinate conversion unit 25 converts the image data g ^(OUT) _XYZ (m, n) generated in step S126 into image data g ^(OUT) _RGB (m, n) defined in the RGB color system. (Step S128), the converted image data g ^(OUT) _RGB (m, n) is output.

<Operational effects of the present embodiment>
According to the second embodiment of the present invention, it is possible to obtain the same effects as those of the first embodiment described above, and by selecting an appropriate autocorrelation matrix according to the imaging situation of the subject OBJ, etc. The spectral radiance of light can be estimated more accurately.

[Modification of Embodiment 2]
In the above-described second embodiment, any one of a plurality of autocorrelation matrices is selected in response to a selection command SEL by a user or the like, and the first estimation matrix is further based on the selected autocorrelation matrix. The configuration in which W ⁽¹⁾ is generated has been illustrated. The first estimation matrix W ⁽¹⁾ is generated using the spectral transmittance f ⁽¹⁾ of the diffusing member 402 and the spectral sensitivity S of the imaging device 400. These values are used for the imaging device 400 and the diffusing member 402. As long as is not exchanged. Therefore, as a modification of the second embodiment, a plurality of auto-correlation matrix _B _1, B 2, · · ·, a plurality of are calculated from _{B M} first estimation matrix ^{_{^{W (1) 1, W (}}} 1) 2 ,..., W ⁽¹⁾ An example of a configuration in which _m is calculated in advance will be described.

<Overall configuration>
FIG. 7 is a functional configuration diagram of an image processing device 1B according to a modification of the second embodiment of the present invention.

Referring to FIG. 7, the image processing apparatus 1 B includes an illumination spectrum estimation unit 100 B instead of the illumination spectrum estimation unit 100 in the image processing apparatus 1 shown in FIG. 1. On the other hand, color reproduction unit 200 is the same as that of image processing apparatus 1 shown in FIG. 1, and therefore detailed description will not be repeated.

The illumination spectrum estimation unit 100B includes an estimation matrix storage unit 17 instead of the estimation matrix calculation unit 12 and the light source data storage unit 13 in the illumination spectrum estimation unit 100 shown in FIG. Since other parts are the same as those in the first embodiment, detailed description will not be repeated.

The estimation matrix storage unit 17 includes first estimation matrices W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ ,... That are calculated in advance based on spectral radiances of a plurality of light source candidates that can be used to provide an illumination environment. ., W ⁽¹⁾ _M is stored in advance. And the estimation matrix storage part 17 respond | corresponds to the external instruction | command from a user etc. among 1st estimation matrix W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ , ..., W ⁽¹⁾ _M stored beforehand. The selected one is output to the spectral radiance calculation unit 11.

The first estimation matrix ^{_{^{_{W (1) 1, W (}}}} 1) 2, ···, W (1) M is, _B 1 is stored in the source data storage section 13A of the image processing apparatus 1A according to the second embodiment, Calculated from B ₂ ,..., B _M. First estimation matrix ^{_{^{_{W (1) 1, W (}}}} 1) 2, ···, W (1) when calculating the _M, the spectral sensitivity of the known spectral transmittance ^{f (1)} of the diffusion member 402 and the image pickup apparatus 400 S Is used.

In the modification of the second embodiment, when at least one of the imaging device 400 and the diffusing member 402 is replaced, B ₁ , B ₂ ,..., B _M stored in the light source data storage unit 13A. Need to be calculated again.

Other processes are basically the same as those in the first or second embodiment described above, and thus detailed description will not be repeated.

<Processing procedure>
FIG. 8 is a flowchart showing an overall processing procedure in image processing apparatus 1B according to the modification of the second embodiment of the present invention. Of the steps in the flowchart shown in FIG. 8, steps having the same contents as those in the flowchart shown in FIG. 4 are denoted by the same reference numerals.

With reference to FIGS. 7 and 8, first, the input unit 10 captures first imaging data g ⁽¹⁾ _RGB (m, n) obtained by imaging at least a part of the light incident on the subject OBJ through the diffusing member 402. Is accepted (step S100). Subsequently, the input unit 10 generates imaging data g ⁽¹⁾ _RGB representing the received first imaging data g ⁽¹⁾ _RGB (m, n) (step S102). Note that the input unit 10 linearizes the first imaging data as necessary.

Next, the estimation matrix storage unit 17 selects one of the first estimation matrices W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ ,..., W ⁽¹⁾ _M stored in advance according to the selection command SEL. One estimation matrix is output to the spectral radiance calculation unit 11 as a first estimation matrix W ⁽¹⁾ (step S105). Subsequently, the spectral radiance calculation unit 11 uses the first estimation matrix W ⁽¹⁾ selected in step S105, and the spectral radiance E ⁽ ^{1) of} the illumination light incident on the subject OBJ from the imaging data g ⁽¹⁾ _RGB. ¹⁾ is calculated (step S106).

<Operational effects of the present embodiment>
According to the modification of the second embodiment of the present invention, the same operational effects as those of the first and second embodiments can be obtained, and a plurality of first estimation matrices W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ ,..., W ^{(1) Since} the calculation process of _M can be simplified, the calculation process can be further speeded up.

[Embodiment 3]
In Embodiment 2 described above, a plurality of autocorrelation matrices are stored in advance for each type of light source data, and the spectral radiance (spectrum) of illumination light is estimated using an arbitrarily selected autocorrelation matrix. The configuration in which is performed is illustrated. On the other hand, in Embodiment 3 described below, the most appropriate estimation result is output after evaluating the estimation result of the spectral radiance of the illumination light using each of the plurality of autocorrelation matrices. The configuration will be exemplified.

<Overall configuration>
The image processing apparatus according to the third embodiment of the present invention is the image processing apparatus 1 according to the first embodiment shown in FIG. 1 except that an illumination spectrum estimation unit 100C is provided instead of the illumination spectrum estimation unit 100. On the other hand, color reproduction unit 200 is the same as that of image processing apparatus 1 shown in FIG. 1, and therefore detailed description will not be repeated.

FIG. 9 is a functional configuration diagram of illumination spectrum estimation unit 100C of the image processing device according to the third embodiment of the present invention. The color reproduction unit 200 included in the image processing apparatus according to the present embodiment is not shown.

Referring to FIG. 9, the illumination spectrum estimation unit 100C includes an input unit 10, spectral

radiance calculation units

11A, 11B, 11C, and 11D, a selection unit 18, an evaluation unit 19, and a tristimulus value conversion unit 14. The coordinate conversion unit 15 and the white balance calculation unit 16 are further included. Among these, since the input unit 10, the tristimulus value conversion unit 14, the coordinate conversion unit 15, and the white balance calculation unit 16 have been described in the first embodiment (FIG. 1), detailed description will be repeated. Absent.

Spectral

radiance calculation units

11A, 11B, 11C, and 11D calculate first estimation matrices W ⁽¹⁾ ₁ , W ⁽ based on spectral radiances of a plurality of light source candidates that can be used to provide an illumination environment. ¹⁾ Spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E of illumination light incident on subject OBJ from imaging data g ⁽¹⁾ using ₂ , W ⁽¹⁾ ₃ , W ⁽¹⁾ ₄ ⁽¹⁾ ₃ and E ⁽¹⁾ Calculate ₄ respectively. The first estimation matrix W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ , W ⁽¹⁾ ₃ , W ⁽¹⁾ ₄ is the first estimation matrix W ⁽¹⁾ stored in the estimation matrix storage unit 17 shown in FIG. ₁ , W ⁽¹⁾ ₂ , W ⁽¹⁾ ₃ , and W ⁽¹⁾ ₄ are substantially the same. That is, the first estimation matrix W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ , W ⁽¹⁾ ₃ , W ⁽¹⁾ ₄ is previously stored for each type of a plurality of light source candidates that can be used to provide an illumination environment. Based on the determined autocorrelation matrices B ₁ , B ₂ , B ₃ , B ₄ , the calculation is performed according to the same procedure as described above. In addition, although four types of 1st estimation matrix W ⁽¹⁾ is illustrated in FIG. 9, as long as there are multiple 1st estimation matrices W ⁽¹⁾ , the number is not restrict | limited. FIG. 9 illustrates a configuration in which the first estimation matrix W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ , W ⁽¹⁾ ₃ , W ⁽¹⁾ ₄ is calculated in advance. Like the illumination spectrum estimation unit 100A, these may be dynamically calculated for each calculation process.

In the present embodiment, as an example, the first estimation matrix W ⁽¹⁾ ₁ is calculated from the autocorrelation matrix B ₁ created based on the fluorescent lamp statistical data, and the first estimation matrix W ⁽¹⁾ ₂ is calculated from the autocorrelation matrix B ₂ created based on the incandescent lamp statistical data, and the first estimation matrix W ⁽¹⁾ ₃ is the autocorrelation matrix created based on the xenon lamp statistical data. It assumed to be calculated from the B _3. Furthermore, it is assumed that the first estimation matrix W ⁽¹⁾ ₄ is calculated from an autocorrelation matrix B ₄ created based on statistical data including all of the fluorescent lamp, the incandescent lamp, and the xenon lamp.

Spectral

radiance calculation units

11A, 11B, 11C, and 11D select the calculated spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , and E ⁽¹⁾ ₄ of the illumination light, respectively. 18 respectively.

The selection unit 18 selects ^one of the spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , and E ⁽¹⁾ ₄ of the input illumination light according to the evaluation result by the evaluation unit 19 described later. Are output as the spectral radiance E ⁽¹⁾ of the illumination light.

The evaluation unit 19 includes spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , E ⁽¹⁾ _{4 of} the illumination light calculated by the spectral

radiance calculation units

11A, 11B, 11C, and 11D, respectively. Of these, the one that is most appropriately estimated is evaluated. More specifically, the evaluation unit 19 compares the spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , E ⁽¹⁾ of the illumination light by comparing with a reference pattern defined in advance. ₄ is evaluated.

In the present embodiment, as an example, calculation of first estimation matrices W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ , W ⁽¹⁾ ₃ (or corresponding autocorrelation matrices B ₁ , B ₂ , B ₃ ), respectively. The reference patterns E ⁽¹⁾ _1AVE , E ⁽¹⁾ _2AVE , E ⁽¹⁾ _3AVE calculated from the spectral radiance (statistical value or actually measured value) of the light source used at times are used. More specifically, for example, a reference pattern ^E _{(1) 1AVE} corresponding to the first estimation matrix ^{W ₍₁₎} _1, the generation of the autocorrelation matrix _{B 1} used in the calculation of the first estimated matrix ^{W ₍₁₎} ₁ It is calculated by averaging each element of the original light source group matrix E _st (see FIG. 3). That is, as shown in FIG. 3, when the group matrix E _st of the light source candidate i is composed of the component values e _i (λ _j ) {1 ≦ i ≦ N, 1 ≦ j ≦ k}, the corresponding reference pattern E ^₍₁₎ When each of the _AVE sampling wavelength lambda _j component values in (1 ≦ j ≦ k) and _{e AVE} (lambda _j), the following relationship is established.

Component value e _AVE (λ _j ) = {e ₁ (λ _j ) + e ₂ (λ _j ) +... + E _N (λ _j )} / N
In the present embodiment, according to such a calculation procedure, spectral radiance (spectrum) representative of each of a fluorescent lamp, an incandescent lamp, and a xenon lamp is calculated in advance as a reference pattern. Note that the reference pattern corresponding to the first estimation matrix W ⁽¹⁾ ₄ is not necessarily calculated. This is because the autocorrelation matrix B ₄ corresponding to the first estimation matrix W ⁽¹⁾ ₄ is created based on statistical data including all of the fluorescent lamp, the incandescent lamp, and the xenon lamp. even create a reference pattern from the B _4, will blurred characteristics of each light source, because the effect of the reference pattern Usumaru.

Next, with reference to FIG. 10 and FIG. 11, the evaluation unit 19 evaluates the spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , E ⁽¹⁾ ₄ of the illumination light. explain.

FIG. 10 shows the spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ and the reference pattern E ⁽¹⁾ _1AVE , E ⁽¹⁾ _2AVE , E ⁽¹⁾ of the illumination light by the evaluation unit 19. It is a figure for _{demonstrating} the comparison process with _3AVE . FIG. 11 is a diagram for explaining the similarity calculation process in FIG. 10.

Referring to FIG. 10, the evaluation unit 19 includes spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ _{3 of} illumination light calculated by the spectral

radiance calculation units

11A, 11B, and 11C, respectively. The reference patterns E ⁽¹⁾ _1AVE , E ⁽¹⁾ _2AVE , E ⁽¹⁾ _3AVE are respectively compared, and the comparison result (similarity as an example) is calculated. The spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ and the reference pattern E ⁽¹⁾ _1AVE , E ⁽¹⁾ _2AVE , E ⁽¹⁾ _3AVE are all 0. It is assumed that it is standardized in the range of ~ 1.

As shown in FIG. 10A, in the reference pattern E ⁽¹⁾ _1AVE created based on the statistical data of fluorescent lamps, there is a peak at a specific wavelength. On the other hand, as shown in FIG. 10B, in the reference pattern E ⁽¹⁾ _2AVE created based on the statistical data of the incandescent lamp, there is no peak, and the luminance increases as the wavelength increases. I understand. In addition, as shown in FIG. _10C , in the reference pattern E ⁽¹⁾ _3AVE created based on the statistical data of the xenon lamp, _there is a slight peak and it is high over almost the entire visible light region. It turns out that it has a brightness | luminance.

The evaluation unit 19 evaluates how similar each spectral radiance is to the corresponding reference pattern. Typically, the evaluation unit 19 calculates the similarity based on the deviation between the waveforms on the wavelength region.

FIG. 11 is a diagram for explaining a comparison process between the spectral radiance E ⁽¹⁾ _{1 of} the illumination light calculated by the spectral radiance calculation unit 11A and the reference pattern E ⁽¹⁾ _1AVE . FIG. 11A shows a state where the spectral radiance E ⁽¹⁾ ₁ of the illumination light and the reference pattern E ⁽¹⁾ _1AVE are plotted on the same wavelength region, and FIG. The process which calculates a deviation is shown.

The evaluation unit 19 calculates the deviation (standardized value) err _j between the spectral radiance E ⁽¹⁾ ₁ of the illumination light and the reference pattern E ⁽¹⁾ _{1AVE at} each sampling wavelength λ _j (1 ≦ j ≦ k). (1 ≦ j ≦ k) is calculated sequentially. Subsequently, the evaluation unit 19 calculates an evaluation result (similarity) by calculating a total average of all the deviations err _j of the sampling wavelength λ _j . That is, the similarity SM can be calculated by the following arithmetic expression using the deviation err _j of the sampling wavelength λ _j .

Similarity SM = {1- (err ₁ + err ₂ +... + Err _k ) / k} × 100 [%]
FIG. 10 shows the measured similarity when the image processing method according to the present embodiment is performed under the illumination environment of a fluorescent lamp. As a result of the similarity calculation process, the similarity of the spectral radiance E ⁽¹⁾ ₁ of the illumination light estimated based on the first estimation matrix W ⁽¹⁾ ₁ is the highest. The spectral radiance E ⁽¹⁾ ₁ of the illumination light is output as the spectral radiance E ⁽¹⁾ . Moreover, this evaluation result agrees with the fact that it was actually measured under the fluorescent lamp illumination environment.

Further, when the illumination environment of the subject OBJ is provided by combining a fluorescent lamp, an incandescent lamp, and a xenon lamp, or provided by a light source other than these, the spectral radiance E ⁽¹⁾ of the illumination light. ⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ For any of ₃ , it is assumed that the degree of similarity is not sufficiently high. In such a case, the spectral radiance E ⁽¹⁾ ₄ of the illumination light estimated based on the first estimation matrix W ⁽¹⁾ ₄ reflecting all the characteristics of the fluorescent lamp, the incandescent lamp, and the xenon lamp is used as the illumination light. It may be appropriate to output as the spectral radiance E ⁽¹⁾ . Therefore, the evaluation unit 19 is used when the evaluation results (similarities) for the spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ of the illumination light are all below the allowable value. outputs the spectral radiance ^{E ₍₁₎} ₄ as the spectral radiance ^E of the illumination light ^(1).

In the above description, the spectral

radiance calculation units

11A, 11B, 11C, and 11D perform spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , E ⁽¹⁾ _{4 of} illumination light. However, the calculation process of the spectral radiance of the illumination light and the calculation process of the similarity degree are sequentially executed for each of the first estimation matrices. May be. In this case, when it is determined that the similarity with respect to the spectral radiance of the illumination light estimated by any of the first estimation matrices exceeds a predetermined threshold (for example, 95%), Since the spectral radiance of the dimension can be output without performing the process, the process can be simplified.

Further, there is no necessity to estimate the spectral radiance E ⁽¹⁾ ₄ of the illumination light based on the first estimation matrix W ⁽¹⁾ ₄ . In other words, if the similarities of the spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , and E ⁽¹⁾ ₃ of the illumination light are all below the allowable value, the estimation impossible is output. Good.

Also, instead of calculating the similarity based on the deviation described above, the similarity may be calculated using a correlation coefficient or the like.

<Processing procedure>
The processing procedure in the image processing apparatus according to the present embodiment is summarized as follows.

FIG. 12 is a flowchart showing an overall processing procedure in the image processing apparatus according to the third embodiment of the present invention. Of the steps in the flowchart shown in FIG. 12, steps having the same contents as those in the flowchart shown in FIG. 4 are denoted by the same reference numerals. FIG. 13 is a flowchart showing the procedure of the evaluation subroutine shown in step S108 shown in FIG.

With reference to FIGS. 9 and 12, first, the input unit 10 captures at least part of the light incident on the object OBJ through the diffusion member 402. First imaging data g ⁽¹⁾ _RGB (m, n) Is accepted (step S100). Subsequently, the input unit 10 generates imaging data g ⁽¹⁾ _RGB representing the received first imaging data g ⁽¹⁾ _RGB (m, n) (step S102). Note that the input unit 10 linearizes the first imaging data as necessary.

Next, the spectral

radiance calculation units

11A, 11B, 11C, and 11D capture images using the first estimation matrices W ⁽¹⁾ ₁ , W ⁽¹⁾ ₂ , W ⁽¹⁾ ₃ , and W ⁽¹⁾ ₄ , respectively. Data g ⁽¹⁾ Spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , E ⁽¹⁾ ₄ of the illumination light incident on the subject OBJ from _RGB is calculated (step S107). Subsequently, the evaluation unit 19 executes an evaluation subroutine, and the spectral radiances E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ ₃ , and E ⁽¹⁾ ₄ of the illumination light calculated in step S107. Among them, the one with the highest estimation accuracy is evaluated (step S108). Further, the selection unit 18, according to the evaluation result in step S108, one of the spectral radiance ^E of the illumination light _{^{_{^{(1) 1, E (1}}}} ) 2, E (1) 3, E (1) 4, lighting Output as spectral radiance E ⁽¹⁾ of light (step S109).

Referring to FIG. 13, the evaluation unit 19 compares the spectral radiance E ⁽¹⁾ ₁ of the illumination light with a predetermined reference pattern E ⁽¹⁾ _1AVE , thereby determining the similarity SM ₁ between the two. Is calculated (step S200). Similarly, the evaluation unit 19 compares the spectral radiance E ⁽¹⁾ ₂ of the illumination light with a predetermined reference pattern E ⁽¹⁾ _2AVE to calculate the similarity SM ₂ between them ( Step S202). Similarly, the evaluation unit 19 compares the spectral radiance E ⁽¹⁾ ₃ of the illumination light with a predetermined reference pattern E ⁽¹⁾ _3AVE to calculate a similarity SM ₃ between them ( Step S204).

Subsequently, the evaluation unit 19 extracts the one having the highest value among the similarities SM ₁ , SM ₂ , SM ₃ calculated in steps S200, S202, S204 (step S206). Furthermore, the evaluation unit 19 determines whether or not the similarity extracted in step S206 is greater than or equal to a predetermined allowable value (step S208).

If the similarity is equal to or greater than a predetermined allowable value (YES in step S208), the evaluation unit 19 evaluates that the spectral radiance corresponding to the similarity extracted in step S206 has the highest estimation accuracy ( Step S210).

On the other hand, when the similarity is not equal to or greater than the predetermined allowable value (NO in step S208), the evaluation unit 19 evaluates the spectral radiance E ⁽¹⁾ ₁ , E ⁽¹⁾ ₂ , E ⁽¹⁾ of the illumination light. _It is evaluated that the spectral radiance E ⁽¹⁾ _{4 of} illumination light other than ₃ has the highest estimation accuracy (step S212).

Thereafter, the processing proceeds to step S109 in FIG.
<Operational effects of the present embodiment>
According to the third embodiment of the present invention, the same operational effects as those of the above-described first embodiment can be obtained, and the spectral radiance of illumination light is high even for a user who does not have any prior knowledge. It can be obtained with estimation accuracy. Therefore, even if the subject OBJ is performed under various conditions, the estimation accuracy of the spectral radiance of the illumination light can be maintained.

[Modifications of Embodiments 1 to 3]
In the above-described first to third embodiments, the configuration using the image processing apparatus mainly configured by hardware is exemplified, but all or a part thereof may be realized by software. That is, the processing in the image processing apparatus may be realized using a computer.

FIG. 14 is a schematic configuration diagram of a computer that realizes an image processing apparatus 1 # according to a modification of the embodiment of the present invention.

Referring to FIG. 14, the computer includes a computer main body 150 equipped with an FD (Flexible Disk) driving device 166 and a CD-ROM (Compact Disk-Read Only Memory) driving device 168, a monitor 152, a keyboard 154, a mouse. 156.

The computer main body 150 further includes a CPU (Central Processing Unit) 160 that is an arithmetic device, a memory 162, a fixed disk 164 that is a storage device, and a communication interface 170 that are connected to each other via a bus.

FD 166a is mounted on the FD drive device 166, and CD-ROM 168a is mounted on the CD-ROM drive device 168. Image processing apparatus 1 # according to the modification of the present embodiment can be realized by CPU 160 executing software using computer hardware such as memory 162. In general, such software is stored in a recording medium such as the FD 166a or the CD-ROM 168a, or distributed via a network or the like. Such software is read from the recording medium by the FD driving device 166 or the CD-ROM driving device 168 or received by the communication interface 170 and stored in the fixed disk 164. Further, it is read from the fixed disk 164 to the memory 162 and executed by the CPU 160.

The monitor 152 is a display unit for displaying information output by the CPU 160, and includes, for example, an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), and the like. The mouse 156 receives a command from a user corresponding to an operation such as click or slide. The keyboard 154 receives a command from the user corresponding to the input key. The CPU 160 is an arithmetic processing unit that executes various arithmetic operations by sequentially executing programmed instructions. The memory 162 stores various types of information according to the program execution of the CPU 160. The communication interface 170 converts the information output from the CPU 160 into, for example, an electrical signal and sends it to another device, and receives the electrical signal from the other device and converts it into information that can be used by the CPU 160. Fixed disk 164 is a non-volatile storage device that stores programs executed by CPU 160 and predetermined data. In addition, other output devices such as a printer may be connected to the computer as necessary.

Furthermore, the program according to the present embodiment is a program module that is provided as a part of a computer operating system (OS) and calls necessary modules in a predetermined arrangement at a predetermined timing to execute processing. There may be. In that case, the program itself does not include the module, and the process is executed in cooperation with the OS. A program that does not include such a module can also be included in the program according to the present invention.

Further, the program according to the present embodiment may be provided by being incorporated in a part of another program. Even in this case, the program itself does not include the module included in the other program, and the process is executed in cooperation with the other program.

The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

Claims

An image processing apparatus (1) capable of performing image processing on imaging data captured by an imaging apparatus (400),
An input unit (10) that receives first imaging data obtained by imaging at least a part of light incident on the subject (OBJ) through the diffusing member (402) under the illumination environment using the imaging device. )When,
A first estimation matrix calculated based on an autocorrelation matrix of spectral radiance of a light source candidate that can be used to provide the illumination environment, a spectral transmittance of the diffusing member, and a spectral sensitivity of the imaging device And a first calculation unit (11, 12, 13) that calculates a spectral radiance of illumination light incident on the subject from the first imaging data.
The image processing apparatus according to claim 1, wherein the spectral radiance of the light source candidate is a characteristic value acquired in advance for each type of light source (300).
The diffusing member is disposed on the optical axis (Ax1) of the imaging device,
The image processing apparatus according to claim 1, wherein the incident intensity of the diffusing member is indicated by a predetermined function value with respect to an angle with respect to the optical axis.
The image processing device according to claim 3, wherein the function value is a cosine function with respect to an angle with respect to the optical axis.
The imaging device is configured to output coordinate values defined in the RGB color system as the imaging data,
The image processing apparatus includes:
A second calculation unit (14, 15) for calculating coordinate values in the RGB color system corresponding to the spectral radiance of the illumination light using the spectral radiance of the illumination light and the color matching function;
The image processing device according to claim 1, further comprising a third calculation unit (16) that calculates a white balance in the imaging device based on a ratio of coordinate values calculated in the second calculation unit. .
Using the second estimation matrix calculated based on the spectral radiance of the illumination light, the spectral sensitivity of the imaging device, and the autocorrelation matrix of the spectral reflectance of colors that can be included in the subject, the illumination The first calculation unit (20, 21, 22) further calculating a spectral reflectance of the subject from second imaging data obtained by imaging the subject with the imaging device under an environment. The image processing apparatus according to item.
A generation unit (24, 25) that generates image data acquired when the subject is imaged under a predetermined illumination environment based on the spectral reflectance of the subject calculated by the fourth calculation unit. The image processing apparatus according to claim 6.
An image processing apparatus (1A) capable of performing image processing on imaging data captured by an imaging apparatus (400),
An input unit (10) that receives first imaging data obtained by imaging at least part of light incident on the subject (OBJ) through the diffusing member (402) using the imaging device under an illumination environment. )When,
A selection unit (13A) that selects one of a plurality of calculation matrices predetermined for each type of light source candidates that can be used to provide the illumination environment according to an external command;
Using the first estimation matrix calculated based on the calculation matrix selected by the selection unit, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device, the subject from the first imaging data A first calculation unit (11, 12) for calculating the spectral radiance of the illumination light incident on
Each of the calculation matrices is an autocorrelation matrix of a matrix indicating a spectral radiance of the light source candidate.
An image processing device (1B) capable of performing image processing on imaging data imaged by an imaging device (400),
An input unit (10) that receives first imaging data obtained by imaging at least part of light incident on the subject (OBJ) through the diffusing member (402) using the imaging device under an illumination environment. )When,
A selection unit (17) that selects one of a plurality of first estimation matrices calculated in advance based on spectral radiances of a plurality of light source candidates that can be used to provide the illumination environment according to an external command. When,
A first calculation unit (11) that calculates a spectral radiance of illumination light incident on the subject from the first imaging data using the first estimation matrix selected by the selection unit;
The first estimation matrix is calculated based on an autocorrelation matrix of a matrix indicating spectral radiance of a corresponding light source candidate, a spectral transmittance of the diffusing member, and a spectral sensitivity of the imaging device. apparatus.
An image processing apparatus capable of performing image processing on imaging data captured by an imaging apparatus (400),
An input unit (10) that receives first imaging data obtained by imaging at least part of light incident on the subject (OBJ) through the diffusing member (402) using the imaging device under an illumination environment. )When,
By using a plurality of first estimation matrices respectively calculated based on spectral radiances of a plurality of light source candidates that can be used to provide the illumination environment, illumination light incident on the subject from the first imaging data A first calculation unit (11A, 11B, 11C, 11D) for calculating spectral radiance candidates,
An evaluation unit (18, 19) that evaluates each calculated spectral radiance candidate by comparison with a predetermined reference pattern, and outputs one of them as a spectral radiance of illumination light in the illumination environment; With
The first estimation matrix is calculated based on an autocorrelation matrix of a matrix indicating spectral radiance of a corresponding light source candidate, a spectral transmittance of the diffusing member, and a spectral sensitivity of the imaging device. apparatus.
Using the imaging device (400), acquiring at least a part of light incident on the subject (OBJ) under an illumination environment through the diffusion member (402) to obtain first imaging data (S100, S102) )When,
A first estimation matrix calculated based on an autocorrelation matrix of spectral radiance of a light source candidate that can be used to provide the illumination environment, a spectral transmittance of the diffusing member, and a spectral sensitivity of the imaging device And (S104, S106) calculating the spectral radiance of the illumination light incident on the subject from the first imaging data.
Using the imaging device (400), acquiring at least a part of light incident on the subject (OBJ) under an illumination environment through the diffusion member (402) to obtain first imaging data (S100, S102) )When,
Selecting one of a plurality of operation matrices predetermined for each type of light source candidates that can be used to provide the illumination environment (S103);
Illumination incident on the subject from the first imaging data using a first estimation matrix calculated based on the selected calculation matrix, the spectral transmittance of the diffusing member, and the spectral sensitivity of the imaging device Calculating spectral radiance of light (S104, S106),
Each of the calculation matrices is an image processing method, which is an autocorrelation matrix of matrices indicating spectral radiances of light source candidates.
Using the imaging device (400), acquiring at least a part of light incident on the subject (OBJ) under an illumination environment through the diffusion member (402) to obtain first imaging data (S100, S102) )When,
Selecting one of a plurality of first estimation matrices calculated in advance based on spectral radiances of a plurality of light source candidates that may be used to provide the illumination environment (S105);
Calculating a spectral radiance of illumination light incident on the subject from the first imaging data using the selected first estimation matrix (S106),
The first estimation matrix is calculated based on an autocorrelation matrix of a matrix indicating spectral radiance of a corresponding light source candidate, a spectral transmittance of the diffusing member, and a spectral sensitivity of the imaging device. Method.
Using the imaging device (400), acquiring at least a part of light incident on the subject (OBJ) under an illumination environment through the diffusion member (402) to obtain first imaging data (S100, S102) )When,
By using a plurality of first estimation matrices respectively calculated based on spectral radiances of a plurality of light source candidates that can be used to provide the illumination environment, illumination light incident on the subject from the first imaging data Calculating each spectral radiance candidate (S107);
Evaluating each of the calculated spectral radiance candidates by comparison with a predetermined reference pattern, and outputting one of them as spectral radiance of illumination light in the illumination environment (S108, S109); Prepared,
The first estimation matrix is calculated based on an autocorrelation matrix of a matrix indicating spectral radiance of a corresponding light source candidate, a spectral transmittance of the diffusing member, and a spectral sensitivity of the imaging device. Method.