JPH04361386A - Highlight area extraction device of color image - Google Patents

Abstract

PURPOSE:To correctly extract an highlight area even when a texture whose color is cognate with the color component of a light source and which has a high luminance exists on the surface of an object. CONSTITUTION:A highlight candidate detection means 10 consisting of a means similar to a conventional one outputs the candidate of the highlight area from a stereo image inputted by RGB camera parts 2A and 2B. On the other hand, a distance image generating part 11 converts the stereo image into a distance image, a normal vector calculation part 12 obtains a normal vector and a light direction calculation part 13 infers the direction of the light source by using it. The number of inference times calculation part 14 obtains the position of the light source from the concentrating rate of lines showing the obtained light source direction and a highlight area decision part 15 desides a correct highlight area from the candidates of the highlight based on the obtained position of the light source.

Description

[Detailed description of the invention]

0001

[Industrial Application Field] The present invention estimates the direction of a light source from an image containing a mixture of highlights and areas with colors and brightness similar to the highlights, using distance information possessed by stereo images. , relates to a highlight region extracting device that extracts a correct highlight region, and relates to a technique necessary for measuring the surface reflectance of an object.

0002

[Prior Art] When measuring the surface reflectance of an object by photographing with a camera, it is possible to measure the diffusely reflected portion, but the highlight area, which results in a measurement result that reflects the color of the light source, cannot be measured due to the light source. The correct measurement result can be obtained by subtracting the resulting light source color. Therefore, a technique to correctly extract highlight areas is required.

[0003] Conventionally, methods for extracting highlight areas from images have been based on the differences in the reflection characteristics of light on the surface of an object, that is, specular reflection, which reflects the spectral distribution of the light source as it is;
A typical method utilizes two reflection components: one that enters the object and is refracted by pigment particles, and the other is diffuse reflection that exits on the surface (see Reference [1] below). This method maps the value of each pixel to a three-dimensional color space,
Use that distribution. As a three-dimensional color space, red (R
), green (G), and blue (B) as the three primary colors is common, so this will be used in the following description. When the R, G, and B intensities are measured for each pixel, if the area has the same surface reflectance, R
It forms a linear distribution that can be approximated by one locus 10 in the GB color space. When a highlight region exists in the target image, this locus 10 can be divided into approximately two straight lines: one due to specular reflection (straight line 11) and one due to diffuse reflection (straight line 12). Here, the straight asymptote (indicated by a broken line) due to specular reflection indicates the component of the light source. A highlight region is obtained by extracting a region corresponding to specular reflection on the locus l0.

An example of the configuration of a conventional color image highlight region extracting device is shown in the block diagram of FIG. In FIG. 2, 2 is an RGB camera section, 3 is a clustering section, 4 is a trajectory detection section, 5 is a component separation section, and 7 is a highlight area detection section.

Next, the operation will be explained. First, R.G.
A color image is input by the B camera section 2. The clustering unit 3 maps the intensities of the RGB components from the color image onto the RGB color space, and divides the obtained trajectory into several clusters (references [1] to [3] below).
]reference). A locus detector 4 finds one locus l0 representative of each cluster, and a component separator 5 separates it into two straight lines, a straight line l1 due to specular reflection and a straight line l2 due to diffuse reflection. Finally, the highlight area detection unit 7 detects a set of points approximated by the straight line l1 due to specular reflection as a highlight area.

[0006]

[Problem to be Solved by the Invention] As mentioned above, the conventional highlight region extraction device extracts highlight regions from images taken with a single camera.
The brightness is somewhat high (same as the true highlight area),
and objects that are similar to the light source color, that is, conventionally target objects with uniform surface reflectance without texture, and when there is a texture on the surface of the object that has a similar color to the color component of the light source and has high luminance. had the disadvantage that the area was detected as a highlight area.

The present invention was made to solve the above-mentioned drawbacks, and even if there is a texture on the surface of an object that is similar in color to the color component of the light source and has high brightness, it may be confused with the texture. It is an object of the present invention to provide a highlight region extracting device for a color image that can correctly extract highlight regions without having to do so.

[0008]

[Means for Solving the Problems] A color image highlight region extraction device according to the present invention uses highlight regions obtained by conventional methods as highlight region candidates, and calculates distances for converting a color stereo image into a distance image. An image generation section, a normal vector calculation section that calculates a normal vector from a distance image for each pixel, and a light source direction calculation section that estimates the direction of the light source using the normal vector obtained from this normal vector calculation section. and an estimated frequency calculation unit that determines the position of the light source from the concentration rate of straight lines indicating the light source direction obtained by the light source direction calculation unit, and a point where the light source direction estimated from among the highlight area candidates is the same. The highlight area determination unit extracts only highlight area candidates concentrated in the area as correct highlight areas.

[0009]

[Operation] In the present invention, a target scene is inputted from two cameras to obtain a color stereo image. This is converted into a distance image (see Reference [4] below), and the shape of the object included in the image is obtained from this distance image. In addition, from the same color stereo image, the R, G, and B intensities at each pixel are automatically mapped to the RGB color space,
The distribution is clustered by grouping similar items together, and then one trajectory l0 is created for each cluster.
is determined as a representative (see Figure 3). If the reflected light distribution represented by this trajectory l0 includes a specular reflection component, it can be separated from the diffuse reflection component.
1) and the diffuse reflection component (straight line 12). Of the two straight lines obtained, a set of points approximated by the straight line l1 having a specular reflection component is selected as a highlight area candidate. These candidates include portions that have colors similar to the color components of the light source and have high intensity due to the influence of texture. Therefore, by estimating the direction of the light source, only the correct highlight area is extracted. Note that FIG. 3 shows that the chromaticity plane intersects perpendicularly with the straight line (broken line) representing the component of the light source.

[0010] The principle of generation of highlight areas will be explained below using FIG. 4. In Figure 4, part of the incident light is reflected as is from the object surface (specular reflection, shown by the solid line), and the rest is refracted by the internal pigment and exits the surface (
Diffuse reflection, shown as a dashed line). When looking at points A and B from the observation point, at point A, the angle of reflection (θ) is equal to the angle of incidence (θ) from the light source, and the direction of the reflected light matches the direction of the observation point. , a highlight area occurs here, but point B
In this case, the direction of the reflected light and the direction of the observation point do not match, so if the surface is of uniform color, no highlight area will occur here.

Now, since the observation direction is obvious, in order to find the light source direction in the highlight area, it is sufficient to know the normal direction. The normal vector can be obtained by spatially differentiating the distance image. The light source direction is estimated so that the angle between the normal vector obtained in this way and the light source direction is equal to the angle between the normal vector and the observation direction.

Among the candidate light source directions, the higher the probability that they are concentrated at the same point, the more likely it is the direction of the true light source. Estimate the height of the rate that is the area. If the number of light sources is known, it is adopted as a true highlight area and displayed on the screen.

Next, using FIG. 5, the direction of the light source is estimated from the observation direction and normal vector at a certain point on the surface,
The steps to obtain the correct highlight area are shown. In Figure 5, the observation point is V, and the points C, D, E, F, G on the object surface
are points that have been selected as highlight area candidates, and N represents the normal vector at each point. Direction of light source lC, lD, lE from observation direction and normal vector at each point
,lF,lG are estimated. Straight line l representing the direction of the light source
D , lF , and lG intersect at one point I, but lE only intersects lD , and lC has no intersection. Therefore, in FIG. 5, point I is determined to be the position of the light source. Points D, F, and G, where the direction of point I is estimated to be the light source direction, are assumed to be true highlight areas, and points C and E are areas that were incorrectly selected as highlight area candidates due to texture. Reject from the light area candidates. [References] [1] Gudrun J. Klinker, Steve
nA. Shafer, and Takeo Kane
de “USING A COLOR REFLECTI
ON MODEL TO SEPARATE HIGH
LIGHTS FROM OBJECT COLOR
”International Conference
on Computer Vision (ICCV)
pp. 145-150, 1987 [2] Gudrun
J. Klinker, Steven A. Shafer
, and Takeo Kanede “The Me
assurance of Highlights
in Color images”International
onal Journal on Computer
Vision (IJCV) 2(1), 1988 [3] Gudrun J. Klinker, Steve
nA. Shafer, and Takeo Kane
de “Color Image Analysis
With an Intrinsic Reflect
ion Model ”International
Conference on Computer Vis
ion (ICCV) Tarpon Springs, F
L, Dec. 5-8, 1988 [4] Yuichi Ota, Yasuyuki Masai, Katsuo Ikeda "Segmental correspondence method for stereo images using dynamic programming" Journal of the Institute of Electronics and Communication Engineers '85/4 Vol. J68-D
No. 4

[0014]

Embodiments Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows a block diagram of an embodiment of a color image highlight region extracting apparatus according to the present invention. Here, for simplicity, a case where there is one point light source will be described. In Figure 1, the highlight area extraction device is RGB
Camera units 2A, 2B, clustering unit 3, trajectory detection unit 4, component separation unit 5, highlight candidate detection unit 6, distance image generation unit 11, normal vector calculation unit 12, light source direction calculation unit 13, estimated frequency calculation unit 14, Highlight area determination unit 15
Consisting of 2A and 2B constitute a stereo image input means 1, and 3 to 6 constitute a highlight candidate detection means 10.

Color stereo images are input by the RGB camera sections 2A and 2B and sent to the clustering section 3 and the distance image generation section 11. In the clustering unit 3, the intensity of R, G, and B at each pixel of the color stereo image is
The distribution is mapped to the GB color space, similar distributions are grouped together into clusters, and the trajectory detecting section 4 determines one representative trajectory for each cluster. Then, the component separation unit 5 converts the trajectory obtained by the trajectory detection unit 4 into 2
The highlight candidate detection unit 6 selects a set of points that are approximated by a straight line extending toward the higher intensity one of the two straight lines obtained by the component separation unit 5 in the RGB color space. List it as a candidate for the light area.

On the other hand, the distance image generation section 11 receives the output from the RGB camera sections 2A and 2B, and converts the color stereo image into a color stereo image using a dynamic programming method (see reference [4] mentioned above) or a triangulation method. Obtain a distance image and normal vector calculation unit 1
2, find the normal vector for each pixel. The light source direction calculation unit 13 estimates the light source direction from the normal vector obtained by the normal vector calculation unit 12. Now, assuming that there is one light source, the estimated power calculation section 14 uses the light source direction calculation section 1.
The point where the light source directions estimated in step 3 intersect with each other is considered to be the light source position. Finally, the highlight area determination unit 15
Then, among the highlight region candidates obtained by the highlight candidate detection means 10, only those whose estimated light source direction is toward the light source position estimated by the estimated frequency calculation section 14 are extracted as correct highlight regions. .

[0017] Here, the number of light sources is one, but the method can be applied even if there are multiple light sources if the colors of the light sources are exactly the same or if each cluster obtained by the clustering section 3 can be clearly separated. . Furthermore, the space in which the intensities of R, G, and B in a color stereo image are mapped by the clustering unit 3 does not need to have three orthogonal components, that is, R, G, and B, and can be applied even if it has only two components. can.

[0018]

Effects of the Invention As explained above, the present invention calculates the distance from images taken by two cameras to an imaging point, creates a distance image by adding the calculated distance information to each pixel, and The color components of each pixel of an image captured by a camera are mapped spatially, and a set of points that can be approximated by a straight line due to specular reflection is selected as a candidate for a highlight area. Find the normal vector for the point on the object corresponding to the pixel from the distance information of each pixel in the distance image, and based on the principle of reflection, add the angle between that normal vector and the reflected light to the opposite side of the normal. Obtain a straight line that can be regarded as the direction of the light source. If these straight lines are densely packed to a certain extent, it is estimated as the position of the light source, and the area where this light source is obtained is determined to be the highlight area among the highlight area candidates. When extracting a highlight area using specular reflection and diffuse reflection, it is possible to correctly extract the highlight area without being affected by texture or the like. In other words, even if the brightness and color are similar, if the light source direction cannot be determined by estimation, it will not be detected as a highlight region, and if it is not a true highlight region, the light source direction estimated from the normal vector will not be detected. Since the area is likely to be scattered and the light source is not fixed, the probability of incorrectly determining that this area is a highlight area due to reflection from the light source is reduced. In this manner, by correctly extracting the highlight region, it is possible to correctly estimate the color of the light source and estimate the surface reflectance with the influence of the light source removed.

[Brief explanation of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

FIG. 2 is a block diagram showing the configuration of a conventional color image cutting device.

[Figure 3] RGB values of each pixel in a color stereo image
It is a diagram when mapping to color space.

FIG. 4 is a diagram showing the principle of generation of highlight areas.

FIG. 5 is a diagram showing a procedure for extracting a correct highlight area from estimation of a light source direction.

[Explanation of symbols]

1 Stereo image input means 2A RGB camera section 2B RGB camera section 3 Clustering section 4 Trajectory detection section 5 Component separation section 6 Highlight candidate detection section 7 Highlight candidate detection section 10 Highlight candidate detection means 11 Distance image generation unit 12 Normal vector calculation section 13 Light source direction calculation section 14 Estimated frequency calculation section 15 Highlight area determination section

Claims (1)

[Claims] [Claim 1] A stereo image input means using two cameras, and a highlight generated due to specular reflection on a glossy surface from the intensity of three primary color components in each pixel of the image obtained by the stereo image input means. A highlight candidate detection means for detecting region candidates and a stereo image obtained by the stereo image input means calculate the distance between the camera and the object, and add the distance information to each pixel of the photographed image. a distance image generation section; a normal vector calculation section that calculates a normal vector of a point on the object corresponding to each pixel from the distance information of the distance image generation section; and a point on the object corresponding to the stereo image input means. , a light source direction calculation unit that calculates a straight line indicating the light source direction from the positional relationship with the normal vector; and an estimation unit that determines the position of the light source from the concentration rate of the straight lines indicating the light source direction obtained by the light source direction calculation unit. A frequency calculation section and highlight region determination, from among the highlight region candidates obtained by the highlight candidate detection means, a highlight region that is directed toward the light source position estimated by the estimated frequency calculation section. 1. A color image highlight region extraction device comprising: