JP2004021388A - Image processing apparatus and photographing system including the same - Google Patents

Abstract

An object of the present invention is to provide a technique capable of sequentially generating images under predetermined lighting conditions in real time.
A reflection coefficient calculating means for calculating a reflection coefficient of a subject imaged as an input image, a new lighting condition setting means for setting an illumination condition different from the input image, the set illumination condition and the reflection coefficient An illumination condition adding unit that generates an image of the illumination condition set by the new illumination condition setting unit from the input image based on the input image, wherein the input image includes an image including depth information of the subject. The reflection coefficient calculation means comprises: a correction means for specifying a specular reflection area from the input image based on the depth information of the subject to correct specular reflection pixels; and a reflection coefficient of the subject from the corrected input image. And a reflection coefficient calculating means for calculating the coefficient.

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus and an imaging system including the same, and more particularly, to a technique for generating a captured image that can be expected under another illumination condition from a captured image of a subject continuously captured under a predetermined illumination condition. It is.
[0002]
[Prior art]
In a conventional virtual studio system, a prepared video (background video) serving as a synthesis source and lighting conditions equivalent to the background video are reproduced in the studio, and performers (subjects) photographed under the lighting conditions are reproduced. By combining the video with the video of (2), a synthesized video without a sense of incongruity was obtained in real time.
[0003]
However, it is impossible to continuously reproduce the lighting conditions that change every moment in the studio, and the lighting conditions that can be reproduced in the studio are limited. For this reason, in a virtual studio system using lighting conditions that change every moment or background images that cannot be reproduced in the studio, typical lighting conditions are reproduced, and the performers perform in these typical lighting conditions. As a result, it is configured to obtain a composite image having a small uncomfortable feeling.
[0004]
On the other hand, with the progress of CG (computer graphic) technology in recent years, lighting conditions expressed in a background image have become complicated, and it is difficult to reproduce lighting conditions that match the background image in a studio. .
[0005]
As a technique for solving this problem, generation of a video under illumination conditions equivalent to a background video by performing three-dimensional CG processing on a video of a performer has been studied. According to the method using the three-dimensional CG technique, the shape and the reflection characteristics (texture) of the performer are created from the performer's video taken under a predetermined lighting condition, and the performance of the performer is adjusted to match the lighting condition of the background video. This is to process the performer's video.
[0006]
[Problems to be solved by the invention]
The present inventor has found the following problems as a result of studying the above-mentioned conventional technology.
Creating the shape of the subject and the reflection characteristics of the surface requires extremely complicated work, and the creation of realistic and realistic three-dimensional CG images requires the creator's high ability and enormous work time. It was.
[0007]
For this reason, the present invention can be applied to a case where the time required for creating a three-dimensional CG image can be ensured during a period from shooting to broadcasting (broadcasting), such as a movie or a recorded video. However, in the case of sequentially broadcasting video shot by a television camera as in live broadcasting, it is necessary to finish creating a three-dimensional CG video before the TV camera finishes shooting the next video. There was a problem that could not be applied to.
[0008]
However, the conventional method of measuring the reflection characteristics on the surface of a subject is, for example, to place the subject in a dark room and measure it with a measuring device whose installation position is known in an environment in which light source information is known at one point. there were. As a conventional method of estimating the reflection characteristic of the subject surface, for example, there has been an apparatus for measuring the reflection characteristic of the subject surface disclosed in Japanese Patent Application No. 2001-145192 (hereinafter referred to as Document 1).
[0009]
An object of the present invention is to provide a technique capable of sequentially generating images under predetermined lighting conditions in real time.
The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.
[0010]
[Means for Solving the Problems]
The following is a brief description of an outline of typical inventions disclosed in the present application.
[0011]
(1) reflection coefficient calculating means for calculating a reflection coefficient of a subject imaged as an input image, new lighting condition setting means for setting lighting conditions different from the input image, the set lighting conditions and the reflection coefficient An illumination condition adding unit that generates an image of illumination conditions set by the new illumination condition setting unit from the input image based on the input image. The reflection coefficient calculating means comprises: a correction means for specifying a specular reflection area from the input image based on the depth information of the subject and correcting specular reflection pixels; and a reflection coefficient of the subject from the corrected input image. And a diffuse reflection coefficient calculating means for calculating
[0012]
(2) photographing means for photographing a photographed image including depth information of a subject, reflection coefficient computing means for computing a reflection coefficient of the subject imaged in the photographed image, and new lighting for setting lighting conditions different from the image A photographing system comprising: a condition setting unit; and an illumination condition adding unit configured to generate an image of the illumination condition set by the new illumination condition setting unit from the captured image based on the set illumination condition and the reflection coefficient. In the reflection coefficient calculating means, a correction means for specifying a specular reflection region from the captured image based on the depth information of the subject and correcting specular reflection pixels, and a reflection coefficient of the subject from the corrected captured image And a diffuse reflection coefficient calculating means for calculating.
[0013]
According to the above-described means, when an image including depth information of a subject is input as an input image, first, based on the depth information of the subject, the correction unit specifies a specular reflection region from the input image, and within this region, The specular reflection pixel which is a pixel is corrected. Next, the diffuse reflection coefficient calculation means calculates the reflection coefficient of the subject from the input image corrected by the correction means. Thereafter, based on the illumination condition set by the new illumination condition setting means and the reflection coefficient calculated by the diffuse reflection coefficient calculation means, the illumination condition adding means sets a new illumination from the input image in which the influence of the specular reflection has been corrected. Since it is configured to generate an image of the condition, that is, the configuration does not require the calculation processing of the specular reflection coefficient which requires an enormous amount of operation, the image of the predetermined illumination condition can be generated in real time (real time). This can be sequentially generated, and image processing within one frame period can be realized.
[0014]
Therefore, by using a photographed image photographed by the photographing means for photographing a photographed image including depth information of a subject as an input image, images under predetermined lighting conditions are sequentially generated in real time (real time). An imaging system capable of performing image processing on a computer can be configured.
[0015]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, the present invention will be described in detail with reference to the drawings together with embodiments (examples) of the present invention.
In all the drawings for describing the embodiments of the present invention, components having the same functions are denoted by the same reference numerals, and their repeated description will be omitted.
[0016]
(Embodiment 1)
FIG. 1 is a diagram for explaining a schematic configuration of the imaging system according to the first embodiment of the present invention. However, in the following description, a device that sequentially captures and outputs two-dimensional luminance distribution images of the same red, blue, and green colors as in a known video camera, for example, every 1/30 second is referred to as an RGB camera. Write.
1, reference numeral 101 denotes a subject, 102 denotes a measurement unit, 103 denotes a semi-transparent mirror (half mirror), 104 denotes an illumination light source, 105 denotes a lens, 106 denotes a depth extraction RGB camera, 107 denotes a normal line estimation unit, and 108 denotes an incident light amount estimation. Unit, 109 a threshold setting unit, 110 a lighting effect removing unit, 111 a new lighting condition setting unit, 112 a new lighting condition adding unit, 113 a spectrocolorimeter, and 114 an area A.
[0017]
As is clear from FIG. 1, in the imaging system according to the first embodiment, an illumination light source 104 for illuminating a performer serving as a subject 101, and illumination light emitted from the illumination light source 104 is applied to the performer. A half mirror 103, a spectrocolorimeter 113 for measuring the spectral characteristics of each color of RGB of the illumination light emitted from the illumination light source 104, and a ray reflected by the body surface of the subject 101 and incident via the half mirror 103 A measurement unit 102 is formed by a lens 105 for forming an optical image of the performer and a depth extracting RGB camera 106 for capturing an optical image of the performer.
[0018]
However, the depth extraction RGB camera 106 of the first embodiment can sequentially obtain the depth distance from the lens principal point to the subject in real time, for example, every 1/30 second for each coordinate of the image obtained by the RGB camera. It is a well-known camera. Also, a group of depth information corresponding to at least one of the three RGB images will be referred to as a photographed depth image. Further, in the first embodiment, it is not a necessary condition that the depth distance is obtained every 1/30 second, and a higher speed or a lower speed can be applied other than every 1/60 second. Needless to say, For a camera having this function (the depth extraction RGB camera 106), several methods have already been put into practical use, and a television camera called a well-known axivision and a stereo camera in which two cameras are arranged at a predetermined interval are used. Although there is a method using a camera and the like, it goes without saying that the first embodiment can be applied to a camera to which another method is applied. However, the depth information of each pixel of the obtained RGB image needs to be obtained in a one-to-one correspondence. That is, the principal point of the lens for acquiring the RGB image and the principal point of the lens for obtaining the depth information need to be at the same position.
[0019]
Further, in the imaging system of the first embodiment, a normal estimating unit 107 that calculates a normal vector at each pixel position from a depth image (photographed depth image) captured by the depth extraction RGB camera 106, An incident light amount estimating unit 108 that calculates an incident light amount of illumination light on the performer's body surface based on the performer; and an image of each color of RGB captured by the depth extracting RGB camera 106 and RGB values measured by the spectrophotometer 113. An illumination effect removing unit 110 that calculates reflection coefficient information based on spectral characteristics of each color, incident light amount information and threshold information of each pixel, a threshold setting unit 109 that sets a threshold, and a normal line information and a reflection coefficient information. A new lighting condition adding unit 112 for generating an image based on a new lighting condition based on the new lighting condition, and a new lighting condition setting for setting a new lighting condition in the new lighting condition adding unit 112 Processor of the image information is formed from 111..
[0020]
Therefore, in the imaging system according to the first embodiment, based on an optical image (an imaging depth image having depth information) captured by the illumination light emitted from the illumination light source 104, the imaging depth is first determined by the normal estimation unit 107. A normal vector at each pixel of the image is calculated, and the obtained normal vector is input to the incident light amount estimation unit 108. The normal vector input to the incident light amount estimating unit 108 is used as reference data when calculating the incident light amount of the illumination light on the performer's body surface, and is converted into a shooting depth image and a normal vector by the incident light amount estimating unit 108. The amount of incident light of the illumination light is calculated based on the calculated amount of light, and the obtained amount of incident light is output to the illumination effect removing unit 110. Lighting is performed based on the incident light amount obtained by the incident light amount estimating unit 108, an RGB optical image (photographed RGB image) captured by the illumination light emitted from the illumination light source 104, and a threshold from the threshold setting unit 109. The effect removing unit 110 calculates a reflection coefficient for each pixel in the captured RGB image, and outputs the reflection coefficient information to the new illumination condition adding unit 112.
[0021]
At this time, in the imaging system of the first embodiment, the normal line information from the normal line estimating unit 107 and the new lighting condition information from the new lighting condition setting unit 111 as the new lighting condition are transmitted to the new lighting condition adding unit 112. It is configured to be input. Therefore, the new illumination condition adding unit 112 according to the first embodiment calculates saturation and luminance information for each pixel of the captured RGB image based on the reflection coefficient, normal line information, and the new illumination condition for each pixel. , An image (output image) suitable for the new lighting condition is generated.
[0022]
As described above, in the imaging system according to the first embodiment, the incident light amount and the reflection coefficient at each pixel are calculated based on the normal vector calculated based on the depth information of the subject 101, and the normal vector information is calculated. Is configured to generate a captured RGB image under a new lighting condition based on. That is, it is possible to realize image processing within one frame period in real time.
[0023]
Next, a detailed configuration of each unit of the imaging system according to the first embodiment will be described.
[0024]
(Measurement unit)
FIG. 2 is a diagram for explaining the optical configuration of the measurement unit according to the first embodiment. However, in the following description, the pinhole position when the RGB camera (depth extraction RGB camera 106) is assumed to be a pinhole camera is described as a lens principal point 201. The light source position when the illumination light source 104 is assumed to be a point light source is referred to as an illumination principal point 202.
[0025]
The measurement unit 102 according to the first embodiment has a configuration in which a depth extraction RGB camera 106 and an illumination light source 104 that virtually photograph the subject 101 on the same optical axis using the half mirror 103 are provided for the subject 101. Has become. At this time, the lens principal point 201 of the depth extraction RGB camera 106 and the illumination principal point 202 of the illumination light source 104 have the optical path length from the subject 101 using L1, L2, and L3 in FIG. As shown in the figure, they are installed to be the same.
[0026]
(Equation 1)
L1 + L2 = L1 + L3 (Formula 1)
As described above, in the imaging system according to the first embodiment, the measurement unit 102 is formed in an arrangement in which the lens principal point 201 and the illumination principal point 202 coincide with each other. Is shielded to prevent a shadow from being photographed on the surface of the subject 101. In other words, the shadow of the subject 101 itself occurs on the subject in the captured image due to the fact that the principal points 201 and 202 of the lens 105 and the illumination light source 104 do not match. This is to prevent a decrease in calculation accuracy of a hidden part in an image processing process described later.
[0027]
Here, one of the purposes of using the half mirror 103 is to irradiate the subject 101 with illumination light emitted from the illumination light source 104 via reflection by the half mirror 103. Another purpose of using the half mirror 103 is to use the transmission of the subject image by the half mirror 103 to capture an image with the depth extraction RGB camera 106. The depth extraction RGB camera 106 has a method that needs to project infrared light from the camera side to the subject 101 and take an image of the reflected light, and this point also uses the property that the half mirror 103 transmits. Therefore, the function of the depth extraction RGB camera 106 is not limited.
[0028]
In the first embodiment, the case where the half mirror 103 is used will be described. However, the same function can be realized by using a known optical prism instead of the half mirror 103. In addition, when a condition is obtained such that the lens principal point 201 and the illumination principal point 202 are the same, the anteroposterior relationship with respect to the subject such as the half mirror 103 and the camera lens 105 differs from the configuration shown in FIG. No problem. For example, as shown in FIG. 3, even when the half mirror 103 is disposed between the lens 105 and the depth extraction camera 106, the measurement unit 102 according to the first embodiment can be configured.
[0029]
In particular, as shown in FIG. 3, when the lens 105, the half mirror 103, and the depth extraction RGB camera 106 are arranged, compared with the configuration shown in FIG. 2, the illumination principal point 202 which is the principal point of the illumination light source 104, Since the amount of movement of the illumination light source 104 required to match the lens principal point 201 which is the principal point of the lens 105 for forming an optical image on the depth extraction RGB camera 106 can be reduced, the entire apparatus can be downsized. It is also possible to obtain the effect of being able to do so. Therefore, when configured as a dedicated system, as shown in FIG. 3, the measurement unit 102 is preferably arranged in the order of the lens 105, the half mirror 103, and the depth extraction RGB camera 106 from the subject 101 side.
[0030]
On the other hand, by incorporating and connecting a part other than the area A114 indicated by the dotted line in FIG. Can be used for multiple purposes.
[0031]
(Normal estimation unit)
The normal estimating unit 107 according to the first embodiment is a unit that estimates a normal vector of each part of the subject 101 in the captured depth image obtained from the measuring unit 102. However, in the following description, the normal vector is a vector perpendicular to the surface of the subject 101 and represents the direction of the surface. In particular, in the first embodiment, the normal vector at each pixel constituting the captured depth image is estimated. The normal vector of the pixel can be calculated by using the depth information between the pixel and peripheral pixels.
[0032]
For example, in the case where the photographed depth image obtained by the measuring unit 102 is composed of 640 pixels horizontally and 480 pixels vertically as shown in FIG. 4A, the lower left two-dimensional coordinates of the image are used as the reference position. (0, 0). Assume that given pixels are given coordinates at a distance of the number of pixels based on the lower left coordinates.
[0033]
When calculating the normal vector of the two-dimensional coordinates (100, 100), as shown in FIG. 4B, the depth information of the pixel (100, 100) and the pixel adjacent to the pixel are calculated. Information of four surrounding pixels (coordinates (100, 101), (99, 100), (101, 100), (100, 99)). First, since the respective two-dimensional coordinates and depth information can be converted into three-dimensional coordinates with the lens principal point 201 as the origin, the first embodiment converts them into three-dimensional coordinates in advance. Next, line segments formed by the three-dimensional coordinates corresponding to the pixel and the surrounding pixels are obtained. Further, by obtaining an outer product between adjacent line segments, a normal vector of a plane formed by the line segments is obtained. Four normal vectors are obtained by this operation. In the first embodiment, the normal vector of the pixel is obtained by averaging and normalizing the four normal vectors. However, the calculation method of the normal vector is not limited, but can be realized by a general calculation method used in the computer graphics field as described above. The normal vector information and the three-dimensional coordinates obtained by the normal estimating unit 107 are sent to the incident light amount estimating unit 108 at the next stage.
[0034]
(Incident light quantity estimation unit)
The incident light amount estimating unit 108 according to the first embodiment is means for estimating the amount of incident light on each part of the subject 101. However, the estimation of the amount of incident light requires the distance from the illumination light source 104 and the angle of incidence of the illumination light on the subject 101.
[0035]
Therefore, in the imaging system according to the first embodiment, the incident angle and the distance from the illumination light source 104 are determined by using the normal vector information and the three-dimensional coordinates of each part of the subject 101 obtained by the normal estimating unit 107 in the preceding stage. Configuration. At this time, the three-dimensional coordinates obtained by the radiation estimating unit 107 in the former stage have the origin at the lens principal point 201, that is, the illumination principal point 202, and when used together with normal vector information, the incident angle of illumination can be easily determined. Can be estimated. The distance from the illumination light source 104 to each part of the subject 101 can also be estimated by obtaining the distance between the three-dimensional coordinates of each part and the origin (0, 0, 0). For example, the coordinates of a predetermined part of the subject 101 are represented by (x₁, Y₁, Z₁) And its normal vector to (x₂, Y₂, Z₂), The incident angle θ is (x₁, Y₁, Z₁), (X₂, Y₂, Z₂) As a vector.
[0036]
First, the distance L1 from the illumination light source 104 is represented by the following equation (2).
[0037]
(Equation 1)
L1 = (X₁ ²+ Y₁ ²+ Z₁ ²)^1/2... (Equation 2)
Since the light intensity of the illumination at a distance L1 from the illumination light source 104 is inversely proportional to the square of the distance L1, the light intensity of the illumination light source 104 at a location L1 away from the light intensity p₁Becomes the following Expression 3.
[0038]
(Equation 2)
p₁= P / L1²... (Equation 3)
Further, the amount of incident light I on the subject is expressed by the following equation 4 based on Lambert's cosine law, which is classic in the field of computer graphics.
[0039]
(Equation 3)
I = p₁/ Cos θ (Equation 4)
The incident light amount estimating unit 108 calculates the incident light amount I for all the pixels of the image by the above-described calculation, and sends the obtained incident light amount I to the illumination effect removing unit 110 at the next stage.
[0040]
(Lighting effect removal section)
The illumination effect removal unit 110 according to the first embodiment uses the captured RGB image obtained by the measurement unit 102 and the incident light amount I obtained by the incident light amount estimation unit 108 in the preceding stage to remove the illumination effect of the subject 101. Is a means for determining the reflection coefficient of the surface.
[0041]
The components of the reflected light from the subject 101 are classified differently for each of various reflection models proposed so far. However, many of them are roughly classified into specular reflection components and diffuse reflection components based on a dichroic reflection model widely used empirically. The specular reflection component causes a phenomenon that is extremely dominant in some object regions as compared with the diffuse reflection component, and hinders the removal of the illumination effect. This area is called a highlight area.
[0042]
These reflection components (specular reflection component and diffuse reflection component) are expressed differently for each of various reflection models proposed. As for the diffuse reflection component, Lambert's cosine law depending on the incident angle of illumination light is used in many reflection models, and is used in the first embodiment in the same manner. On the other hand, the specular reflection component is not easily represented only by the relationship with the incident angle of the illumination light, and even if it is applied to any reflection model, a high-quality, high-speed and universally applicable removal method regardless of the material of the subject 101 is required. Not yet. Therefore, the illumination effect removal unit 110 according to the first embodiment has a configuration in which a mechanism for estimating a highlight region and interpolating the same with pixel values around the highlight region is incorporated as a method not considering the reflection model. However, it goes without saying that if a higher quality and more accurate specular reflection component removal method is proposed in the future, it can be incorporated in the first embodiment.
[0043]
The feature of the portion that is specularly reflected is relatively bright in the image, and the light source color is dominant over the color of the object. For example, a part of the surface of a red object color under white illumination that causes specular reflection appears to glow white. Such an area is called a highlight area and interferes with the estimation of the reflection coefficient of the subject 101. Further, depending on the material of the subject 101, if the incident angle of the illumination light on the surface of the subject 101 and the exit angle of the reflected light from the surface of the subject 101 to the camera side are close to each other, mirror reflection easily occurs. It is known to be. That is, under the optical conditions of the first embodiment, as shown in FIG. 5, the more the surface of the subject 101 is perpendicular to the straight line L4 extending from the lens principal point 201 to each part of the subject 101, the more the specular reflection occurs. Tends to occur.
[0044]
Therefore, the illumination effect removing unit 110 according to the first embodiment is configured to determine the specular reflection portion, that is, the highlight region, by using the following three determination criteria in combination from the above-described properties.
(Value used for determination on straight line L4)
First, an angle between a straight line L4 extending from the lens principal point 201 to each part of the subject 101 and a normal vector of the surface of the subject 101 is defined as θa. This θa can be calculated from the normal vector information of the subject 101 and the three-dimensional coordinates.
(Value used for color judgment)
Next, the comparison between the color of the subject part and the light source color is performed as follows. The red, blue and green spectral characteristics of the light primary colors are measured in advance, and as shown in FIG. 6, the points S are normalized on three-dimensional coordinates with the intensities of red, blue and green as axes.₁Projected as On the other hand, the red, blue, and green components of the relevant portion of the subject 101 are normalized on three-dimensional coordinates around the red, blue, and green intensities from the captured RGB image captured by the depth extraction RGB camera 106, Point S₂Projected on three-dimensional coordinates. The light source color point S at this time₁And the color point S of the subject₂And the distance on the three-dimensional coordinate with₁And Thus the distance n₁By setting the distance n₁Is smaller, it can be determined that the colors are more similar. However, in the first embodiment, the spectral characteristics of the light source color are measured in advance, and the measured values are used. However, the comparison standard is not limited to this. It is also possible to incorporate the spectrocolorimeter 113 in an appropriate part and automatically measure and use the spectral characteristics without direct effort.
(Value used to judge brightness)
As shown in FIG. 7, the peak value I of the luminance value of the captured RGB image captured by the depth extraction RGB camera 106_maxAnd a histogram for the luminance values of all the pixels, and the brightest luminance value I_maxBrightness value I that becomes the peak on the first histogram_peakIs calculated. Then I_peak− (I_max-I_peak) Is I_sAnd Thus, the luminance value I_sIs set, the highlight portion due to the specular reflection becomes the luminance value I_peakCan be approximated to a distribution that can be expressed by a Gaussian function with the center value and the maximum value._peakI around_peak-I_maxThe distribution is symmetrical with the space turned back. However, the luminance value I_sIs the luminance value I_maxCorresponds to the folded part.
(Use of three criteria)
The first to third threshold values Th are manually determined in advance for the three determination criteria.₁, Th₂, Th₃Is set by the threshold setting unit 109 in advance.
[0045]
First, the angle θa of each pixel of the captured RGB image captured by the depth extraction RGB camera 106 and the first threshold Th₁And the angle θa is equal to the first threshold Th.₁If it is smaller, it is considered that the reflection by specular reflection is the dominant part, and the information R of the region satisfying this condition is considered.₁Save.
[0046]
Next, R₁, The brightness value I of all pixels in the region_outAnd luminance value I_sAnd the difference value I_diffIs calculated by the following equation (5).
[0047]
(Equation 4)
I_diff= I_out-I_s... (Equation 5)
Difference value I_diffIs the second threshold Th₂If it is larger than this, it is considered that the reflection by the specular reflection is dominant, and the information R of the region satisfying this condition is considered.₂Save.
[0048]
Finally R₂Distance n for all pixels in the region₁Ask for. Where the distance n₁Is the third threshold Th₃The smaller portion is regarded as a portion where the reflection by specular reflection is dominant, and information R of a region satisfying this condition is obtained.₃Save.
[0049]
Finally R₃Is regarded as a region where specular reflection is dominant. Where R₃Is subjected to a segmentation process, which is a general image processing technique,₃Is divided into closed sections having the minimum area in the image, and the pixel values of the area are replaced with the surrounding pixel values for each area.
[0050]
With respect to the captured RGB image in which the reflection coefficient caused by specular reflection has been processed as described above, the imaging system of the first embodiment calculates the reflection coefficient caused by diffuse reflection in the following procedure.
Diffuse reflection light I_dMeans that the incident light is I and the diffuse reflection coefficient is K_dAre represented by the following Expressions 6 and 7.
[0051]
(Equation 5)
I_d= K_d× I (Equation 6)
K_d= I_d/ I (Equation 7)
Diffuse reflection light I_dThe red, blue, and green components of the observed captured RGB image corresponding to_r, C_b, C_g), And the red, blue, and green components of the incident light I are represented by (I_r, I_b, I_g), The diffuse reflection coefficient K_dRed component, blue component, green component (K_r, K_b, K_g) Is calculated by the following Expressions 8 to 10.
[0052]
(Equation 6)
K_r= C_r/ I_r... (Equation 8)
K_b= C_b/ I_b... (Equation 9)
K_g= C_g/ I_g... (Equation 10)
The obtained diffuse reflection coefficient (K_r, K_b, K_g) Is sent to the new lighting condition adding unit 112 at the next stage as information from which the lighting effect has been removed.
[0053]
FIG. 8 is a diagram for explaining a schematic configuration of the illumination effect removing unit according to the first embodiment. Here, reference numeral 801 denotes an irradiation incident angle determination unit (first region specifying unit), 802 denotes a color determination unit (spectral characteristic calculation unit, second region specification unit), and 803 denotes a brightness determination unit (brightness value calculation unit; 3, an area interpolating unit (correction unit) 804, and a diffuse reflection light removing unit (diffuse reflection coefficient calculation unit) 805.
[0054]
8, the illumination incident angle determination unit 801 includes: an incident light amount information from the incident light amount estimation unit 108; a normal vector and three-dimensional position information from the normal line estimation unit 107; a captured RGB image from the depth extraction RGB camera 106; And the first threshold Th from the threshold setting unit 109₁Is a means for performing the determination processing described in “Value used for determination on straight line L4” described above. Region information R obtained by this illumination incident angle determination unit 801₁Is input to the color determination unit 802.
[0055]
The color determination unit 802 receives the incident light amount information from the incident light amount estimation unit 108, the normal vector and three-dimensional position information from the normal line estimation unit 107, the captured RGB image from the depth extraction RGB camera 106, and the threshold setting unit 109. Of the second threshold Th₂Is a means for performing the determination processing described in the above-mentioned “value used for color determination”. However, as described above, the color determination unit 802 uses the region information R input from the illumination incident angle determination unit 801.₁, The determination process is performed only on the region obtained by the illumination incident angle determination unit 801. The region information R obtained by the color determination unit 802₂Is input to the next-stage brightness determination unit 803.
[0056]
The brightness determination unit 803 includes the incident light amount information from the incident light amount estimation unit 108, the normal vector and three-dimensional position information from the normal line estimation unit 107, the captured RGB image from the depth extraction RGB camera 106, and the threshold setting unit 109. From the third threshold Th₃Is a means for performing the determination processing described in the above “value used for determining brightness”. However, as described above, the brightness determination unit 803 uses the region information R input from the color determination unit 802.₂, The determination process is performed only on the region obtained by the color determination unit 802. The region information R obtained by the brightness determination unit 803₃Is input to the area interpolation unit 804 at the next stage.
[0057]
The region interpolation unit 804 outputs the region information R obtained by the brightness determination unit 803.₃Is used to interpolate the area of the captured RGB image with the pixel values around the area based on the above. This interpolation processing is the processing described in the above-mentioned “Use of Three Determination Criteria”. The captured RGB image after interpolation processing obtained by the area interpolation unit 804 is input to the diffuse reflection light removal unit 805 at the next stage.
[0058]
The diffuse reflection light removing unit 805 is a unit that calculates the diffuse reflection coefficient based on the above-described equations 8, 9, and 10, and obtains the obtained diffuse reflection coefficient (K_r, K_b, K_g) Is input to the new lighting condition adding unit 112 as an output of the lighting effect removing unit 110.
[0059]
(Threshold setting part)
First to third threshold values Th used in the lighting effect removing unit 110₁, Th₂, Th₃Is a means for setting. For example, three knob-shaped volumes are arranged as threshold setting buttons, and these thresholds can be manually set by operating these knobs. The first to third threshold values Th₁, Th₂, Th₃Can be set at the discretion of the operator while viewing the output image before or during use. However, the present invention does not limit the user interface. Also, the luminance value I of the illumination is obtained from the spectral colorimeter in the measuring section.₁And obtain the luminance value I₁The second threshold value Th is proportional to₂It is needless to say that these thresholds can be automatically set, for example, by automatically setting the threshold.
[0060]
(New lighting condition setting section)
The new lighting condition setting unit 111 is a means for setting a required lighting condition. In the new lighting condition setting unit 111 according to the first embodiment, the setting items can set the number of lights, arrangement in a three-dimensional space, light distribution characteristics, color temperature, and the like.
[0061]
These setting items are necessary for producing an image by well-known computer graphics, and a function equivalent to a part of the function of general computer graphics software is added to the new lighting condition setting unit 111. It can also be realized by incorporating. In particular, when it is not necessary to sequentially change the lighting conditions in real time, a mechanism that continues to send predetermined lighting conditions may be used. In addition, in a case where even a person who is not used to the operability of setting the lighting conditions can intuitively operate, for example, only the direction of the lighting is designated by a well-known joystick, and information of the designated direction is obtained. It is also possible to adopt a configuration in which the information is sequentially sent to the new lighting condition adding unit 112 at the next stage. Further, the setting items can be limited.
[0062]
Further, it can be used in combination with a virtual studio system which is a video production method of a television program. However, the virtual studio system generates a computer graphics image serving as a background under the same conditions as the camera that is shooting the subject 101, and combines the generated image with the subject image shot by the camera, This is a system that generates a video effect as if the subject 101 exists on a video generated by computer graphics. Since the lighting conditions of the background image generated by the computer graphics are set in the computer, the setting conditions can be given to the new lighting condition setting unit 111 of the present method.
It is also possible to estimate outdoor lighting conditions according to the actual date and time or the date and time designated by the user in accordance with physical laws, and to send it to the new lighting condition adding unit 112 at the next stage.
[0063]
In the first embodiment, the user interface is not limited, but it is needless to say that the lighting conditions need to be sequentially transmitted in real time to the new lighting condition adding unit 112 at the next stage.
[0064]
(New lighting condition addition section)
The new lighting condition adding unit 112 includes normal vector information from the normal estimating unit 107, the reflection coefficient (specular reflection coefficient and diffuse reflection coefficient) from the lighting effect removing unit 110, and the new lighting from the new lighting condition setting unit 111. This is a means for inputting condition information, and for generating and outputting a subject image expected under new lighting conditions based on the input information. However, since the generation of the subject image under the new condition can be realized by a general method used in known computer graphics, a detailed description is omitted.
[0065]
Here, when the lighting conditions obtained from the virtual studio system are given to the new lighting condition setting unit 111, a subject image under the same lighting conditions as the computer graphics video generated by the virtual studio system can be generated. At this time, by using the depth information, means for determining the order of the subject 101 and the set of computer graphics used for the background of the virtual studio is provided, so that the subject 101 located before the set of computer graphics is provided. Since only the area can be combined with the background video by the set of computer graphics, even when the subject 101 is in a place different from the location where the background video was captured, the subject 101 can capture the background video. An effect (compositing effect) as if shooting at a position is obtained.
[0066]
The camera and the synchronization generator of the apparatus described above have been described on the basis of 1/30 second or 1/60 second. However, when the equipment of the NTSC signal is used, the synchronization signal is 59. Since it is 94 Hz, it goes without saying that it is 1 / 59.94 second or 2 / 59.94 second. In addition, there are video cameras of various frequencies such as a high-speed camera and a long-time exposure camera, but the present invention does not limit the frequency, and the processing is sequentially repeated for each minimum unit constituting an image. , The present invention can be applied to a photographing system using a photographing camera of another frequency.
[0067]
In the above-described imaging system according to the first embodiment, it is needless to say that the subject 101 needs to be captured in an environment where there is no illumination other than the illumination in the measurement unit 102 in the dark room. Nor.
[0068]
As described above, in the imaging system according to the first embodiment, the normal estimating unit based on the captured depth image captured using the depth extraction RGB camera 106 under the illumination light emitted from the illumination light source 104. 107 calculates a normal vector at each pixel of the photographed depth image, and an incident light amount estimating unit 108 calculates an incident light amount of illumination light on the body surface of the subject 101 as a performer based on the input normal vector. The removal unit 110 detects the incident light amount obtained by the incident light amount estimation unit 108, the captured RGB image captured by the illumination light emitted from the illumination light source 104, and the first to third threshold values Th from the threshold value setting unit 109.₁, Th₂, Th₃, A reflection coefficient for each pixel in the captured RGB image is calculated, and the new lighting condition adding unit 112 generates and outputs a subject image under new lighting conditions based on the reflection coefficient.
[0069]
At this time, in the imaging system according to the first embodiment, the illumination effect removing unit 110 that calculates the reflection characteristic of the subject 101, that is, the reflection coefficient, includes the irradiation incident angle determination unit 801, the color determination unit 802, the brightness determination unit 803, An insertion section 804 and a diffuse reflection light removing section 805 are provided.
[0070]
Here, as an operation related to specular reflection, first, the illumination incident angle determination unit 801 sets the incident light amount information, the normal vector, the three-dimensional position information, the captured RGB image, and the first threshold Th.₁The region is specified based on. Next, for the area specified by the illumination incident angle determination unit 801, the color determination unit 802 performs the incident light amount information, the normal vector, the three-dimensional position information, the captured RGB image, and the second threshold Th.₂The region is specified based on. Next, for the area specified by the color determination unit 802, the brightness determination unit 803 determines the incident light amount information, the normal vector, the three-dimensional position information, the captured RGB image, and the third threshold value Th.₃The region is specified based on. Next, with respect to the area specified by the brightness determination unit 803, the area interpolation unit 804 interpolates the area of the captured RGB image with the pixel values around the area. Thereafter, the diffuse reflection light removing unit 805 calculates a reflection coefficient related to the diffuse reflection light with respect to the captured RGB image in which the highlight region is interpolated (corrected) by the region interpolation unit 804.
[0071]
As described above, the configuration in which the illumination effect removing unit 110 according to the first embodiment generates a captured RGB image in which the influence of specular reflection is removed from the captured RGB image obtained by the depth extraction RGB camera 106, that is, an enormous amount of calculation is required. Since the configuration does not require the calculation processing of the specular reflection coefficient, it is possible to sequentially generate an image under a predetermined illumination condition in real time (real time). In particular, image processing within one frame period is required. To live broadcasts.
[0072]
(Embodiment 2)
FIG. 9 is a diagram for explaining a schematic configuration of the imaging system according to the second embodiment of the present invention. Here, reference numeral 901 denotes an optical shutter, 902 denotes a synchronization generator, 903 denotes a frequency divider, 904 denotes an image memory unit, and 905 denotes a difference image generation unit. Further, in the following description, operations and effects related to the optical shutter 901, the synchronization generator 902, the frequency divider 903, the image memory unit 904, and the difference image generation unit 905, which are different in configuration from the imaging system of the first embodiment. Will be described in detail.
[0073]
As is apparent from FIG. 9, in the imaging system according to the first embodiment, an illumination light source 104 for illuminating a performer who is an object 101 and illumination light emitted from the illumination light source 104 is applied to the object 101 side. A half mirror 103, an optical shutter 901 disposed between the illumination light source 104 and the half mirror 103 to control transmission and blocking of illumination light emitted from the illumination light source 104, and illumination emitted from the illumination light source 104. A spectrocolorimeter 113 for measuring the spectral characteristics of each of the RGB colors of light, and a lens 105 for forming a light beam (optical image of the subject 101) reflected by the body surface of the subject 101 and incident via the half mirror 103. And a depth extracting RGB camera 106 for photographing an optical image of the subject 101, and an external synchronization input terminal of the depth extracting RGB camera 106. A sync generator 902 for generating a synchronization signal, the measurement unit 102 from the frequency divider 903 for dividing the synchronization signal from sync generator 902 is formed.
[0074]
As described above, the measuring unit 102 according to the second embodiment is configured to control the shooting timing of the depth extraction RGB camera 106 in synchronization with the synchronization signal generated by the synchronization generator 902. Further, the operation of the optical shutter 901 is controlled by a signal (frequency-divided signal) obtained by dividing the synchronization signal by the frequency divider. Therefore, it is possible to synchronize the shooting of the subject 101 with the depth extraction RGB camera 106 and the illumination of the subject 101. The details of the measuring unit 102 will be described later.
[0075]
Further, in the imaging system according to the first embodiment, a normal estimating unit 107 that calculates a normal vector at each pixel position from a depth image (captured depth image) captured by the depth extraction camera 106, and An incident light amount estimating unit 108 for calculating an incident light amount of illumination light on the performer's body surface, an image memory unit 904 for sequentially storing captured RGB images captured by the depth extracting RGB camera 106, and reading from the image memory unit 904. A difference image generation unit 905 that generates a difference image for each color of RGB from an image (delayed image) delayed by one frame (one shooting cycle) and the captured RGB image captured by the depth extraction RGB camera 106; The image for each color of RGB generated by the generation unit, the spectral characteristics of each color of RGB measured by the spectral colorimeter 113, and the incidence for each pixel. A lighting effect removing unit 110 that calculates reflection coefficient information based on the amount information and the threshold information, a threshold setting unit 109 that sets a threshold, and a new image generating unit that generates an image under new lighting conditions based on the normal information and the reflection coefficient information. An image information processing unit is formed by the lighting condition adding unit 112 and the new lighting condition setting unit 111 for setting a new lighting condition in the new lighting condition adding unit 112.
[0076]
As described above, the processing unit according to the second embodiment temporarily stores the captured RGB image captured by the depth extraction RGB camera 106 in the image memory unit 904, and stores the stored captured RGB image and the captured RGB image captured in real time. Are sequentially generated by the difference image generation unit 905, and the difference image is used by the illumination effect removal unit 110.
[0077]
In FIG. 9, a synchronization generator 902 is a well-known synchronization signal generator that generates a pulse of a synchronization signal in units of 1/30 second or 1/60 second, which is the imaging frequency of the camera image of the depth extraction RGB camera 106. Means. However, the synchronization signal has a waveform suitable for an external synchronization input terminal of the video camera constituting the depth extraction RGB camera 106, and this synchronization signal is input to the video camera. This video camera adjusts so as to capture an image in synchronization with the input synchronization signal. However, a general commercial TV camera has a terminal for inputting an external synchronization signal called an external synchronization input terminal or a genlock terminal, and an external synchronization signal is output by a built-in PLL circuit (PhaseLocked @ Loop: phase synchronization circuit). A circuit for oscillating to generate a synchronization signal for a camera internal circuit in synchronism with the circuit is incorporated. Therefore, the synchronization processing in the second embodiment can be easily realized.
[0078]
The frequency divider 903 is a means for dividing the synchronization signal of 1/30 or 1/60 second by 2, and has a period of 1/15 second or 1/30 second, which is half of the synchronization signal. Generate a signal (divided signal). In particular, in the second embodiment, the duty ratio of the frequency-divided signal is set to a digital signal of 50%, and the waveform is shaped so that the determination at each unit to which the frequency-divided signal is input is easy.
[0079]
An optical shutter 901 arranged perpendicularly to the illumination projection direction is a well-known optical shutter that can be opened and closed by an electric signal, and blocks light in a closed state and transmits light in an open state. Has become. In addition, the optical shutter 901 according to the second embodiment is in a closed state when a frequency-divided signal input as a drive signal or a control signal is High (1), and is opened in a Low (0) period. The optical shutter of the second embodiment can be realized by using a mechanical shutter or a liquid crystal shutter as the optical shutter 901.
[0080]
The image memory unit 904 is a unit for sequentially storing captured RGB images captured by the depth extraction RGB camera 106. In the second embodiment, at the input timing of the next captured RGB image, It is configured to output the captured RGB images stored at the input timing. In other words, when the captured RGB image has been input and the next captured RGB image is sent from the depth extraction RGB camera 106, the image memory unit 904 stores the content stored in the image memory, that is, the captured RGB image at that timing. This is sent to the difference image generation unit 905 at the next stage. That is, the captured RGB image that has passed through the image memory unit 904 is delayed by one image from the RGB image output from the depth extraction RGB camera 106.
[0081]
The difference image generation unit 905 outputs the current captured RGB image G directly input from the depth extraction RGB camera 106._pAnd an image G delayed by one image input from the image memory unit 904_dImage (difference image) G from the difference_diffIs a means for generating. In particular, in the difference image generation unit 905 of the second embodiment, 1/30 second or 1/60 of the following equation 11 and equation 12 are obtained based on the signal obtained by dividing the synchronization generator by 2 using the frequency divider. By switching every second, the difference image G_diffIs obtained.
[0082]
(Equation 7)
G_diff= G_p-G_d··· (Equation 11)
G_diff= G_d-G_p... (Equation 12)
In particular, in the difference image generation unit 905 of the second embodiment, the image when the illumination light source 104 of the measurement unit 102 is turned on becomes the left variable on the right side, and the image when it is turned off becomes the right variable on the right side. Control. That is, by subtracting the non-lighted image from the lighted image, only the reflected light from the illumination light source 104 in the measuring unit 102 is converted into the difference image G_diffIt is configured to take out as.
[0083]
In order to realize this, when the frequency-divided signal from the frequency divider 903 is 0, Expression 11 is selected, and when the frequency-divided signal is 1, Expression 12 is selected to obtain a difference image. Output to the effect removing unit 110.
[0084]
By performing such a contrivance, even when the subject 101 is illuminated by a light source other than the measuring unit 102, the same effect as the imaging system of the first embodiment is obtained.
[0085]
FIG. 10 is a diagram for explaining the collection timing of captured RGB images and the irradiation timing of illumination light by the depth extraction camera according to the second embodiment. However, the image names shown in FIG. 10 indicate captured RGB images sequentially captured by the depth extraction RGB camera 106, and in this specification, natural numbers of 1, 2, 3, 4, 5, 6,. The consecutive numbers indicated by are designated as image names.
[0086]
First, the photographing operation of the photographed RGB image in the measurement unit 102 will be described based on FIG.
[0087]
As described above, the measuring unit 102 according to the second embodiment has a configuration in which a synchronization signal is input to the depth extraction RGB camera 106, whereas the synchronization signal is divided into two by the optical shutter 901. The divided frequency signal is inputted. Therefore, in the measurement unit 102 according to the second embodiment, the depth extraction RGB camera 106 shoots in synchronization with the input of the synchronization signal in synchronization with the synchronization signal.
[0088]
On the other hand, the optical shutter 901 is configured to switch between the closed state and the open state in order according to the High (1) period and the Low (0) period of a frequency-divided signal having a duty ratio of 50% obtained by dividing the synchronization signal by two. Has become. That is, in the optical shutter 901 of the second embodiment, transmission and shielding of the illumination light emitted from the illumination light source 104 are controlled to be switched in synchronization with the synchronization signal. The irradiation of the illumination light is also synchronized with the synchronization signal.
[0089]
As a result, as shown in FIG. 10, shooting of captured RGB images continuous with images 1, 2, 3, 4, 5, 6,... Was performed by a depth extraction RGB camera operating in synchronization with a synchronization signal. In this case, the frequency-divided signal output from the frequency divider 903 becomes 1 while, for example, images 1, 3, 5,... Are taken, and becomes 0 while images 2, 4, 6,. . .., I.e., while the frequency-divided signal is 1, the optical shutter 901 is closed to block the illumination light from the illumination light source 104, and the illumination light illuminates the subject 101. Will not be performed. On the other hand, during shooting of images 2, 4, 6,..., That is, during the period in which the frequency-divided signal is 0, the optical shutter 901 is opened and the illumination light from the illumination light source 104 is transmitted. 101 will be illuminated.
[0090]
As described above, in the measuring unit 102 according to the second embodiment, the captured RGB image obtained by illuminating the subject 101 with the illumination light from the illumination light source 104 and the illumination light source 104 for each imaging cycle (frame period) of the depth extraction RGB camera 106 And a captured RGB image that does not illuminate the subject 101 with illumination light from the camera.
[0091]
Next, an operation of generating an output image in the processing unit will be described with reference to FIG. However, the operation is the same as the operation of generating the output image in the processing unit of the first embodiment except that the image data input to the illumination effect removing unit 110 is different. Accordingly, in the following description, the difference image G by the image memory unit 904 and the difference image generation unit 905 will be described._diffOnly the generating operation will be described in detail.
[0092]
The captured RGB images captured by the depth extraction RGB camera 106 are sequentially stored in the image memory unit 904. At this time, the same captured RGB image is also input to the difference image generation unit 905. Accordingly, when the captured RGB image of the image name 1 shown in FIG. 10 is captured, the image memory unit 904 and the difference image generation unit 905 store the captured RGB image of the image name 1 in the current captured RGB image G._pWill be entered as At this time, since the previous image is not stored in the image memory unit 904, the difference image G from the difference image generation unit 905 is output._diffIs not output.
[0093]
When the captured RGB image of the image name 2 is captured in the next capturing cycle, the captured RGB image of the image name 2 is stored in the image memory unit 904 and the difference image generating unit 905._pWill be entered as At this time, first, the difference image generation unit 905 outputs the image G obtained by delaying the captured RGB image of the image name 1 by one image from the image memory unit 904, which is an image captured in the immediately preceding imaging cycle._dRead as Next, the difference image generation unit 905 outputs the captured RGB image of the image name 1 from the image memory unit 904 (the image G delayed by one image)._d) And the captured RGB image of the image name 2 from the depth extraction RGB camera 106 (the current captured RGB image G_p) And the difference image G_diffGenerate The calculation at this time is, as described above, the photographed RGB image of the image name 2 (the current photographed RGB image G), which is an image photographed by irradiating the irradiation light from the illumination light source 104 according to Expression 11._p), The photographed RGB image of the image name 1 photographed when the irradiation light from the illumination light source 104 is not irradiated (the image G delayed by one image)_d), The difference image G_diffGenerate
[0094]
When the captured RGB image of the image name 3 is captured in the next capturing cycle, the captured RGB image of the image name 3 is stored in the image memory unit 904 and the difference image generating unit 905._pIs entered as At this time, the difference image generation unit 905 outputs the image G obtained by delaying the captured RGB image of the image name 2 by one image from the image memory unit 904, which is an image captured in the immediately preceding imaging cycle._dRead as Next, the difference image generation unit 905 outputs the captured RGB image of the image name 2 from the image memory unit 904 (the image G delayed by one image)._d) And a captured RGB image of the image name 3 from the depth extraction RGB camera 106 (current captured RGB image G_p) And the difference image G_diffGenerate The calculation at this time is, as described above, a captured RGB image of image name 2 (an image G delayed by one image), which is an image captured by irradiation with the illumination light from the illumination light source 104, according to Expression 12._d), The photographed RGB image of the image name 3 photographed when the irradiation light from the illumination light source 104 is not irradiated (the current photographed RGB image G_p), The difference image G_diffGenerate
[0095]
The currently captured RGB image G stored in the image memory unit 904 described above_pAnd a photograph taken when the irradiation light from the illumination light source 104 is not irradiated, from the photographed RGB image photographed by irradiation with the irradiation light according to the expression 11 or 12 by the differential image generation means 905. Difference image G obtained by sequentially performing subtraction processing of RGB images_diffIs output to the illumination effect removing unit 110, and the same processing as in the first embodiment is performed. Therefore, the same effect as in the imaging system of the first embodiment can be obtained.
[0096]
In the second embodiment, the currently captured RGB image G stored in the image memory unit 904 described above is stored._pAnd a photograph taken when the irradiation light from the illumination light source 104 is not irradiated, from the photographed RGB image photographed by irradiation with the irradiation light according to the expression 11 or 12 by the differential image generation means 905. By sequentially performing the subtraction processing of the RGB images, only the light reflected by the illumination light source 104 of the measurement unit 102 is subtracted from the difference image G._diffCan be taken out. As a result, even when the subject 101 is illuminated by the illumination light source 104 other than the measurement unit 102, the same effect as that of the imaging system according to the first embodiment in which the subject 101 is placed in a dark room and imaged is obtained. It has a special effect.
[0097]
(Embodiment 3)
FIG. 11 is a diagram for explaining a schematic configuration of an illumination light source in the imaging system according to the third embodiment of the present invention. However, the configuration of the imaging system according to the third embodiment is the same as that of the imaging system according to the first or second embodiment except for the configurations of the depth extracting RGB camera 106 and the illumination light source 104 that configure the measuring unit 103. Configuration. Therefore, in the following description, the configuration of the measurement unit 102 will be described in detail.
[0098]
11, 1101 is a television camera, 1102 is a lens remote controller, 1103 is a first A / D converter, 1104 is a second A / D converter, 1105 is a lookup table search unit, 1106 is principal point position information A voltage conversion unit, 1107 is a servomotor, 1108 is a rotating shaft, 1109 is a sliding unit, and 1110 is a light source lamp.
[0099]
As shown in FIG. 11, in the imaging system according to the third embodiment, a television camera 1101 including a depth-extracting RGB camera 106 and a lens 105 according to a remote operation signal (for example, a voltage signal or the like) from a lens remote controller 1102 is illustrated. The depth extraction camera 106 and the lens 105 are configured by using the television camera 1101 in which the zoom mechanism and the focus mechanism that do not operate and the shooting field of view (angle of view) of the television camera 1101 can be arbitrarily set. Further, a remote operation signal for controlling the zoom mechanism and the focus mechanism is input to the first and second A / D converters 1103 and 1104.
[0100]
In the first and second A / D converters 1103 and 1104, remote operation signals (the zoom amount and the focus amount of the lens) from the lens remote controller 1102 are converted into digital signals and output to the lookup table search unit 1105. . The lookup table search unit 1105, to which the zoom amount and the focus amount converted into the digital signal are input, refers to a table (not shown) that stores the position information of the illumination principal point 202 corresponding to the zoom amount and the focus amount, An illumination principal point 202 suitable for the combination of the zoom amount and the focus amount is searched. The position information of the illumination principal point 202 obtained by the search is converted into driving power for driving the servo motor 1107 by the principal point position information voltage converter 1106, and the servo motor 1107 is driven.
[0101]
Here, in the third embodiment, a thread is formed on the rotating shaft 1108 of the servomotor 1107, and the thread of the rotating shaft 1108 and the thread of the sliding portion 1109 are fitted. As a result, the sliding portion 1109 to which the light source lamp 1110 is attached moves in the direction indicated by the arrow in the drawing (the direction in which the illumination principal point 202 of the illumination light source 104 increases or decreases) in accordance with the amount of rotation of the rotation shaft 1108, The position of the light source lamp 1110, that is, the position of the illumination principal point 202 is moved.
[0102]
Therefore, by driving the servo motor 1107, the sliding portion 1109 moves along the rotation axis 1108 together with the light source lamp 1110, and the light source lamp 1110 is moved to the illumination principal point 202 according to the zoom amount and the focus amount.
[0103]
As described above, in the imaging system according to the third embodiment, the positions of the principal points (illumination principal points) with respect to the focus amount and the zoom amount are stored in the look-up table in advance, and the lens remote controller which is a remote controller for a so-called remote control zoom lens is used. From the remote control signal output from 1102, the position of the illumination principal point 202 according to the focus amount and the zoom amount is obtained. Next, since the movement of the light source lamp 1110 is controlled to the lighting principal point 202 based on the obtained position information of the lighting principal point 202, this is the case where the television camera 1101 with a zoom function is used. In addition, it is possible to take a picture without blind spots of the lighting.
[0104]
In the imaging system according to the third embodiment, the light source lamp 1110 is moved to the lens principal point 201 corresponding to the zoom amount and the focus amount of the lens 105 by monitoring a remote operation signal output from the lens remote controller 1102. However, the present invention is not limited to this, and the configuration may be such that the zoom amount and the focus amount of the television camera 1101 are directly detected, and the illumination principal point 202 is moved according to the detected amounts.
[0105]
FIG. 12 is a diagram for explaining a schematic configuration of an illumination light source provided with a mechanism for directly detecting a zoom amount and a focus amount of a television camera. Hereinafter, a mechanism for detecting the zoom amount and the focus amount and the movement of the illumination principal point 202 according to the detected amount will be described with reference to FIG. Note that the illumination light source 104 illustrated in FIG. 12 has another configuration except for a focus ring 1201, a first rotary encoder 1202, a zoom ring 1203, a second rotary encoder 1204, a first counter 1205, and a second counter 1206. Is the same as the illumination light source 104 of the third embodiment shown in FIG.
[0106]
As shown in FIG. 12, in the imaging system according to the third embodiment, the television camera 1101 includes first and second rotary encoders 1202 and 1204 for detecting the movement amounts of the focus ring 1201 and the zoom ring 1203. Has become. That is, the focus amount is detected by detecting the rotation amount of the focus ring 1201, and the zoom amount is detected by detecting the rotation amount of the zoom ring 1203.
[0107]
The output of the first rotary encoder 1202 is counted by a first counter 1205, and the counted value is input to the lookup table search unit 1105. The output of the second rotary encoder 1204 is counted by a second counter 1206, and the counted value is also input to the lookup table search unit 1105.
[0108]
Here, based on the count values from the first and second counters 1205 and 1206, the lookup table search unit 1105 stores a table (not shown) in which the position information of the illumination principal point 202 according to the zoom amount and the focus amount is stored. The illumination principal point 202 suitable for the combination of the zoom amount and the focus amount is searched with reference to the reference. The position information of the illumination principal point 202 obtained by the search is converted into drive power for driving the servo motor 1107 by the principal point position information voltage conversion unit 1106, and the servo motor 1107 is driven. The light source lamp 1110 moves to the main illumination point 202 in the same manner as the illumination light source 104 of the third embodiment. As a result, it is possible to take a picture without eliminating the blind spot of the illumination.
[0109]
At this time, as shown in FIG. 12, the configuration in which the rotation amounts of the focus ring 1201 and the zoom ring 1203 are read out mechanically is more effective than the configuration of the third embodiment shown in FIG. Since the position of the lens principal point 201 can be accurately grasped, accurate control becomes possible.
[0110]
However, the configuration shown in FIG. 12 requires a mechanism for extracting the rotation of the focus ring 1201 and the zoom ring 1203, the rotary encoders 1202 and 1204, and the like. Therefore, the configuration of the third embodiment shown in FIG. It can be configured.
[0111]
It is needless to say that the illumination light source 104 according to the third embodiment can be applied to the imaging systems according to the first and second embodiments.
[0112]
In the imaging systems of the first to third embodiments, the illumination effect removing unit 110 corrects a pixel affected by specular reflection with a pixel value around the pixel, but it is assumed that there is no specular reflection. Needless to say, only the diffuse reflection coefficient may be calculated and used as the reflection coefficient.
[0113]
Further, the photographing systems according to the first to third embodiments are also applicable to a system that combines a video of a person who is the subject 101 and a background video prepared in advance, such as a sticker printing apparatus. In this case, a print sticker without a sense of incongruity can be generated by generating a person image under lighting conditions that match the background image and combining it with the background image. In this case, by utilizing the effect that the lighting conditions can be changed in real time and sequentially, the user can check the degree of synthesis before printing on the sticker, and since there is no blind spot of lighting at the time of shooting, the entire image of the person is A uniform amount of light can be taken.
[0114]
In a simple photographing system such as an existing sticker printing apparatus, a sheet that easily reflects a flashlight is installed behind a person who is the subject 101, and a person area and a background area are determined and synthesized based on the brightness of a photographed image. However, in the present invention, since the distance information is obtained at the same time, the person area can be determined based on the distance information without arranging a special sheet behind the person.
[0115]
In addition, it is difficult to set up good-looking and natural lighting in ordinary households in terms of space, for example, in the conventional telephone or video conference using a communication medium such as the Internet or a telephone line, which is currently spreading, In practice, the lighting installed on the ceiling of the room or the lighting near the videophone is used. However, by applying the invention of the present application, the illumination conditions can be changed in various ways, so that the illumination can be set to a natural normal light.
[0116]
Furthermore, by applying the present invention, it is possible to change the lighting conditions in the new lighting condition setting unit according to the shooting date and time, so that even when indoors, the sense of season, time (for example, autumn evening), etc. The effect of can be expressed automatically. In addition, by using the depth information, an area that is deeper than the person is replaced with another image, so that privacy can be protected by not showing the room to the other party.
[0117]
As described above, the invention made by the inventor has been specifically described based on the embodiment of the present invention. However, the present invention is not limited to the embodiment of the present invention, and does not depart from the gist of the invention. It goes without saying that various changes can be made in.
[0118]
【The invention's effect】
The following is a brief description of an effect obtained by a representative one of the inventions disclosed in the present application.
[0119]
(1) Since the configuration does not require the calculation processing of the specular reflection coefficient which requires an enormous amount of calculation, it is possible to sequentially generate images under predetermined illumination conditions in real time (real time).
[0120]
(2) Lighting effect removal based on a difference image obtained by sequentially performing a subtraction process of a photographed RGB image photographed when irradiation light is not irradiated from a photographed RGB image photographed by irradiation light irradiation. Since the unit calculates the reflection coefficient which does not require the calculation processing of the specular reflection coefficient, even if the subject is illuminated, it is possible to sequentially generate images under predetermined illumination conditions in real time (real time). it can.
[0121]
(3) Since the light source lamp can be moved to the main lighting point according to the zoom amount and the focus amount of the TV camera, even when a zoom lens is used for the TV camera, it is possible to take a picture without eliminating the blind spot of the lighting. it can.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining a schematic configuration of an imaging system according to a first embodiment of the present invention.
FIG. 2 is a diagram illustrating an optical configuration of a measurement unit according to the first embodiment.
FIG. 3 is a diagram for explaining another optical configuration of the measurement unit according to the first embodiment.
FIG. 4 is a diagram for explaining an operation of a normal estimation unit according to the first embodiment;
FIG. 5 is a diagram for explaining an operation of a lighting effect removing unit according to the first embodiment.
FIG. 6 is a diagram for explaining the principle of measuring spectral characteristics in the illumination effect removing unit according to the first embodiment.
FIG. 7 is a diagram for explaining a procedure of calculating a value used for determining brightness in the illumination effect removing unit according to the first embodiment.
FIG. 8 is a diagram for explaining a schematic configuration of a lighting effect removing unit according to the first embodiment.
FIG. 9 is a diagram for explaining a schematic configuration of an imaging system according to a second embodiment of the present invention.
FIG. 10 is a diagram for explaining the collection timing of captured RGB images and the irradiation timing of illumination light by a depth extraction camera according to Embodiment 2.
FIG. 11 is a diagram for explaining a schematic configuration of an illumination light source in a photographing system according to a third embodiment of the present invention.
FIG. 12 is a diagram for explaining a schematic configuration of another illumination light source in the imaging system according to the third embodiment of the present invention.
[Explanation of symbols]
101 subject 102 measuring unit
103: semi-transparent mirror (half mirror) 104: illumination light source
105: Lens # 106: Depth extraction RGB camera
107: normal line estimation unit # 108: incident light amount estimation unit
109: threshold setting unit # 110: lighting effect removing unit
111: New lighting condition setting unit # 112: New lighting condition adding unit
113: spectral colorimeter 測 114: area A
201: Lens principal point 202: Illumination principal point
801: Irradiation incident angle determination unit # 802: Color determination unit
803: brightness determination unit 804: region interpolation unit
805: Diffuse reflected light removal unit
901: Optical shutter # 902: Synchronous generator
903: frequency divider $ 904: image memory unit
905: Difference image generation unit
1101 ... TV camera $ 1102 ... Lens remote control
1103: first A / D converter # 1104: second A / D converter
1105 Lookup table search unit
1106: Principal point position information voltage converter # 1107: Servo motor
1108: Rotating shaft # 1109: Sliding part
1110 ... Light source lamp
1201: Focus ring # 1202: First rotary encoder
1203: Zoom ring # 1204: Second rotary encoder
1205: first counter 第 1206: second counter

Claims (10)

A reflection coefficient calculating means for calculating a reflection coefficient of a subject imaged as an input image, a new lighting condition setting means for setting a lighting condition different from the input image, and based on the set lighting condition and the reflection coefficient. An illumination condition adding unit that generates an image of the illumination condition set by the new illumination condition setting unit from the input image, wherein the input image comprises an image including depth information of the subject, A reflection coefficient calculation unit configured to specify a specular reflection region from the input image based on the depth information of the subject and correct a specular reflection pixel; and a diffusion unit configured to calculate a reflection coefficient of the subject from the corrected input image. An image processing apparatus comprising: a reflection coefficient calculating unit. 2. The image processing apparatus according to claim 1, wherein the correction unit estimates a normal vector of each part of the subject based on depth information of the subject imaged in the input image; An image processing apparatus, comprising: first area specifying means for specifying the specular reflection area in the input image based on a normal vector. 3. The image processing apparatus according to claim 2, wherein the input image includes red, green, and blue captured images captured using illumination light whose spectral characteristics have been measured in advance, and wherein the correction unit performs pixel-by-pixel correction on the input image. A spectral characteristic calculating unit that calculates the spectral characteristics of the pixels, and a region in which the spectral characteristics of each of the pixels and the spectral characteristics of the illumination light are close to each other from the region specified by the first region specifying unit. An image processing apparatus comprising: a second area specifying unit for specifying. 4. The image processing apparatus according to claim 3, wherein the correction unit generates a histogram of luminance values for the input image, and focuses on a luminance value at a peak on a high luminance side of the histogram. A luminance value computing unit that computes a reference luminance value obtained by folding back to a lower luminance side, and a region that is equal to or larger than the reference luminance value from the region specified by the second region specifying unit and is specified as the specular reflection region. An image processing apparatus comprising: a third area specifying unit. The image processing apparatus according to any one of claims 2 to 4, further comprising a threshold setting unit that sets a threshold of the first to third region identification units. Photographing means for photographing a photographed image including depth information of a subject; reflection coefficient calculating means for calculating a reflection coefficient of the subject imaged in the photographed image; and new lighting condition setting means for setting lighting conditions different from the image And an illumination condition adding unit configured to generate an image of an illumination condition set by the new illumination condition setting unit from the captured image based on the set illumination condition and the reflection coefficient. A reflection coefficient calculation unit configured to specify a specular reflection area from the captured image based on the depth information of the subject and correct a specular reflection pixel; and a diffusion unit configured to calculate a reflection coefficient of the subject from the corrected captured image. An imaging system comprising: a reflection coefficient calculating unit. 7. The imaging system according to claim 6, wherein the correction unit estimates a normal vector of each part of the subject based on depth information of the subject imaged in the captured image, and the normal vector estimation unit. An imaging system comprising: a first area identification unit that identifies the specular reflection area in the captured image based on a line vector. 8. The image capturing system according to claim 7, wherein the image capturing unit includes a unit that captures red, green, and blue captured images under illumination light whose spectral characteristics have been measured in advance, and the correction unit includes a pixel for the captured image. A spectral characteristic calculating unit that calculates spectral characteristics for each pixel; and a region where the spectral characteristics of each pixel and the spectral characteristics of the illumination light are close to each other from the region specified by the first region specifying unit, and And a second area specifying means for specifying the image data. 9. The imaging system according to claim 8, wherein the correction unit generates a histogram of luminance values for the photographed image, and calculates a last luminance value of the histogram with a luminance value at a peak on a high luminance side of the histogram as a center. A brightness value calculating means for calculating a reference brightness value folded back to the low brightness side; and a third means for selecting an area having the reference brightness value or more from the area specified by the second area specifying means and specifying the selected area as the specular reflection area. An imaging system comprising: an area specifying unit. The photographing system according to any one of claims 6 to 9, further comprising a threshold setting unit that sets a threshold of the first to third region specifying units.

Images