JP4862004B2 - Depth data generation apparatus, depth data generation method, and program thereof - Google Patents

Description

  The present invention relates to a depth data generation apparatus, a depth data generation method, and a program for generating depth data used when generating stereoscopic data.

  Conventionally, various stereoscopic display devices have been invented, and a method of reproducing stereoscopic data with a stereoscopic display device has been tried. For example, in Patent Document 1, when stereoscopic data is displayed on a three-dimensional display using glasses such as a barrier-type three-dimensional display or polarizing glasses / shutter-type glasses, images of two viewpoints corresponding to the left and right eyes in advance are displayed. A technique is disclosed that allows humans to perceive stereoscopically by preparing and synthesizing on a three-dimensional stereoscopic display. Patent Document 2 discloses a technique for expressing a three-dimensional effect by contrasting two video display surfaces in a DFD (Depth-fused 3-D) display.

To create a 3D image to be displayed on these displays, shoot multiple cameras and synthesize a multi-viewpoint video, or measure depth information to the object displayed in the captured video. It is necessary to take one of the methods of giving three-dimensional information. In particular, three-dimensional information based on depth information is essential for a DFD display. In addition, when it is difficult to shoot with a plurality of cameras arranged side by side, a method of recombining multi-view images based on depth information is often used. As a method for measuring depth information for stereoscopic images, a method of measuring distance by measuring the arrival speed of a reflected wave such as a laser (Patent Document 3), a method of measuring from a stereo image (Patent Document 4), and the like are disclosed. Yes.
JP-A-8-248355 Japanese Patent No. 3022558 JP2007-327956 Japanese Patent No. 3242529

  By the way, the depth data required for the generation of the three-dimensional data is characterized in that it is desired that the depth data is smooth and has no noise region where the depth value is extremely large and small. However, in the prior art, in order to acquire depth information to be displayed on a three-dimensional display, the depth information is acquired by computer vision technology such as stereo measurement, or is measured by a distance sensor or the like. It was mainstream. Here, the former has many measurement constraints, etc., and a high calculation cost is required to improve the quality. If the calculation cost is reduced, the measurement cannot be performed and the depth value is lost. . There is also a problem that noise is likely to occur due to errors caused by measurement. In the latter case, accurate depth data can be acquired in real time, but there are problems that the resolution of depth data is low and dedicated hardware is required.

  Therefore, the present invention provides a depth data generation device, a depth data generation method, and a program thereof that can reduce the loss and noise of the depth information of each pixel of the target image as a stereoscopic image and can generate depth data at high speed. The purpose is to provide.

  The present invention has been made to solve the above-described problem, and includes a separating unit that separates a background image and an object image in an image, and a first outer pixel located on the outer edge among the pixels of the object image. And, in order from the first outline pixel to the inside direction, outline pixel specifying means for repeating the process of specifying the outline pixel of the next outline excluding the specified pixel, and the inside direction from the first outline pixel Depth value assigning means for sequentially assigning depth values in ascending order or descending order to the contour pixels sequentially specified in the above, and depth data for generating depth value data that holds the assigned depth value for each pixel of the object image A depth data generating apparatus comprising: a generating means;

  The present invention also provides a separating unit that separates a background image and an object image in an image, a skeleton pixel extracting unit that extracts a skeleton pixel by performing a thinning process on the object image, and a pixel for each of the skeleton pixels. A distance specifying means for specifying a distance to a depth value calculation target pixel in the object pixel in a normal direction of a line indicated by the skeleton pixel, and a predetermined depth value is assigned to each of the skeleton pixels A depth value assigning unit that calculates and assigns a depth value based on a ratio of the distance in the normal direction from the skeleton pixel for each pixel in the normal direction of a certain pixel of the skeleton pixels; and the target A depth data generating unit configured to generate depth value data that holds the assigned depth value for each pixel of an object image. .

  The present invention is also a depth data generation method of a depth data generation device, wherein a background image and an object image in an image are separated, and a first outline pixel located outside the outline of each pixel of the object image is determined. The process of specifying and repeating the process of specifying the next outline pixel excluding the specified pixel in order from the first outline pixel to the inside direction, and sequentially specifying the outline pixel from the first outline pixel in the inside direction. On the other hand, a depth data generation method is characterized in that depth values are sequentially assigned in ascending order or descending order, and depth value data that retains the assigned depth value is generated for each pixel of the object image.

  Further, the present invention is a depth data generation method of a depth data generation apparatus, wherein a background image and an object image in an image are separated, a thinning process is performed on the object image to extract a skeleton pixel, For each skeleton pixel, specify the distance to the depth value calculation target pixel in the object pixel in the normal direction of the line indicated by the skeleton pixel, and set a predetermined depth value for each skeleton pixel. For each pixel in the normal direction of a certain pixel of the skeleton pixel, a depth value is calculated and assigned based on the ratio of the distance in the normal direction from the skeleton pixel, and each pixel of the object image A depth data generation method characterized by generating depth value data that holds the assigned depth value for a pixel.

  According to another aspect of the present invention, the computer of the depth data generation device identifies a first outline pixel located in the outline of each pixel of the object image, separating means for separating the background image and the object image in the image, Outer pixel specifying means for repeating the process of specifying the next outer outline pixel excluding the specified pixel in order from the first outer pixel in the inner direction, and the outer outline sequentially specified from the first outer pixel in the inner direction. Depth value allocating means for sequentially assigning depth values to pixels in ascending or descending order, and depth data generating means for generating depth value data that holds the assigned depth value for each pixel of the object image. It is a program for.

  Further, the present invention provides a computer of the depth data generation apparatus, separating means for separating a background image and an object image in an image, skeleton pixel extracting means for performing thinning processing on the object image and extracting skeleton pixels, For each of the skeleton pixels, distance specifying means for specifying a distance to the depth value calculation target pixel in the target pixel in the normal direction of the line indicated by the skeleton pixel, each of the skeleton pixels is predetermined. Depth value assignment for each pixel in the normal direction of a certain pixel of the skeleton pixels, and calculating and assigning the depth value based on the ratio of the distance in the normal direction from the skeleton pixel Means for generating depth value data for holding the assigned depth value for each pixel of the object image, Is a program.

  According to the present invention, since the depth value is given to all the pixels of the object image in the video, there is no loss of the depth value that is inconvenient for stereoscopic viewing. Further, since the depth value is given in units of object images, noise can be suppressed. In addition, real-time processing can also be expected because there is almost no calculation cost as in conventional stereo measurement. On the other hand, all that is needed is a camera and no dedicated hardware such as a distance sensor is required. According to the present invention, depth data suitable for generating a stereoscopic image can be acquired in real time without requiring dedicated hardware.

(First embodiment)
First, a depth data generating apparatus according to a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a depth data generating apparatus according to the first embodiment.
In this figure, reference numeral 1 denotes a depth data generation device. The depth data generation device 1 includes processing units such as a video reception unit 11, a background separation unit 12, an outline image specification unit 13, a depth value assignment unit 14, a depth data generation unit 15, and an output unit 16.

  In the first embodiment, in the depth data generation device 1, in the video reception unit 11, the captured video is received, and the background separation unit 12 is an object that is a background image and a subject in each image of the received video. Separate images. Then, the outline pixel specifying unit 13 specifies the first outline pixel located outside the outline of each pixel of the object image, and sequentially removes the specified pixels in the inner direction from the first outline pixel. The process of specifying the outline pixel of the outline is repeated. The depth value assigning unit 14 sequentially assigns depth values in ascending order or descending order to the outline pixels sequentially specified in the inner direction from the first outline pixel. Then, the depth data generation unit 15 generates depth value data that holds the depth value assigned to each pixel of the object image, and the output unit 16 outputs the depth data of each image in the video. This depth data is used when generating a stereoscopic image in a video display device or the like.

FIG. 2 is a diagram illustrating a processing flow of the depth data device according to the first embodiment.
FIG. 3 is a diagram illustrating a processing outline of the depth data device according to the first embodiment.
Next, details of processing of the depth data device 1 according to the first embodiment will be described with reference to FIGS. 2 and 3.
First, it is assumed that the depth value D _n of n = 0 and the depth value D ₀ of n = 0 are input to the depth data device 1 (step S101) and recorded in a memory or the like. The value of the depth value _{D n} is a value calculated by a particular function f (n) {n = 0 , ···, N}. Then, the video reception unit 11 of the depth data device 1 receives video data (step S102).

  Then, as shown in FIG. 3A, the background separation unit 12 separates the background image and the object image that is the subject for each image of the video data (step S103). For this separation technique, various methods can be used. For example, the separation is performed by performing background separation processing or chroma key processing. In the background separation processing, video data obtained by photographing only the background is stored in advance before the video is photographed, and is read from the image in the video data accepted by the video accepting unit 11 and the image data of only the background. This is a technique of comparing a background image with a pixel unit and separating it as a background image when the pixel values are equal, and as an object image that is a subject when the pixel values are different. Also, chroma key processing stores chroma key data (blue back image data) corresponding to the background image in advance, and sets a background surface that matches the color of the chroma key data on the back of the object that is the subject at the time of shooting. To do. Then, the subject is photographed, and the image in the video data received by the video receiving unit 11 is compared with the chroma key data in units of pixels. When the pixel values are equal, the background image is used. This is a technique for separating an image as an object image.

  Note that the background difference processing has the advantage that you can shoot without preparing a chroma key background in advance, but noise remains during separation due to subtle differences in lighting between the background shooting and shooting and the strength of the outline of the background video There is a drawback. On the other hand, the chroma key processing has the advantage that it is easy to separate the background and the foreground without generating noise, but the chroma key background must be prepared in advance and the foreground of the same color as the chroma key background can be obtained. There is a drawback that you have to be careful not to. In addition, there are existing techniques such as Japanese Patent Application No. 2001-194610 that only separate the foreground image and the object image, and either method may be used.

When the background separation unit 12 finishes the separation of the background image and the object image, the outer image specifying unit 13 next specifies a pixel in which at least one side of the pixel of the object image is in contact with the pixel of the background image. The set of these pixels is identified as the outermost pixel (first outer pixel) (step S104). Then, as shown in FIG. 3B, the depth value assigning unit 14 assigns an initial depth value D ₀ to the outermost pixel (step S105). Note that when the object image is two or more, based on the size and brightness of each object image is assigned a depth value D ₀ differ in the initial value set in advance by statistical processing, respectively. For example, the depth value D ₀ that varies depending on the size and brightness is recorded in the table, and the depth value D ₀ of the object image may be calculated by interpolation calculation.

Then, the outline image specifying unit 13 determines whether or not the depth values have been assigned to all the pixels in the object image (step S106). If no depth value has been assigned to all the pixels, n is increased by one. Then (step S107), the next outermost pixel is specified by the same process among the pixels of the object image excluding the outermost pixel (step S108). Then, as shown in (FIG. 3 (c)), the depth value assigning unit 14 assigns a depth value _{D 1} relative to the outermost pixels of the second (step S109). The value of the depth value D ₁ is a value obtained by substituting n = 1 to the function f (n). Then, the outline image specifying unit 13 repeats the processes of steps S106 to S109 until it determines that the depth value has been assigned to all the pixels n (n = 0 to N) in the object image.

When depth values are assigned to all the pixels of the object image (FIG. 3D), the depth data generation unit 15 then corresponds to the object image with a depth value of 0 for the pixels corresponding to the background image. for pixels to generate depth data that holds information about the depth value D _n assigned respectively the processing (step S110). Then, the output unit 16 outputs the generated depth data (step S111). In the above-described processing, the background separation unit 12, the outline image specification unit 13, the depth value assignment unit 14, the depth data generation unit 15, and the output unit 16 are described above for all images in the video received by the video reception unit 11. Repeat the process. The depth data is used when generating a stereoscopic image in a video display device that is an output destination.

(Second Embodiment)
Next, a depth data generating apparatus according to a second embodiment of the present invention will be described with reference to the drawings.
FIG. 4 is a block diagram showing the configuration of the depth data generating apparatus according to the second embodiment.
As shown in this figure, the depth data device according to the second embodiment is replaced with a skeleton pixel extraction unit 17, a distance specification unit 18, instead of the outline image specification unit 13 of the depth data device 1 according to the first embodiment. The processing part is provided. The other processing units are the same.

  In the second embodiment, in the depth data generation device 1, the video reception unit 11 receives a captured video, and the background separation unit 12 displays a background image and an object image that is a subject in each image of the received video. To separate. Then, the skeleton pixel extracting unit 17 performs thinning processing on the object image and extracts skeleton pixels. Further, the distance specifying unit 18 specifies the distance to each pixel of the skeleton pixel to the depth value calculation target pixel in the target pixel in the normal direction of the line indicated by the skeleton pixel. Then, the depth value assigning unit 14 assigns a predetermined depth value to each of the skeleton pixels, and for each pixel in the normal direction of a certain pixel of the skeleton pixels, the distance in the normal direction from the skeleton pixel is set. A depth value is calculated and assigned based on the ratio, and the depth data generation unit 15 generates depth value data that holds the assigned depth value for each pixel of the object image. As in the first embodiment, this depth data is used when generating a stereoscopic image in a video display device or the like.

FIG. 5 is a diagram illustrating a processing flow of the depth data device according to the second embodiment.
FIG. 6 is a diagram showing a processing outline of the depth data device according to the second embodiment.
Next, details of processing of the depth data device 1 according to the second embodiment will be described with reference to FIGS. 5 and 6.
First, it is assumed that the depth value D _n of n = 0 and the depth value D ₀ of n = 0 are input to the depth data device 1 (step S201) and recorded in a memory or the like. The value of the depth value _{D n} is a value calculated by a particular function f (n) {n = 0 , ···, N}. Similar to the first embodiment, when there are two or more object images, different depth values D ₀ that are initial values predetermined by statistical processing based on the size, brightness, and the like of each object image. May be assigned respectively. For example, the depth value D ₀ that varies depending on the size and brightness is recorded in the table, and the depth value D ₀ of the object image may be calculated by interpolation calculation. Then, the video reception unit 11 of the depth data device 1 receives video data (step S202).

  Then, the background separation unit 12 separates the background image and the object image that is the subject for each image of the video data (step S203). This process is the same as step S103 of the first embodiment. When the separation of the background image and the subject object image is completed, the skeleton pixel extraction unit 17 performs thinning processing on the object image and extracts the skeleton pixels as shown in FIG. (Step S204). The thinning process is a process of taking out the object image body and adjusting a line having a uniform thickness to a line having the same thickness. Specifically, within the set of pixels of the object image, the image is scanned from the upper left and the pixels outside are sequentially deleted while retaining the pixel connectivity (4-connected or 8-connected), and the process is repeated. Finally, a process in which one pixel is connected in contact with another pixel on at least one of the four sides, or a line connected in contact with another pixel at at least one of the four vertexes of the pixel. It is. For example, a thinning method of Hilditch is used. Thereby, the skeleton pixel as shown in FIG. 6A can be specified.

Next, the distance specifying unit 18 specifies one pixel i of the skeleton pixels (step S205), and the depth value assigning unit 14 has a depth value D that is an initial value for the pixel i of the skeleton pixels. ₀ is assigned (step S206). Further, as shown in FIG. 6B, the distance specifying unit 18 specifies a pixel j in the object image in the normal direction of the line that is a set of skeleton pixels from the specified pixel i (step S207). ), The distance L (ij) from the pixel i to the pixel j is specified (step S208). Further, the depth value assigning unit 14 uses the distance L (ij) from the pixel i to the pixel j and the proportionality constant R to set the depth value Dj of the pixel j by Dj = D ₀ + R · L (ij). Calculate (step S209). Then, the depth value assigning unit 14 assigns the depth value Dj calculated in step S209 to the pixel j in the normal direction with respect to the pixel i (step S210).

  Then, the depth value assigning unit 14 determines whether or not the processing from step S207 to step S210 has been performed on all the pixels in the object image in the normal direction from the pixel i identified in step S205 (step S205). If not, the process from step S207 to step S210 is performed for the pixel j in the next normal direction. The normal direction is, for example, when the skeleton pixel to be processed is in contact with at least another skeleton pixel at the vertex, and the two vertices other than the vertex that is in contact with the other pixel and the opposite vertex If the skeleton pixel to be processed is in contact with another skeleton pixel on both the side where the skeleton pixel is to be processed and the side opposite to that side, the direction of the pixel in contact with The direction of the pixels in contact with the two sides other than the sides (that is, the direction of the vertical or horizontal pixels) is determined depending on how the normal pixels are determined according to the form in which each pixel in the skeleton pixel is connected. Just decide.

  Next, the distance specifying unit 18 determines whether or not the above-described processing from step S205 to step S210 has been performed for all the skeleton pixels (step S212). The pixel i is specified, and the processing from S205 to S210 is repeated (FIG. 6C). Here, the same depth value as an adjacent pixel is given to the pixel in the object image that is not in the normal direction of the skeleton pixel.

After the depth value is assigned to all the pixels of the object image, the depth data generation unit 15 assigns the depth value to 0 for the pixel corresponding to the background image and assigns the pixel corresponding to the object image in the above step. Depth data holding the information of the obtained depth value D _j is generated (step S213). Then, the output unit 16 outputs the generated depth data (step S214). In the above-described processing, the background separation unit 12, the outline image specification unit 13, the depth value assignment unit 14, the depth data generation unit 15, and the output unit 16 are described above for all images in the video received by the video reception unit 11. Repeat the process. The depth data is used when generating a stereoscopic image in a video display device that is an output destination.

  As described above, the embodiment of the present invention has been described. However, according to the processes of the first embodiment and the second embodiment described above, since the depth value is given to all the pixels of the object image in the video, the stereoscopic view is obtained. Defects in depth values that are inconvenient to do are eliminated. Further, since the depth value is given in units of object images, noise can be suppressed. In addition, real-time processing can also be expected because there is almost no calculation cost as in conventional stereo measurement. On the other hand, all that is needed is a camera and no dedicated hardware such as a distance sensor is required. According to the present invention, depth data suitable for generating a stereoscopic image can be acquired in real time without requiring dedicated hardware.

  The depth data generation apparatus described above has a computer system inside. Each process described above is stored in a computer-readable recording medium in the form of a program, and the above process is performed by the computer reading and executing the program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of the depth data generation apparatus by 1st Embodiment. It is a figure which shows the processing flow of the depth data apparatus by 1st Embodiment. It is a figure which shows the process outline | summary of the depth data apparatus by 1st Embodiment. It is a block diagram which shows the structure of the depth data generation apparatus by 2nd Embodiment. It is a figure which shows the processing flow of the depth data apparatus by 2nd Embodiment. It is a figure which shows the process outline | summary of the depth data apparatus by 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Depth data generation apparatus 11 ... Video | video reception part 12 ... Background separation part 13 ... Outer image specific part 14 ... Depth value allocation part 15 ... Depth data generation part 16 ... Output unit 17 ... skeleton pixel extraction unit 18 ... distance specifying unit

Claims (6)

Separating means for separating the background image and the object image in the image;
A process of specifying a first outline pixel located in the outline of each pixel of the object image, and specifying an outline pixel of the next outline excluding the specified pixels in order from the first outline pixel inward. A contour pixel specifying means for repeating
Depth value assigning means for sequentially assigning depth values in ascending order or descending order to the outline pixels sequentially specified in the inner direction from the first outline pixel;
Depth data generating means for generating depth value data for holding the assigned depth value for each pixel of the object image;
A depth data generating apparatus comprising: Separating means for separating the background image and the object image in the image;
Skeleton pixel extraction means for performing thinning processing on the object image and extracting skeleton pixels;
For each pixel of the skeleton pixel, distance specifying means for specifying a distance to a depth value calculation target pixel in the target pixel in the normal direction of the line indicated by the skeleton pixel;
A predetermined depth value is assigned to each of the skeleton pixels, and the depth of each pixel in the normal direction of a certain pixel of the skeleton pixels is determined based on the ratio of the distance in the normal direction from the skeleton pixel. Depth value assigning means for calculating and assigning values;
Depth data generating means for generating depth value data for holding the assigned depth value for each pixel of the object image;
A depth data generating apparatus comprising: A depth data generation method of a depth data generation device,
Separate the background image and the object image in the image,
A process of specifying a first outline pixel located in the outline of each pixel of the object image, and specifying an outline pixel of the next outline excluding the specified pixels in order from the first outline pixel inward. Repeat
Sequentially assign depth values in ascending or descending order to the contour pixels sequentially identified in the inner direction from the first contour pixel;
Depth data generation method for generating depth value data that holds the assigned depth value for each pixel of the object image. A depth data generation method of a depth data generation device,
Separate the background image and the object image in the image,
Perform thinning processing on the object image to extract skeleton pixels,
For each pixel of the skeleton pixel, specify the distance to the depth value calculation target pixel in the object pixel in the normal direction of the line indicated by the skeleton pixel,
A predetermined depth value is assigned to each of the skeleton pixels, and the depth of each pixel in the normal direction of a certain pixel of the skeleton pixels is determined based on the ratio of the distance in the normal direction from the skeleton pixel. Calculate and assign values,
Depth data generation method for generating depth value data that holds the assigned depth value for each pixel of the object image. Depth data generator computer
Separating means for separating the background image and the object image in the image,
A process of specifying a first outline pixel located in the outline of each pixel of the object image, and specifying an outline pixel of the next outline excluding the specified pixels in order from the first outline pixel inward. Outer pixel identification means that repeats,
Depth value assigning means for sequentially assigning depth values in ascending order or descending order to the outline pixels sequentially specified in the inner direction from the first outline pixel;
Depth data generating means for generating depth value data for holding the assigned depth value for each pixel of the object image;
Program to function as. Depth data generator computer
Separating means for separating the background image and the object image in the image,
Skeleton pixel extracting means for performing thinning processing on the object image and extracting skeleton pixels;
For each pixel of the skeleton pixel, a distance specifying means for specifying a distance to a depth value calculation target pixel in the target pixel in a normal direction of a line indicated by the skeleton pixel;
A predetermined depth value is assigned to each of the skeleton pixels, and the depth of each pixel in the normal direction of a certain pixel of the skeleton pixels is determined based on the ratio of the distance in the normal direction from the skeleton pixel. Depth value assigning means for calculating and assigning values,
Depth data generating means for generating depth value data for holding the assigned depth value for each pixel of the object image;
Program to function as.

Images