JP5928024B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for generating a depth map of a captured image.

  2. Description of the Related Art Conventionally, there is known an image processing apparatus that generates a depth map representing depth information of a subject from a plurality of captured images captured by changing the focus position at the same angle of view (see, for example, Patent Document 1). When a depth map is generated, by capturing an image focused on the subject and an image shifted in focus on the background side, and comparing the focus state of each area based on the strength of the edge, etc. It is determined whether the image area is a subject image area or a background image area. Then, based on the determination result, depth information is assigned to each pixel in each region.

JP 2009-232382 A

However, when a moving subject is imaged, it is difficult to appropriately determine whether the region that has disappeared due to the movement and becomes a background is the image region of the subject or the background image region.
In addition, even for an area where a flat subject such as an unpatterned wall is shown, the in-focus state cannot be clearly distinguished, and an appropriate determination is made as to whether it is the subject image area or the background image area. It may not be possible.

Conventionally, an area where it is not possible to clearly determine whether the image area of the subject or the background image area is an area having an intermediate depth between the image area of the subject and the background image area. If not, it will not be an accurate depth map.
Thus, in the conventional technique for generating a depth map, it is difficult to generate an accurate depth map.

  The present invention has been made in view of such a situation, and an object thereof is to generate a more accurate depth map.

In order to achieve the above object, one aspect of the image processing apparatus according to the present invention is to shift the focus to the data of a subject focused image captured by focusing on a specific subject and the background side of the specific subject. Each of the regions included in the subject in-focus image is based on the image acquisition means for acquiring the data of the captured background in-focus image, and the data of the subject in-focus image and the data of the background in-focus image. , determination and subject area having a depth corresponding to the specific object, and a background region having a depth corresponding to the background, and the object area and the non-specific region other than the background area, which one of the regions of A depth value representing a depth corresponding to the specific subject is set in the subject region, and the background region corresponds to the background based on a determination result of the region determining unit A depth value setting means for setting a depth value representative of the go, based on the image of the non-specific region of said subject focused image and the background focus image, of the object area set by the depth value setting means the determined value based on any of the depth value of the depth value or the background region, wherein the depth value expansion processing means for setting a depth value representing the depth of the non-specific region, Bei example, said depth value setting means Sets a tentative depth value for each pixel in the subject area, the background area, and the non-specific area, and the depth value expansion processing means targets each pixel in the non-specific area for the non-specific area. An extended depth value is obtained based on a temporary depth value of at least one extended source pixel that satisfies a predetermined condition from a peripheral region other than the non-specific region adjacent to the extended depth value; Respectively set as the depth value of each pixel of the non-specific region, and wherein the.
In order to achieve the above object, according to one aspect of the image processing method of the present invention, a subject focused image data picked up by focusing on a specific subject and a focus on the background side of the specific subject. A region included in the subject in-focus image based on the image acquisition step of acquiring the data of the background in-focus image captured by shifting, the data of the subject in-focus image and the data of the background in-focus image Whether each of the subject region has a depth corresponding to the specific subject, a background region having a depth corresponding to the background, or a non-specific region other than the subject region and the background region And a depth value representing a depth corresponding to the specific subject is set in the subject region based on the judgment result in the region judging step, and the background In the depth value setting step for setting a depth value representing the depth corresponding to the background in the area, and in the depth value setting step based on the image of the non-specific area in the subject focused image and the background focused image A depth value expansion processing step for setting a value determined based on either the set depth value of the subject area or the depth value of the background area as a depth value representing the depth of the non-specific area, In the depth value setting step, a temporary depth value is set for each pixel in the subject area, the background area, and the non-specific area, and in the depth value expansion processing step, each pixel in the non-specific area is set. And at least one extension source that satisfies a predetermined condition from a peripheral region other than the non-specific region adjacent to the non-specific region It obtains an extended depth value based on the temporary depth value of the unit, respectively set the extended depth value as the depth value of each pixel of the non-specific region, and wherein the.
In order to achieve the above object, an aspect of the program according to the present invention is that a computer focuses on data of a subject in-focus image captured by focusing on a specific subject and on the background side of the specific subject. Included in the subject in-focus image based on the image acquisition function for acquiring the background in-focus image data captured by shifting the image, the subject in-focus image data, and the background in-focus image data Each region is any one of a subject region having a depth corresponding to the specific subject, a background region having a depth corresponding to the background, and a non-specific region other than the subject region and the background region. A depth value representing a depth corresponding to the specific subject is set in the subject region based on a judgment result of the region judgment function and the judgment result of the region judgment function, and the background region A depth value setting function for setting a depth value representing the depth corresponding to the background, and a depth value setting function based on the image of the non-specific region in the subject focused image and the background focused image. A depth value expansion processing function for setting a value determined based on either the depth value of the subject area or the depth value of the background area as a depth value representing the depth of the non-specific area, and The depth value setting function sets a temporary depth value for each pixel in the subject area, the background area, and the non-specific area, and the depth value expansion processing function targets each pixel in the non-specific area. The extended depth based on a temporary depth value of at least one extension source pixel that satisfies a predetermined condition from a peripheral region adjacent to the non-specific region that is adjacent to the non-specific region The calculated, respectively set the extended depth value as the depth value of each pixel of the non-specific region, and wherein the.

  According to the present invention, it is possible to generate a more accurate depth map.

1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram which shows the functional structure for performing a depth map production | generation process among the functional structures of the image processing apparatus of FIG. It is a schematic diagram which shows the specific example of the image referred in a depth map production | generation process. 3 is a flowchart for explaining an example of a flow of depth map generation processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG. 2.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Constitution)
FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus according to an embodiment of the present invention.
The image processing apparatus 1 can be configured by a digital camera, for example.

  The image processing apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an imaging unit 16, and an operation unit. 17, a display unit 18, a storage unit 19, a communication unit 20, and a media drive 21.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 19 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An imaging unit 16, an operation unit 17, a display unit 18, a storage unit 19, a communication unit 20, and a media drive 21 are connected to the input / output interface 15.

  Although not shown, the imaging unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the imaging unit 16.
Hereinafter, the output signal of the imaging unit 16 is referred to as “captured image data”. The captured image data is appropriately supplied to the CPU 11 or the like.

The operation unit 17 is configured with various buttons and the like, and inputs various information according to user instruction operations.
The display unit 18 includes a display, a speaker, and the like, and outputs an image and sound.
The storage unit 19 is composed of a hard disk or a DRAM (Dynamic Random Access Memory) or the like, and stores various image data.
The communication unit 20 has a communication interface such as USB (Universal Serial Bus) or infrared communication, and controls communication performed with other devices (not shown) via a network such as the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the media drive 21. The program read from the removable medium 31 by the media drive 21 is installed in the storage unit 19 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

FIG. 2 is a functional block diagram showing a functional configuration for executing the depth map generation process among the functional configurations of the image processing apparatus 1.
Depth map generation processing is to capture a captured image focused on a subject and a captured image shifted in focus on the background side, determine a subject region and a background region, A series of processing until a depth map is generated that represents the depth distribution in each area in the image by setting the depth information determined based on the depth information (depth value) of the subject area or the background area Say.

When the depth map generation process is executed, in the CPU 11, the imaging control unit 51, the image acquisition unit 52, the image deformation processing unit 53, the region determination unit 54, the expansion source region specification unit 55, and the depth value extension The processing unit 56 functions.
Note that at least some of the functions of the imaging control unit 51 to the depth value expansion processing unit 56 may be transferred to other components (for example, a graphic accelerator (not shown)) other than the CPU 11.

The imaging control unit 51 controls the imaging unit 16 triggered by the operation of the shutter button or the like of the operation unit 17. For example, the imaging control unit 51 performs an operation of pressing down to the middle of the shutter button (a predetermined position that does not reach the lower limit) (hereinafter referred to as “half-pressing operation” or simply “half-pressing”), or the shutter button When there is an operation of pressing down to the lower limit (hereinafter referred to as “full press operation” or simply “full press”), the focusing and exposure in the imaging unit 16 are controlled.
In other words, the imaging control unit 51 controls the imaging unit 16 to focus the focus lens on the subject when a half-press operation is performed.
In addition, when there is a full-press operation, the imaging control unit 51 notifies the imaging unit 16 of imaging instruction information, and an image in which the focus lens is focused on the subject and an image in which the focus is shifted to the background side are displayed. The imaging unit 16 is controlled to continuously capture images.

  When the imaging unit 16 receives information on the imaging instruction from the imaging control unit 51, the imaging image in which the focus lens is focused on the subject (hereinafter referred to as “subject focused image”) and the captured image in which the focus is shifted toward the background side ( (Hereinafter referred to as “background focused image”) and the data of the subject focused image and the background focused image are supplied to the image acquisition unit 52.

  When the image of the subject in-focus image and the background in-focus image is supplied from the image capturing unit 16, the image acquisition unit 52 stores the data in the storage unit 19 and performs image transformation processing on the data of the object in-focus image and the background in-focus image. To the unit 53.

When the image focus processing unit 53 receives data of the subject focused image and the background focused image from the image acquisition unit 52, the image deformation processing unit 53 executes image deformation processing for matching the shift between the two images. That is, the image deformation processing unit 53 performs alignment so that the feature points of the subject focused image and the background focused image overlap. Further, the image deformation processing unit 53 performs image deformation by affine transformation or the like so that the subject focused image and the background focused image overlap more accurately.
Data of the subject focused image and the background focused image subjected to the image deformation processing by the image deformation processing unit 53 is supplied to the region determination unit 54.

The region determination unit 54 includes a moving body region extraction unit 54a, an edge strength calculation unit 54b, and a region type determination unit 54c.
When the moving object region extraction unit 54a receives the data of the subject focused image and the background focused image after the image transformation processing from the image transformation processing unit 53, the luminance and color difference of the subject focused image and the background focused image, etc. Based on the information, the moving object region is extracted.
Here, the moving object region refers to a region in which an imaged object is different as the subject moves from the imaging timing of the subject focused image to the imaging timing of the background focused image. For example, the moving object region is a region where the subject is captured in the subject focused image and the background is captured in the background focused image. Since the pixels in the moving object region have the pixel value (luminance and color difference values) of the subject on one side of the subject focused image and the background focused image and the background pixel value on the other side, the subject focused image and the background focused image There is a large difference in pixel values. Therefore, in the present embodiment, the moving object region extraction unit 54a determines the moving object region based on the difference between the pixel values at corresponding positions of the subject focused image and the background focused image. That is, the pixel values of the pixels at corresponding positions of the subject focused image and the background focused image are compared, and if the pixel value difference is larger than the set threshold value, it is determined that the pixel is a moving object region pixel. .
The moving object region is extracted by repeating the above processing for all the pixels.
As a result of the above processing, information for specifying the moving object region (hereinafter referred to as “moving object region data”) is generated.
The moving body region extraction unit 54a supplies the moving body region data to the region type determination unit 54c.

The edge strength calculation unit 54 b calculates the edge strength of the subject focused image and the background focused image after the image deformation processing supplied from the image deformation processing unit 53.
The edge strength is an index representing the degree of change in luminance and color difference in the image, and the edge strength calculation unit 54b extracts the distribution of the edge strength in the subject focused image and the background focused image. In the present embodiment, the edge strength is calculated by applying a filter capable of detecting an edge, such as a high-pass filter, to the luminance data of the target pixel to be processed and its surrounding pixels.
Data indicating the edge strength of the subject focused image and the background focused image calculated by the edge strength calculating unit 54b is supplied to the region type determining unit 54c.

The region type determination unit 54c performs subject alignment based on the moving object region data supplied from the moving object region extraction unit 54a and the edge strength data of the subject focused image and the background focused image acquired from the edge strength calculation unit 54b. The type of each area included in the focused image and the background focused image is determined.
Specifically, the area type determination unit 54c determines that each area included in the subject focused image and the background focused image includes a moving object area, a subject image area (hereinafter referred to as a “subject area”), and a background image area. (Hereinafter referred to as “background region”) or an image region of a flat part (hereinafter referred to as “flat region”).
Here, the edge strength of the focused object image is larger than the edge strength of the object image when not focused. Therefore, it is possible to determine that the region in which the subject focused image has a higher edge strength than the background focused image is the subject region, and the background focused image has a higher edge strength than the subject focused image is determined to be the background region. it can. Further, it is possible to determine that a region having a small difference in edge strength between the subject focused region and the background focused region and a small absolute value of the edge strength is a flat region.

Therefore, in this embodiment, the region type determination unit 54c compares the edge strength of the subject focused image with the edge strength of the background focused image for each pixel, and the edge strength of the subject focused image is larger ( For example, if it is larger than a set threshold value, the pixel is determined to be a subject area. Similarly, the region type determination unit 54c compares the edge strength of the subject focused image with the edge strength of the background focused pixel, and the edge strength of the background focused image is larger (for example, larger than the set threshold value). ), The pixel is determined to be a background area. Furthermore, when the difference between the edge intensity of the subject focused image and the edge intensity of the background focused pixel is small (for example, below the set threshold), the absolute value of the edge intensity is below the set threshold. In this case, it is determined that the pixel is a flat region. Note that the region type determination unit 54c determines that the region indicated by the moving region data as the moving region is a moving region.
By repeating the above determination for all pixels, information (hereinafter referred to as “determination area data”) in which any one of the subject area, background area, moving object area, and flat area is associated with all pixels. Is generated).
The area type determination unit 54 c supplies the determination area data to the extension source area specifying unit 55.

The extension source area specifying unit 55 acquires the determination area data from the area type determining unit 54c.
The extension source region specifying unit 55 includes a provisional depth value setting unit 55a, a color distance calculation unit 55b, and a related region determination unit 55c.

The provisional depth value setting unit 55a generates a provisional depth map composed of provisional depth values of each pixel (hereinafter, referred to as “temporary depth values” as appropriate) based on the moving object region data and the determination region data. In the temporary depth map, for example, the pixel of the moving object region is 128, the pixel of the subject region is 255, the pixel of the background region is 0, and the pixel of the flat region is associated with each pixel as a temporary depth value. Here, since both the moving object region and the flat region are regions where it is difficult to determine whether the object region is a subject region or a background region, the provisional depth value is an intermediate value between them. Since the moving body area and the flat area are subject areas or background areas, that is, it is difficult to specify the depth value, the moving body area and the flat area are hereinafter collectively referred to as appropriate. This is called “non-specific area”.
The data of the temporary depth map is stored in the RAM 13 and can be referred to as necessary.

  The related area determination unit 55c targets pixels in the moving object area and the flat area (non-specific area) in the subject in-focus image, and the color distance is within a set threshold from surrounding pixels other than the moving object area and the flat area. Search for pixels in the relevant area. Here, the color distance refers to the distance in the color space of the pixel values of two pixels. For example, when the pixel value is represented by luminance and color difference, the color distance can be obtained by adding the square of the luminance difference between two pixels and the square of the color difference difference.

Then, the related area determination unit 55c specifies a pixel whose searched color distance is within a threshold as a pixel from which the depth value is extended to the pixels in the moving object area and the flat area in the subject focused image.
Specifically, the related area determination unit 55c is a pixel other than the moving object area and the flat area located around the target pixel in the moving object area and the flat area in the subject focused image, and the color distance is equal to or less than the threshold value. At least one or more pixels (hereinafter referred to as “extension source pixels”) are identified.
In addition, as a specific condition of the extension source pixel, the pixel whose color distance is equal to or smaller than the threshold is that a pixel having a color close to the moving object region and the flat region is likely to be the same as the moving object region and the flat region. This is because it is considered appropriate as a pixel from which to expand the depth value of the region and the flat region.

The depth value expansion processing unit 56 acquires information on the expansion source pixel from the related region determination unit 55c, and calculates the depth value of the pixel of interest in the moving object region and the flat region based on the provisional depth value of the identified expansion source pixel. (Hereinafter referred to as “extended depth value” as appropriate).
In the present embodiment, when the depth value expansion processing unit 56 acquires the information of the expansion source pixel from the related region determination unit 55c, the depth value of the target pixel in the moving object region and the flat region is specified for the target pixel. Replace with the average of the temporary depth values of the expansion source pixels.
When the above depth value replacement processing is completed for all the pixels in the moving object region and the flat region by the depth value expansion processing unit 56, the expanded depth value based on the temporary depth value of the expansion source pixel in the subject region or the background region Is expanded to the moving object region, and the temporary depth value of the moving object region is replaced.
In other words, the temporary depth value becomes the depth value in the subject area and the background area, and the extended depth value becomes the depth value in the moving object area and the flat area.
The depth value expansion processing unit 56 causes the storage unit 19 to store depth map data in which depth values for the subject area, the background area, the moving object area, and the flat area are set.

Hereinafter, various images referred to in the depth map generation processing will be described with reference to FIG.
FIG. 3 is a schematic diagram illustrating a specific example of an image referred to in the depth map generation process.
In FIG. 3, for the sake of simplicity, a case where the captured image includes a subject area, a background area, and a moving object area, but does not include a flat area will be described as an example.

FIG. 3A is a schematic diagram showing a subject focused image 81.
In FIG. 3A, a subject focused image 81 is captured in a state where the focus lens of the image processing apparatus 1 is focused on the face 92 of the person who is the subject and is not focused on the background forest 91. ing.
FIG. 3B is a schematic diagram showing the background focused image 82.
The background focused image 82 is captured in a state where the focus lens of the image processing apparatus 1 focuses on the background forest 91 and does not focus on the human face 92. Here, in FIG. 3B, the human face 92 has moved to the right since the subject in-focus image 81 in FIG.

FIG. 3C is a schematic diagram showing a provisional depth map 83.
The moving object region extraction unit 54a refers to the subject focused image 81 in FIG. 3A and the background focused image 82 in FIG. 3B, and a pixel whose difference in pixel values (for example, luminance and color difference) is greater than or equal to a threshold value. Are extracted as moving object areas A2 and A4 (non-specific areas) in FIG. Further, the region type determination unit 54c determines the background region A1 and the subject region A3 in FIG. Then, the temporary depth value setting unit 55a sets a temporary depth value for each pixel of the moving object region, the background region, and the subject region.
For example, a depth value 0 is set in the background area A1 in FIG. 3C, a depth value 255 is set in the subject area A3, and a depth value 128 is set in the moving object areas A2 and A4.
As a result, a temporary depth map as shown in FIG.

FIG. 3D is a schematic diagram showing the depth map 84.
The related area determination unit 55c adds pixels other than the moving object area around the pixel whose color distance is close to the pixel values of the moving object areas A2 and A4 in the subject focused image 81 of FIG. Is determined.
That is, in the example illustrated in FIG. 3A, the related area determination unit 55c determines that the extension source pixel whose color distance is close to the pixel value of the pixel in the moving object area A2 is the subject area (the area of the human face 92). judge. Therefore, the depth value expansion processing unit 56 sets the depth value of each pixel in the moving object region A2 to the average value of the depth values of the expansion source pixels in the subject region A3. Further, the related area determination unit 55c determines that the extension source pixel whose color distance is close to the pixel value of the pixel of the moving object area A4 is the background area (the area of the background forest 91). Therefore, the depth value expansion processing unit 56 sets the depth value of each pixel in the moving object region A4 to the average value of the depth values of the expansion source pixels in the background region A1.
As a result, a depth map 84 is generated from the temporary depth map 83. That is, the depth value of the moving object area A2 is an average value of the depth values of the subject area A3, and the depth value of the moving object area A4 is the average value of the depth values of the background area A1, so that the depth value of the object area or the background area is expanded. And becomes a moving body region A5.
Conventionally, a depth value such as an intermediate value between the background and the subject has been set in the moving object region, but a more accurate depth value is set as the depth value of the moving object region by the above processing.

(Operation)
Next, the operation will be described.
FIG. 4 is a flowchart for explaining an example of the flow of depth map generation processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG.

  The depth map generation process is started when an operation for generating a depth map is performed on the operation unit 17 by the user, and the following process is executed.

In step S1, the imaging control unit 51 determines whether or not the user has operated the shutter button.
If the user does not operate the shutter button, it is determined as NO in Step S1, and the process returns to Step S1.
On the other hand, when the user operates the shutter button, it is determined as YES in Step S1, and the process proceeds to Step S2.

In step S2, the imaging control unit 51 determines whether or not the operation of the shutter button by the user is half-pressed.
If the user operates the shutter button halfway, YES is determined in step S2, and the process proceeds to step S3.
If the shutter button operation by the user is not half-pressed, that is, if it is fully pressed, NO is determined in step S2, and the process proceeds to step S4.

  In step S <b> 3, the imaging control unit 51 controls the imaging unit 16 so as to follow and focus on the subject. Thereafter, the process returns to step S1.

  In step S <b> 4, the imaging unit 16 captures the subject focused image focused on the subject and supplies the subject focused image data to the image acquisition unit 52.

  In step S <b> 5, the imaging unit 16 captures the background focused image with the focus shifted to the background side, and supplies the background acquisition image data to the image acquisition unit 52.

  In step S <b> 6, the image transformation processing unit 53 performs image transformation processing for matching the background shift between the subject focused image and the background focused image supplied from the image acquisition unit 52.

  In step S7, the moving object region extraction unit 54a extracts a moving object region (non-specific region) based on information such as luminance and color difference between the subject focused image and the background focused image.

  In step S8, the edge strength calculation unit 54b calculates the edge strength of the subject focused image and the background focused image for each pixel.

In step S9, the region type determination unit 54c compares the edge strength of each pixel in the subject focused image and the background focused image.
When the edge strength of the subject focused image is larger than the edge strength of the background focused image (for example, larger than the set threshold value), YES is determined in step S9, and the process proceeds to step S10.
If the edge strength of the subject focused image is not greater than the edge strength of the background focused image, NO is determined in step S9, and the process proceeds to step S11.

  In step S10, the region type determination unit 54c sets the target pixel as a subject region.

In step S11, the region type determination unit 54c compares the edge strength of each pixel in the subject focused image and the background focused image.
When the edge strength of the subject focused image is smaller than the edge strength of the background focused image (for example, smaller than the set threshold value), YES is determined in step S11, and the process proceeds to step S12.
If the edge strength of the subject focused image is not smaller than the edge strength of the background focused image, NO is determined in step S11, and the process proceeds to step S13.

  In step S12, the region type determination unit 54c sets the target pixel as a background region.

In step S13, the region type determination unit 54c determines the edge strength of the subject focused image and the background focused image for each pixel.
If the edge strength of the subject focused image and the edge strength of the background focused image are equal to or less than the threshold value, YES is determined in step S13, and the process proceeds to step S14.
If the edge intensity of the subject focused image and the edge intensity of the background focused image are not less than or equal to the threshold value, NO is determined in step S13, and the process proceeds to step S15.

  In step S14, the region type determination unit 54c sets the target pixel as a flat region (non-specific region).

  In step S <b> 15, the related area determination unit 55 c specifies an expansion source pixel that is a source for extending the expansion depth value, as the temporary depth value of the pixels in the moving object area and the flat area (non-specific area).

In step S16, the depth value expansion processing unit 56 sets the temporary depth value of the target pixel in the moving object region and the flat region to the extended depth value obtained from the depth value of the expansion source pixel corresponding to the target pixel. Generate a depth map.
In step S <b> 17, the depth value expansion processing unit 56 causes the storage unit 19 to store the generated depth map data. Thereafter, the depth map generation process ends.

As described above, in the present embodiment, when a predetermined operation of the operation unit 17 by the user (a full press operation of the shutter button) is performed, the imaging control unit 51 causes the imaging unit 16 to focus on the subject. And control to capture a background focused image.
When the image acquisition unit 52 acquires the data of the subject focused image and the background focused image from the imaging unit 16, the image acquisition unit 52 supplies the data to the image deformation processing unit 53.
The image deformation processing unit 53 performs image deformation processing so as to match the data shift of the subject focused image and the background focused image.
When the moving object region extraction unit 54a receives the data of the subject focused image and the background focused image after the image transformation processing from the image transformation processing unit 53, the luminance, the color difference, and the like of the subject focused image and the background focused image, etc. Based on the information, the moving object region is extracted.

The edge strength calculation unit 54b calculates the edge strength of the subject focused image and the background focused image.
The region type determination unit 54c selects any one of the subject region, the background region, and the flat region for all pixels based on the edge strength data of the subject focused image and the background focused image acquired from the edge strength calculation unit 54b. It is determined whether it is.
The related area determination unit 55c replaces the depth values of the pixels in the moving object area and the flat area (non-specific area) with the extended depth values calculated from the depth values of surrounding pixels other than the moving object area and the flat area. Specify the original pixel.
The depth value expansion processing unit 56 replaces the provisional depth value of the pixel of interest in the moving object region and the flat region with the average value (expansion depth value) of the depth value of the expansion source pixel specified for the pixel of interest.

In short, in the present embodiment, the moving object region, the subject region, the background region, and the flat region are extracted from the subject focused image and the background focused image, and the depth values of the moving object region and the flat region (non-specific region) are set to the peripheral values. This is set to information (extended depth value) calculated from the depth value of the subject area or the background area.
As a result, conventionally, an unreliable value is stored as a depth value of a region that cannot be clearly determined as a subject region or a background region, such as a moving object region or a flat region, regardless of the actual depth. On the other hand, a highly reliable depth value can be set using the depth value of the subject area or the background area.
Therefore, according to the present invention, it is possible to generate a more accurate depth map.

In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention. For example, in the above-described embodiment, the background focused image may be a captured image that is captured while being focused on the background of the subject, or may be a captured image that is captured while shifting the focus on the background side of the subject. Good.
In addition, when replacing the temporary depth values of the moving object region and the flat region with the extended depth value calculated from the depth value of the extension source pixel, the average value of the depth values of the extension source pixel specified for the target pixel is used. Even a value other than the average value can be replaced with a representative value such as a maximum value, a minimum value, or a mode value.
In addition, the provisional depth values of the moving object region and the flat region are replaced with values calculated from the depth values of pixels other than the moving object region and the flat region around the target pixel. Or it is good also as an average value of the whole moving body area | region.
In the above-described embodiment, the related area determination unit 55c uses the data represented by the color space represented by the luminance and the color difference when calculating the color distance. It is also possible to use data expressed in the color space.

In the above-described embodiment, a digital camera has been described as an example of the image processing apparatus 1 to which the present invention is applied. However, the present invention is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having an image processing function. Specifically, for example, the present invention can be applied to a notebook personal computer, a television receiver, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 2 is merely an example and is not particularly limited. That is, it is sufficient that the image processing apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG. .
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which a program is recorded, the hard disk included in the storage unit 19 in FIG.

In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the order, but is not necessarily performed in chronological order, either in parallel or individually. The process to be executed is also included.
Further, in the present specification, the term “system” means an overall apparatus configured by a plurality of devices, a plurality of means, and the like.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalent scope thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Image acquisition means for acquiring data of a subject in-focus image captured by focusing on a specific subject, and data of a background in-focus image captured by shifting the focus to the background side of the specific subject;
Based on the data of the subject focused image and the data of the background focused image, each of the areas included in the focused subject image corresponds to a subject area having a depth corresponding to the specific subject and the background. Area determining means for determining which area is a background area having a depth to be determined and an area of the subject area and a non-specific area other than the background area;
Based on the determination result of the area determination unit, a depth value representing a depth corresponding to the specific subject is set in the subject area, and a depth value representing a depth corresponding to the background is set in the background area. Setting means;
Related area determination means for determining whether the non-specific area is related to the subject area or the background area based on the image of the non-specific area in the subject focused image and the background focused image; ,
Based on the determination result of the related area determination means, a value determined based on either the subject area set by the depth value setting means or the depth value of the background area is set as the depth of the non-specific area. Depth value expansion processing means for setting as a depth value representing
An image processing apparatus comprising:
[Appendix 2]
The area determination unit specifies, as the non-specific area, at least one of a moving object area moved by the subject focused image and the background focused image and a flat area where a flat subject is imaged. The image processing apparatus according to appendix 1, characterized by:
[Appendix 3]
The related area determination means compares the degree of approximation between the pixel data in the non-specific area and the pixel data in the subject area and the background area adjacent to the non-specific area. The image processing apparatus according to appendix 1 or 2, wherein the depth value of the non-specific region is determined based on the depth value of a region to which pixel data closer to the data belongs.
[Appendix 4]
The related area determination unit compares the degree of approximation based on a color distance between a pixel in the non-specific area and a pixel area in the subject area and the background area adjacent to the non-specific area. The image processing apparatus according to appendix 3.
[Appendix 5]
An image acquisition step of acquiring data of a subject in-focus image captured by focusing on a specific subject, and data of a background in-focus image captured by shifting the focus to the background side of the specific subject;
Based on the data of the subject focused image and the data of the background focused image, each of the areas included in the focused subject image corresponds to a subject area having a depth corresponding to the specific subject and the background. A region determination step for determining which region is a background region having a depth to be determined and a non-specific region other than the subject region and the background region;
Based on the determination result in the region determination step, a depth value representing a depth corresponding to the specific subject is set in the subject region, and a depth value representing a depth corresponding to the background is set in the background region. Configuration steps;
A related region determination step for determining whether the non-specific region is related to the subject region or the background region based on the image of the non-specific region in the subject focused image and the background focused image; ,
Based on the determination result in the related area determination step, a value determined based on either the subject area set in the depth value setting step or the depth value of the background area is set to the depth of the non-specific area. A depth value expansion processing step to set as a depth value representing
An image processing method comprising:
[Appendix 6]
On the computer,
An image acquisition function for acquiring data of a subject in-focus image captured by focusing on a specific subject and data of a background in-focus image captured by shifting the focus to the background side of the specific subject;
Based on the data of the subject focused image and the data of the background focused image, each of the areas included in the focused subject image corresponds to a subject area having a depth corresponding to the specific subject and the background. A region determination function for determining whether the region is a background region having a depth to be determined, and a non-specific region other than the subject region and the background region;
Based on the determination result of the area determination function, a depth value representing a depth corresponding to the specific subject is set in the subject area, and a depth value representing a depth corresponding to the background is set in the background area. Setting function,
A related region determination function for determining whether the non-specific region is related to the subject region or the background region based on the image of the non-specific region in the subject focused image and the background focused image; ,
Based on the determination result of the related area determination function, a value determined based on either the subject area set by the depth value setting function or the depth value of the background area is set to the depth of the non-specific area. Depth value expansion processing function to be set as a depth value representing
A program characterized by realizing.

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Bus, 15 ... Input / output interface, 16 ... Imaging part, 17 ... Operation unit, 18 ... display unit, 19 ... storage unit, 20 ... communication unit, 21 ... media drive, 31 ... removable media, 51 ... imaging control unit, 52 ... Image acquisition unit, 53... Image deformation processing unit, 54... Region determination unit, 54 a... Moving object region extraction unit, 54 b. ... Extension source area specifying section, 55a ... Temporary depth value setting section, 55b ... Color distance calculation section, 55c ... Related area determination section, 56 ... Depth value expansion processing section

Claims (8)

Image acquisition means for acquiring data of a subject in-focus image captured by focusing on a specific subject , and data of a background in-focus image captured by shifting the focus to the background side of the specific subject;
Based on the data of the subject focused image and the data of the background focused image, each of the areas included in the focused subject image corresponds to a subject area having a depth corresponding to the specific subject and the background. and a background area having a depth of, said object area and the non-specific region other than the background area, and the determining area determination means for determining which of the areas of,
Based on the determination result of the area determination unit, a depth value representing a depth corresponding to the specific subject is set in the subject area, and a depth value representing a depth corresponding to the background is set in the background area. Setting means;
Based on either the depth value of the subject region or the depth value of the background region set by the depth value setting means based on the image of the non-specific region in the subject focused image and the background focused image. a depth value expansion processing means for the determined value is set as a depth value representing the depth of the non-specific region,
Bei to give a,
The depth value setting means sets a temporary depth value for each pixel of the subject area, the background area, and the non-specific area,
The depth value expansion processing unit targets at least one pixel in the non-specific region as a target of at least one extension source pixel satisfying a predetermined condition from a peripheral region adjacent to the non-specific region other than the non-specific region. Obtaining an extended depth value based on the depth value, and setting the extended depth value as the depth value of each pixel in the non-specific region,
An image processing apparatus. The region determination means determines at least one of a region of a moving object moved between the subject focused image and the background focused image and a flat region where a flat subject is captured as the non-specific region. The image processing apparatus according to claim 1, characterized by: The depth value expansion processing means further includes related area determination means for determining whether the non-specific area is related to the subject area or the background area,
The related area determining unit based on the determination result, the value determined based on either the de TOPS of the depth value or the background region of the object region set by the depth value setting means, wherein the non-specific The image processing apparatus according to claim 1, wherein the image processing apparatus is set as a depth value representing a depth of the region. The related area determination means compares the degree of approximation between the data of each pixel in the non-specific area and the data of pixels in the peripheral area other than the non-specific area adjacent to the non-specific area, and the approximation The image processing apparatus according to claim 3, wherein at least one or more extension source pixels whose degree satisfies the predetermined condition are specified . The related area determination unit is configured to calculate the approximation based on a color distance between data of each pixel in the non-specific area and data of pixels in the peripheral area other than the non-specific area adjacent to the non-specific area. The image processing apparatus according to claim 4, wherein the degrees are compared. 6. The image processing apparatus according to claim 1, wherein the depth value expansion processing unit sets an average value of provisional depth values of the expansion source pixels as the extended depth value. An image acquisition step of acquiring data of a focused subject image captured by focusing on a specific subject and data of a focused background image captured with a focus shifted to the background side of the specific subject;
Based on the data of the subject focused image and the data of the background focused image, each of the areas included in the focused subject image corresponds to a subject area having a depth corresponding to the specific subject and the background. and a background area having a depth of, said object area and the non-specific region other than the background area, and the determining area determination step of determining which of the areas of,
Based on the determination result in the region determination step, a depth value representing a depth corresponding to the specific subject is set in the subject region, and a depth value representing a depth corresponding to the background is set in the background region. Configuration steps;
Based on either the depth value of the subject region or the depth value of the background region set in the depth value setting step based on the image of the non-specific region in the subject focused image and the background focused image. a depth value expansion processing step of a determined value is set as a depth value representing the depth of the non-specific region,
Only including,
In the depth value setting step, a temporary depth value is set for each pixel of the subject area, the background area, and the non-specific area,
In the depth value expansion processing step, for each pixel in the non-specific region, at least one or more expansion source pixels satisfying a predetermined condition from a peripheral region other than the non-specific region adjacent to the non-specific region. Obtaining an extended depth value based on the provisional depth value, and setting the extended depth value as a depth value of each pixel in the non-specific region,
An image processing method. On the computer,
An image acquisition function for acquiring data of an in- focus image captured by focusing on a specific subject and data of a background in-focus image captured by shifting the focus to the background side of the specific subject;
Based on the data of the subject focused image and the data of the background focused image, each of the areas included in the focused subject image corresponds to a subject area having a depth corresponding to the specific subject and the background. and a background area having a depth of, said object area and the non-specific region other than the background area, and determines the area judgment function which one of the areas of,
Based on the determination result of the area determination function, a depth value representing a depth corresponding to the specific subject is set in the subject area, and a depth value representing a depth corresponding to the background is set in the background area. Setting function,
Based on either the depth value of the subject region or the depth value of the background region set by the depth value setting function based on the image of the non-specific region in the subject focused image and the background focused image. a depth value expansion processing function of the determined value is set as a depth value representing the depth of the non-specific region,
Realized ,
The depth value setting function sets a temporary depth value for each pixel of the subject area, the background area, and the non-specific area,
The depth value extension processing function targets at least one extension source pixel that satisfies a predetermined condition from a peripheral region other than the non-specific region adjacent to the non-specific region for each pixel in the non-specific region. Obtaining an extended depth value based on the depth value, and setting the extended depth value as the depth value of each pixel in the non-specific region,
A program characterized by that.

Images