JP6336148B2 - Image processing apparatus, image processing method, image processing program, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing technique for estimating a subject distance from a plurality of captured images.

  A technique for generating an image with a shallow depth of field from an image with a deep depth of field by image processing is known (see, for example, Patent Documents 1 and 2). Generation of an image with a shallow depth of field can be realized by creating a distance map that represents the distance distribution of the subject, separating the main subject and the background based on the created distance map, and blurring only the background image. it can.

Japanese Patent Application Laid-Open No. 07-021365 JP 09-181966 A

  In general, the distance map is created by analyzing a plurality of captured images. Specifically, the distance is estimated based on the DFD method (Depth from Defocus method) that estimates the distance from the difference in the blur of multiple images with different focus positions and the principle of triangulation from the correspondence of pixels between images. There is a stereo method. In contrast to the active method of acquiring the distance by irradiating the subject with ultrasonic waves or infrared rays at the time of photographing, these methods are called passive methods and are often used because they do not require a special device.

However, since these passive methods analyze a captured image, the subject distance may not be estimated correctly depending on the shooting scene. The DFD method described above has a feature that there is no need to search for correspondence between pixels as in the stereo method, but since the blur in the image is used as a clue for calculating the distance, an area lacking in the clue is included in the image. If there is, there is a problem that an incorrect distance map is generated.
If an incorrect distance map is used to generate an image with a shallow depth of field, a correct blur effect cannot be obtained in an area where the distance map is not correctly obtained. That is, there is a problem in that the image quality is significantly deteriorated visually, such as a main subject area that is not desired to be blurred or a background area that is not blurred.

  In order to solve the above-described problem, the present invention reduces an error in an estimated subject distance, thereby reducing image quality deterioration in blur addition processing, an image processing method, an image processing program, and An object is to provide an imaging device.

In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus for processing a plurality of images having different blurring methods, and a displacement detection means for detecting a displacement direction between the plurality of images. And distance map generating means for determining distance information of the subject from the plurality of images and generating a distance map, and determining reliability information indicating reliability for each distance information of the distance map based on the displacement direction, And a reliability map generating means for generating a reliability map corresponding to the distance map .

The image processing method according to the present invention is an image processing method for processing a plurality of images with different blurring methods, a detection step for detecting a displacement direction between the plurality of images, and a subject from the plurality of images. A distance map generating step for generating a distance map by determining distance information of the distance, and determining reliability information representing reliability for each distance information of the distance map based on the positional deviation direction, and reliability corresponding to the distance map And a reliability map generation step of generating a map.

  According to the present invention, it is possible to provide an image processing device, an image processing method, an image processing program, and an imaging device that can reduce image quality degradation in the blur addition processing by reducing an error in the estimated subject distance. Can do.

1 is a block diagram illustrating a configuration of an imaging device. The block diagram which shows the structure of a depth-of-field control circuit. The operation | movement flowchart figure of an imaging device. The figure which shows the example of a focus image and a defocus image. The figure explaining the production | generation process of a reliability map. The figure explaining the outline | summary of the alignment process. The figure explaining the outline | summary of a distance map correction process. The figure explaining the principle which estimates a distance by DFD method.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the examples illustrated in the description of the embodiments.
The imaging apparatus according to the present embodiment has a function of capturing a plurality of images with different focus positions and a function of generating an image with a shallow depth of field by processing the captured images.

<System configuration>
FIG. 1A is a system configuration of the imaging apparatus according to the present embodiment. The imaging device 1 is a digital still camera, a digital video camera, or the like that includes the image processing device according to the present invention.

Reference numeral 100 denotes a photographic lens that guides subject light to the image sensor 102. Reference numeral 101 denotes an exposure control member including an aperture and a shutter. Subject light incident via the photographing lens 100 is incident on the image sensor 102 via the exposure control member 101.
The image sensor 102 is an image sensor that converts subject light into an electrical signal and outputs it, and is typically composed of an image sensor such as a CCD or CMOS.
The image forming circuit 103 is an image forming circuit for digitizing an analog signal output from the image sensor 102 to form an image. The image forming circuit 103 includes an analog / digital conversion circuit, an auto gain control circuit, an auto white balance circuit, a pixel interpolation processing circuit, a color conversion processing circuit, etc. (not shown). It also includes a depth of field control circuit for changing the depth of field of the formed image. The depth-of-field control circuit corresponds to the image processing apparatus according to the present invention.

The exposure control unit 104 is means for controlling the exposure control member 101. The distance measurement control unit 105 is a means for controlling the focusing of the photographic lens 100. The exposure control unit 104 and the distance measurement control unit 105 are controlled using, for example, a TTL method (Through the Lens, a method for controlling exposure and focus by measuring light actually passing through a photographing lens). .
The system control circuit 106 is a control circuit that controls the operation of the entire imaging apparatus 1. Control of the optical system for photographing and control for digital processing of the photographed image are performed.
The memory 107 is a memory using a flash ROM or the like for recording operation control data used in the system control circuit 106 and a processing program. The nonvolatile memory 108 is an electrically erasable and recordable nonvolatile memory such as an EEPROM that stores information such as various adjustment values.
The frame memory 109 is a frame memory that stores several frames of images generated by the image forming circuit 103. The memory control circuit 110 is a memory control circuit that controls image signals input to and output from the frame memory 109. The image output unit 111 is an image output unit for displaying an image generated by the image forming circuit 103 on an image output device (not shown).

Next, the configuration of the depth of field control circuit included in the image forming circuit 103 will be described with reference to FIG.
Note that the image processing apparatus according to the present invention, that is, the depth-of-field control circuit in the present embodiment, may be realized using a dedicated circuit or may be realized by a computer. When implemented by a computer, each unit illustrated in FIG. 1B functions by loading a program stored in the auxiliary storage device into the main storage device and executing it by the CPU. (The CPU, auxiliary storage device, and main storage device are not shown)

  The alignment unit 201 is a unit that aligns a plurality of captured images, and includes a positional deviation detection unit according to the present invention. In this embodiment, since the DFD method is used for distance estimation, processing is performed using two images, a focus image and a defocus image. Since these two images are images taken continuously, the positions of the subjects may not be completely matched. The positioning unit 201 is a means for acquiring a shift width and / or shift direction between images for each region, detecting a positional shift, and aligning the positions. In the following description, the focus image refers to an image focused on the subject, and the defocus image refers to an image in which the focus image and the subject are blurred.

The distance map estimation unit 202 is means for applying a DFD method to two images, estimating the subject distance for each region, and creating a distance map representing the distribution of the distance on the image. Details of the DFD method will be described later.
The reliability map calculation unit 203 determines, for each region, reliability indicating how reliable the distance map created by the distance map estimation unit 202 is, and calculates a reliability map that represents the distribution of the reliability on the image. It is a means to create. Details of the reliability calculation method will be described later. The reliability map calculation unit 203 is reliability information calculation means in the present invention.

The distance map correction unit 204 is a unit that corrects the distance map based on the reliability map created by the reliability map calculation unit 203 and creates a corrected distance map. A method for correcting the distance map will be described later. The distance map estimation unit 202 and the distance map correction unit 204 are distance information calculation means in the present invention.
The depth-of-field control unit 205 is a unit that adds blur to a captured image based on the corrected distance map created by the distance map correcting unit 204. The depth of field control unit 205 is depth of field control means in the present invention.

<Process flowchart>
Next, the operation of the imaging apparatus 1 will be described in detail using a processing flowchart.
FIG. 2A is a flowchart illustrating processing from when the imaging device 1 captures a subject to adding predetermined blur to the captured image, that is, processing for generating and displaying an image with a shallow depth of field. is there.

When a shooting start button (not shown) is pressed, step S201 is started.
In step S201, first, exposure control, focus control, and the like are performed by the exposure control unit 104 and the distance measurement control unit 105, and shooting conditions are determined. Thereafter, the image sensor 102 photoelectrically converts the subject image formed through the optical system 100 to generate an analog signal corresponding to the subject luminance. Thereafter, the analog image signal generated by the image sensor 102 is converted into a digital image via the image forming circuit 103 and recorded in the frame memory 109. The image obtained in step S201 is a focus image.

  In step S202, shooting is performed in the same manner by changing shooting conditions such as a focus position, an aperture stop, and a focal length so that the subject blurs with respect to the image shot in step S201. The shooting conditions may be changed in any way as long as different degrees of blur can be obtained for the same subject. The captured image is similarly recorded in the frame memory 109. The image obtained in step S202 is a defocused image. Note that it is preferable that the shooting in step S201 and the shooting in step S202 are performed continuously in a short time so that there is no deviation in the position of the subject.

In step S <b> 203, the image forming circuit 103 performs image processing on the image recorded in the frame memory 109. The image processing is processing such as white balance, pixel interpolation, color conversion processing, noise reduction processing, and the like.
In step S203, the depth-of-field control circuit included in the image forming circuit 103 estimates the distance map for the image recorded in the frame memory 109, and the subject of the image based on the estimated distance map. Performs processing to change the depth of field. Hereinafter, these two processes are referred to as a depth of field control process. The execution order of the depth-of-field control process and other image processes may be set so that the obtained image is optimal, and is not particularly limited. The same applies to other parameters in image processing.

FIG. 2B is a flowchart showing details of the depth-of-field control process performed in step S203.
First, in step S211, the reliability map calculation unit 203 analyzes the focus image and creates a reliability map. The reliability map is a map representing the reliability of the distance map created in step S213, and is a map representing how accurate the distance to the subject estimated by the DFD method is. This reliability map may be a map divided for each pixel included in the captured image, or may be a map divided into specific small blocks such as a rectangular area. There is no restriction on the division size or shape. In the present embodiment, the captured image is divided into rectangular areas.

  A specific example will be described with reference to FIGS. 3 (a) and 3 (b). FIG. 3A shows the focus image taken in step S201, and FIG. 3B shows the defocus image taken in step S202.

The reliability map is generated based on the focus image shown in FIG. Specifically, the reliability is determined for each divided rectangular area and assigned to the reliability map. Specifically, the reliability is a value indicating how easily the influence of defocusing appears in the area. The fact that it is an area where the influence of defocusing is difficult to occur means that it is an area that lacks clues for calculating the distance by the DFD method, so an area with low reliability is an area where the calculated distance is not accurate. It means that.
For example, in the case of the example in FIG. 3A, a portion with high luminance (reference numeral 302) such as the sun and a portion with no luminance change such as sky (reference numeral 303) are not easily affected by defocusing. Since it is a part where it is difficult to detect the difference, it is considered that the distance cannot be accurately estimated, that is, a part with low reliability.

The reliability corresponding to the area, that is, the ease of detecting the difference in blur can be determined based on the luminance value, saturation value, hue, frequency characteristic of the area, and the like. Specifically, the following areas can be determined to be areas with low reliability.
Example 1) The case where the average of the luminance values of the pixels in the region is out of the predetermined range. Example 2) The case where the luminance value of the pixels in the region is not sufficiently changed. Example 3) The spatial frequency in the region is predetermined. When lower than the value As described above, a plurality of evaluation criteria can be used for determining the reliability. Moreover, you may use evaluation criteria other than what was illustrated.

An example in which a plurality of evaluation criteria are applied to the focus image in FIG. 3A to determine reliability for each region and a reliability map is generated will be described with reference to FIG. Here, for the sake of explanation, it is assumed that the reliability map is divided into 4 × 4 16 regions.
FIG. 4A shows a reliability map generated based on the evaluation criterion “giving low reliability to a region where the average luminance is higher than the threshold”. In addition, FIG. 4B shows a reliability map generated based on the evaluation criterion “giving low reliability to a region where the luminance change width is lower than the threshold”. The reliability value to be assigned may be expressed by an arbitrary numerical value, or may be expressed by a numerical value of 0 to 1 by normalizing the maximum value to the minimum value. There is no particular rule on the expression method.
In the present embodiment, in step S211, the reliability map shown in FIG. 4A and the reliability map shown in FIG. 4B are generated. In the present embodiment, two reliability maps are generated based on different evaluation criteria. However, the number of generated reliability maps may be any number or may be integrated and output.

Next, in step S212, the alignment unit 201 detects a misalignment between the focus image and the defocus image, and the reliability map calculation unit 203 further generates a reliability map based on the detection result.
The alignment is a process for minimizing the influence of a change in image magnification due to a change in shooting conditions and a movement of the camera. In addition, the reliability map generated in step S212 using the positional deviation amount (movement amount) and / or the positional deviation direction (movement direction) at the time of alignment, that is, the positional deviation between the focus image and the defocused image. Generate another confidence map. The reason why the reliability map is generated from the misalignment is that when the corresponding areas of the focus image and the defocus image are greatly separated, the distance estimated by the DFD method is expected to be inaccurate in the area. is there.
The alignment process is performed by dividing an image into a plurality of regions and calculating a motion vector for each region. FIG. 5 is a diagram showing an outline of the alignment process.

  FIG. 5A shows the result of block matching performed by dividing an image into small regions (not shown). Each arrow indicated by reference numeral 501 represents a motion vector calculated for the minute area. Then, these motion vectors are collected for each 4 × 4 large area indicated by reference numeral 502, and a representative motion vector 503 that is a motion vector for the area is calculated. FIG. 5B shows a representative motion vector for each region. The representative motion vector can be calculated using an average value, a median value, or the like. This calculation method is not limited to a specific method.

Here, consider a case where only the head of the main subject 301 in FIG. 3 moves to the right between the focus image and the defocus image. Then, only in the region to which the head belongs, the direction of the representative motion vector 504 is different from the surroundings. Therefore, it can be determined that the region to which the head belongs is a region where the correspondence between the focus image and the defocus image is not sufficient, and the reliability of the estimated distance is low.
In step S212, as the direction and magnitude of the representative motion vector in a certain area differ from the surrounding representative motion vectors, the reliability assigned to the area is reduced. For example, an average value of representative motion vectors of a plurality of regions around the target region may be calculated, and the reliability may be reduced as the difference from the average value increases.
FIG. 4C is a reliability map generated by the alignment process. As described above, the portion of the region 403 where the representative motion vector is different from the surrounding area has low reliability.
In the present embodiment, the reliability is calculated after calculating the representative motion vector. However, the reliability is not necessarily calculated as such, and the reliability may be directly calculated from the motion vector of the minute region. In addition, the movement of the subject may be detected by obtaining the difference between the focus image and the defocus image, and the reliability of the region where the movement is large compared to the surroundings may be lowered.

Next, in step S213, the distance map estimation unit 202 creates a distance map.
In the present embodiment, the DFD method is used for estimating the distance map. Here, the principle of the DFD method will be briefly described with reference to FIG. In FIG. 7, the distance u to the target object P can be calculated using Formula 1 from the focal length f of the lens if the position v at which the target object P is in focus is known.

In the DFD method, on the assumption that the imaging surface position and the in-focus position of the target object P are different, the in-focus position v is obtained from the degree of blur of the image projected on the imaging surface, and this is substituted into Equation 1. Thus, the distance to the target object P is calculated. When the imaging surface is at s ₁ , the points on the object surface at the distance u are diffused into a circle called a circle of confusion on the imaging surface to form an image i ₁ expressed by Equation 2.

Here, * indicates a convolution operation, and i ₀ indicates an image at a focused position. H ₁ is a PSF (Point Spread Function), which depends on the diameter d ₁ of the circle of confusion proportional to the distance vs-s ₁ between the imaging surface and the in-focus position. Therefore, a confusion circle diameter d ₁ is estimated from the image i ₁ by assuming a PSF model using the confusion circle diameter as a parameter.
However, as can be seen from Equation 2, since the observed image i ₁ depends on the image i _{0 of the} target object, the confusion circle diameter d ₁ cannot be obtained as it is. Therefore, the observed image i ₂ at a different imaging plane position s ₂ is imaged, and the ratio between the observed image i ₁ and the observed image i ₂ in the frequency domain of the image is taken, so that the relationship between the observed image and the PSF is given by Equation 3. Can be derived.

Here, I ₁ , I ₂ , I ₀ , H ₁ , and H ₂ represent the Fourier transform results of the observed images i ₁ , i ₂ , the focused image i ₀ , and the PSFs h ₁ and h ₂ , respectively. A ratio of the Fourier transform of PSF is calculated in advance from the optical system parameters to create a table, and the distance value can be calculated from what is actually calculated.
In the present embodiment, the image is divided into 4 × 4 regions, and a distance map is created in which the calculated distance value to the subject is assigned to each region in the image.

The description of the process flowchart will be continued. In step S214, the calculated plurality of reliability maps are integrated to create an integrated reliability map.
The integrated reliability map creation process will be described with reference to FIG. 4A and 4B are reliability maps obtained based on the focus image in step S211. FIG. 4C is a reliability map calculated by the alignment process performed in step S212. These reliability maps are integrated as shown in FIG.
Specifically, each reliability is normalized to a predetermined value range and then calculated by multiplying each region. The method of integration is not limited to a specific method. For example, the reliability may be weighted according to a difference in shooting modes (such as a scene with a lot of movement or a scene where a still object is shot). In this embodiment, in order to simplify the description, an integrated reliability map is generated by binarizing the hatched portion as “low reliability” and the other portions as “high reliability”.

In step S215, the distance map correction unit 204 corrects the distance map created in step S213 using the integrated reliability map created in step S214.
The outline of this process will be briefly described with reference to FIG. First, a process of setting a plurality of regions in which the brightness and hue of the image are within a predetermined range is performed on the focus image (FIG. 3A) acquired in step S201. In this example, as shown in FIG. 6A, the image is divided into 4 × 4 blocks, and four areas 601 to 604 indicated by different hatching are set. Each of the four set areas is referred to as a correction area. Here, for simplification of description, the size of the block divided in this step is the same as the size of the area included in the reliability map and the distance map, but the sizes may be different from each other.

Next, the distance map calculated in step S213 is corrected using the integrated reliability map generated in step S214. Here, a process is performed in which a set distance value is deleted for a region with low reliability in the reliability map, and a distance value possessed by a region with high reliability in the surrounding area is substituted.
FIG. 6B is a diagram illustrating a state in which the distance value of the region with low reliability is deleted from the created distance map. Since the region to which the x mark is given is a region with low reliability, it means that the distance value has been deleted. When there is a region having a distance value in the vicinity of a region having no distance value, the distance value is substituted. At this time, priority is given to areas belonging to the same correction area.

For example, since the area 606 belonging to the same correction area 602 as the area 607 from which the distance value has been deleted is an area for which the distance has been calculated, the distance value is given to the area 607. Similarly, the area 608 is given the distance value of the area 605 belonging to the same correction area 603. Since no region belonging to the same correction region exists in the region 609, the distance value of the peripheral region (for example, the region 606 or the region 607 after the distance value is given) is given.
Through the above processing, a corrected distance map as shown in FIG. 6C can be obtained. Then, in order to separate the foreground and the background, threshold processing is performed on the distance map to obtain the binary distance map shown in FIG. The distance map obtained in this way is the correction distance map in the present invention. In this example, priority is given to areas belonging to the same correction area in the distance value acquisition. However, the distance values corresponding to a plurality of surrounding areas may be acquired preferentially from areas close to each other. The average may be used.

In step S216, the depth-of-field control unit 205 performs a process of changing the depth of field using the distance map corrected in step S215.
Specifically, the depth of field is obtained by applying a predetermined two-dimensional filter to a region corresponding to the background region 609 in FIG. 6D of the focus image in FIG. Get a shallow image. As this filter, a Gaussian function or the like may be used, or a filter imitating a specific lens may be applied. Further, the calculation method is not limited to the product-sum calculation, and may use a frequency space such as FFT. There is no limitation on the calculation method. Here, an appropriate transition region may be added between the foreground region 610 and the background region 609 in FIG. 6D to avoid a sudden change.

  Next, the process proceeds to step S204, and the system control circuit 106 outputs the image having a shallow depth of field created in step S203 to an image display device (not shown) via the image output unit 111.

  As described above, the imaging apparatus according to the present embodiment calculates a distance map from a focused image and a defocused image, corrects the distance map in consideration of reliability, and uses the corrected distance map. Generate a blur added image. With this configuration, it is possible to reduce an error in the estimated distance, and to avoid image deterioration in the blur addition process.

  It should be noted that the description of the embodiment is an example for explaining the present invention, and the present invention can be implemented with appropriate modifications or combinations without departing from the spirit of the invention. For example, in the description of the embodiment, the mode in which the image processing apparatus according to the present invention is incorporated in the imaging apparatus has been described. However, if two images of the focus image and the defocus image can be acquired, the present invention is not limited to the other apparatus. It can also be applied to. For example, the present invention can be applied to a video reproduction device or an image display device, or can be implemented as image processing software for adding blur to an image or a moving image, moving image editing software, or the like.

In the description of the embodiment, the distance value of the area with low reliability is deleted and the distance value is complemented from the surrounding area, but the distance value may be directly corrected.
In addition, the intensity parameter for controlling the depth of field may be changed depending on the reliability. For example, for an area with low reliability, a process for reducing visual image degradation may be performed by applying a gain to the amount of blur added and adjusting the blur to be weak.
In the description of the embodiment, the reliability map is generated from the motion vector. However, when the misalignment between a plurality of images with different blurs includes enlargement / reduction and / or rotation, the enlargement / reduction and / or rotation between the images is misaligned from the detected misalignment amount and misalignment direction. And reliability information may be created based on the calculation result.

  In the description of the embodiment, a mode is described in which a process for generating a reliability map by analyzing an image (step S211) and a process for generating a reliability map by a result of alignment (step S212) are performed. Only the latter processing may be performed. The object of the present invention can be achieved without performing image analysis on the focus image.

  In addition, the present invention can be implemented as an image processing method including at least a part of the above processing, or can be implemented as an image processing program that causes a computer to execute these methods. The above processes and means can be freely combined and implemented as long as no technical contradiction occurs.

  DESCRIPTION OF SYMBOLS 201 Position alignment part, 202 Distance map estimation part, 203 Reliability map calculation part, 204 Distance map correction | amendment part, 205 Depth of field control part

Claims (5)

An image processing apparatus that processes a plurality of images with different blurring methods,
A displacement detection means for detecting a displacement direction between the plurality of images;
Distance map generating means for determining distance information of a subject from the plurality of images and generating a distance map;
Reliability map generation means for obtaining reliability information representing reliability for each distance information of the distance map based on the positional deviation direction, and generating a reliability map corresponding to the distance map. An image processing apparatus. The positional deviation detection means further detects the positional deviation amount between the plurality of images,
The image processing apparatus according to claim 1, wherein the reliability map generation unit generates the reliability map based on the positional deviation amount and the positional deviation direction. An imaging apparatus comprising: an imaging unit; and the image processing apparatus according to claim 1. An image processing method for processing a plurality of images with different blurring methods,
A detection step of detecting a displacement direction between the plurality of images;
A distance map generating step for determining distance information of a subject from the plurality of images and generating a distance map;
A reliability map generation step of determining reliability information representing reliability for each distance information of the distance map based on the positional deviation direction, and generating a reliability map corresponding to the distance map. Image processing method. A program for causing a computer to execute each step of the image processing method according to claim 4.

Images