JP7133979B2 - Image processing device, image processing method, image processing program, and storage medium - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, a program, and a storage medium storing the program, which perform composition based on first and second images that have different exposure conditions and correspond to each other.

A general imaging device has a limited range of brightness (dynamic range) that can be acquired in one imaging. Therefore, in a scene (landscape) including both dark and bright areas that are extremely different from each other, the captured image may suffer from crushed shadows and blown highlights. When the exposure time is shortened, the overexposure in bright areas can be suppressed, but the underexposure occurs in dark areas, resulting in an image with reduced visibility. On the other hand, when the exposure time is lengthened, the blocked-up shadows in the dark areas can be suppressed, but the blown-out highlights occur in the bright areas, resulting in an image with lost texture.

Therefore, a high dynamic range (HDR) synthesis technique has been proposed for synthesizing a plurality of images with different exposure conditions to generate an image with a wide dynamic range. For example, in Patent Document 1, a short-exposure time image and a long-exposure time image are acquired in the same scene, the composition ratio of each image is determined for each pixel based on the luminance values of these two images, and the determined composition ratio Composite the two images based on . At this time, the composition ratio is determined so that the composition ratio of the long-exposure-time image increases in the dark portion, and the composition ratio of the short-exposure-time image increases in the bright portion.

However, with the HDR synthesis technology as described above, when there is a positional deviation between a plurality of images before synthesis, there is a problem that adverse effects such as blurring and double images (ghosts) occur in the synthesized image obtained by synthesizing these images. There is In particular, when the same subject is imaged using sensors at different positions with multiple images with different exposure conditions, or when the same subject is imaged by time division using an imaging device mounted on a moving object. , parallax occurs between a plurality of images. In other words, if the depth distance of the subject with respect to the imaging device differs when multiple images are captured, the position of the subject locally shifts between the multiple images, resulting in blurring and double images (ghosts). ), etc. will occur.

In order to suppress such an adverse effect, a technique is known that detects the positional deviation of a plurality of images at a high density (for example, for each pixel), corrects the positional deviation of the plurality of images, and synthesizes the images. In Patent Document 2, a plurality of input images with parallax and different exposure amounts are subjected to image brightness adjustment and then matching is performed, matching information is obtained for all pixels, and based on the obtained matching information. A technique for correcting the positions of a plurality of images and synthesizing them has been disclosed.

JP 2013-12999 A Japanese Patent No. 5367640

However, in the technique described in Patent Document 2, the positional deviation search is performed for all pixels including areas with crushed shadows and areas with blown highlights. In other words, even if it is sufficient to simply perform a compositing process that replaces the blocked-up shadow areas or blown-out highlights areas in the reference image with an image under other exposure conditions, the misregistration search is performed for all pixels. conduct. Therefore, the image processing is inefficient, and there is room for improvement in shortening the processing time.

Accordingly, the present invention has been made in view of the above problems, and aims to provide a technique capable of reducing the processing time required for searching for misregistration while maintaining the quality of an image generated by synthesis. aim.

An image processing apparatus according to the present invention is an image processing apparatus that combines a first image and a second image that have different exposure conditions and that correspond to each other. a first synthesis ratio determination unit that determines a first synthesis ratio for each pixel in synthesis of one of the first image and the second image when the image and the second image are synthesized; a second synthesis ratio determination unit that determines a second synthesis ratio for each pixel in synthesis of the one image when the first image and the second image are synthesized based on the luminance values of the two images; , based on the first composition ratio and the second composition ratio, the displacement between the first image and the second image is generated when the first image and the second image are composited. A misalignment saliency determination unit that determines, for each pixel, the degree of influence on the luminance of a synthesized image to be processed as misalignment saliency; a weighted motion vector calculator for calculating a motion vector for each pixel between the first image and the second image by the positional displacement search; the first image, the second image, and the motion vector; and an image generation unit that generates a third image by performing synthesis based on the above.

According to the present invention, the displacement between the first image and the second image is generated when the first image and the second image are synthesized based on the first synthesis ratio and the second synthesis ratio. The degree of influence on the brightness of the composite image is determined for each pixel as the salience of positional deviation, and based on the salience of positional deviation, while changing the amount of processing required for searching for positional deviation, the positional deviation search is performed to obtain the first image. A motion vector for each pixel between and the second image is calculated. According to such a configuration, it is possible to reduce the processing time required for searching for misregistration while maintaining the quality of the image generated by combining.

1 is a block diagram showing the configuration of an image processing apparatus according to Embodiment 1; FIG. 4 is a graph showing the input/output relationship in the first combining ratio determining section according to Embodiment 1; 9 is a graph showing the input/output relationship in the second combining ratio determining section according to Embodiment 1; 4 is a contour map showing the input/output relationship in the positional deviation saliency determination unit according to the first embodiment; FIG. 4 is a diagram for explaining operations of a first synthesis ratio determination unit, a second synthesis ratio determination unit, and a positional deviation salience determination unit according to Embodiment 1; FIG. 4 is a diagram for explaining the operation of a weighted motion vector calculator according to Embodiment 1; FIG. 4 is a diagram for explaining the operation of the image generator according to Embodiment 1; FIG. 2 is a block diagram showing the configuration of a computer system that executes processing of the image processing apparatus according to Embodiment 1; FIG.

<Embodiment 1>
FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to Embodiment 1 of the present invention.

The image processing apparatus according to the first embodiment generates and outputs the third image ImgC by synthesizing the first image ImgA and the second image ImgB that correspond to each other under different exposure conditions. The image processing device of FIG. Prepare.

The first combining ratio determining unit 1a determines the value of one of the first image ImgA and the second image ImgB when combining the first image ImgA and the second image ImgB based on the luminance value of the first image ImgA. determines a first synthesis ratio WeiA for each pixel in the synthesis of . The second synthesis ratio determination unit 1b performs second synthesis for each pixel in synthesis of the one image when the first image ImgA and the second image ImgB are synthesized based on the luminance value of the second image ImgB. Determine the ratio WeiB. In the first embodiment, it is assumed that one of the first image ImgA and the second image ImgB is the first image ImgA. In the following description, the image formed by the first synthesis ratio WeiA for each pixel may be referred to as the first synthesis ratio image WeiA, and the image formed by the second synthesis ratio WeiB for each pixel may be referred to as the second synthesis ratio image WeiA. It may also be referred to as composite ratio image WeiB. In addition, in the following description, when the first synthesis ratio WeiA and the second synthesis ratio WeiB are not distinguished, they may simply be referred to as synthesis ratios.

Based on the first synthesis ratio WeiA and the second synthesis ratio WeiB, the positional displacement saliency determination unit 2 determines whether the positional displacement between the first image ImgA and the second image ImgB is the first image ImgA and the second image ImgB The degree of influence on the brightness of the synthesized image generated when performing the synthesis of is determined for each pixel as the positional deviation conspicuity Imp. The misregistration conspicuity Imp substantially corresponds to the extent to which the influence of the misregistration is conspicuous in the synthesized image. Further, the positional deviation conspicuity Imp corresponds to the extent to which the brightness of the image generated when combining the first synthesis ratio image WeiA and the second synthesis ratio image WeiB is affected. In the following description, an image formed by misalignment saliency Imp for each pixel may also be referred to as a misregistration saliency image Imp.

The weighted motion vector calculation unit 3 changes the amount of processing required for the positional displacement search (positional displacement detection) based on the positional displacement salience Imp, and calculates the first image ImgA and the second image ImgB by the positional displacement search. A motion vector W for each pixel between is calculated. It should be noted that changing the amount of processing required for the positional deviation search corresponds to changing the granularity of the positional deviation search.

The image generator 4 generates a third image ImgC by combining the first image ImgA, the second image ImgB, and the motion vector W. FIG. In Embodiment 1, the image generating section 4 includes a position correcting section 4a and a synthesizing section 4b.

The position correction unit 4a corrects the position of the other image of the first image ImgA and the second image ImgB based on the motion vector W for each pixel. The other image of the first image ImgA and the second image ImgB is an image different from the one image described above. In the first embodiment, it is assumed that the other image is the second image ImgB. The position-corrected second image ImgBw is generated by the correction of the position correction unit 4a.

The synthesizing unit 4b generates a third image by synthesizing one image and the other position-corrected image with weighted addition based on the first synthesis ratio WeiA or the second synthesis ratio WeiB. do. In the first embodiment, the synthesizing unit 4b synthesizes the first image ImgA and the position-corrected second image ImgBw with weighted addition based on the first synthesis ratio WeiA to obtain the third image. Generate ImgC.

Each component will be described in detail below.

<First Image ImgA and Second Image ImgB>
The image processing apparatus according to the first embodiment receives two images Ia and Ib that correspond to each other under different exposure conditions. The exposure conditions are set values such as the exposure time, aperture, device sensitivity, and neutral density filter in the imaging device, and refer to all the conditions for controlling the amount of exposure. In the first embodiment, two images corresponding to each other are two images with parallax therebetween. Such two images are, for example, two images obtained by imaging the same object (subject, scene, etc.) using sensors at different positions, or using an imaging device mounted on a moving object. For example, two images obtained by imaging the same subject in a time division manner. If the depth distance of the subject with respect to the imaging device differs when the two images are captured, the position of the subject locally shifts between the two images. The image processing apparatus according to the first embodiment synthesizes a plurality of images with localized positional deviations after correcting the positional deviations. However, the two images corresponding to each other may be two images with no parallax between them.

In the first embodiment, of the two images, an image that is not subjected to position correction and serves as a position reference (hereinafter also referred to as a reference image) is the first image. On the other hand, the image (hereinafter also referred to as the reference image) that has undergone position correction and is deformed so as to match the position of the reference image is referred to as the second image.

Here, either the first image or the second image may be selected as the standard image (or the reference image) as long as the exposure conditions of the captured images are different. That is, the less exposed image may be used as the first image (reference image), and the more exposed image may be used as the second image (reference image), or vice versa. However, it is preferable to select, as the first image (reference image), the image in which the area of the blocked-up shadow area and the area of the blown-out area are smaller. In the following description, for the sake of simplicity, the image with low exposure is the first image (reference image), and the image with high exposure is the second image (reference image).

In addition, prior to the alignment and composition processing, the two images whose brightness is adjusted are selected as the first image and the second image in consideration of the exposure condition of the captured image and the camera response function (CRF). may be

A camera response function describes the relationship between the pixel values of a captured image and the radiance of a scene. In general imaging devices, the image detector often outputs a linear signal with respect to the radiance of the scene. It may also be converted. A function that integrates the transformations in these processes and transforms the radiance of the scene into the pixel values of the final captured image is called the camera response function. A function that is the inverse of the camera response function and converts the pixel values of the captured image into the radiance of the scene is called the inverse camera response function. The camera response function and its inverse function are obtained by photographing a subject whose radiance is known and calibrating the function based on the captured image, imaging the subject by changing only the exposure time, and calculating the function based on the captured image. can be obtained by a known method such as a method of estimating .

In the following description, it is assumed that an inverse camera response function corresponding to each captured image is obtained in advance by calibration or the like, and that a LUT (Look Up Table) or polynomial representing the inverse camera response function is given.

The process of generating the images Ia' and Ib' by matching the brightness of the images Ia and Ib captured under different exposure conditions is represented by the following equations (1) and (2), for example. According to these formulas, the brightness of the images Ia and Ib is substantially normalized.

In equations (1) and (2), p is the pixel position, and I _a (p) and I _b (p) represent pixel values at pixel position p in images Ia and Ib (hereinafter referred to as image The same notation is used when representing pixel values of data). In addition, f _a and f _b are camera response inverse functions of the images Ia and Ib, E _a and E _b are the exposure amounts Ea and Eb of the images Ia and Ib, and K is an arbitrary constant.

In the following, for the sake of simplicity, it is assumed that images whose brightness is adjusted by converting the pixel values of the captured image into (linear values of) radiance are input as the first and second images. That is, assuming that the image Ia' represented by equation (1) is the first image ImgA in FIG. 1, and the image Ib' represented by equation (2) is the second image ImgB in FIG. It is explained below.

<First Synthesis Ratio Determination Unit 1a and Second Synthesis Ratio Determination Unit 1b>
Based on the luminance value of the first image ImgA, the first synthesis ratio determining unit 1a performs the first synthesis for each pixel in the synthesis of the first image ImgA when the first image ImgA and the second image ImgB are synthesized. A ratio WeiA is determined to generate a first composite ratio image WeiA.

Note that the pixel values of the first composite ratio image WeiA and the pixel values of a second composite ratio image WeiB, which will be described later, are both the values of the first image ImgA when the first image ImgA and the second image ImgB are composited. Represents the composite ratio (weighting).

In Embodiment 1, the synthesis ratio takes a value between 0 and 1 for each pixel. is (1-α), and the pixel value of the synthesized image is α×ImgA+(1-α)×ImgB. As a result, a pixel with a composition ratio of 1 has a pixel value of 100% of the image ImgA (a pixel value of 0% of the image ImgB) in the composite image, and a pixel with a composition ratio of 0 has a pixel value of 100% in the composite image. will have 0% pixel values of image ImgA (100% pixel values of image ImgB).

Note that the definition of the pixel value of the composition ratio may be reversed. That is, both the pixel values of the first composite ratio image WeiA and the pixel values of the second composite ratio image WeiB, which will be described later, are the values of the second image ImgB when the first image ImgA and the second image ImgB are composited. It may be expressed as a composite ratio (weighting) of . However, in that case, the magnitudes of these values are also inverted in the processing described below.

When the first image ImgA is the image with the lower exposure, the first synthesis ratio determining unit 1a reduces the first synthesis ratio WeiA of the pixel with a small luminance value in the first image ImgA, For a pixel with a large luminance value in one image ImgA, the first synthesis ratio WeiA for that pixel is increased.

The first synthesis ratio determination unit 1a determines the first synthesis ratio WeiA for each pixel based on the luminance value of the first image ImgA, for example, using the following equation (3). Note that TH _bright and TH _dark are threshold values in the following equation (3). The threshold TH _bright is determined based on the saturation level as described later, and the threshold TH _dark is determined based on the noise level as described later.

FIG. 2 is a graph showing the luminance value of the first image ImgA on the horizontal axis and the first synthesis ratio WeiA on the vertical axis. In the example shown in Equation (3) and the solid line in FIG. 2, when the brightness value of the first image ImgA is equal to or greater than the threshold TH _bright , the first synthesis ratio WeiA is 1, and the brightness value of the first image ImgA is the threshold. If it is equal to or less than TH _dark , the first synthesis ratio WeiA is zero. Then, in a range where the luminance value of the first image ImgA is greater than the threshold TH _dark and less than the threshold TH _bright , the first synthesis ratio WeiA gradually increases according to the luminance value of the first image ImgA.

Therefore, in a bright portion of the composite image where the brightness value of the first image ImgA is equal to or greater than the threshold TH _bright , the first composite ratio WeiA is 1, and the first image ImgA with lower exposure is adopted 100%. On the other hand, in a dark portion of the composite image where the luminance value of the first image ImgA is equal to or less than the threshold TH _dark , the first composite ratio WeiA is 0, and the second image ImgB with higher exposure is adopted at 100%. In the composite image, the first composite ratio WeiA is greater than 0 and greater than 1 in a portion where the brightness value of the first image ImgA is greater than the threshold TH _dark and less than the threshold TH _bright . , and the lower exposed first image ImgA and the higher exposed second image ImgB are blended.

The threshold TH _bright is set to a value equal to or slightly smaller than the luminance value SatLv of saturated pixels in the image with higher exposure among the captured images before brightness adjustment. The luminance value SatLv of a saturated pixel is the upper limit value I _MAX (I _MAX = 255) is calculated by the following equation (4) that performs the same transformation as in equation (2).

The threshold TH _bright is set, for example, to a value equal to or greater than 95% and equal to or less than 100% of SatLv. As a result, when the brightness of the luminance value of the first image ImgA is at a brightness level at which the second image ImgB is saturated, the composition ratio of the first image ImgA is surely 1, and the second image is saturated. ImgB can be prevented from being adopted for the third image ImgC. In the following description, it is assumed that the threshold TH _bright is set to a value equal to SatLv.

The threshold TH _dark is determined to be a value smaller than the threshold TH _bright and sufficiently larger than the luminance value NoiLv corresponding to the noise level of the captured image before brightness adjustment. When the standard deviation of noise in the original captured image (for example, the less exposed image Ia) is _σn , the luminance value NoiLv corresponding to the noise level is obtained by formula (1) and It is calculated by the following equation (5) that performs a similar conversion.

The threshold TH _dark is set to a value sufficiently larger than NoiLv, for example, a value about three times NoiLv. As a result, in the portion of the first image ImgA where the brightness of the luminance value is the brightness of the noise level, the composition ratio of the first image ImgA is surely 0, and the noise of the first image ImgA is reduced to the noise level of the third image ImgC. can be prevented from being adopted by

As described above, the first synthesis ratio determination unit 1a determines the pixel A first synthesis ratio image WeiA is generated by determining a first synthesis ratio WeiA for each pixel.

On the other hand, based on the luminance value of the second image ImgB, the second synthesis ratio determination unit 1b determines the number of pixels in the synthesis of the first image ImgA when the first image ImgA and the second image ImgB are synthesized. 2 A composite ratio WeiB is determined to generate a second composite ratio image WeiB.

The second combining ratio determining unit 1b differs from the first combining ratio determining unit 1a only in that it determines the combining ratio according to the luminance value of the second image ImgB. Same as part 1a.

That is, the second synthesis ratio determination unit 1b reduces the second synthesis ratio WeiB for pixels with low luminance values in the second image ImgB, and reduces the second synthesis ratio WeiB for pixels with high luminance values in the second image ImgB. The second synthesis ratio WeiB of pixels is increased. Therefore, the second composition ratio WeiB determined by the second composition ratio determination unit 1b is not the composition ratio for composition of the second image ImgB, but an estimated value of the composition ratio for composition of the first image ImgA.

In the first embodiment, the second synthesis ratio determination unit 1b uses, for example, the following equation (6) similar to the equation (3), based on the luminance value of the second image ImgB, the second synthesis ratio for each pixel: Determine WeiB. FIG. 3 is a graph showing the luminance value of the second image ImgB on the horizontal axis and the second composition ratio WeiB on the vertical axis. Note that the second composition ratio WeiB is not the composition ratio of the second image ImgB indicated by the dashed line, but the composition ratio of the first image ImgA indicated by the solid line.

In Equation (6), TH _bright and TH _dark are threshold values, and the same values as those of TH _bright and TH _dark of the first combining ratio determining unit 1a can be used for these values. The common relational expression can be used in the first synthesis ratio determination unit 1a and the second synthesis ratio determination unit 1b because the images whose brightness is adjusted in the equations (1) and (2) are the first image and the second image. This is because it is input as the second image. In the configuration in which the functions for performing the conversions of equations (1) and (2) are incorporated into the first composition ratio determination unit 1a and the second composition ratio determination unit 1b, respectively, the picked-up images before brightness adjustment are converted into the first image and the second image. It can be input as two images.

As described above, the second synthesis ratio determination unit 1b determines the pixel A second composite ratio image WeiB is generated by determining a second composite ratio WeiB for each image.

As described above, the first composition ratio determination unit 1a and the second composition ratio determination unit 1b perform gradation conversion on the luminance values of the first image ImgA and the second image ImgB. As a result, the first composite ratio image WeiA and the second composite ratio image WeiA normalized to a value between 0 and 1 have a value of 0 in an underexposure region and a value of 1 in an overexposed region, and have approximately the same brightness. A ratio image WeiB is generated.

<Displacement saliency determination unit 2>
Based on the first composite ratio image WeiA and the second composite ratio image WeiB, the positional displacement conspicuity determination unit 2 of FIG. Then, the degree of influence on the luminance of the synthesized image generated when the second image is synthesized is determined for each pixel to generate the positional deviation saliency image Imp.

The positional deviation saliency determination unit 2 increases the positional deviation saliency Imp as the first synthesis ratio image WeiA (first synthesis ratio WeiA) decreases, and the first synthesis ratio image WeiA (first synthesis ratio WeiA) and the second synthesis ratio WeiA As the absolute value of the difference from the composite ratio image WeiB (second composite ratio WeiB) increases, the positional deviation conspicuity Imp is increased. The misalignment saliency determination unit 2 calculates the misalignment salience Imp using, for example, the following equation (7).

The positional deviation conspicuity Imp in Expression (7) is the inversion of the composition ratio of the first image ImgA in the composite image (that is, the composition ratio of the second image ImgB) (1−WeiA), and the first composition ratio WeiA. It is represented by the product of the absolute difference |WeiB-WeiA| with the second synthesis ratio WeiB.

The first term (1-WeiA) on the right side of equation (7) takes a value from 0 to 1. When the first synthesis ratio WeiA of the first image ImgA is 1, the first term on the right side is 0, and as a result, the salience of displacement Imp is 0. Further, when the composition ratio of the first image ImgA is less than 1, the first term (1-WeiA) on the right side indicating the composition ratio of the second image ImgB becomes a value greater than 0. As a result, the salience of positional deviation Imp increases in proportion to this value.

The second term |WeiB-WeiA| on the right side of equation (7) takes a value from 0 to 1. When the pixel values of the pixels at the same position are equal in the first synthesis ratio image WeiA and the second synthesis ratio image WeiB, the positional shift conspicuity Imp is zero. In addition, when the pixel values of the pixels at the same position are different between the first synthesis ratio image WeiA and the second synthesis ratio image WeiB, the positional deviation conspicuity Imp increases in proportion to the difference absolute value.

FIG. 4 is a diagram plotting the values of the misalignment conspicuity Imp calculated by the equation (7) with contour lines for the combinations of the values of the first synthesis ratio WeiA and the values of the second synthesis ratio WeiB. The positional deviation saliency Imp decreases toward the right side of the contour map, that is, as the first synthesis ratio WeiA increases. Further, the closer the diagonal line (straight line WeiB=WeiA) indicated by the dashed line in the contour map, that is, the smaller the absolute value of the difference between the first synthesis ratio WeiA and the second synthesis ratio WeiB, the smaller the misalignment conspicuity Imp. Further, the positional deviation conspicuity increases as the first synthesis ratio WeiA decreases and the absolute difference between the first synthesis ratio WeiA and the second synthesis ratio WeiB increases as the contour map moves from the center to the upper left region. Imp increases.

Note that the positional deviation saliency determination unit 2 may calculate the positional deviation salience Imp using the following equation (8) instead of calculating the positional deviation saliency Imp using the equation (7). .

In Expression (8), r ₁ and r ₂ are multipliers for adjusting the contribution of each term, and are real numbers of 0 or more. In equation (8), if r1 is set to ₀ , the misregistration saliency Imp will not decrease even if the first synthesis ratio WeiA increases. _The greater the value of r1, the greater the extent to which the displacement saliency Imp is reduced when the first synthesis ratio WeiA is large. That is, by adjusting r1, the contribution of the _first synthesis ratio WeiA to the misalignment saliency Imp can be adjusted. Similarly, by adjusting r2, the contribution of the absolute value of the difference between the first synthesis ratio WeiA and the _second synthesis ratio WeiB to the misalignment saliency Imp can be adjusted.

As described above, the positional deviation conspicuity Imp increases when the composition ratio of the first image ImgA represented by the first composition ratio image WeiA is small, and the difference between the first composition ratio image WeiA and the second composition ratio image WeiB. It has the characteristic of increasing when the absolute value is large.

In the following description, it is assumed that the value calculated by Equation (7) is used as the positional deviation saliency Imp.

<Weighted motion vector calculator 3>
The weighted motion vector calculator 3 of FIG. 1 changes the amount of processing required for the positional displacement search based on the positional displacement saliency Imp, and calculates the difference between the first image ImgA and the second image ImgB by the positional displacement search. A motion vector W for each pixel of is calculated.

As shown in FIG. 1, the weighted motion vector calculator 3 according to the first embodiment includes a first synthesis ratio image WeiA formed by the first synthesis ratio WeiA and a second synthesis ratio image WeiA formed by the second synthesis ratio WeiB. A motion vector W between the first image ImgA and the second image ImgB is indirectly calculated by searching for the positional deviation between the two composite ratio images WeiB.

As motion vector calculation methods, for example, a positional deviation search based on a block matching method and a positional deviation search based on a gradient method are known.

In a misregistration search based on the block matching method, for example, for each block set in units of pixels in a reference image, the correlation with a plurality of misregistration candidate position blocks in the reference image is calculated, and the misregistration with the highest correlation is determined. A motion vector for each pixel is calculated by searching.

In the positional displacement search based on the gradient method, the motion vector for each pixel is calculated by sequentially updating the motion vector that minimizes the evaluation function of the gradient method through iterative processing.

The motion vector calculation method of the weighted motion vector calculation unit 3 is not limited to these positional deviation searches, and may be other known methods, and is not limited to a specific motion vector calculation method.

Here, the weighted motion vector calculator 3 changes the amount of processing required for the positional displacement search based on the positional displacement conspicuity Imp. In other words, the weighted motion vector calculator 3 substantially changes the granularity of positional displacement search based on the positional displacement saliency Imp.

In Embodiment 1, the weighted motion vector calculator 3 increases the amount of processing required for the positional displacement search as the positional displacement saliency Imp increases. That is, the weighted motion vector calculator 3 reduces the amount of processing required for the positional displacement search as the positional displacement saliency Imp decreases.

Reducing the amount of processing required for positional displacement search is achieved by, for example, processing elements that can be executed independently and in parallel for each pixel in each method and each step of motion vector calculation, as shown in some specific examples below. corresponds to omitting the execution of the processing element for a particular pixel or reducing the precision of the processing element for a particular pixel when there is a

In the first specific example, the weighted motion vector calculation unit 3 assigns a predetermined value to a pixel with a low positional deviation saliency Imp, without performing a positional deviation search using the block matching method. Reduce the number of pixels for which the offset search is performed.

In the second specific example, the weighted motion vector calculation unit 3 reduces the interval at which the positional displacement search using the block matching method is performed to 2 pixels as the positional displacement saliency Imp of the pixels on which the positional displacement search should be performed decreases. It is coarsened once, once every four pixels, and so on. Then, the weighted motion vector calculation unit 3 obtains the pixels for which the positional displacement search has not been performed by performing linear interpolation from the results of the neighboring pixels for which the positional displacement search has been performed. reduce the number of

In the third specific example, the weighted motion vector calculation unit 3 adjusts the increments (intervals) of the positional deviation candidate positions using the block matching method as the positional deviation saliency Imp of the pixels for which the positional deviation search should be performed becomes smaller. 2 pixels, 4 pixels, and so on. That is, the weighted motion vector calculator 3 reduces the number of positional deviation candidate positions.

In the fourth specific example, the weighted motion vector calculator 3 reduces the number of iterative processes in positional deviation search using the gradient method.

As described above, the weighted motion vector calculation unit 3 performs the positional displacement search for each pixel in an area with a large positional displacement saliency Imp, and performs the processing required for the positional displacement search in an area with a small positional displacement salience Imp. A motion vector W for each pixel is calculated by the displacement search.

<Position corrector 4a>
The position correction unit 4a in FIG. 1 corrects the position of the second image ImgB for each pixel based on the motion vector W to generate the position-corrected second image ImgBw.

The motion vector W is a displacement map representing to which pixel on the second image ImgB each pixel of the first image ImgA has moved. The position correction unit 4a interpolates the pixel value of the pixel position p+W(p) moved from the pixel position p on the second image ImgB according to the motion vector W(p), thereby obtaining the second image ImgBw after the position correction. Find the pixel value of . Any interpolation method such as a bilinear method, a bicubic method, or a Lanczos method is used for the interpolation.

The position correction unit 4a interpolates the pixel values of the movement destination positions in the second image ImgB for all pixel positions p to generate the position-corrected second image ImgBw.

<Combining unit 4b>
The synthesizing unit 4b in FIG. 1 generates the third image ImgC by synthesizing the first image ImgA and the position-corrected second image ImgBw with weighted addition based on the first synthesis ratio WeiA. . The synthesizing unit 4b in FIG. 1 calculates the third image ImgC using, for example, the following equation (9).

Note that, prior to the weighted addition, the synthesizing unit 4b obtains WeiA′ by performing filter processing such as a smoothing filter and a maximum value filter on the first synthesis ratio WeiA. may be used as weights for weighted addition.

As described above, when the first image ImgA and the second image ImgB that correspond to each other under different exposure conditions are input, the image processing apparatus according to the first embodiment corrects the displacement of the second image ImgB. A third image ImgC is generated by synthesizing the first image ImgA with the first image ImgA. This enables high dynamic range synthesis that generates an image with a wide dynamic range.

<Action>
Next, the operation of the image processing apparatus according to the first embodiment will be described with reference to FIGS. 5 to 7. FIG.

First, the operations of the first synthesis ratio determination unit 1a, the second synthesis ratio determination unit 1b, and the positional deviation salience determination unit 2 will be described with reference to FIG. 5 to 7, the first image ImgA is a short exposure time image, and the second image ImgB is a long exposure time image. In Fig. 5, a person is in the foreground of a dark room, and the bright scenery outside the window is visible in the background. Two sensors were installed to create parallax, and the exposure time was changed to capture the image. The images are shown schematically as a first image ImgA and a second image ImgB.

As shown in FIG. 5, in the first image ImgA, which is a short-exposure-time image, the area of the scenery outside the window has appropriate brightness and is clear, but the area of the person in the foreground is too dark and is blackened out. The S/N in this area is low. Also, in the second image ImgB, which is a long-exposure-time image, the region of the person in the foreground has appropriate brightness and is clear, but the region of the scenery outside the window is too bright and overexposed. Detail is lost.

In addition, there is a parallax between the first image ImgA and the second image ImgB, and the positional deviation of the subject (person in front, window frame, scenery outside the window) with different depth distances varies locally. there is In FIG. 5, the person in the second image ImgB is displaced to the left with respect to the same person in the first image ImgA, and there is a positional displacement between the window frames of the first image ImgA and the second image ImgB. Instead, the view outside the window of the second image ImgB is shifted to the right with respect to the same view of the first image ImgA.

The image processing apparatus according to the first embodiment first converts the pixel values of the two captured images into values linear to the radiance to match the brightness (not shown in FIG. 5), so that the first image Acquire ImgA and a second image ImgB. Then, based on the luminance value of the first image ImgA, the first synthesis ratio determination unit 1a determines the value for each pixel in the synthesis of the first image ImgA when the first image ImgA and the second image ImgB are synthesized. (arrow A1a in FIG. 5). In addition, the second synthesis ratio determination unit 1b determines the number of pixels in the synthesis of the first image ImgA when the first image ImgA and the second image ImgB are synthesized based on the luminance value of the second image ImgB. 2 A second composite ratio image WeiB representing the composite ratio WeiB is generated (arrow A1b in FIG. 5).

As a result, as shown in FIG. 5, the first composite ratio image WeiA becomes 0 in the brightness region corresponding to blocked-up shadows (the torso portion of the person in the foreground) of the first image ImgA, and the brightness region corresponding to blown-out highlights (the window The image becomes 1 in the outside scenery area), and the image takes a linear value with respect to the luminance value in other areas.

Further, as shown in FIG. 5, the second composite ratio image WeiB becomes 0 in the brightness area corresponding to blocked-up shadows (the torso portion of the person in front) of the second image ImgB, and the brightness area corresponding to blown-out highlights (outside the window). The image becomes 1 in the scenery part), and the image takes a linear value with respect to the luminance value in other areas.

In the white portions of the first synthesis ratio image WeiA and the second synthesis ratio image WeiB in FIG. 5, the synthesis ratio of the first image ImgA is 1, and the first image ImgA and the second image ImgB are synthesized. In the corresponding part of the synthesized image to be processed, the synthesis ratio of the first image ImgA is 100% and the synthesis ratio of the second image ImgB is 0%. Also, in the black portion, the synthesis ratio of the first image ImgA is 0, in the corresponding portion of the synthesized image, the synthesis ratio of the first image ImgA is 0%, and the synthesis ratio of the second image ImgB is 100%. . Furthermore, gray portions that are neither white nor black have a synthesis ratio between 0 and 1 for the first image ImgA. It means that

Based on the luminance values of the input first image ImgA and second image ImgB, the first composition ratio determination unit 1a and the second composition ratio determination unit 1b set the composition ratio of the brightness region corresponding to blocked-up shadows to 0 and the The first synthesis ratio image WeiA and the second synthesis ratio image WeiB are generated so that the synthesis ratio of the equivalent luminance region is 1 and the synthesis ratio of the other regions is a linear value with respect to the luminance value. For this reason, the first synthesis ratio image WeiA and the second synthesis ratio image WeiB each have not only brightness, but also a brightness area corresponding to blocked-up shadows (where the synthesis ratio is 0) and a luminance area equivalent to oversaturated highlights (where the synthesis ratio is 1). ) will be an image that substantially matches.

However, the positions of the subject in the first composite ratio image WeiA and the second composite ratio image WeiB are the same as those in the first image ImgA and the second image ImgB. Therefore, the positional deviation between the first image ImgA and the second image ImgB is stored as the positional deviation between the first composite ratio image WeiA and the second composite ratio image WeiB.

Next, the misalignment saliency determining unit 2 generates a misalignment salience image Imp based on the first synthetic ratio image WeiA and the second synthetic ratio image WeiB (arrow A2 in FIG. 5). In FIG. 5, the values of the misregistration salience Imp are shown in gray scale, with dark colors representing portions with low misregistration salience Imp and bright colors representing large misregistration saliency Imp.

As shown in FIG. 5, the generated positional displacement saliency image Imp is relatively large in a portion where the synthesis ratio of the first image ImgA represented by the first synthesis ratio image WeiA is small, and is relatively large in the portion where the synthesis ratio is small. The absolute value of the difference between the synthesis ratio of one image ImgA and the synthesis ratio of the first image ImgA represented by the second synthesis ratio WeiB is relatively large in areas where the difference is large, and is relatively small in other areas.

In FIG. 5 , the portions where the misalignment saliency Imp is large are the edge side portion of the misalignment saliency image Imp (indicated by symbol Ra) of the torso of the person in the foreground, and the eyes, nose, and mouth. (indicated by symbol Rb). In these regions, the value of the first composite ratio image WeiA is small, and the second image ImgB is adopted at a high ratio in the composite image generated when the first image ImgA and the second image ImgB are composited. Moreover, in these areas, the second image ImgB is displaced from the first image ImgA, and the difference absolute value is large. Therefore, in these areas, the effect is conspicuous in the composite image. Therefore, the misregistration salience determining unit 2 determines the misregistration salience Imp such that the misregistration salience Imp of such an area increases.

On the other hand, in the scenery area outside the window (indicated by symbol Rc), although there is a positional shift between the first image ImgA and the second image ImgB, the second image ImgB is saturated and the first image ImgB is saturated. The value of the composite ratio image WeiA is 1, and the first image ImgA is adopted 100% in the composite image. Therefore, the displacement of the second image ImgB in these regions does not affect the synthesized image. Therefore, the misregistration salience determination unit 2 determines the misregistration saliency Imp such that the misregistration salience Imp of such an area becomes small.

In addition, the value of the first synthesis ratio image WeiA is 1 in the area of the wall of the room (indicated by symbol Rd) and the portion of the torso of the person in the foreground inside the misalignment saliency image Imp (indicated by symbol Re). Although the second image ImgB is used in the synthesized image, it is almost inconspicuous in the synthesized image because the luminances of the first image ImgA and the second image ImgB are close to each other. Therefore, the misregistration salience determination unit 2 determines the misregistration saliency Imp such that the misregistration salience Imp of such an area becomes small.

As described above, the positional deviation salience determination unit 2 determines the positional deviation saliency Imp in consideration of the first composite ratio image WeiA and the difference between the first composite ratio image WeiA and the second composite ratio image WeiB. do.

Next, the operation of the weighted motion vector calculator 3 will be described with reference to FIG. The weighted motion vector calculator 3 changes the amount of processing required for the positional displacement search based on the positional displacement conspicuity Imp, and calculates the difference between the first composite ratio image WeiA and the second composite ratio image WeiB by the positional displacement search. A motion vector W for each pixel between is calculated (arrow A3 in FIG. 6).

In the example of FIG. 6, the weighted motion vector calculation unit 3 performs the positional displacement search using the block matching method only for the portion where the positional displacement saliency Imp is large. On the other hand, the weighted motion vector calculator 3 assigns a predetermined value, here a zero vector (no positional deviation), to a portion where the positional deviation salience Imp is small, without performing a positional deviation search using the block matching method. . The portion of FIG. 6 where the displacement saliency Imp is large is the region of the person in the foreground (body edge portion, etc.), and the weighted motion vector calculator 3 performs the position displacement search for this portion on a pixel-by-pixel basis.

More specifically, the weighted motion vector calculator 3 sequentially sets blocks at positions of pixels where the positional deviation conspicuity Imp is equal to or greater than a predetermined threshold value in the first composite ratio image WeiA, and sets blocks at each pixel position. with blocks of a plurality of displacement candidate positions in the second composite ratio image WeiB. Then, the weighted motion vector calculation unit 3 calculates the positional deviation with the highest correlation as the motion vector of the pixel. On the other hand, the weighted motion vector calculator 3 does not perform positional displacement search for a pixel whose positional displacement salience Imp is less than a predetermined threshold, and forcibly assigns a zero vector as the motion vector of the pixel. As a result, an image of motion vector W as shown in FIG. 6 is obtained.

In the image of the motion vector W in FIG. 6, the displacement amounts of the pixels of the second synthesis ratio image WeiB corresponding to the pixels of the first synthesis ratio image WeiA are indicated by arrows (vectors) on equally spaced grid points. It is In the case of FIG. 6, the image of the motion vector W is an image in which a vector in the left direction corresponding to the positional deviation amount is stored in the region of the person in the foreground, and a zero vector is stored in the other regions.

Finally, the operation of the image generator 4 including the position corrector 4a and the synthesizer 4b will be described with reference to FIG. The position correction unit 4a corrects the position of the second image ImgB for each pixel based on the motion vector W to generate the position-corrected second image ImgBw (arrow A4 in FIG. 7). In the motion vector W of FIG. 6, only the region of the person in the foreground has a positional shift. It is corrected so that it is at the same position as the image ImgA. As a result, like the second image ImgBw after the position correction in FIG. 7, the region of the person in the foreground moves from the position indicated by the broken line to the position indicated by the solid line. Therefore, the second image ImgBw is an image in which not only the area such as the scenery outside the window but also the area of the person in the foreground is aligned with the first image ImgA.

Based on the first synthesis ratio WeiA represented by the first synthesis ratio image WeiA, the synthesizing unit 4b synthesizes the first image ImgA and the position-corrected second image ImgBw with weighted addition to obtain a third image. Generate image ImgC (arrow A5 in FIG. 7).

Since the synthesis ratio of the first synthesis ratio image WeiA in FIG. 7 is 1 in the scenery area outside the window, 100% of the first image ImgA is adopted for that area in the third image ImgC. On the other hand, since the composition ratio is smaller than 1 in the region of the person in the foreground, the region in the third image ImgC includes the first image ImgA and the position-corrected second image ImgBw. are employed in some proportion. As a result, the third image ImgC in FIG. 7 is an image in which there is no overexposure in the scenery area outside the window and the person area in the foreground is clear. Furthermore, in the region of the person in front of the third image ImgC, the second image ImgBw that has been positionally corrected so as to be aligned with the first image ImgA is combined. The image is free of blurring and double images.

The above is the description of the operation of the image processing apparatus according to the first embodiment.

<Summary of Embodiment 1>
In the image processing apparatus according to the first embodiment, when synthesizing images under a plurality of exposure conditions with different viewpoints, the positional deviation search is not uniformly performed for all pixels, but in an area where the conspicuousness of positional deviation is small, the position Since the amount of calculation required for the displacement search is reduced, it is possible to reduce the processing time required for the positional displacement search for each pixel while maintaining the quality of the third image ImgC. This effect will be described in order below.

In the image processing apparatus according to the first embodiment, two images with different exposure conditions are defined as the first image ImgA and the second image ImgB, and the first composition ratio determination unit 1a and the second composition ratio determination unit 1b A composition ratio for each pixel was determined based on the luminance values of the first image ImgA and the second image ImgB.

Then, in the positional deviation saliency determination unit 2, based on the first synthesis ratio WeiA and the absolute difference between the first synthesis ratio image WeiA and the second synthesis ratio image WeiB, the first image ImgA and the second image ImgB The extent to which the luminance of a synthesized image generated by synthesis is affected is determined for each pixel as the positional deviation conspicuity Imp.

Furthermore, in the weighted motion vector calculation unit 3, the smaller the displacement conspicuity Imp, the smaller the amount of processing required for the displacement search, and the displacement search results in the difference between the first image ImgA and the second image ImgB. A motion vector W for each pixel was calculated. Further, as a specific example of means for reducing the amount of processing required for the positional deviation search, for processing elements that can be executed independently and in parallel for each pixel, the number of target pixels for which the positional deviation search is performed is reduced, or the positional deviation search is performed. The number of positional deviation candidate positions in the search is reduced, and the number of iterations is reduced.

Finally, the third image ImgC is generated by synthesizing the first image ImgA, the second image ImgB, and the motion vector W in the image generating unit 4 including the position correcting unit 4a and the synthesizing unit 4b. .

Generally, misregistration search using a block matching method requires a process of searching for the optimum misregistration for a plurality of misregistration candidate regions for each pixel. A misregistration search using the gradient method requires an iterative process for each pixel in order to solve the simultaneous equations. Therefore, the process of calculating a motion vector for each pixel is a process with a very high calculation cost (calculation amount) per pixel. Therefore, when implemented so that the calculation for each pixel is performed sequentially by a CPU (Central Processing Unit), or when implemented as parallel processing using a parallel processor such as a GPU (Graphics Processing Unit), the pixel to be processed The amount of processing required for the misregistration search, such as the number of vertices and the number of iterations per pixel, challenges the processing time.

In the image processing apparatus according to the first embodiment, the synthesized image generated by synthesizing the first image ImgA and the second image ImgB has a small conspicuity of positional deviation. The processing amount of the above-described misregistration search, which is computationally costly, is reduced for pixels determined to have an insignificant effect on the composite image. As a result, it is possible to reduce the processing time required for searching for positional deviation in units of pixels while maintaining the quality of the third image ImgC.

Further, in the image processing apparatus according to the first embodiment, the weighted motion vector calculation unit 3 searches for a positional shift between the first synthetic ratio image WeiA and the second synthetic ratio image WeiB, thereby indirect Specifically, a motion vector W between the first image ImgA and the second image ImgB is calculated.

Here, the first synthesis ratio image WeiA and the second synthesis ratio image WeiB each have not only brightness, but also a brightness area equivalent to blocked-up shadows (where the synthesis ratio is 0) and a brightness area equivalent to overexposed highlights (where the synthesis ratio is 1). ) are images that substantially match each other. Also, the positional deviation between the first image ImgA and the second image ImgB is stored as the positional deviation between the first composite ratio image WeiA and the second composite ratio image WeiB. For this reason, the first composite ratio image WeiA and the second composite ratio image WeiB are images suitable for searching for misregistration, in which the effects of image differences (such as blocked-up shadows and blown-out highlights) due to different exposure conditions are suppressed. I can say. In the first embodiment, since the positional displacement search is performed on the first composite ratio image WeiA and the second composite ratio image WeiB, robust positional displacement search can be performed for portions with image differences such as blocked-up shadows and blown-out highlights. be possible.

Furthermore, when the underexposure region and overexposed region have a considerable area, it is theoretically impossible to perform a high-precision displacement search in these regions, which may cause errors. . However, in such an area, the synthesis ratio of the first image ImgA is 1, or the difference between the first synthesis ratio image WeiA and the second synthesis ratio image WeiB is 0, and the positional deviation saliency is 0. Therefore, it is possible not to search for positional deviation. Thus, according to the first embodiment, in addition to the effect of reducing the processing time, it is possible to suppress positional deviation search errors in blocked-up shadows and blown-out highlights.

Note that, as described above, the positional deviation saliency determination unit 2 may calculate the positional deviation salience based on the relationship shown in Equation (8). By adjusting the values of r ₁ and r ₂ in Equation (8), the contribution of the first synthesis ratio WeiA to the salience of misalignment, and the contribution of the first synthesis ratio WeiA and the second It is possible to adjust the contribution of the absolute value of the difference from the composite ratio WeiB. This makes it possible to adjust the balance between the extent to which positional deviation is conspicuous in the third image ImgC and the processing time.

As described above, according to the first embodiment, the amount of processing required for searching for misregistration is reduced in regions where the conspicuousness of misregistration is small. processing time can be reduced.

Each part of the image processing apparatus shown in FIG. 1 (the part shown as a functional block) is implemented by a processing circuit. The processing circuitry may be dedicated hardware or a processor executing a program. For example, the function of each part in FIG. 1 may be realized by a processing circuit, or the functions of a plurality of parts may be collectively realized by a processing circuit.

When the processing circuit is a processor, the function of each part of the image processing device is implemented by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in a storage medium (recording medium). The processing circuit implements the function of each part by reading a program stored in a storage medium into a memory and executing the program. It can be said that this program causes a computer to execute processing in an image processing method implemented in an image processing apparatus.

Some of the functions of each part of the image processing apparatus may be implemented by dedicated hardware, and some may be implemented by software or firmware.

Thus, the processing circuitry may implement each of the functions described above through hardware, software, firmware, or a combination thereof.

FIG. 8 is a diagram showing an example of a configuration in which all the functions of the image processing apparatus of FIG. 1 are implemented by a computer 60a including a single processor, together with the imaging device 10a and storage device 11a. In the example of FIG. 8, the computer 60a, the imaging device 10a, and the storage device 11a constitute an imaging system.

A computer 60a shown in FIG. 8 includes a processor 61a, a memory 62a, an input interface 63a, and an output interface 64a, which are connected by a bus 65a.

The input interface 63a receives images Ia and Ib captured under different exposure conditions by the imaging device 10a and information on the exposure amounts Ea and Eb of each image. Alternatively, the images Ia and Ib captured under different exposure conditions by the imaging device 10a and the exposure amounts Ea and Eb of each image are temporarily stored in the storage device 11a, and the images Ia and Ib read out from the storage device 11a, and Information on the exposure amounts Ea and Eb of each image is input to the input interface 63a.

The processor 61a operates according to the program stored in the memory 62a, and performs the processing of each part of the image processing apparatus of Embodiment 1 on the images Ia and Ib input via the input interface 63a. The resulting output signal is output from the output interface 64a. Alternatively, the processor 61a saves the output signal obtained as a result of processing in the storage device 11a via the output interface 64a.

As described above, a program for causing a computer to execute part or all of the processing in any one of the image synthesizing device, the image synthesizing device method, and the image processing device and method of the first embodiment, and the program are recorded, A computer-readable storage medium also forms part of the first embodiment.

<Modification>
In the above description, among the captured images, an image with low exposure is defined as the reference image and the first image ImgA, and an image with high exposure is defined as the reference image and the second image ImgB. However, as described above, conversely, an image with high exposure may be used as the reference image and the first image ImgA, and an image with low exposure may be used as the reference image and the second image ImgB. In this case, the relationship between the luminance value and the synthesis ratio is reversed in the first synthesis ratio determination unit 1a and the second synthesis ratio determination unit 1b. In other words, the first composite ratio image WeiA and the second composite ratio image WeiB are created such that the composite ratio for the first image ImgA is large on the low luminance side and the composite ratio for the first image ImgA is small on the high luminance value side. should be generated. Other processing may be performed in the same manner as the processing described above.

Also, in the above description, the first synthesis ratio is the synthesis ratio for each pixel in synthesizing the first image ImgA based on the luminance value of the first image ImgA. However, the first synthesis ratio may be a synthesis ratio for each pixel in synthesizing the image of the second image ImgB based on the luminance value of the first image ImgA. Similarly, the second synthesis ratio may be a synthesis ratio for each pixel in synthesizing the image of the second image ImgB based on the luminance value of the second image ImgB.

Then, the positional deviation salience determination unit 2 increases the positional deviation saliency as the second synthesis ratio decreases, and increases the positional deviation saliency as the absolute difference between the first synthesis ratio and the second synthesis ratio increases. You can make it bigger. In addition, the position correction unit 4a corrects the position of the first image ImgA for each pixel based on the motion vector W, and the synthesizing unit 4b corrects the position of the second image ImgB based on the second synthesis ratio. A third image ImgC may be generated by weighted addition of the first image ImgA thus obtained. The above effects can also be obtained with the configuration described above.

In addition, within the scope of the invention, the embodiments can be appropriately modified or omitted.

1a first synthesis ratio determination unit 1b second synthesis ratio determination unit 2 displacement salience determination unit 3 weighted vector calculation unit 4 image generation unit 4a position correction unit 4b synthesis unit ImgA first image; ImgB second image, ImgBw second image after position correction, ImgC third image, Imp positional deviation saliency image, W motion vector, WeiA first composite ratio image, WeiB second composite ratio image.

Claims (11)

An image processing device that performs synthesis based on a first image and a second image that have different exposure conditions and correspond to each other,
a first synthesis for each pixel in synthesizing one of the first image and the second image when the first image and the second image are synthesized based on the luminance value of the first image; a first synthesis ratio determining unit that determines the ratio;
Determining a second synthesis ratio for determining a second synthesis ratio for each pixel in synthesis of the one image when the first image and the second image are synthesized based on the luminance value of the second image. Department and
Based on the first composition ratio and the second composition ratio, the displacement between the first image and the second image is generated when the first image and the second image are composited. a misregistration saliency determination unit that determines, for each pixel, the degree of influence on the luminance of a synthesized image that is based on the misregistration saliency;
Weighted motion for calculating a motion vector for each pixel between the first image and the second image by searching for positional displacement while changing the amount of processing required for searching for positional displacement based on the conspicuity of positional displacement. a vector calculator;
An image processing apparatus comprising: an image generation unit configured to generate a third image by combining the first image, the second image, and the motion vector. The image processing device according to claim 1,
The misalignment saliency determination unit
The positional deviation conspicuity is increased as one of the first synthesis ratio and the second synthesis ratio decreases, and the positional deviation is conspicuous as the absolute difference between the first synthesis ratio and the second synthesis ratio increases. increase sexuality,
The weighted motion vector calculator,
The image processing device, wherein the smaller the positional deviation conspicuity, the smaller the amount of processing required for the positional deviation search. The image processing device according to claim 1 or claim 2,
The weighted motion vector calculator,
calculating the motion vector by performing a position shift search between a first composite ratio image formed by the first composite ratio and a second composite ratio image formed by the second composite ratio; processing equipment. The image processing device according to any one of claims 1 to 3,
The weighted motion vector calculator,
An image processing device that changes the amount of processing required for the positional deviation search by changing the number of pixels for which the positional deviation search using a block matching method is performed based on the positional deviation saliency. The image processing device according to any one of claims 1 to 3,
The weighted motion vector calculator,
An image processing device that changes the amount of processing required for the positional displacement search by changing the number of positional displacement candidate positions in the positional displacement search using a block matching method based on the positional displacement conspicuity. The image processing device according to any one of claims 1 to 3,
The weighted motion vector calculator,
An image processing device that changes the amount of processing required for the positional deviation search by changing the number of iterative processes in the positional deviation search using a gradient method based on the positional deviation saliency. The image processing device according to any one of claims 1 to 6,
The misalignment saliency determination unit
Let WeiA(p), WeiB(p), and Imp(p) be the first synthesis ratio, the second synthesis ratio, and the positional deviation salience, respectively, where the pixel position is p, and r ₁ and r ₂ are 0 or more. An image processing apparatus that determines the conspicuity of positional deviation according to the following equation (11) based on the first synthesis ratio and the second synthesis ratio when real numbers are used.

The image processing device according to any one of claims 1 to 7,
The image generator is
a position correction unit that corrects the position of the other image of the first image and the second image based on the motion vector for each pixel;
A synthesizing unit that generates a third image by synthesizing the one image and the position-corrected other image with weighted addition based on the first synthesis ratio or the second synthesis ratio. and an image processing device. An image processing method for synthesizing based on a first image and a second image that have different exposure conditions and correspond to each other,
a first synthesis for each pixel in synthesizing one of the first image and the second image when the first image and the second image are synthesized based on the luminance value of the first image; determine the ratio,
determining a second synthesis ratio for each pixel in synthesis of the one image when the first image and the second image are synthesized based on the luminance value of the second image;
Based on the first composition ratio and the second composition ratio, the displacement between the first image and the second image is generated when the first image and the second image are composited. The degree of influence on the brightness of the synthesized image is determined for each pixel as the salience of positional deviation
calculating a motion vector for each pixel between the first image and the second image by the positional displacement search while changing the processing amount required for the positional displacement search based on the positional displacement conspicuity;
An image processing method, wherein a third image is generated by combining the first image, the second image, and the motion vector. An image processing program that causes a computer to execute each process in the image processing method according to claim 9 . A computer-readable storage medium storing the image processing program according to claim 10 .

Images