JP2009206585A - Image processing apparatus, image processing program and digital camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of optimally acquiring an image having blurrs to a photographic scene in any photographing scene, and to provide an image processing program and a digital camera. <P>SOLUTION: This image processing apparatus is provided with: a separating means (20) for separating a subject region (A region) and a region of parts other than the subject (B region), from a captured image; and a blurring means (20) for applying blur processing on at least the B region as a target of the A region and the B region, separated by the separating means. In the apparatus, the blurring means changes the blurring processing, in response to information regarding the brightness of a photographing scene acquired in photographing the photographed image, or about the kind of photographic mode. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing of a captured image captured by a camera, an image processing program, and a digital camera including them.

  When shooting a scene in which a main subject is present, such as portrait shooting or close-up shooting, it is desirable to be able to acquire a photographed image that makes the main subject stand out as much as possible.

As an example of a conventional technique that makes a main subject of a captured image stand out, for example, Patent Document 1 discloses a technique that blurs the background of a captured image in order to make the main subject stand out.
JP 2000-307935 A

  However, in the prior art, depending on the shooting scene, it has not been possible to optimally blur the shot image. As a result, it was not possible to acquire a photographed image that made the main subject stand out and reflected in the appearance.

  The present invention is to solve the above-mentioned problems of the prior art. It is an object of the present invention to provide an image processing apparatus, an image processing program, and a digital camera that can acquire an image that is optimally blurred for any shooting scene.

  An image processing apparatus according to a first aspect of the present invention includes a separating unit that separates a subject area (A region) and a region other than the subject (B region) from a captured image, and among the A region and the B region separated by the separating unit, An image processing apparatus including a blurring unit that performs blurring processing on at least the B region, and the blurring unit performs blurring according to information on the brightness of the shooting scene or the type of shooting mode acquired when shooting the shot image. Change processing.

  According to a second aspect, in the first aspect, the blur means determines whether the scene of the captured image is bright or dark based on the information, and performs blur processing for each of the bright and dark cases. Change to blur processing.

  In a third aspect based on the second aspect, the bright scene blur processing is an area in which the brightness of the bright spot area in the blur processing target area is substantially equal to the brightness of the center position of the bright spot area. The blur is performed so that the (equal luminance region) is narrow and the luminance change in the peripheral portion of the equal luminance region is moderate.

  According to a fourth aspect of the present invention, in the second or third aspect, the blur processing for a dark scene is substantially equal to the luminance at the center position of the bright spot region with respect to the bright spot region in the blur processing target region. The blur is performed so that the region (equal luminance region) becomes wide and the luminance change in the peripheral portion of the equal luminance region is steep.

  According to a fifth invention, in any one of the first to fourth inventions, the blur means includes the luminance component of the blur processing target area on which the blur processing has been performed on the captured image after the gamma correction. Is amplified at an amplification factor that is substantially the same before and after the blur processing.

  According to a sixth invention, in any one of the first to fifth inventions, the blur means determines in advance an image signal of a saturated bright spot region in the blur processing target region that has been subjected to the blur processing. Amplify with the amplification factor.

  According to a seventh invention, in any one of the first to sixth inventions, the blur means includes a saturated luminescent spot region image signal in a blur processing target region subjected to the blur processing. Amplification is performed at an amplification factor according to the size of the area of the point region.

  In an eighth aspect based on any one of the first to seventh aspects, the blur means performs a contrast enhancement process on the blur processing target area that has been subjected to the blur process.

  A digital camera according to a ninth aspect includes a photographing unit that obtains a photographed image by photographing a scene and the image processing device according to any one of the first to eighth aspects.

  An image processing program according to a tenth aspect of the invention is a separation procedure for separating a subject area (A area) and a non-subject area (B area) from a captured image, and at least B of the separated A area and B area. An image processing program for causing a computer to execute a blur procedure for performing blur processing on an area, and in accordance with information on the brightness of a shooting scene or the type of shooting mode acquired at the time of shooting a shot image Change the blur processing.

  In an eleventh aspect based on the tenth aspect, in the blurring procedure, it is determined whether the scene of the captured image is bright or dark based on the information, and the blurring process is performed for each of the bright and dark cases. Change to blur processing.

  In a twelfth aspect based on the eleventh aspect, the blur processing for a bright scene is an area having a luminance substantially equal to the luminance at the center position of the bright spot area with respect to the bright spot area in the blur processing target area. The blur is performed so that the (equal luminance region) is narrow and the luminance change in the peripheral portion of the equal luminance region is moderate.

  In a thirteenth aspect based on the eleventh or twelfth aspect, the blur processing for a dark scene is substantially the same as the luminance at the center position of the bright spot area for the bright spot area in the blur target area. The blur is performed so that the region (equal luminance region) becomes wide and the luminance change in the peripheral portion of the equal luminance region is steep.

  In a fourteenth aspect based on any one of the tenth to the thirteenth aspects, in the blur step, the luminance component of the blur processing target area that has been subjected to the blur processing in the captured image after the gamma correction Is amplified at an amplification factor that is substantially the same before and after the blur processing.

  According to a fifteenth aspect, in any one of the tenth to fourteenth aspects, in the blurring procedure, an image signal of a saturated bright spot region in the blur processing target region subjected to the blurring process is determined in advance. Amplify with the amplification factor.

  In a sixteenth aspect based on any one of the tenth to fifteenth aspects, in the blurring procedure, an image signal of a saturated bright spot region in the blur processing target region that has been subjected to the blurring process is converted into a saturated bright spot. Amplification is performed at an amplification factor according to the size of the area of the point region.

  In a seventeenth aspect based on any one of the tenth to sixteenth aspects, in the blurring procedure, a contrast enhancement process is performed on the blurring processing target area subjected to the blurring process.

  In the present invention, the blurring process is changed according to the brightness of the shooting scene acquired at the time of shooting the shot image or the information on the type of shooting mode. Slow processing is applied. Therefore, in any shooting scene, it is possible to acquire an image that is optimally blurred for the shooting scene.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described. The present embodiment is an embodiment of a digital camera.

  FIG. 1 is a block diagram of a digital camera of the present invention.

  The digital camera includes an imaging lens 11 and a lens driving unit 12, an aperture 13, an imaging device 14, an analog signal processing unit 15, a timing generator (TG) 16, a buffer memory 17, an image processing unit 18, and a compression. / Decoding unit 19, control unit 20, photometry unit 21, monitor 22, recording medium 23, light emitting unit 24, operation unit 25, and bus 26. Here, the buffer memory 17, the image processing unit 18, the compression / decoding unit 19, the control unit 20, the photometry unit 21, the monitor 22, and the recording medium 23 are connected via a bus 26. Further, the lens driving unit 12, the diaphragm 13, the analog signal processing unit 15, the TG 16, the light emitting unit 24, and the operation unit 25 are each connected to the control unit 20.

  The imaging lens 11 includes a plurality of lens groups including a focus lens and a zoom lens. For simplicity, the imaging lens 11 is illustrated as a single lens in FIG.

  The lens driving unit 12 generates a lens driving signal in accordance with an instruction from the control unit 20, performs focus adjustment and zoom adjustment by moving the imaging lens 11 in the optical axis direction, and subjects the light flux that has passed through the imaging lens 11 to the subject. An image is formed on the light receiving surface of the image sensor 14.

  The diaphragm 13 adjusts the amount of the light beam reaching the light receiving surface of the image sensor 14 by adjusting the opening amount in accordance with an instruction from the control unit 20.

  The imaging element 14 is a CCD type or CMOS type area image sensor, and is arranged on the image space side of the imaging lens 11. The image sensor 14 photoelectrically converts a subject image formed on the light receiving surface to generate an analog image signal. The output of the image sensor 14 is connected to an analog signal processing unit 15.

  The analog signal processing unit 15 performs analog signal processing such as CDS (correlated double sampling), gain adjustment, and A / D conversion on the analog image signal output from the image sensor 14 in accordance with an instruction from the control unit 20. And the processed image signal is output. Further, the analog signal processing unit 15 sets an adjustment amount for gain adjustment based on an instruction from the control unit 20, and thereby adjusts photographing sensitivity corresponding to ISO sensitivity. The output of the analog signal processing unit 15 is connected to the buffer memory 17.

  The TG 16 supplies timing pulses to the image sensor 14 and the analog signal processing unit 15 based on instructions from the control unit 20. The drive timing of the image sensor 14 and the analog signal processing unit 15 is controlled by the timing pulse.

  The buffer memory 17 temporarily stores the image signal output from the analog signal processing unit 15 as image data. The buffer memory 17 temporarily stores image data created in the course of processing by the control unit 20.

  The image processing unit 18 performs image processing such as white balance adjustment, color separation (interpolation), and gamma correction on the image data in the buffer memory 17 in accordance with an instruction from the control unit 20. Note that the image processing unit 18 is configured as an ASIC or the like.

  The compression / decoding unit 19 performs a compression process on the image data in accordance with an instruction from the control unit 20. The compression process is performed in JPEG (Joint Photographic Experts Group) format or the like.

  In accordance with an instruction from the control unit 20, the photometry unit 21 uses a known photometry method such as multi-division photometry (multi-pattern photometry), center-weighted photometry, or spot photometry based on the image data recorded in the buffer memory 17. And an evaluation value indicating the brightness of the entire shooting scene and the brightness distribution of the shooting scene.

  The monitor 22 is an LCD monitor provided on the back surface of the digital camera housing or the like, an electronic viewfinder having an eyepiece, and the like, and various images such as photographed images and digital cameras according to instructions from the control unit 20. A GUI screen or the like for allowing the user to input information required by the user is displayed.

  Image data or the like of the captured image is recorded on the recording medium 23 by the control unit 20. The recording medium 23 is a memory card incorporating a semiconductor memory, a small hard disk, or the like.

  The light emitting unit 24 drives an electronic flash or the like provided in the digital camera housing based on an instruction from the control unit 20 to emit strobe light for illuminating the subject toward the object scene.

  The operation unit 25 includes operation members such as various mode setting buttons and release buttons, and sends operation signals to the control unit 20 according to the contents of member operations by the user.

  The control unit 20 comprehensively controls each unit of the digital camera according to the operation signal sent from the operation unit 25. For example, when the digital camera is set to the shooting mode, the control unit 20 drives the lens driving unit 12, the diaphragm 13, the analog signal processing unit 15, and the TG 16 to start shooting a through image. At this time, the image sensor 14 is driven in the draft mode, and the image data of the through image is sequentially recorded in the buffer memory 17 via the analog signal processing unit 15. The control unit 20 performs focus adjustment control (AF) of the imaging lens 11 in cooperation with the lens driving unit 12 based on the image data of the through image. Further, when the control unit 20 drives the photometry unit 21 to calculate the evaluation value of the shooting scene based on the image data of the through image in the buffer memory 17, the setting contents of the analog signal processing unit 15 and the like are set based on the evaluation value. Adjust.

  At the time of shooting, the control unit 20 determines shooting conditions (whether flash emission, aperture value, shutter speed, etc.) based on the evaluation value calculated by the photometry unit 21, and the lens driving unit 12 under the determined shooting conditions. The diaphragm 13, the analog signal processing unit 15, and the TG 16 (and the light emitting unit 24 depending on the shooting conditions) are driven to perform shooting. At this time, the image sensor 14 is driven in the frame mode, and the image data of the captured image is recorded in the buffer memory 17 via the analog signal processing unit 15. After this recording, the control unit 20 drives the image processing unit 18 and the compression / decoding unit 19 to perform image processing and compression processing on the image data of the photographed image recorded in the buffer memory 17 and perform these processing. The subsequent image data is recorded on the recording medium 23.

  When the digital camera is set to the image playback mode, the control unit 20 drives the monitor 22 to display the content of the captured image recorded on the recording medium 23 on the LCD monitor or the like.

  Hereinafter, the blur processing operation of the digital camera of this embodiment will be described with reference to the flowchart of FIG. The flowchart in FIG. 2 is executed, for example, when the user performs shooting with the digital camera mode set to the “background blur processing mode”.

  Step 101: The control unit 20 determines photographing conditions (whether or not strobe light is emitted, aperture value, shutter speed, etc.) based on the evaluation value calculated by the photometry unit 21.

  Step 102: The control unit 20 performs shooting under the determined shooting conditions. The image data of the captured image acquired by this shooting is recorded in the buffer memory 17.

  Step 103: The control unit 20 separates a subject region (subject region) and a region other than the subject from the captured image in the buffer memory 17. Here, it is assumed that the area other than the subject is, for example, a background area (background area). For the separation of the subject area and the background area, a conventional subject extraction technique such as “Japanese Patent Application No. 2007-096724” can be used. In the case where these areas are separated using this subject extraction technique, another image (preliminary image) is captured when the captured image is captured in step 102. However, when the shooting scene at the time of shooting is, for example, bright and continuous light shooting, the preliminary image is shot under the same shooting conditions as the shot image and forcibly emitting strobe light. Then, in this step 103, luminance comparison (luminance difference or luminance ratio comparison) between the captured image and the preliminary image is performed, and the subject area and the background area may be separated from the captured image based on the result.

  Step 104: The control unit 20 uses the information on the presence or absence of strobe light emission of the shooting condition determined in the above step 101 as the brightness information of the shooting scene, and based on this information, the brightness of the shooting scene, that is, the shooting scene is “bright”. Or “dark”. Specifically, if the information on the presence or absence of strobe emission is “no strobe emission”, the control unit 20 determines that the shooting scene is “bright” (Yes side), and proceeds to step 105. On the other hand, when the information on the presence or absence of strobe light is “strobe light is present”, the control unit 20 determines that the shooting scene is “dark” (No side), and proceeds to step 106.

  Step 105: The control unit 20 performs blur processing for a bright scene on the separated background area. When the blur processing ends, the control unit 20 proceeds to step 107.

  Step 106: The control unit 20 performs blur processing for a dark scene on the separated background area.

  Here, the blur process for a bright scene and the blur process for a dark scene will be described.

  When blur processing is performed on an image (in this embodiment, a background area of a captured image), how to blur bright spots in the image is important. This is because the brightness of the bright spot often affects the overall image impression. Also, in order to obtain a suitable blurred image, the shooting scene is bright (for example, shooting outdoors on a sunny day) or dark (for example, shooting a night view). It is necessary to change the blurring method (blurring degree) of the bright spot.

  For example, in an image obtained by photographing a person on a sunny day as shown in FIG. 3A, a white flower field (2a) in the background is a bright spot, but such a scene is bright. In the above image, it is preferred that blurring is performed so that the outline of the bright spot (white flower field) is unclear. If you do not do so, the bright spots will appear and you will feel uncomfortable.

  On the other hand, in an image obtained by taking a portrait of a person as shown in FIG. 3 (b), the street lamp (2b) in the background is a bright spot, but in an image in which such a shooting scene is dark, It is preferable that blurring is performed so that the bright spot portion extends in a circular shape, that is, the outline of the bright spot (streetlight) portion becomes clear. If this is not done, the bright spot will be blurred and the user will feel uncomfortable.

  For this reason, in the blurring process performed when the shooting scene is bright (bokeh processing for a bright scene) in order to obtain an image that does not feel uncomfortable, as shown in FIG. 4A, a bright spot ( For the area of bright spot 1), an area (equal brightness area) having a brightness substantially equal to the brightness of the center position of the bright spot (bright spot 1) area is narrow, and a peripheral portion of the equal brightness area is Defocusing is performed so that the luminance change in the region is gradual (bright spot 2).

  Further, in the blurring process (dark scene blurring process) performed when the shooting scene is dark, as shown in FIG. 4B, the bright spot (bright spot 1) in the image has its bright spot. Bokeh is performed so that a region (equal luminance region) having a luminance substantially equal to the luminance of the center position of the region of the point (bright point 1) is wide and the luminance change in the peripheral portion of the equal luminance region is steep. (Bright spot 3).

  As described above, by applying special blur processing to the image (in this embodiment, the background area of the captured image) with different blurring methods (degrees of blur) depending on whether the shooting scene is bright or dark, in any shooting scene This makes it possible to obtain a blurry image suitable for a shooting scene without any sense of incongruity.

  Note that the blur processing is executed by, for example, performing a convolution operation using a point spread function on an image.

  Further, the change of the blur method such as the above-described bright scene blur process and dark scene blur process is executed by using different point spread functions.

  Similar blur processing can be performed by using various digital filters such as a low-pass filter instead of the point spread function.

  Step 107: The control unit 20 drives the monitor 22 to display a photographed image with a blurred background portion on an LCD monitor or the like. Thereby, the user can confirm the performance of the captured image.

  Step 108: The control unit 20 drives the compression / decoding unit 19 to perform a compression process on the photographed image in which the background portion is blurred, and records the compressed image on the recording medium 23.

In addition, in order to obtain an image with a better looking appearance, any of the following (1) to (4) is performed when performing the blurring process on the image (the background area of the captured image in the present embodiment). It is preferable to apply such a process.
(1) Processing to reduce blurring feeling caused by gamma correction Generally, an output device such as a monitor or a printer has a non-linear γ characteristic. In order to obtain a characteristic image, the image is subjected to gamma correction so that the total γ characteristic with the output device is linear (γ = 1). For this reason, the image after gamma correction has a γ characteristic opposite to that of the output device (see FIG. 5).

  As described above, since an image photographed by a digital camera has a non-linear γ characteristic, if a convolution calculation process using a point spread function, that is, a blur process, is performed on the photographed image, the image is captured before and after the blur process. This causes a problem that the luminance information is not maintained. Therefore, as a result, an image with a sense of incongruity in brightness of the bright spot is created as compared with an image blurred by the lens effect as seen in an image taken with a single-lens reflex camera. Note that the blur due to the lens effect means that the background of an image to be captured is optically blurred by adjusting the aperture of the camera and the focal length of the lens to reduce the depth of field. Point to.

  Therefore, in blurring processing, it is desirable to amplify the luminance component of the image that has been nonlinearly gamma corrected in the digital camera at an amplification factor different from that of the color component.

  For example, after performing blur processing on an image that has been nonlinearly gamma corrected as described above, the color space of the image after the blur processing is converted from the RGB color space to the YCbCr color space, and the converted image The image signal is separated into a luminance component (Y signal) and a color difference component (Cb signal and Cr signal). Then, after performing blur processing on each of the Y signal, Cb signal, and Cr signal, only the Y signal is amplified. At this time, the amplification of the Y signal is performed at an amplification factor at which the level of the luminance component of the image is substantially the same before and after the blurring process. In this way, by amplifying the Y signal at an amplification factor that maintains the level of the luminance component before and after blur processing, the brightness of the bright spot in the image after blur processing is blurred due to the lens effect. Since the level can be substantially the same as that in the image, the uncomfortable feeling of blur caused by gamma correction can be reduced. According to this method, the image signal is separated into a luminance component signal (Y signal) and a color difference component signal (Cb signal, Cr signal), and only one signal (Y signal) is amplified. Since the luminance component of the image is corrected to a preferred level, the processing load can be reduced as compared with the case where the luminance component is corrected using three signals (R signal, G signal, and B signal).

In the above description, the example in which the luminance component of the image is amplified after the blur processing is performed on the image has been described. However, the present invention is not limited to this, and the image is subjected to the blur processing after the luminance component of the image is amplified. May be.
(2) Processing 1 for reducing the uncomfortable feeling of blur caused by white clips
FIG. 6 is a diagram for explaining a change in the luminance distribution of an image when blur processing is performed on an image including a bright clipped white point. Note that white clipping refers to a function of clipping a luminance portion exceeding the dynamic range of the image sensor in a captured image of a digital camera.

  FIG. 6A shows the luminance distribution of the image before the blur processing is performed. In FIG. 6A, for simplicity, the image includes a bright spot A1 having a signal level equal to or higher than the white clip level and a bright spot B1 having a signal level lower than the white clip level. It is assumed that the signal level is very low in portions other than the bright spot B1. The signal level represents the brightness (brightness) of the bright spot. When the signal level is high, the bright spot becomes bright, and when the signal level is low, the bright spot becomes dark. Moreover, such an image is an image acquired when, for example, a scene such as a night view including two bright spots having different brightness is photographed.

  In the digital camera, a portion having a luminance exceeding the dynamic range of the image sensor such as the bright spot A1 in FIG. 6A is white clipped, and luminance information above the white clip level is lost.

  Here, consider a case where blurring is performed by a lens effect.

  FIG. 7 is a diagram for explaining a change in the luminance distribution of an image captured when blurring is performed by the lens effect. FIG. 7A shows the luminance distribution of the light beam from the object scene incident on the lens of the digital camera. In this case, for the sake of simplicity, it is assumed that the object scene includes two bright spots (A1 'and B1').

  As shown in FIG. 7A, the luminous flux of the bright spot A1 ′ has a luminance exceeding the dynamic range of the image sensor, but the subject image by the luminous flux of the bright spot A1 ′ is blurred due to the lens effect. Since it is formed on the light receiving surface of the image pickup device later, when the image of the subject image is taken, the signal level becomes lower than the white clip level as shown by the bright spot A2 ′ in FIG. For this reason, luminance information is not lost.

  However, when blurring is performed as image processing, blurring processing is performed on an image including the bright spot A1 (FIG. 6A) from which luminance information has been lost due to the white clip. Therefore, the blur of the bright spot A1 is performed. The bright point A2 (FIG. 6B) indicating the state after processing becomes darker as the signal level is lowered by the amount of the lost luminance information as compared with the case where blurring is performed due to the lens effect.

  Therefore, it is desirable to perform an amplification process as shown in FIG. 6C on the image signal of the bright spot A2 whose signal level has decreased. By doing so, the signal level of the bright spot A2 can be increased like the bright spot A3 shown in FIG. Note that the amplification of the image signal of the bright spot A2 is performed at a predetermined amplification factor or an amplification factor designated by the user. As a result, it is possible to reduce the uncomfortable feeling of blur caused by the white clip, and it is possible to create an image that is visually blurry.

In the above description, an example in which amplification processing is performed on a bright spot image signal in an image after blur processing has been described. However, the present invention is not limited to this, and the bright spot image signal in the image before blur processing is amplified. After that, blur processing may be performed on the image.
(3) Process 2 for reducing the uncomfortable feeling of blur caused by white clips
The process (3) is intended to further increase the degree of the effect of the process (2), that is, the degree to reduce the uncomfortable feeling of blur.

  FIG. 8 is a diagram for explaining a change in luminance distribution of an image when blurring processing is performed on an image including a bright clipped point.

  FIG. 8A shows the luminance distribution of the image before the blur processing is performed. In FIG. 8A, for the sake of simplicity, it is assumed that the image includes three bright spots (C1, D1, E1) having a signal level equal to or higher than the white clip level and having different area areas. . Further, it is assumed that the signal level is extremely low in the portions other than the three bright spots in the image.

  In general, when shooting a subject having the same brightness with a digital camera, the closer the distance between the digital camera and the subject, the larger the area occupied by the subject in the shot image, and the brightness also Get higher. From this principle, it can be estimated that the amount of luminance information lost by the white clip increases as the area of the three bright spots in FIG. 8A increases. That is, the three bright spots in FIG. 8 (a) have a large area in the image in the order of “bright spot C1 <bright spot E1 <bright spot D1”. It can be estimated that the brightness increases in the order of “bright spot C1 <bright spot E1 <bright spot D1”. Further, based on the above estimation, considering the case where blurring is performed on the three bright spots shown in FIG. 8A due to the lens effect, the signal level of each bright spot in the image obtained by photographing them is “bright spot C1. It is considered that the brightness increases in the order of <bright spot E1 <bright spot D1>.

  However, when blur processing is performed on an image including these three bright spots, the brightness information is lost due to the white clip in any of the bright spots, so that the bright spots C1, D1, and E1 (FIG. 8A). The bright points C2, D2, and E2 (FIG. 8B) indicating the state after the blur processing of () are signal levels corresponding to the lost luminance information as compared with the case where the blur is performed due to the lens effect. Will fall and become darker.

  Therefore, it is desirable to perform an amplification process as shown in FIG. 8C on the image signals in the respective areas of the bright spots C2, D2 and E2 where the signal level is lowered. However, the amplification process is performed at an amplification factor determined in accordance with the size of each area of the white clipped bright spots C1, D1, E1 (FIG. 8A). For example, since the area of each of the bright spots C1, D1, and E1 is larger in the order of “bright spot C1 <bright spot E1 <bright spot D1”, the bright spots C2, D2, and E2 after blur processing (FIG. 8B). ) Is determined as “amplification factor of bright spot C2 = small, amplification factor of bright spot E2 = medium, amplification factor of bright spot D2 = high”. Then, amplification processing is performed on the image signals in the respective areas of the bright spots C2, D2, and E2 at the determined amplification factors. By doing so, the signal levels of the bright spots C2, D2, and E2 whose signal levels have been lowered can be increased like the bright spots C3, D3, and E3 shown in FIG. Can be further reduced.

  In this manner, by changing the amplification factor according to the size of the area of the white clipped bright spot region, by performing amplification processing on the image signal of the bright spot region after blur processing at the amplification factor, Even when blur processing is performed as image processing, it is possible to create an image closer to the case where blurring is performed due to the lens effect.

In the above description, the example in which the amplification process is performed on the image signal of the bright spot area in the image after the blur process has been described. However, the present invention is not limited thereto, and the image signal of the bright spot area in the image before the blur process is described. The image may be subjected to blur processing after amplification.
(4) Processing for further improving the uncomfortable feeling of blur When blur processing is applied to an image, the signal level of the image is shifted in the direction of averaging. That is, the high luminance signals in the images such as the bright spots F1 and G1 shown in FIG. 9A have the signal level lowered after the blur processing, and the bright spots F2 and G2 shown in FIG. 9B, respectively. It becomes a state. Further, the low luminance signal in the image as shown by the portions other than the bright spots F1 and G1 in FIG. 9A has a signal level that increases after the blur processing, and other than the bright spots F2 and G2 in FIG. 9B. The state is as shown by the part. For this reason, the image subjected to the blurring process has a low contrast, and is an image that is not realistic and does not appear to the eye.

  Therefore, it is desirable to perform contrast enhancement processing as shown in FIG. 9C on the image after blur processing. However, in contrast enhancement processing, for example, gradation processing using a tone curve as shown in FIG. 10 is performed. That is, in order to be opposite to the change in the signal level due to the blur processing, the bright points F2 and G2 (FIG. 9B) included in the image after the blur processing are converted into high luminance signals (signals with a high signal level). On the other hand, gradation processing is performed to increase the signal level by gain amplification so that the bright points F3 and G3 shown in FIG. Further, for a low luminance signal (a signal having a low signal level) as indicated by a portion other than the bright spots F2 and G2 of FIG. 9B included in the image after the blurring process, the luminance of FIG. Tone processing is performed to reduce the signal level by suppressing the gain so that the portions other than the points F3 and G3 are in the state shown. By doing so, the bright spots F2 and G2 whose signal levels have been lowered by the blur processing (the portions that are originally desired to be bright like night street lamps) are each gain-amplified and shown in FIG. The brightness can be brought close to the original brightness like the bright spots F3 and G3. In addition, the gains of the portions other than the bright spots F2 and G2 whose signal levels are increased by the blurring process (portions that should originally be dark like black in the night scene) are suppressed, and the bright spots in FIG. As shown by the portions other than F3 and G3, the original darkness can be obtained. As a result, the sense of incongruity is further improved, and an image that is more realistic and can be seen visually can be created.

  Note that the above-described contrast enhancement processing is not limited to the tone curve illustrated in FIG. 10, and any method may be used as long as it is a method of gain-amplifying a high luminance signal and suppressing a gain of a low luminance signal. .

  The tone curve used for the contrast enhancement process may be selected by a user from a plurality of tone curves prepared in advance. Also, prepare a tone curve to be used as a template in advance and let the user freely adjust the tone curve contents based on the template, and let the user create a tone curve to be used for contrast enhancement processing. Also good.

  Note that the processes (1) to (4) may be performed alone or in combination of two or more processes.

(Operational effects of the first embodiment)
As described above, in the digital camera of the first embodiment, the shooting condition is determined based on the photometric evaluation value, and the blurring process applied to the shot image is changed according to the brightness of the shooting scene determined based on the shooting condition. . Then, the blurring process is performed on the captured image (in this embodiment, the background area of the captured image).

  In this case, the optimal blur according to the brightness of the shooting scene is applied to the background area of the shot image after the blur processing. In addition, such a blurred photographed image is a photographed photographed image that makes the main subject stand out and appears to the eye.

  For example, the digital camera of the first embodiment determines the brightness of the photographic scene, and when the photographic scene is “bright”, changes the blurring process applied to the photographic image to the blurring process for the bright scene. On the other hand, when the shooting scene is “dark”, the blurring process applied to the shot image is changed to the blurring process for the dark scene. The digital camera according to the first embodiment performs the blurring process on the captured image (in this embodiment, the background area of the captured image).

  As described above, in the digital camera according to the first embodiment, the blurring process performed on the captured image (in this embodiment, the background area of the captured image) is changed to an optimum one according to the brightness of the captured scene.

  Therefore, in any shooting scene, it is possible to reliably acquire a shot image that is optimally blurred for the shooting scene.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described. This embodiment is also an embodiment of a digital camera. In the following description, the same reference numerals are given to the components of the digital camera common to the first embodiment, and the duplicate description will be omitted.

  In the first embodiment, the brightness of the shooting scene is determined based on the shooting conditions determined based on the photometric evaluation value, and the blurring process is performed according to the brightness of the shooting scene (in the first embodiment, the shot image). For the background area). On the other hand, in the second embodiment, the type of the shooting mode (scene mode) selected by the user is determined, and blurring processing is performed on the shot image according to the brightness of the shooting scene estimated from the type of the scene mode. .

  Here, the scene mode is a shooting mode in which various conditions for shooting (shooting conditions) such as exposure and color settings are determined in advance to be optimal for each shooting scene. Such types include portrait mode, backlight mode, close-up mode, night view portrait mode, and the like.

  If the type of scene mode selected by the user is detected by the digital camera, the scene that the user intends to shoot (here, the shooting scene is “bright” or “dark”, that is, the brightness of the shooting scene)). Can be estimated. That is, if the type of the selected scene mode is “portrait mode”, “backlight mode”, or “close-up mode”, the shooting scene can be estimated to be “bright”, and the scene mode type is “ If it is “night view portrait mode”, it can be estimated that the shooting scene is “dark”. Accordingly, if the blurring process applied to the shot image is changed according to the brightness of the shot scene estimated from the type of the scene mode, an image with a suitable blur condition for each shot scene can be created.

  Hereinafter, this operation will be described with reference to the flowchart of FIG. The flowchart in FIG. 11 is executed, for example, when the user performs shooting with the digital camera mode set to the “background blur processing mode”. Note that the processing of Steps 206, 207, 209, and 210 of the present embodiment is the same as the processing of Steps 105, 106, 107, and 108 of the first embodiment, and therefore redundant description is omitted.

  Step 201: The control unit 20 determines whether or not a scene mode is selected by the user. When the scene mode is selected (Yes side), the control unit 20 proceeds to Step 202. On the other hand, when the scene mode is not selected (No side), the control unit 20 proceeds to Step 214.

  Step 202: The control unit 20 performs shooting under the shooting conditions for the selected scene mode. The image data of the captured image acquired by this shooting is recorded in the buffer memory 17.

  Step 203: The controller 20 determines whether the type of the selected scene mode is “portrait mode”, “backlight mode”, “close-up mode”, or “night scene portrait mode”. In the case of any one of these four (Yes side), the control unit 20 proceeds to Step 204. On the other hand, if the scene mode is other than these (No side), the control unit 20 proceeds to Step 211.

  Step 204: The control unit 20 separates a subject area (subject area) and a non-subject area from the captured image in the buffer memory 17. Here, it is assumed that the area other than the subject is, for example, a background area (background area). For the separation of the subject area and the background area, a conventional subject extraction technique such as “Japanese Patent Application No. 2007-096724” can be used. In the case where these regions are separated using this subject extraction technique, another image (preliminary image) is captured when the captured image is captured in step 202 described above. However, when the selected scene mode is, for example, “night scene portrait mode”, the preliminary image is captured under the same shooting conditions as the captured image, that is, the shooting conditions for “night scene portrait mode” and the strobe. It is carried out without forcibly emitting light. Then, in this step 204, luminance comparison (luminance difference or luminance ratio comparison) between the captured image and the preliminary image is performed, and the subject area and the background area may be separated from the captured image based on the result.

  Step 205: The controller 20 determines whether the type of the selected scene mode is “portrait mode”, “backlight mode”, or “close-up mode”. In the case of any one of these three (Yes side), the control unit 20 estimates that the shooting scene is “bright”, and proceeds to Step 206 to perform blur processing for a bright scene. On the other hand, in the case of a scene mode other than these, that is, the “night scene portrait mode” (No side), the control unit 20 estimates that the shooting scene is “dark” and performs blur processing for the dark scene. To step 207.

  Step 208: The control unit 20 performs contrast enhancement processing on the separated background region. In the background area, the blur processing according to the brightness of the shooting scene estimated from the type of the scene mode by the processing in step 206 or step 207, that is, the blur processing for a bright scene or the blur for a dark scene. Any one of the processes is performed. Further, the above-described process (4) described in the first embodiment is applied to the contrast enhancement process.

  Step 211: When the type of the selected scene mode is not “portrait mode”, “backlight mode”, “close-up mode”, or “night scene portrait mode”, the control unit 20 performs “background blur”. After warning the user that it is a scene mode that is not subject to processing, the processing ends. The warning to the user is performed by, for example, displaying a message on the monitor 22 that a scene mode that is not subject to background blur processing is selected.

  Step 212: The control unit 20 drives the monitor 22 to display the captured image (not subjected to the blurring process) in the buffer memory 17 on the LCD monitor or the like. Thereby, the user can confirm the performance of the captured image.

  Step 213: The control unit 20 drives the compression / decoding unit 19 to perform the compression process on the captured image (not subjected to the blur process) in the buffer memory 17, and the compressed image is recorded on the recording medium. 23.

  Step 214: When the scene mode is not selected, the control unit 20 performs the processing of Step 101 to Step 106 in the flow of FIG. 2 described in the first embodiment, and performs a photographic scene on the photographic image. The blur processing according to the brightness of the is performed. Note that the blur processing is performed on the background region of the captured image, for example. When the blur processing ends, the control unit 20 proceeds to step 208.

  In addition, in order to obtain an image with better appearance, the above-described (1) described in the first embodiment is performed when performing the blurring process on the image (the background area of the captured image in the present embodiment). It is preferable to apply one or more treatments (1) to (3).

(Operational effect of the second embodiment)
As described above, in the digital camera of the second embodiment, the blurring process performed on the captured image is changed according to the type of the shooting mode (scene mode) selected by the user. Then, the blurring process is performed on the captured image (in this embodiment, the background area of the captured image).

  In this case, the optimum blur according to the type of the scene mode is applied to the background area of the photographed image after the blur processing. In addition, such a blurred photographed image is a photographed photographed image that makes the main subject stand out and appears to the eye.

  For example, in the digital camera according to the second embodiment, when the user selects any one of “portrait mode”, “backlight mode”, and “close-up mode” as the scene mode, the shooting scene is estimated to be “bright”. Then, the blurring process applied to the captured image is changed to the blurring process for a bright scene. On the other hand, when the user selects the “night scene portrait mode” as the scene mode, the shooting scene is estimated to be “dark” and the blurring process applied to the shot image is changed to the blurring process for the dark scene. The digital camera according to the second embodiment performs the blurring process on the captured image (in this embodiment, the background area of the captured image).

  As described above, in the digital camera of the second embodiment, the blurring process performed on the photographed image (in this embodiment, the background area of the photographed image) is performed on the brightness of the photographed scene estimated from the type of the scene mode selected by the user. It will be changed to the optimal one according to your needs.

  Therefore, it is possible to reliably acquire a photographed image that is optimally blurred for the photographing scene that the user intends to photograph.

  Further, in the digital camera of the second embodiment, the contrast enhancement process is performed after the blurring process is performed on the captured image (in this embodiment, the background area of the captured image).

  In this case, the brightness of the bright spot included in the background area of the photographed image whose signal level has been lowered by the blurring process is brought close to the intended brightness by gain amplification. Further, the brightness of the portion other than the bright spot in the background area of the captured image whose signal level has been increased by the blurring process is brought closer to the original darkness by the gain suppression.

  Therefore, the sense of incongruity caused by the blurring process is further improved, and it is possible to reliably acquire a captured image that is closer to reality and looks visually.

(Other)
In step 103 of the first embodiment (step 204 in the second embodiment), as a technique for separating the subject and the background from the photographed image, the brightness of the strobe light emission image and the non-flash light image of the prior art is used. Although an example using the “subject extraction technique based on comparison” has been shown, usable subject extraction methods are not limited to this. For example, conventional methods such as a method based on multiple frames (images) with different focal lengths and in-focus states, a method based on spatial frequency analysis, a method based on face recognition, a method based on a specific color such as skin color, a method based on edge information, etc. Any subject extraction technique can be used.

  In the description of step 104 of the first embodiment, an example of automatic determination in which the brightness of the shooting scene is determined based on the evaluation value calculated by the photometry unit 21 is shown. However, the determination method is not limited to this. Alternatively, manual determination may be made to determine the brightness of the shooting scene based on a user input operation.

  Further, the processing after step 103 of the first embodiment and the processing of steps 204 to 210 of the second embodiment may be executed when the digital camera is set to a non-shooting mode such as an image playback mode. In such a case, the subject / background separation information obtained in step 103 of the first embodiment or step 204 of the second embodiment and the photographing conditions obtained in step 101 of the first embodiment (based on these photographing conditions). The brightness of the shooting scene is determined) or the type of shooting mode (scene mode) obtained in step 201 of the second embodiment is recorded on the recording medium 23 together with the shot image as an image file. If the image file is based on the format of the Exif standard, the subject and background separation information, the shooting condition or the shooting mode (scene mode) type, the manufacturer's note (MakerNote) tag of the image file, etc. Record as information. Then, when the digital camera is set to the image playback mode or the like, the image file recorded on the recording medium 23 is read out, and the processing after step 103 of the first embodiment and the processing of steps 204 to 210 of the second embodiment. It is good to execute. Alternatively, the processing performed by reading out the image file may be executed by an external processing device such as a computer instead of being executed by the digital camera.

  Further, in step 107 of the first embodiment (step 209 in the second embodiment), after an image after blur processing is displayed on an LCD monitor or the like, an instruction to redo the blur processing may be received from the user. . At the time of acceptance, the user uses the parameters used in the process for reducing or improving the above-mentioned blurring feeling, that is, the gain of the Y signal in the process (1), and the bright spot image in the process (2). The signal amplification factor, the amplification factor of the image signal in each region of the bright spot in the process (3), and the tone curve used for the contrast enhancement process are designated in the process (4). When the user instructs to redo, the process is executed again from step 105 or 106 of the first embodiment (step 206 or 207 in the second embodiment) using the above parameters specified by the user. It is also possible to redo the blur processing. Further, such a process may be repeatedly executed until a result desired by the user is obtained. By doing so, it is possible to create a more effective picture-making image that the user desires and has high performance.

  In the second embodiment, four scene modes of “portrait mode”, “backlight mode”, “close-up mode”, and “night scene portrait mode” have been described as the types of scene modes. Types are not limited to these four types.

  In the above, the subject and the background are separated from the photographed image and the blurring process is performed only on the background. However, the subject and the foreground are separated, or the subject, the background, and the foreground are separated at the same time. The blurring process may be performed on one or more of the subject, the background, and the foreground. In addition, when performing a blurring process with respect to two or more of these, it is good to apply so that the intensity | strength of a blur may be changed by each. For example, when blur processing is performed on the subject and the background, the blur processing may be performed on the subject at a lower intensity than the background.

  In addition to the above-described blur processing for the background region of the captured image, various image processing such as brightness correction processing, saturation correction processing, contrast enhancement processing, and sharpening processing may be performed on the subject region. Therefore, it is possible to make the subject stand out more, and it is possible to create a picture-making image with high performance.

  Further, the method of changing the blur processing according to the brightness of the shooting scene or the type of shooting mode (scene mode) is not limited to the contents shown in the flowcharts of FIGS.

It is a block diagram of the digital camera of the present invention. It is a flowchart which shows the operation | movement of the blurring process of the digital camera of 1st Embodiment. It is a figure which shows the image of the luminescent spot in a picked-up image. It is a figure which shows the difference in the blurring process in a bright scene and a dark scene. It is a figure which shows the (gamma) characteristic of the image image | photographed with a digital camera. It is a figure explaining the process (process 1) which reduces the discomfort of the blur which arises by a white clip. It is a figure which shows the change of the luminance distribution of an image in the case of performing blur by a lens effect. It is a figure explaining the process (process 2) which reduces the discomfort of the blur which arises by a white clip. It is a figure explaining the process which further improves the discomfort of blur. It is a figure which shows the example of the tone curve used for contrast emphasis processing. It is a flowchart which shows the operation | movement of the blurring process of the digital camera of 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Imaging lens, 12 ... Lens drive part, 13 ... Aperture, 14 ... Imaging element, 15 ... Analog signal processing part, 16 ... Timing generator (TG), 17 ... Buffer memory, 18 ... Image processing part, 19 ... Compression / Decoding section, 20 ... control section, 21 ... photometry section, 22 ... monitor, 23 ... recording medium, 24 ... light emitting section, 25 ... operation section, 26 ... bus

Claims (17)

Separating means for separating a region of the subject (A region) and a region other than the subject (B region) from the captured image;
An image processing apparatus comprising blur means for performing blur processing on at least the B area of the A area and the B area separated by the separating means,
The blur processing unit changes the blur processing according to information on brightness of a shooting scene or type of shooting mode acquired when shooting the shot image. The image processing apparatus according to claim 1.
The blur means determines whether the scene of the captured image is bright or dark based on the information, and changes the blur processing to dedicated blur processing for the bright case and the dark case, respectively. A featured image processing apparatus. The image processing apparatus according to claim 2,
The bright scene blur processing is such that, with respect to the bright spot area in the target area, an area (equal brightness area) having a brightness substantially equal to the brightness of the center position of the bright spot area is narrowed, and An image processing apparatus characterized by performing blurring so that a change in luminance in a peripheral portion of an equal luminance region is moderated. The image processing apparatus according to claim 2 or 3,
The blur processing for the dark scene has a wide area (equal luminance area) having a luminance substantially equal to the luminance of the central position of the bright spot area with respect to the bright spot area in the target area, and An image processing apparatus characterized in that blurring is performed so that a luminance change in a peripheral portion of an equal luminance region is sharp. The image processing apparatus according to any one of claims 1 to 4,
The blur means has a luminance component of the target area subjected to the blur processing in the captured image after the gamma correction, and the level of the luminance component is substantially the same before and after the blur processing is performed. An image processing apparatus characterized by amplifying with an amplification factor. In the image processing device according to any one of claims 1 to 5,
The image processing apparatus characterized in that the blur means amplifies an image signal of a saturated bright spot region in the target region subjected to the blur processing at a predetermined amplification factor. The image processing apparatus according to any one of claims 1 to 6,
The blur means amplifies an image signal of a saturated bright spot area in the target area subjected to the blur processing at an amplification factor according to the size of the area of the saturated bright spot area. An image processing apparatus. The image processing apparatus according to any one of claims 1 to 7,
The image processing apparatus, wherein the blur means performs a contrast enhancement process on the target area on which the blur process has been performed. Photographing means for obtaining a photographed image by photographing a scene;
A digital camera comprising: the image processing apparatus according to claim 1. A separation procedure for separating a region of the subject (A region) and a region other than the subject (B region) from the captured image;
An image processing program for causing a computer to execute a blur procedure for performing blur processing on at least the B area of the separated A area and B area,
In the blurring procedure, the blurring process is changed in accordance with information on the brightness of a shooting scene or the type of shooting mode acquired at the time of shooting the shot image. The image processing program according to claim 10,
In the blur procedure, it is determined whether the scene of the captured image is bright or dark based on the information, and the blur processing is changed to dedicated blur processing for the bright case and the dark case, respectively. A characteristic image processing program. The image processing program according to claim 11,
The bright scene blur processing is such that, with respect to the bright spot area in the target area, an area (equal brightness area) having a brightness substantially equal to the brightness of the center position of the bright spot area is narrowed, and An image processing program characterized by performing blurring so that a change in luminance in a peripheral portion of an equal luminance region is moderated. The image processing program according to claim 11 or 12,
The blur processing for the dark scene has a wide area (equal luminance area) having a luminance substantially equal to the luminance of the central position of the bright spot area with respect to the bright spot area in the target area, and An image processing program for performing blurring so that a luminance change in a peripheral portion of an equiluminance region is sharp. The image processing program according to any one of claims 10 to 13,
In the blur procedure, the luminance component of the target area that has been subjected to the blur processing in the photographed image after gamma correction is substantially the same before and after the blur processing. An image processing program characterized by amplification at an amplification factor. The image processing program according to any one of claims 10 to 14,
In the blur procedure, an image signal of a saturated bright spot region in the target region that has been subjected to the blur processing is amplified at a predetermined amplification factor. The image processing program according to any one of claims 10 to 15,
In the blur procedure, an image signal of a saturated bright spot region in the target region subjected to the blur processing is amplified with an amplification factor according to the size of the area of the saturated bright spot region. An image processing program. The image processing program according to any one of claims 10 to 16,
In the blur procedure, a contrast enhancement process is performed on the target area on which the blur process has been performed.

Images