JP6089476B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, various techniques using subject extraction and subject extraction results have been considered. For example, in the invention of Japanese Patent Laid-Open No. 2004-228688, the autofocus is optimally focused on the subject even when the subject moves in the camera direction by deforming the AF distance measurement frame according to the amount of change in the subject shape extracted by subject extraction. An imaging device for adjustment is disclosed.

JP 2009-069748 A

  By the way, subject extraction is generally performed based on color information in many cases. However, actual subjects have various hues. For example, there is a problem that subject extraction is difficult for an achromatic subject with little hue or an image generated by imaging a subject biased to a specific hue.

  The present invention has been made in view of the above points, and an object of the present invention is to perform extraction of a suitable subject area in accordance with the distribution of subject color information.

The image processing apparatus of the present invention calculates the feature amount of the image, calculates a region detecting section that detects a subject region that is determined based on the feature quantity, the evaluation value indicating the distribution of color information before outs image evaluation A value calculation unit; and a control unit that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit , wherein the region detection unit includes the feature amount And calculating the plurality of feature amounts, detecting the subject region based on the plurality of feature amounts, and determining the priority of the plurality of feature amounts based on the evaluation value Based on the priority, the area detection unit detects the subject area.

  In addition, the area detection unit calculates at least a luminance-related feature amount as the feature amount, and the evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value, and the control The unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold value, and if the image is an achromatic image, the feature relating to the luminance The object region may be detected by the region detection unit by setting the priority of the amount higher than when the image is not an achromatic image.

  In addition, the area detection unit calculates at least a color-related feature amount as the feature amount, and the evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value, and the control The unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold, and if the image is an achromatic image, the color feature The object region may be detected by the region detection unit by setting the priority of the amount lower than when the image is not an achromatic image.

  The area detection unit calculates at least a color-related feature amount as the feature amount, and the evaluation value calculation unit calculates an evaluation value indicating a degree of bias to an arbitrary hue in the image as the evaluation value. Then, the control unit determines whether or not the image is biased to the arbitrary hue by comparing the evaluation value with a predetermined threshold, and the image is biased to the arbitrary hue. If the image is an image, the priority of the feature amount related to the arbitrary hue among the feature amounts related to the color is set higher than when the image is not an image biased to the arbitrary hue, The subject region may be detected by the region detection unit.

  In addition, the region detection unit calculates a plurality of feature amounts as the feature amount, and among the plurality of feature amounts, a first region determined based on a first feature amount and the first feature amount An overlapping area that overlaps with a second area determined based on a second feature amount different from the first characteristic area is detected as the subject area, and the control unit detects the first area and the second area based on the evaluation value. The detection method of at least one of the regions may be determined.

  In addition, the area detection unit calculates a feature value related to luminance as the first feature value, and the evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value, The control unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold, and when the image is an achromatic image, The overlapping area may be detected based on only one area.

In addition, the area detection unit calculates a color-related feature amount as the second feature amount, and the evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value, the control unit, by comparing the evaluation value with a predetermined threshold value, the image is determined whether the image of the achromatic, when the image is an image of an achromatic, the first The method of detecting the area 2 may be changed.

  In addition, the area detection unit calculates a color-related feature amount as the second feature amount, and the evaluation value calculation unit uses the evaluation value as an evaluation value indicating a degree of bias to an arbitrary hue in the image. The control unit determines whether or not the image is an image biased to the arbitrary hue by comparing the evaluation value with a predetermined threshold, and the image is the arbitrary hue. In the case where the image is biased to the image, the weight of the second region based on the feature amount related to the arbitrary hue among the second feature amounts is set to the image, and the image is biased to the arbitrary hue. The area of the subject may be detected by the area detection unit by setting it higher than when the image is not an image.

Imaging apparatus of the present invention includes an imaging unit for imaging a subject, calculates the feature amount of the image obtained by the imaging unit, and a region detecting section that detects a subject region that is determined based on the feature quantity, pre-outs An evaluation value calculation unit that calculates an evaluation value indicating a distribution of color information of an image, and a control unit that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit And the region detection unit calculates a plurality of feature amounts as the feature amount, detects the subject region based on the plurality of feature amounts, and the control unit, based on the evaluation value, Priorities of the plurality of feature amounts are determined, and the subject region is detected by the region detection unit based on the determined priorities.

The image processing program of the present invention, the computer calculates the feature amount of the image, a region detecting step of detecting a subject region that is determined based on the feature quantity, the evaluation value indicating the distribution of color information before outs image An evaluation value calculation procedure to be calculated; and a control procedure for determining a method for detecting the subject region by the region detection procedure based on the evaluation value calculated in the evaluation value calculation procedure; and the region detection procedure Includes a procedure of calculating a plurality of feature amounts as the feature amount, and detecting the subject region based on the plurality of feature amounts, wherein the control procedure is based on the evaluation value in the plurality of feature amounts. It includes a procedure for determining priority and detecting the subject region by the region detection procedure based on the determined priority.
The image processing apparatus of the present invention includes an evaluation value calculation unit that calculates an evaluation value of an image based on color information, and an area detection unit that detects a subject area included in the image, and the region detection unit includes: A first detection mode for detecting a subject region included in the image based on luminance information rather than color information based on the evaluation value calculated by the evaluation value calculation unit and included in the image based on color information rather than luminance information One of the second detection modes for detecting the subject area to be detected is selected.
In this case, the region detection unit compares the evaluation value calculated by the evaluation value calculation unit with a predetermined threshold value, and thereby selects one of the first detection mode and the second detection mode. Select.
Further, the evaluation value calculation unit calculates the evaluation value that becomes a smaller value as the image becomes achromatic, and the region detection unit determines that the evaluation value calculated by the evaluation value calculation unit is lower than a predetermined threshold value. If it is smaller, the first detection mode is selected.
Further, the imaging apparatus of the present invention includes an imaging unit that images a subject via a lens, an evaluation value calculation unit that calculates an evaluation value of an image of the subject captured by the imaging unit based on color information, and the image A region detection unit that detects a subject region to be detected, and a drive unit that drives the lens by the subject region detected by the region detection unit, wherein the region detection unit is evaluated by the evaluation value calculation unit A first detection mode that detects a subject area included in the image based on luminance information rather than color information, and a second detection mode that detects a subject area included in the image based on color information rather than luminance information. And one of them is selected.
In this case, the region detection unit compares the evaluation value calculated by the evaluation value calculation unit with a predetermined threshold value, and thereby selects one of the first detection mode and the second detection mode. Select.
Further, the evaluation value calculation unit calculates the evaluation value that becomes a smaller value as the image becomes achromatic, and the region detection unit determines that the evaluation value calculated by the evaluation value calculation unit is lower than a predetermined threshold value. If it is smaller, the first detection mode is selected.

  According to the present invention, a suitable subject region can be extracted according to the distribution of subject color information.

2 is a block diagram illustrating configurations of a lens barrel 10, an imaging device 20, and a storage medium 40. FIG. It is a flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is a figure explaining mask extraction processing (I). It is another figure explaining a mask extraction process (I). It is another figure explaining a mask extraction process (I). It is another figure explaining a mask extraction process (I). It is another figure explaining a mask extraction process (I). It is a figure explaining mask extraction processing (II). It is a figure explaining mask extraction processing (III). It is another figure explaining mask extraction process (III). It is a figure explaining mask extraction processing (IV).

  Hereinafter, embodiments will be described in detail with reference to the drawings.

  In the present embodiment, a description will be given by taking as an example an apparatus including a lens barrel 10, an imaging device 20, and a recording medium 40 as shown in FIG.

  The imaging device 20 captures an optical image incident from the lens barrel 10. The obtained image is stored in the storage medium 40 as a still image or a moving image.

  The lens barrel 10 includes a focus adjustment lens (hereinafter referred to as an “AF (Auto Focus) lens”) 11, a lens driving unit 12, an AF encoder 13, and a lens barrel control unit 14. The lens barrel 10 may be detachably connected to the imaging device 20 or may be integrated with the imaging device 20.

  The imaging device 20 includes an imaging unit 21, an image processing device 22, a display unit 23, a buffer memory unit 24, a storage unit 25, a CPU 26, an operation unit 27, and a communication unit 28. The imaging unit 21 includes an imaging element 29 and an A / D (Analog / Digital) conversion unit 30. The imaging unit 21 is controlled by the CPU 26 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).

  In the lens barrel 10, the AF lens 11 is driven by the lens driving unit 12 and guides an optical image to the light receiving surface (photoelectric conversion surface) of the imaging element 29 of the imaging device 20. The AF encoder 13 detects the movement of the AF lens 11 and outputs a signal corresponding to the movement amount of the AF lens 11 to the lens barrel control unit 14. Here, the signal corresponding to the movement amount of the AF lens 11 may be, for example, a sine wave signal whose phase changes according to the movement amount of the AF lens 11.

  The lens barrel control unit 14 controls the lens driving unit 12 in accordance with a drive control signal input from the CPU 26 of the imaging device 20. Here, the drive control signal is a control signal for driving the AF lens 11 in the optical axis direction. The lens barrel control unit 14 changes, for example, the number of steps of the pulse voltage output to the lens driving unit 12 according to the drive control signal. Further, the lens barrel control unit 14 outputs the position (focus position) of the AF lens 11 in the lens barrel 10 to the CPU 26 of the imaging device 20 based on a signal corresponding to the movement amount of the AF lens 11. Here, the lens barrel control unit 14 integrates, for example, signals according to the movement amount of the AF lens 11 according to the movement direction of the AF lens 11, thereby moving the AF lens 11 in the lens barrel 10 ( Position) may be calculated. The lens driving unit 12 drives the AF lens 11 according to the control of the lens barrel control unit 14 and moves the AF lens 11 in the optical axis direction within the lens barrel 10.

  In the imaging device 20, the imaging element 29 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 10 (optical system) into an electric signal, and supplies the electric signal to the A / D conversion unit 30. Output. The imaging element 29 is configured by a photoelectric conversion element such as a CMOS (Complementary Metal Oxide Semiconductor), for example. Further, the image sensor 29 may convert an optical image into an electric signal for a partial region of the photoelectric conversion surface (image cutout). Further, the image sensor 29 outputs an image obtained when a photographing instruction from the user is received via the operation unit 27 to the storage medium 40 via the A / D conversion unit 30 and the communication unit 28. On the other hand, the image pickup device 29 converts the continuously obtained image into a through image and converts it into a buffer memory unit 24 and the display unit 23 in the state before accepting a shooting instruction from the user via the operation unit 27. The data is output via the unit 30.

  The A / D conversion unit 30 digitizes the electrical signal converted by the image sensor 29 and outputs an image, which is a digital signal, to the buffer memory unit 24 and the like.

  The image processing device 22 performs image processing on the image temporarily stored in the buffer memory unit 24 based on the image processing conditions stored in the storage unit 25. The image after image processing is stored in the storage medium 40 via the communication unit 28. Further, the image processing device 22 performs mask extraction processing on the image temporarily stored in the buffer memory unit 24 (details will be described later). The extracted mask information is output to the CPU 26 and stored in the storage unit 25, the storage medium 40, and the like.

  The display unit 23 is, for example, a liquid crystal display, and displays an image generated by the imaging unit 21, an operation screen, and the like. The buffer memory unit 24 temporarily stores the image generated by the imaging unit 21. The storage unit 25 stores imaging conditions, determination conditions referred to by the CPU 26 in various determinations, and the like.

  The CPU 26 appropriately acquires necessary information from the image processing unit 22, the storage unit 25, and the like, and comprehensively controls each unit in the imaging device 20 based on the acquired information. Control by the CPU 26 includes focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, flash emission amount change setting, subject tracking setting, and various shooting mode settings. , Various image processing settings, various display settings, brightness optimization settings linked to zoom magnification, and the like. In addition, the CPU 26 monitors the operation state of the operation unit 27 and outputs image data to the display unit 23.

  The operation unit 27 includes, for example, a power switch, a shutter button, a multi-selector (cross key), or other operation keys. The operation unit 27 receives a user operation input when operated by the user, and outputs a signal corresponding to the operation input to the CPU 26. Output to.

  The communication unit 28 is connected to a removable storage medium 40 such as a card memory, and writes, reads, or deletes information (image data, area information, etc.) to the storage medium 40.

  The storage medium 40 is a storage unit that is detachably connected to the imaging device 20 and stores information (image data, area information, and the like). Note that the storage medium 40 may be integrated with the imaging device 20.

  The imaging device 20 includes an automatic detection mode for automatically detecting the main subject area in addition to the normal mode for detecting the main subject area based on the focus adjustment information at the time of shooting. The automatic detection mode is a mode for automatically and continuously detecting a main subject area based on a through image for composition confirmation. This automatic detection mode may be set by a user operation via the operation unit 27, or may be automatically set by the CPU.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowchart of FIG.

  In step S <b> 101, the CPU 26 controls the imaging unit 21 to start acquiring a through image. The acquired image information of the through image is temporarily stored in the buffer memory unit 24. This through image is continuously generated at a predetermined time interval. The acquisition of the through image by the CPU 26 is sequentially performed sequentially in time.

  In step S102, the CPU 26 controls the image processing device 22 to perform normal image processing. Normal image processing includes white balance adjustment, interpolation processing, color tone correction processing, gradation conversion processing, and the like. Since the specific method of each process is the same as that of a well-known technique, description is abbreviate | omitted. The image processing device 22 acquires the image data of the target image from the buffer memory unit 24, performs image processing, and then outputs the image data to the buffer memory unit 24 again.

  In step S103, the CPU 26 controls the image processing device 22 to calculate an evaluation value Da indicating the achromatic degree in the image. The evaluation value Da is an evaluation value indicating the distribution of color information in the image. The evaluation value Da is smaller as the target image is closer to an achromatic color, and is larger as the target image is farther from the achromatic color and colorful. The evaluation value Da may be obtained by any method as in the known technique.

  For example, the image processing apparatus 22 reads out the image that has been subjected to the normal image processing in step S102 from the buffer memory unit 24, performs resize processing and color space conversion processing as appropriate, and generates a YUV (for example, YCbCr) image. Then, the above-described evaluation value Da is calculated based on the deviation between the Cb image and the Cr image indicating the color difference. The image processing device 22 calculates a deviation Cbσ in a predetermined area at the center of the Cb image and a deviation Crσ in a predetermined area at the center of the Cr image, and sets it as an evaluation value Da. Note that the small deviation Cbσ and deviation Crσ means that there are few color variations, in other words, it is close to an achromatic color. Further, the predetermined area described above is a predetermined area. This area does not necessarily have to be in the center, and may be any size including the entire image. Furthermore, the size and position of the predetermined area described above may be variable according to the zoom magnification in the AF lens 11.

  In step S104, the CPU 26 determines whether or not the evaluation value Da <T1 calculated in step S103. If the CPU 26 determines that the evaluation value Da <T1, the process proceeds to step S110 described later. On the other hand, when determining that the evaluation value Da ≧ T1, the CPU 26 proceeds to step S105. T1 mentioned above is a predetermined threshold value. T1 includes a threshold value for the deviation Cbσ and a threshold value for the deviation Crσ. Note that the two threshold values may be the same value or different values. The CPU 26 determines that the evaluation value Da <T1 when both the deviation Cbσ and the deviation Crσ are less than the threshold value. The case where the evaluation value Da <T1 is a case where the deviation Cbσ and the deviation Crσ based on the target image are sufficiently small and the target image is very achromatic or achromatic. Specifically, a case where the image is close to a monochrome image or is a monochrome image, or an image such as a snow scene is conceivable.

  In step S105, the CPU 26 determines whether or not the evaluation value Da <T2 calculated in step S103. If the CPU 26 determines that the evaluation value Da <T2, the process proceeds to step S109 described later. On the other hand, when determining that the evaluation value Da ≧ T2, the CPU 26 proceeds to step S106. T2 described above is a predetermined threshold value, and T1 <T2. T2 includes a threshold value for the deviation Cbσ and a threshold value for the deviation Crσ, similarly to T1 described above. Note that the two threshold values may be the same value or different values. The CPU 26 determines that the evaluation value Da <T2 when both the deviation Cbσ and the deviation Crσ are less than the threshold value. The case where the evaluation value Da <T2 is a case where T1 ≦ the evaluation value Da <T2, and the case where the deviation Cbσ and the deviation Crσ based on the target image are relatively small and the target image is close to an achromatic color. . Specifically, an image such as a winter withering landscape can be considered.

  In step S106, the CPU 26 determines whether or not the evaluation value Da <T3 calculated in step S103. If the CPU 26 determines that the evaluation value Da <T3, the process proceeds to step S108 described later. On the other hand, when determining that the evaluation value Da ≧ T3, the CPU 26 proceeds to step S107. T3 described above is a predetermined threshold, and T1 and T2 <T3. T3 includes a threshold value for the deviation Cbσ and a threshold value for the deviation Crσ, similarly to the above-described T1 and T2. Note that the two threshold values may be the same value or different values. The CPU 26 determines that the evaluation value Da <T3 when both the deviation Cbσ and the deviation Crσ are less than the threshold value.

  The case where the evaluation value Da <T3 is a case where T1, T2 ≦ the evaluation value Da <T3, and the case where the deviation Cbσ and the deviation Crσ based on the target image are slightly small. Specifically, an image such as a general landscape can be considered. On the other hand, the case where the evaluation value Da ≧ T3 is a case where the deviation Cbσ and the deviation Crσ based on the target image are average values.

  The threshold values (T1 to T3) described in steps S104 to S106 may be variable according to the zoom magnification of the AF lens 11. For example, when the zoom magnification is high, the target image is an enlarged image. In this case, since the part with a color is also expanded, it is necessary to also increase the value of each threshold value (T1-T3). Moreover, in each determination in step S104 to step S106, although the example which determines whether it is less than a threshold value about both deviation Cb (sigma) and deviation Cr (sigma) was shown, it is determined whether only one of them is less than a threshold value It is good also as composition to do.

  In step S107, the CPU 26 controls the image processing device 22 to perform mask extraction processing (I). The mask extraction process is a technique for calculating a feature amount in an image and detecting a subject area based on the feature amount. For example, the feature extraction in the image is obtained, and the mask extraction is performed by obtaining a continuous region having the same feature quantity.

  The mask extraction process (I) will be described with reference to the schematic diagrams of FIGS. The image processing device 22 creates a mask image based on each of the Y image, Cb image, and Cr image of the YUV image described above.

  As shown in FIG. 3, the image processing device 22 creates a Y mask M [Y], a Cb mask M [Cb], and a Cr mask M [Cr] from each of the Y image, the Cb image, and the Cr image. For example, the image processing device 22 generates a binarized image in a predetermined range (for example, Yave + K · σ) with respect to the average pixel value Yave at the four pixels at the center of the Y image, for example. Then, a labeling process is performed on the generated Y-binarized image, and unnecessary masks are eliminated to create a Y mask M [Y]. Note that K described above is a predetermined coefficient (for example, K = 0.6), and σ is a deviation from Yave. Since the binarization process and the labeling process are the same as those in the known technique, the description thereof is omitted. The image processing apparatus 22 performs the same processing to create a Cb mask M [Cb] from the Cb image, and creates a Cr mask M [Cr] from the Cr image.

  Further, as shown in FIG. 3, the image processing apparatus 22 creates a center mask M [a] from the created Y mask M [Y], Cb mask M [Cb], and Cr mask M [Cr]. The center mask M [a] is a mask formed of an overlapping region where the Y mask M [Y], the Cb mask M [Cb], and the Cr mask M [Cr] overlap. Note that when the center mask M [a] is created, the overlapping region may be obtained by limiting the region to the central region, for example. Or you may limit to the mask containing a center. In the subsequent processing, only the finally created center mask M [a] is used.

  Further, as shown in FIG. 4, the image processing device 22 creates three types of luminance masks from the Y image (however, four or more types of masks may be created from the Y image). The image processing device 22 uses each pixel value of the Y image as an input value and each pixel value after correction (for example, gamma correction) as an output value. Then, binarized images of three sections are generated according to the output value. As shown in FIG. 4, a Y highlight binarized image is generated from the highlight portion of the Y image, and a Y intermediate or higher binarized image is generated from the middle portion or more of the Y image. A Y shadow binary image is generated from the portion. Then, the image processing apparatus 22 performs a labeling process on each of the generated three types of binarized images, and eliminates unnecessary masks, whereby a Y highlight mask M [Y1], a Y intermediate or higher mask M Each brightness mask of [Y2] and Y shadow mask M [Y3] is created.

  Further, as shown in FIG. 5, the image processing device 22 creates three types of color masks from each of the Cb image and the Cr image. The image processing device 22 generates a binarized image of three predetermined sections according to the output value, similarly to the example of FIG. 4 described above, for the Cb image. As shown in FIG. 5, from the Cb image, the Cb blue side binarized image centered on the blue side, the Cb intermediate binarized image centered on the intermediate component, and the Cb yellow side binary centered on the yellow side. A digitized image is generated. Similarly, from the Cr image, a Cr red side binarized image centered on the red side, a Cr intermediate binarized image centered on the intermediate component, and a Cr green side binarized image centered on the green side are generated. Is done. Then, the image processing device 22 performs a labeling process on each of the generated six types of binarized images, and eliminates unnecessary masks, whereby the Cb blue mask M [Cb1] and the Cb intermediate mask M [ Six color masks are created: Cb2], Cb yellow side mask M [Cb3], Cr red side mask M [Cr1], Cr intermediate mask M [Cr2], and Cr green side mask M [Cr3].

  Further, as shown in FIG. 6, the image processing apparatus 22 creates a pure color mask from the Cb image and the Cr image. The pure color mask is a mask obtained by extracting an “absolutely pure subject” having an arbitrary fixed area. Since the creation of the pure color mask is performed in the same manner as in the known technique, description thereof is omitted. As illustrated in FIG. 6, the image processing device 22 creates, for example, a pure red mask M [R] from the Cb image and the Cr image. Note that a pure color mask other than the pure color red mask M [R] may be further created.

  In addition, although the example which produces | generates a binarized image suitably was shown at the time of creation of each mask mentioned above, it is good also as a structure which produces directly the labeling mask with respect to a multi-value image, without producing | generating a binarized image.

  Through the processing described above, the image processing apparatus 22 has one type of mask (center mask M [a]) shown in FIG. 3, three types of luminance masks shown in FIG. 4, and six types shown in FIG. A total of 11 types of masks are prepared, one color mask shown in FIG. 6 and one type pure color mask shown in FIG.

  Next, the image processing apparatus 22 eliminates unnecessary masks from the created eleven types of masks and determines priorities for the remaining masks. The priority indicates a priority when determining a mask that is actually used to extract a subject area from eleven types of masks.

  In the mask extraction process (I) that is a normal mask extraction process for a general image, for example, as shown in FIG. 7, three levels of priority are determined in advance. The first priority is the pure red mask M [R], which is a pure color mask, and the second priority is the Cb blue side mask M [Cb1] and the Cb yellow side mask M [Cb3] of the color masks, Other masks have the third highest priority. Note that the priority shown in FIG. 7 is an example, and any number of stages may be used as long as there are a plurality of stages.

  Further, the assignment of each mask to each priority in FIG. 7 is an example. As shown in FIG. 7, the priority is determined so that the priority of the mask related to the color information is relatively high. Then, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority in accordance with these priorities, and uses the mask as a mask for extracting a subject area. Note that a method for selecting a mask for extracting a subject area may be any method such as a known technique. Finally, the image processing apparatus 22 extracts a subject area based on the selected mask, and proceeds to step S111 described later.

  In step S108, the CPU 26 controls the image processing device 22 to perform mask extraction processing (II). The mask extraction process (II) is a mask extraction process for an image such as a general landscape (an image having a slightly small deviation Cbσ and deviation Crσ based on the target image). Only differences from the mask extraction process (I) described in step S107 will be described below.

  The image processing apparatus 22 creates the 11 types of masks described in step S107. Then, the image processing device 22 eliminates unnecessary masks from the created eleven types of masks and determines the priority of the remaining masks. Here, unlike step S107, the image processing apparatus 22 determines the priority of the Y highlight mask M [Y1] among the luminance masks to be relatively high, and sets the priority of the various color masks to be relatively low. To do. For example, as shown in FIG. 8, the priority of the Y highlight mask M [Y1] is set to the first priority, and the other masks are set to the second priority. The priority is determined in this way because the target image is an image of a general landscape or the like (an image with a slightly small deviation Cbσ and deviation Crσ based on the target image). This is because, in comparison with the extraction process (I), in the mask extraction, the degree of dependence on the color-related feature amount is further reduced. Then, as in step S107, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority, and selects it as a mask for extracting a subject area. Finally, the image processing apparatus 22 extracts a subject area based on the selected mask, and proceeds to step S111 described later.

  In step S109, the CPU 26 controls the image processing device 22 to perform mask extraction processing (III). The mask extraction process (III) is a mask extraction process for an image (an image having a relatively small deviation Cbσ and deviation Crσ based on the target image) such as a wintering scenery where the target image is achromatic. Hereinafter, only differences from the processes described in steps S107 and S108 will be described.

  The image processing apparatus 22 creates 10 types of masks excluding one type of pure color mask among the 11 types of masks described in step S107. However, when creating the Cb mask M [Cb] and the Cr mask M [Cr], the conditions are different from those of the mask extraction process (I) and the mask extraction process (II).

  FIG. 9 is a diagram illustrating the creation of the Cb mask M [Cb] and the Cr mask M [Cr] in the mask extraction process (III). FIG. 9A illustrates a mask extraction process (I) for an image (an image having a relatively small deviation Cbσ and deviation Crσ based on the target image) such as an achromatic landscape close to an achromatic color that is the target of the mask extraction process (III). ) And mask extraction processing (II) will be described. If the deviation Cbσ and the deviation Crσ based on the target image are relatively small, there is little information about the color, and therefore binarization is performed when processing similar to the mask extraction processing (I) and mask extraction processing (II) is performed. The possibility of picking up noise at the stage becomes high. That is, by binarizing an originally narrow color width, a binarized image in which one subject area is separated may be generated. Therefore, in the mask extraction process (III), as shown in FIG. 9B, the binarization range in the Cb image is defined as Cbave (average pixel value in the central four pixels of the Cb image) ± K1 · σ (where K1> K ) To extract a wider appropriate subject area and reduce noise. Similarly, regarding the Cr image, the binarization range in the Cr image is set to Cave (average pixel value in the central four pixels of the Cr image) ± K1 · σ (where K1> K).

  Further, for the center mask M [a], instead of obtaining an overlapping region where the Y mask M [Y], Cb mask M [Cb] and Cr mask M [Cr] overlap, only the Y mask M [Y] is used. A center mask M [a] is created. That is, the Y mask M [Y] itself is set as the center mask M [a] without using (creating) the Cb mask M [Cb] and the Cr mask M [Cr]. At this time, since the center mask M [a] is created based only on the feature quantity related to luminance, the center mask M [a] limited to the luminance space instead of the color space is created. This is due to the fact that the target image is an image such as a wintering scenery close to an achromatic color (an image with a relatively small deviation Cbσ and deviation Crσ based on the target image), and mask extraction processing (I ) And mask extraction processing (II), in order to further reduce the dependency on the color-related feature amount in mask extraction.

  Then, the image processing apparatus 22 excludes unnecessary masks from the created 10 types of masks, and determines the priority of the remaining masks. For example, as shown in FIG. 10, the Y highlight mask M [Y1] of the luminance mask, the Cb blue side mask M [Cb1], the Cb yellow side mask M [Cb3], and the Cr red side mask M [[ The priority of Cr1] is the first priority, and the priority of the other masks is the second priority. Then, as in step S107, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority, and selects it as a mask for extracting a subject area. Finally, the image processing apparatus 22 extracts a subject area based on the selected mask, and proceeds to step S111 described later.

  In step S110, the CPU 26 controls the image processing device 22 to perform mask extraction processing (IV). The mask extraction process (IV) is a black-and-white image in which the target image is very achromatic or achromatic, an image close to a black-and-white image, an image such as a snow scene (deviation Cbσ and deviation Crσ based on the target image). Is a mask extraction process for a sufficiently small image). Hereinafter, only differences from the processes described in steps S107 to S109 will be described.

  The image processing apparatus 22 creates four types of masks excluding six types of color masks and one type of pure color mask among the 11 types of masks described in step S107. That is, a center mask M [a] and three types of luminance masks are created.

  For the center mask M [a], instead of obtaining an overlapping area where the Y mask M [Y], Cb mask M [Cb] and Cr mask M [Cr] overlap, the center mask M [Y] is used only by using the Y mask M [Y]. M [a] is created. That is, the Y mask M [Y] itself is set as the center mask M [a] without using (creating) the Cb mask M [Cb] and the Cr mask M [Cr]. At this time, since the center mask M [a] is created based only on the feature quantity related to luminance, the center mask M [a] limited to the luminance space instead of the color space is created. The center mask M [a] is created in this way for an image whose target image is very achromatic or achromatic (an image with a sufficiently small deviation Cbσ and deviation Crσ based on the target image). This is because the subject area is extracted without using a mask based on the color-related feature amount.

  Then, the image processing device 22 excludes unnecessary masks from the created four types of masks and determines priorities for the remaining masks. For example, as shown in FIG. 11, the priority of the Y highlight mask M [Y1] in the luminance mask is set to the first priority, and the priority of the other masks is set to the second priority. Then, as in step S107, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority, and selects it as a mask for extracting a subject area. Finally, the image processing device 22 extracts a subject area based on the selected mask, and proceeds to step S111.

  In each mask extraction process described in steps S107 to S110, the image processing apparatus 22 may extract only one subject area or a plurality of subject areas. In addition, in each mask extraction process described in steps S107 to S110, a mask selection method that is not used, a center mask M [a] creation method, and a priority change method are examples. In any case, the degree of dependence on the feature amount relating to the color may be adjusted according to the evaluation value Da indicating the achromatic degree of the target image.

  In step S111, the CPU 26 records the result of the mask extraction process in any one of steps S107 to S110 as subject information in the buffer memory unit 24, the storage unit 25, or the like. The subject information includes information such as the extraction conditions (color classification, mask classification, etc.) of the mask extraction process, the position of the mask, the size and shape of the mask, the position of the extracted subject area, and the size and shape of the subject area.

  In step S112, the CPU 26 determines whether or not a shooting instruction has been issued. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S113. On the other hand, if it is determined that the shooting instruction is not performed, the CPU 26 returns to step S102 and performs the processing after step S102 on the image of the next frame. The shooting instruction is given by a user operation via the shutter button of the operation unit 27.

  In step S113, the CPU 26 controls each unit to perform shooting. At this time, the CPU 26 performs shooting based on the subject information recorded in step S113. The CPU 26 performs 3A processing of focus adjustment (AF) setting processing, exposure adjustment (AE) setting processing, and white balance adjustment processing (AWB) based on the subject information recorded in step S113, and the image processing device 22. The conditions of various image processing in are determined.

  In the focus adjustment (AF) setting process, the CPU 26 shapes the extracted subject area and executes a contrast scan. Then, the CPU 26 causes the AF lens 11 to focus based on the result of the contrast scan (focus drive).

  In addition, when a shooting instruction has been issued in a state where mask extraction processing and subject information recording have not been completed, the CPU 26 performs normal 3A processing and performs shooting based on the result of this 3A processing. do it. Alternatively, reception of user operations relating to shooting instructions may be prohibited until mask extraction processing and recording of subject information are completed.

  In step S <b> 114, the CPU 26 records the image generated by the imaging on the storage medium 40 via the communication unit 28 and ends the series of processes.

  As described above, according to the present embodiment, the image data to be processed is acquired, the feature amount indicated by the image data is calculated, and the subject region determined based on the feature amount is detected. Furthermore, an evaluation value indicating the distribution of color information of the image indicated by the image data (an evaluation value indicating the degree of achromatic color) is calculated, and a subject area detection method is determined based on the calculated evaluation value. Therefore, a suitable subject area can be extracted according to the distribution of color information (color density, etc.) of the target image.

  In particular, according to the present embodiment, the mask corresponding to the subject is extracted by changing the calculation method of each mask and the priority of each mask according to the color information distribution (color density, etc.) of the target image. Therefore, the accuracy of subject area extraction can be improved.

  In addition, according to the present embodiment, when the target image is an achromatic image, the priority of the feature amount related to the luminance is set higher than that in the case where the image is not an achromatic image to detect the subject area. I do. Accordingly, it is possible to suppress the dependence on the color information in the detection of the subject area, and to extract a suitable subject area. In addition, when the target image is an achromatic image, the first region determined based on the feature amount related to luminance and the second region determined based on the second feature amount different from the feature amount related to luminance The same effect can be obtained by detecting the overlapping area based only on the first area when detecting the overlapping area where and overlap. In addition, when the target image is an achromatic image, the same priority is set by lowering the priority of the feature amount related to the color than when the image is not an achromatic image, and the object region is detected. An effect can be obtained. In addition, when the target image is an achromatic image, the first region determined based on the first feature amount and the second region determined based on the feature amount relating to a color different from the first feature amount. The same effect can be obtained by changing the detection method of the second region when detecting the overlapping region overlapping with the region.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In the above embodiment, an example in which only the masks to be used are created in each mask extraction process (steps S107 to S110 in FIG. 2) has been described. good.

  In the above embodiment, the evaluation value Da indicating the achromatic degree in the target image is used as the evaluation value indicating the distribution of color information in the target image. However, the present invention is not limited to this example. For example, an evaluation value indicating the degree of bias to an arbitrary hue in the target image (so-called color cast, the color of an arbitrary hue occupying the screen) may be used. In this case, the evaluation value corresponds to, for example, a distribution width (for example, deviation). When this evaluation value is a value that becomes smaller as the target image is biased to an arbitrary hue and becomes a larger value as the spread of the hue becomes larger, for example, when the forest is a subject, The evaluation value is small for an image with a significant degree of bias. The same applies to an image in which the degree of deviation from red to orange hue is significant, such as when the sunset is the subject. On the other hand, an evaluation value is high for an image having a low degree of bias to an arbitrary hue, such as a colorful image including subjects of various colors. And the effect similar to embodiment mentioned above can be acquired by changing the calculation method of each mask and the priority of each mask according to this evaluation value. For example, for an image with a significant bias toward a green hue, the classification is changed by changing the threshold when creating a color mask (see FIG. 5) or a pure color mask (see FIG. 6). It is better to change the relative resolution. For example, the resolution related to the green-based mask may be lowered, and conversely, the resolution related to the mask other than the green-based mask may be increased. In other words, for images that are significantly biased to any hue, you can either reduce the sensitivity to dominant colors (that is, reduce the resolution) or increase the sensitivity to non-dominant colors (that is, increase the resolution). It is better to change the resolution.

  Further, among the color mask (see FIG. 5) and the pure color mask (see FIG. 6), the priority of the green mask may be increased. Conversely, for a colorful image including subjects of various colors, the same process as the mask extraction process (I) of the above-described embodiment may be performed.

  Moreover, although the example which calculates the evaluation value which shows distribution of the color information in a target image based on a YUV (for example, YCbCr) image was shown in the said embodiment, this invention is not limited to this example. For example, an evaluation value indicating the distribution of color information in the target image may be calculated based on the Lab image. When calculating the evaluation value based on the Lab image, the dimension L meaning brightness is used instead of the Y component, and the complementary color dimensions a and b are used instead of the Cb component and Cr component. By performing the copied process, the same effect as in the present embodiment can be obtained.

  In the above-described embodiment, an example in which a series of processing is performed based on a through image for composition confirmation has been described, but the present invention is not limited to this example. For example, the present invention can be similarly applied to a case where a live view image for composition confirmation generated in a single-lens reflex camera or the like is targeted. Further, the present invention can be similarly applied to a case where a moving image recorded on the recording medium 40 or the like is targeted.

  In the above embodiment, an example in which a series of processing is performed for all frames has been described, but the present invention is not limited to this example. For example, a plurality of images generated discretely in time may be targeted. Specifically, a plurality of images subjected to frame thinning as appropriate may be targeted. By performing such processing, the processing load can be reduced.

  Further, the image processing described in the above-described embodiment may be realized by software by a “computer system” including a computer and an image processing program. In this case, a part or all of the processing of the flowchart described in the embodiment may be configured to be executed by the computer system. For example, part or all of the processing from step S101 to step S111 in FIG. 2 may be executed by a computer. By adopting such a configuration, it is possible to perform the same processing as in the above-described embodiment.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a www system is used. The computer-readable recording medium is a writable nonvolatile memory such as a flexible disk, a magneto-optical disk, a ROM, or a flash memory, a portable medium such as a CD-ROM, and a storage such as a hard disk built in the computer system. Refers to the device.

  Further, the computer-readable recording medium is a volatile memory (for example, DRAM (Dynamic Random Access) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory)) that holds a program for a certain period of time is also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 20 ... Imaging device, 21 ... Imaging part, 22 ... Image processing apparatus, 23 ... Display part, 26 ... CPU

Claims (16)

An area detection unit that calculates a feature amount of an image and detects a subject area determined based on the feature amount;
An evaluation value calculation unit for calculating an evaluation value showing the distribution of color information before outs image,
A control unit that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit ;
The region detection unit calculates a plurality of feature amounts as the feature amount, detects the subject region based on the plurality of feature amounts,
The image processing apparatus , wherein the control unit determines priorities of the plurality of feature amounts based on the evaluation values, and detects the subject region by the region detection unit based on the determined priorities . The image processing apparatus according to claim 1.
The region detection unit calculates a feature amount related to at least luminance as the feature amount,
The evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value,
The control unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold, and when the image is an achromatic image, the brightness An image processing apparatus in which the priority of the feature amount related to the image is set higher than that in the case where the image is not an achromatic image, and the subject region is detected by the region detection unit . The image processing apparatus according to claim 1 .
The region detection unit calculates at least a feature amount related to a color as the feature amount,
The evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value,
The control unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold, and when the image is an achromatic image, the color An image processing apparatus in which the priority of the feature amount related to the image is set lower than that in the case where the image is not an achromatic image, and the subject region is detected by the region detection unit . The image processing apparatus according to claim 1 .
The region detection unit calculates at least a feature amount related to a color as the feature amount,
The evaluation value calculation unit calculates an evaluation value indicating a degree of bias to an arbitrary hue in the image as the evaluation value,
The control unit determines whether or not the image is biased to the arbitrary hue by comparing the evaluation value with a predetermined threshold, and the image is biased to the arbitrary hue. The priority of the feature quantity related to the arbitrary hue among the feature quantities related to the color is set higher than when the image is not an image biased to the arbitrary hue, An image processing apparatus for detecting the subject area by a detection unit. The image processing apparatus according to claim 1 .
The region detection unit calculates a plurality of feature amounts as the feature amount, and among the plurality of feature amounts, a first region determined based on a first feature amount and the first feature amount are: An overlapping area overlapping with a second area determined based on a different second feature amount is detected as the subject area;
The control unit is an image processing apparatus that determines a detection method of at least one of the first region and the second region based on the evaluation value . The image processing apparatus according to claim 5 .
The area detection unit calculates a feature quantity related to luminance as the first feature quantity,
The evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value,
The control unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold, and when the image is an achromatic image, An image processing apparatus that detects the overlapping area based on only one area . The image processing apparatus according to claim 5 .
The region detection unit calculates a feature amount related to a color as the second feature amount,
The evaluation value calculation unit calculates an evaluation value indicating an achromatic degree in the image as the evaluation value,
The control unit determines whether or not the image is an achromatic image by comparing the evaluation value with a predetermined threshold, and when the image is an achromatic image, An image processing apparatus that changes the detection method of the area of 2 . The image processing apparatus according to claim 5 .
The region detection unit calculates a feature amount related to a color as the second feature amount,
The evaluation value calculation unit calculates an evaluation value indicating a degree of bias to an arbitrary hue in the image as the evaluation value,
The control unit determines whether or not the image is biased to the arbitrary hue by comparing the evaluation value with a predetermined threshold, and the image is biased to the arbitrary hue. If the image is not an image biased to the arbitrary hue, the weight of the second region based on the characteristic amount related to the arbitrary hue among the second characteristic amounts is set. An image processing apparatus configured to detect the subject region by setting the region detection unit to be high . An imaging unit for imaging a subject;
  A region detection unit that calculates a feature amount of an image obtained by the imaging unit and detects a subject region determined based on the feature amount;
  An evaluation value calculation unit for calculating an evaluation value indicating a distribution of color information of the image;
  A control unit that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit;
  The region detection unit calculates a plurality of feature amounts as the feature amount, detects the subject region based on the plurality of feature amounts,
  The image pickup apparatus, wherein the control unit determines priorities of the plurality of feature amounts based on the evaluation values, and detects the subject region by the region detection unit based on the determined priorities.   On the computer,
  A region detection procedure for calculating a feature amount of an image and detecting a subject region determined based on the feature amount;
  An evaluation value calculation procedure for calculating an evaluation value indicating a distribution of color information of the image;
  A control procedure for determining a method of detecting the subject region by the region detection procedure based on the evaluation value calculated in the evaluation value calculation procedure;
  As well as
  The region detection procedure includes a procedure of calculating a plurality of feature amounts as the feature amount, and detecting the subject region based on the plurality of feature amounts,
  The control procedure includes an image processing program including a procedure for determining a priority in the plurality of feature amounts based on the evaluation value, and detecting the subject region by the region detection procedure based on the determined priority. .   An evaluation value calculation unit for calculating an evaluation value of an image based on color information;
  An area detection unit for detecting a subject area included in the image,
  The region detection unit uses the evaluation value calculated by the evaluation value calculation unit to detect a subject region included in the image based on luminance information rather than color information, and to use color information rather than luminance information. An image processing apparatus that selects one of a second detection mode for detecting a subject area included in the image based on the second detection mode.   The image processing apparatus according to claim 11.
  The region detection unit selects one of the first detection mode and the second detection mode by comparing the evaluation value calculated by the evaluation value calculation unit with a predetermined threshold value. Image processing device.   The image processing apparatus according to claim 11 or 12,
  The evaluation value calculation unit calculates the evaluation value that becomes a smaller value as the image becomes achromatic,
  The area detection unit is an image processing apparatus that selects the first detection mode when the evaluation value calculated by the evaluation value calculation unit is smaller than a predetermined threshold.   An imaging unit for imaging a subject via a lens;
  An evaluation value calculation unit that calculates an evaluation value of an image of a subject imaged by the imaging unit based on color information;
  An area detection unit for detecting a subject area included in the image;
  A drive unit that drives the lens by the subject area detected by the area detection unit,
  The region detection unit uses the evaluation value calculated by the evaluation value calculation unit to detect a subject region included in the image based on luminance information rather than color information, and to use color information rather than luminance information. An imaging apparatus that selects one of a second detection mode for detecting a subject area included in the image based on the second detection mode.   The imaging device according to claim 14, wherein
  The region detection unit selects one of the first detection mode and the second detection mode by comparing the evaluation value calculated by the evaluation value calculation unit with a predetermined threshold value. Image processing device.   In the imaging device according to claim 14 or 15,
  The evaluation value calculation unit calculates the evaluation value that becomes a smaller value as the image becomes achromatic,
  The area detection unit is an imaging apparatus that selects the first detection mode when the evaluation value calculated by the evaluation value calculation unit is smaller than a predetermined threshold.

Images