JP2008147714A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To set suitable correction parameters of an image by using information representing a focus point of an image. <P>SOLUTION: The information representing the focus point of the image is acquired (S201), the image is dynamically divided into areas according to a subject (S202), and an area corresponding to the information representing the focus point is extracted from the divided areas (S204). Statistic of values of pixels included in the extracted area are calculated (S205) and compared with a predetermined threshold to judge whether the image needs to be corrected (S206). When it is judged that the image needs to be corrected, correction parameters for correcting the image are set based upon the statistic (S207). <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to image processing for correcting an image such as a digital photographic image.

  There are various image processes for adjusting the saturation, color tone, contrast, gradation, etc. of a digital image. The operator has specialized knowledge regarding images, and performs these image processing using specialized or dedicated software and empirical knowledge. The final image is obtained by repeating trial and error while confirming the image processing result on the computer monitor.

  In recent years, digital cameras, personal computers, the Internet, household printers, and the like have become widespread, and the opportunity for individuals to handle digital photographic images is increasing against the backdrop of increasing the capacity of storage media such as hard disks. On the other hand, an image taken with a digital camera is not necessarily an image taken under suitable conditions. Of course, even if an image was taken under unfavorable conditions, it cannot be discarded if it is an important image for the photographer. Therefore, image correction for obtaining an appropriate image from an image photographed under unfavorable conditions is desired.

  The color correction method disclosed in Japanese Patent Laid-Open No. 11-17969 uses a dialog with a user through a graphical user interface. If this method is used, the representative color and the color correction parameter in the correction target area can be manually set, and flexible and satisfactory color correction can be performed. However, this method is not preferable when processing a large number of photographic images in a short time, which requires processing time and user effort. In addition, the correction results vary depending on the skill level of the user.

  In order to solve the variation of the correction result due to the difference in the skill level of the user and the problem of the processing time, automatic correction that does not require the user's operation is required.

  As a method for performing automatic correction, the invention of Japanese Patent Application Laid-Open No. 2004-260835 discloses lightness correction means. An image is divided into a plurality of fixed areas, and a plurality of areas located at the center of the image are used as main parts, and a correction reference area is selected from these areas and used for correction. Further, the digital camera performs automatic image correction using a fixed area inside. In this case, the reference fixed area for correction is selected and used for correction by combining the fixed area and the focus position (focus position). As described above, when automatic correction is performed, an image is divided into a plurality of fixed areas, and a correction reference area is selected from these areas to determine a correction amount.

  However, as in the invention of Japanese Patent Application Laid-Open No. 2004-260835, when an image is divided into fixed areas and the main subject is determined only by position information such as the central portion, there may be two or more subjects with similar sizes in the central portion. For example, the correction amount is determined based on these subjects. In addition, the correction amount is determined by selecting one of the subjects based on positional information such as closer to the center. Such a subject selection method may result in a correction result not intended by the user.

  For example, in the original image as shown in FIG. 3A, it is assumed that the user has focused on the flower 301 that appears dark at the time of shooting. When this original image is divided into 8 × 8 fixed regions as shown in FIG. 3C, the brightness histogram of nine regions 313 including white flowers 302 located in the center of the image is referred to. As a result, the area 313 is determined to have appropriate brightness and is not corrected, or the area 313 is determined to be too bright and correction is performed to reduce the brightness. In other words, the flower 301 focused by the user (who has the greatest interest) remains dark or is corrected darker.

  In addition, there is a method of determining a correction amount from information indicating division and focus positions by a plurality of fixed areas and using statistics such as a color distribution of a focused area and a histogram of brightness. In this method, if the subject exists across the boundary of the fixed area, the statistics of the subject itself may not be obtained accurately. For example, as shown in FIG. 3C, it is assumed that an image is divided into 8 × 8 fixed areas, and the in-focus areas are determined as six areas 314. In this case, the area ratio of the bright background portion and the dark subject portion of the region 314 is approximately one to one. Therefore, when the average value of the luminance of the area 314 is obtained, it becomes a neutral value, and as a result, the subject 301 in focus remains dark.

  In other words, the method of dividing an image into a plurality of fixed areas can easily and easily calculate the color distribution and brightness statistics of main subjects, which are important when determining the correction amount, even if position information and focus information are combined. It cannot be accurately grasped.

Japanese Patent Laid-Open No. 11-17969 Japanese Patent Laid-Open No. 2004-260835

  An object of the present invention is to set an appropriate correction parameter for an image using information indicating the focus position of the image.

  The present invention has the following configuration as one means for achieving the above object.

  The image processing according to the present invention acquires information indicating the focus position of the image, dynamically divides the image according to the subject of the image, and converts the area into information indicating the focus position. The corresponding region is extracted, the statistic of the value of the pixel included in the extracted region is calculated, and the calculated statistic is compared with a predetermined threshold value to determine whether the image needs to be corrected. When it is determined that the correction is necessary, a correction parameter for correcting the image is set based on the statistic.

  According to the present invention, it is possible to set an appropriate correction parameter for an image using information indicating the focus position of the image.

  Hereinafter, image processing according to an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus that performs image processing according to the first embodiment.

  The input unit 101 is a keyboard or a pointing device (mouse, trackball, trackpad or tablet) for inputting instructions and data from the user. When the present embodiment is applied to an input device such as a digital camera, buttons, mode dials, switches, and the like correspond to the input unit 101.

  The data storage unit 102 stores image data, and is usually a hard disk, an optical disk, a memory card, or the like. Of course, the data storage unit 102 can also store programs and other data.

  The communication unit 107 is a serial bus interface (I / F) such as USB or IEEE1394, a wired network I / F, or an infrared or wireless wireless network I / F. The image processing apparatus can communicate with an image input / output device, a computer apparatus or a server on a network (not shown) via the communication unit 107.

  The display unit 103 displays an image before or after image processing, or displays a graphic user interface. For the display unit 103, a display device such as a CRT or an LCD is generally used. Of course, the display unit 103 may be a monitor connected to the image processing apparatus via a cable.

  The CPU 104 uses the RAM 106 as a work memory, executes various programs stored in the ROM 105 or the data storage unit 102, controls various components described above, and performs various processes including image processing for setting correction parameters described later. Perform image processing.

  Further, the image processing apparatus may further include an image input unit (not shown) having a lens and a CCD, and the image captured by the image input unit may be stored in the data storage unit 102.

  In the following description, as an example, an original image is expressed in an RGB color space, each channel has an 8-bit integer value, a value from 0 to 255, and the image size is 1600 × 1200 pixels. Also, when indicating the pixel position in the image, the upper left corner of the image is the origin, the right direction (horizontal direction) of the image is the positive direction of the x coordinate, and the lower direction (vertical direction) is the positive direction of the y coordinate.

[Image processing]
Next, image processing executed by the image processing apparatus to correct the brightness of the entire image using information indicating the focus position (hereinafter referred to as focus position information) will be described.

  FIG. 2 is a flowchart for explaining a method for correcting the brightness of an image in accordance with a main subject in focus, which is a process executed by the CPU 104.

  The CPU 104 reads out the image data to be corrected and the additional information from the data storage unit 102 and stores it in a predetermined area of the RAM 106 (S201). Generally, information called Exif (exchangeable image file format) information is added to an image photographed with a digital camera. Therefore, when image data is acquired, the additional information can also be acquired. However, when additional information is stored separately from the image data, the image data and additional information related to the image data are acquired separately. Further, there are various methods for acquiring additional information, such as receiving directly from an application program.

  Next, the CPU 104 divides the image represented by the image data stored in the predetermined area of the RAM 106 into areas corresponding to the shape of the subject by a dynamic area dividing method using a Markov random field (MRF) model ( S202).

  FIG. 3B is a diagram showing an example in which the original image shown in FIG. 3A is divided into regions by the MRF model, and shows how the original image is divided into nine regions. Since the region dividing method using the MRF model is a well-known technique, its detailed description is omitted.

  FIG. 5 is a diagram showing an example of area division data obtained as a result of area division using the MRF model. The number labeled on each pixel indicates the area to which the pixel belongs. The area division data is map data of 1600 × 1200 pixels that is the same as the image size, and is stored in a predetermined area of the RAM 106.

  Next, the CPU 104 extracts focus position information from the additional information (S203). The focus position information can be extracted from the subject area information (Subject Area tag) of the Exif information. The SubjectArea tag indicating the subject area information is defined in “Exchangeable image file format for digital still cameras: Exif version 2.2” JEITACP-3451. Also, since a method for reading out Exif tag data is known, a detailed description thereof is omitted.

  FIG. 4A is a diagram for explaining focus position information. In the following description, it is assumed that rectangular area information indicating the main subject area is obtained from the SubjectArea tag. The rectangular area information includes the center coordinates of the focus position 406, for example, (1200, 800), and the size of the pinch position 406, for example, (200, 150).

  Next, the CPU 104 extracts an in-focus area F from the extracted focus position information and the area division result (S204). A divided area including the center coordinates of the focus position may be extracted as the area F. In other words, the number labeled on the pixel corresponding to the center coordinate of the focus position may be obtained from the area division data shown in FIG.

  FIG. 4B is a diagram showing a state where the focus positions 401 to 406 shown in FIG. 4A are superimposed on the original image shown in FIG. 3A, and shows the relationship between the subject and the focus position. FIG. 4C is a diagram illustrating a state in which the region-divided image illustrated in FIG. 3B and the focus positions 401 to 406 illustrated in FIG. 4A are overlapped. In Example 1, as shown in FIG. 4C, the number indicating the region 305 corresponding to the focus position 406 is acquired, and the region 305 is extracted as the region F in focus.

Next, the CPU 104 obtains an average value of the brightness of all the pixels included in the area F and sets the brightness B of the area F (S205). Note that the RGB value of the pixel may be converted into a YCbCr value, and the converted luminance value Y may be used as the brightness. Conversion from RGB to YCC is performed using equation (1). Instead of the luminance value Y in the YCC space, the brightness value L in the L * a * b * space may be used.
┌ ┐ ┌ ┐┌ ┐
│Y │ │ 0.2290 0.5870 0.1140││R│
│Cb│ = │-0.1687 -0.3313 0.5000││G│… (1)
│Cr│ │ 0.5000 -0.4187 -0.0813││B│
└ ┘ └ ┘└ ┘

Accordingly, the luminance value Yij of the pixel P (i, j) = (rij, gij, bij) at the coordinates (i, j) can be calculated by the equation (2).
Yij = 0.2990 × rij + 0.5870 × gij + 0.1440 × bij… (2)

Alternatively, Expression (3) may be used as an approximate expression of Expression (2).
Yij = 0.3 × rij + 0.6 × gij + 0.1 × bij… (3)

The CPU 104 reads the region number of the pixel P (i, j) from the region division data, and determines whether or not the pixel P (i, j) is included in the region F depending on whether or not it matches the number of the region F. judge. Then, when the brightness of the pixels included in the area F is calculated, the brightness B of the area F, which is an average value of the brightness, is calculated by the equation (4).
B = ΣYij / N (number of pixels in region F) (4)
Here, the range of Σ operation is region F (i, j ∈ region F)
N is the number of pixels in area F

  Next, the CPU 104 compares the brightness B of the region F with the threshold value Th (S206). When B <Th, it is determined that the brightness of the area F is insufficient, and when B ≧ Th, it is determined that the brightness of the area F is sufficient.

  FIG. 6 is a diagram showing an example of the relationship between brightness before and after correction, and is an example in which the threshold value Th = 128. That is, if the brightness B of the region F is 128 or more, the brightness is not corrected. When the brightness B of the area F is less than 128, the brightness is corrected brightly. For example, if the brightness B of the area F is 78, it is less than the threshold value Th, so that the brightness B is corrected to be 90.5. It should be noted that an experience value may be set as the default value for the threshold value Th so that the user can adjust the threshold value Th as necessary.

When determining that the brightness of the area F is insufficient, the CPU 104 sets a correction amount for correcting the brightness B of the area F as shown in FIG. 6 (S207). In the first embodiment, an example will be described in which brightness is corrected by gamma correction and a gamma value is calculated as a correction amount. The gamma correction is a process for applying a conversion according to Expression (5) to each pixel.
ν _out = ν _in (1 / γ)… (5)
Where ν _in is the original pixel value normalized to a range of 0 to 1.0,
ν _out is the pixel value after gamma correction normalized to the range of 0 to 1.0,
γ is a correction parameter (gamma value)

If γ <1.0 in equation (5), the pixel value becomes smaller as ν _out ≦ ν _in and is corrected darkly by gamma correction. On the contrary, when γ> 1.0, the pixel value becomes large as ν _out ≧ ν _in and is brightly corrected by gamma correction.

Therefore, the correction parameter γ is calculated from the brightness B of the region F and the corrected brightness B ′ by Expression (6).
γ = log B / log B '… (6)

For example, the correction parameter γ when the brightness B = 78 in the region F is changed to the brightness B ′ = 90.5 is as follows.
γ = log (78/255) / log (90.5 / 255) ≒ 1.143

  Next, the CPU 104 corrects the brightness of the entire image data with the correction parameter γ (S208). FIG. 7 is a diagram showing a correction gamma curve 702 in the case of γ≈1.143, which is corrected brighter than the characteristic 701 of the original image.

  In the first embodiment, an example of the relationship between the brightness before and after the correction is shown in FIG. 6, but it is preferable that this relationship (characteristic) is determined by an experiment or the like.

  In this way, it is possible to easily correct the subject focused by the photographer to the appropriate brightness by determining the correction amount for correcting the brightness based on the result of dividing the image into the shape of the subject and the focus information. Can do. Also, unlike fixed division, the amount of correction is determined only by the pixels included in the subject, so if there is only information indicating the subject in focus, the main subject will be clipped even if correction processing is applied to the entire image. Even when correction processing is performed, correction is performed with the same correction amount. As a result, variations in correction results are suppressed, and correction quality is stabilized.

  The image processing according to the second embodiment of the present invention will be described below. Note that the same reference numerals in the second embodiment denote the same parts as in the first embodiment, and a detailed description thereof will be omitted.

  In the first embodiment, the focus position information is extracted from the additional information. However, there is a case where the additional information is described in a format that does not have the focus position information or cannot be read. In the second embodiment, image processing for correcting the brightness of the entire image in consideration of the presence / absence of focus position information will be described.

  FIG. 8 is a flowchart for explaining a method for correcting the brightness of an image according to the second embodiment. The same processes as those shown in FIG. 2 are denoted by the same reference numerals, and detailed description thereof is omitted.

  The CPU 104 determines whether or not focus position information can be extracted from the additional information (S801). If extraction is possible, the process proceeds to step S203, and the same processing as in the first embodiment is performed.

  When the focus position information cannot be extracted from the additional information, the CPU 104 calculates the edge strength of each divided area (S802). Since the calculation of the edge strength can be performed using a known edge detection filter such as a Prewitt, Sobel, or Laplacian filter, a detailed description is omitted.

  Next, the CPU 104 extracts a region with the strongest edge strength as a focused region F (S803). Then, the process proceeds to step S205, and the same process as in the first embodiment is performed.

  As described above, even when focus position information cannot be extracted, a main subject region can be predicted from the edge strength of the divided region, and correction suitable for the region can be performed.

  Hereinafter, image processing according to the third embodiment of the present invention will be described. Note that the same reference numerals in the third embodiment denote the same parts as in the first and second embodiments, and a detailed description thereof will be omitted.

  In the first and second embodiments, the method of correcting the brightness from the region division result and the focus position information has been described. In the third embodiment, a method for obtaining white balance from the region division result and the focus position information will be described.

  FIG. 9 is a flowchart for explaining a method for obtaining white balance according to the third embodiment. The same processes as those shown in FIG. 2 are denoted by the same reference numerals, and detailed description thereof is omitted.

  When the area F is extracted, the CPU 104 calculates an average color (Ravr, Gavr, Bavr) and an average value Savr of saturation in the area F (S901). For the average color (Ravr, Gavr, Bavr), the total value of the RGB values (rij, gij, bij) of all the pixels P (i, j) included in the region F may be divided by the number N of pixels in the region F. The average value Savr of saturation is the average value of S values in the HSV space. Since the conversion from the RGB space to the HSV space is a well-known technique, a detailed description is omitted, but the S value of the HSV space is calculated from the average color (Ravr, Gavr, Bavr) to be Savr. In the following description, it is assumed that the average color (Ravr, Gavr, Bavr) = (158, 158, 150) and the saturation average value Savr = 5 are calculated.

  Next, the CPU 104 determines whether or not the average value Savr of saturation is the saturation of the white region (S902). And when it determines with it not being the saturation of a white area | region, a process is complete | finished. Various methods can be considered for the determination of the white region. For example, if Savr ≦ 15, the saturation of the white region may be determined.

If it is determined that the saturation is in the white area, the CPU 104 corrects the white balance of the entire image based on the average color (Ravr, Gavr, Bavr) in the area F (S903), and ends the process. There are various methods for correcting white balance. For example, the RGB values (rij, gij, bij) of each pixel P (i, j) of the image are used by using the average colors (Ravr, Gavr, Bavr) of the region F. The correction may be made using equation (7).
R '(i, j) = rij x Gavr / Ravr
G '(i, j) = gij (7)
B '(i, j) = bij × Gavr / Bavr

  Note that the white region determination method is not limited to the method of determining based on the S value. For example, the saturation information may be determined by an expression difference vector by CbCr in the YCC space, or may be determined based on RGB values obtained by actually photographing a scene that is likely to cause color fogging.

  Similarly to the second embodiment, when focus position information cannot be extracted, the edge strength of each region may be calculated, the region having the strongest edge strength may be regarded as the region F, and the white balance may be corrected. .

  If the white balance is adjusted by the above method, the white balance can be adjusted around the subject that the photographer has paid attention to even if the background or the area of a white object other than the main subject is large. Similarly to the brightness correction, unlike the fixed division, the average value of color and saturation can be calculated without including an area other than a white subject. Therefore, if the image obtained by cutting out the main subject is stored together with the focus position information, the same correction amount as that of the image before cutting out can be obtained, and the correction quality is stabilized.

[Modification]
In the above embodiment, the example in which the original image is divided into regions has been described. However, the region may be divided after the original image is reduced. Compared with the case where the original image is divided into regions, the region division accuracy at the boundary portion of the subject is slightly lowered, and the processing speed can be increased.

  In the above embodiment, the region dividing method using the MRF model has been described. However, any method can be used as long as the region can be divided for each subject.

  In the above-described embodiment, it has been described that the area division data (map) shown in FIG. 5 is created as a result of the area division. However, the present invention is not limited to this information. For example, after the map is created, information such as average luminance and average color of each area, the number of pixels, center coordinates, barycentric coordinates, saturation, and brightness may be output and stored together. By having additional information such as average luminance, the information can be used for subsequent brightness correction and white balance correction.

  Further, in the above-described embodiment, the example in which the subject area information (SubjectArea) in the Exif format is used when acquiring the focus position has been described. However, the present invention is not limited to this method. For example, depending on the model of the digital camera, the shooting information unique to each company may be described in a Makernotes tag that can be used independently by each company in the Exif format, and the focus information in the shooting information may be used.

  Further, the correction of brightness and the correction of white balance may be performed simultaneously by combining the third embodiment and the first or second embodiment.

[Other embodiments]
Note that the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, and a printer), and a device (for example, a copying machine and a facsimile device) including a single device. You may apply to.

  Another object of the present invention is to supply a storage medium (recording medium) that records software for realizing the functions of the above-described embodiments to a system or apparatus, and a computer (CPU or MPU) of the system or apparatus executes the software. Is also achieved. In this case, the software itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the software constitutes the present invention.

  In addition, the above functions are not only realized by the execution of the software, but an operating system (OS) running on a computer performs part or all of the actual processing according to instructions of the software, and thereby the above functions This includes the case where is realized.

  In addition, the software is written in a function expansion card or unit memory connected to the computer, and the CPU of the card or unit performs part or all of the actual processing according to instructions of the software, thereby This includes the case where is realized.

  When the present invention is applied to the storage medium, the storage medium stores software corresponding to the flowchart described above.

FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus that executes image processing according to the first embodiment; A flowchart illustrating a method of correcting the brightness of an image according to a main subject in focus; A diagram showing an example of an original image, A diagram showing an example of dividing an original image by an MRF model, A diagram for explaining the problem when an image is divided into fixed areas. A diagram explaining the focus position information, The figure which shows the state where the focus position is superimposed on the original image, The figure which shows the state where the area division picture and the focus position are superimposed The figure which shows an example of the area | region division data obtained as a result of the area division | segmentation using an MRF model, The figure which shows the example of the relationship of the brightness before and after correction, The figure which shows the gamma curve for correction, Flowchart for explaining a method for correcting the brightness of the image of Example 2, 12 is a flowchart for explaining a method for obtaining white balance according to a third embodiment.

Claims (9)

Get information that shows the focus position of the image,
Dynamically segmenting the image according to the subject of the image,
An area corresponding to the information indicating the focus position is extracted from the divided areas.
Calculating the statistic of the value of the pixels contained in the extracted region;
Compare the calculated statistic with a predetermined threshold value to determine whether the image needs to be corrected,
An image processing method comprising: setting a correction parameter for correcting the image based on the statistic when it is determined that the correction is necessary.   2. The image processing method according to claim 1, wherein the statistic is an average value of brightness of all pixels included in the extracted region.   3. The image processing method according to claim 2, wherein the correction parameter is a parameter for correcting the brightness of the image.   2. The image processing method according to claim 1, wherein the statistic is an average value of saturations of all pixels included in the extracted region.   5. The image processing method according to claim 4, wherein the correction parameter is a parameter for correcting white balance of the image.   Further, when information indicating the focus position cannot be acquired, a region having the strongest edge strength is extracted from the region divided, and the information indicating the extracted region is used as information indicating the focus position. The image processing method according to claim 1. Acquisition means for acquiring information indicating a focus position of the image;
Dividing means for dynamically dividing the image according to the subject of the image;
Extraction means for extracting an area corresponding to the information indicating the focus position from the divided areas;
Calculating means for calculating a statistic of a value of a pixel included in the extracted region;
A determination unit that compares the calculated statistic with a predetermined threshold to determine whether the image needs to be corrected;
An image processing apparatus, wherein when it is determined that the correction is necessary, a correction parameter for correcting the image is set based on the statistic.   7. A computer program for controlling the image processing apparatus to realize the image processing according to any one of claims 1 to 6.   9. A computer-readable storage medium in which the computer program according to claim 8 is recorded.

Images