JP5733588B2 - Image processing apparatus and method, and program - Google Patents

Description

The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program that enable composition confirmation to be more easily confirmed.

  Some imaging devices such as a digital camera include two imaging systems.

  For example, an imaging device including a front camera that images the front of the user and a rear camera that images the rear has been proposed (see, for example, Patent Document 1).

  However, in the above-described imaging apparatus, the user cannot image one subject at the same time with two imaging systems.

  In a general digital camera, the standard mode and the wide-angle mode can be switched in one imaging system, but the user cannot check the composition in each mode at the same time.

  Furthermore, a camera having two display units for displaying images picked up by the respective image pickup systems with respect to the two image pickup systems has been proposed. However, since there are two display units, the user can You must check the composition of the image.

JP 2007-60721 A

  As described above, it is not easy to easily confirm the composition of two images having different imaging ranges.

  The present invention has been made in view of such a situation, and makes it possible to confirm the composition more easily.

The image processing apparatus according to an aspect of the present invention provides the brightness of the second image so that the image quality of the second image having a different angle of view from the first image is different from the image quality of the first image. A first image quality adjusting means for adjusting at least one of color density, hue, and sharpness; an object detecting means for detecting a moving object in the second image; and the first image quality. The position of the subject of the first image and the second image is matched with the second image in which at least one of brightness, color density, hue, and sharpness has been adjusted by the adjusting means. And a combining unit that combines the first image and the object image of the region of the object in the second image detected by the object detecting unit .

The image processing apparatus may further include display means for displaying the synthesized image synthesized by the synthesizing means.
Wherein the image processing apparatus, the image quality of the object in the region before Symbol in the object detecting means is said second detectable by imaging, so that different from the image quality of the second image, the second in the image A second image quality adjusting unit that adjusts at least one of brightness, color density, hue, and sharpness of the area of the object of the object, and the synthesizing unit includes the second image, The second image in which at least one of brightness, color density, hue, and sharpness has been adjusted by the first image and the second image quality adjustment means so that the positions of the subjects coincide with each other. An object image of the region of the object in the image can be synthesized.

The image processing apparatus includes a composition analysis unit that analyzes the composition of the second image, and a composition having an angle of view different from the angle of view of the first image based on the composition analyzed by the composition analysis unit. And a composition extraction means for extracting the image from the second image, and the composition means extracts the first image and the composition extraction so that the position of the subject coincides with the second image. The extracted images of the composition extracted by the means can be synthesized.

The first image quality adjusting means can reduce the brightness, saturation, or brightness of the second image so that the image quality of the second image is lower than the image quality of the first image .

The image processing apparatus has at least one of brightness, color density, hue, and sharpness of the first image so that the image quality of the first image is different from the image quality of the second image. Second image quality adjusting means for adjusting one of them can be further provided.

The second image quality adjusting means can increase the brightness, saturation, or brightness of the first image so that the image quality of the first image is higher than the image quality of the second image .

In the image processing method according to one aspect of the present invention, the brightness of the second image is set so that the image quality of the second image having a different angle of view from the first image is different from the image quality of the first image. An image quality adjustment step for adjusting at least one of color density, hue, and sharpness; an object detection step for detecting a moving object in the second image; and brightness in the image quality adjustment step. The first image and the second image have a subject position that matches the second image in which at least one of color density, hue, and sharpness is adjusted. A synthesis step of synthesizing an image and an object image of the region of the object in the second image detected in the object detection step .

According to another aspect of the present invention, there is provided a program for controlling brightness and color of the second image so that the image quality of the second image having a different angle of view from the first image is different from the image quality of the first image. An image quality adjustment step for adjusting at least one of density, hue, and sharpness; an object detection step for detecting a moving object in the second image ; The first image , so that the position of the subject of the first image and the second image coincides with the second image in which at least one of density, hue, and sharpness is adjusted ; And a synthesis step of synthesizing the object image of the region of the object in the second image detected in the object detection step .

In one aspect of the present invention, the brightness of the second image, the intensity of the color, and the image quality of the second image having a different angle of view from the first image are different from the image quality of the first image. A second object in which at least one of hue and sharpness is adjusted , a moving object in the second image is detected , and at least one of brightness, color strength, hue, and sharpness is adjusted The first image and the object image of the region of the object in the detected second image are combined so that the positions of the subjects of the first image and the second image coincide with the image of.

  According to one aspect of the present invention, the composition can be confirmed more easily.

It is a figure which shows the example of the external appearance of the digital camera as one Embodiment of the imaging device to which this invention is applied. It is a figure which shows the example of a display of the captured image which a digital camera images. It is a block diagram which shows the function structural example of a digital camera. It is a block diagram which shows the structural example of an image process part. It is a block diagram which shows the structural example of an image process part. It is a block diagram which shows the structural example of an image process part. It is a block diagram which shows the structural example of an image process part. It is a block diagram which shows the structural example of an image process part. It is a block diagram which shows the structural example of an image process part. 4 is a flowchart for explaining image display processing of the digital camera of FIG. 3. It is a figure explaining the distortion correction process of a distortion correction part. It is a figure explaining the distortion correction process of a distortion correction part. It is a figure explaining the optical axis correction process of an optical axis correction part. It is a figure explaining the adjustment of the magnitude | size of the main image and sub image in the synthetic | combination process of a synthetic | combination part. It is a block diagram which shows the other structural example of a digital camera. It is a block diagram which shows the structural example of an object detection part. It is a block diagram which shows the structural example of an object detection part. It is a block diagram which shows the structural example of an image process part. 16 is a flowchart for describing image display processing of the digital camera in FIG. 15. It is a flowchart explaining the example of an object detection process. It is a flowchart explaining the other example of an object detection process. It is a figure which shows the example of the synthesized image which highlighted the detected object image. It is a block diagram which shows the further another structural example of a digital camera. It is a flowchart explaining the image display process of the digital camera of FIG. It is a figure which shows the example of a recommendation composition. It is a figure which shows the example of the synthesized image which displayed the recommendation composition. It is a figure explaining the aspect-ratio of an image pick-up element. It is a block diagram which shows the further another structural example of a digital camera. It is a flowchart explaining the image display process of the digital camera of FIG. It is a figure explaining a cut-out image. It is a figure explaining an extracted image. It is a figure explaining the example of the synthesized image displayed on a display part. It is a flowchart explaining the image recording process of the digital camera of FIG. It is a figure explaining the example of the synthesized image recorded on a recording part. It is a block diagram which shows the structural example of the hardware of a computer.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The description will be given in the following order.
1. 1. First embodiment 2. Second embodiment 3. Third embodiment Fourth embodiment

<1. First Embodiment>
[Appearance and image of imaging device]
FIG. 1 shows an example of the appearance of a digital camera as an embodiment of an imaging apparatus to which the present invention is applied.

  The digital camera 11 in FIG. 1 includes two optical systems, a main lens 31 and a sub lens 32. The main lens 31 is a so-called standard lens. The sub lens 32 is a wide-angle lens (for example, a fisheye lens), and its angle of view is sufficiently wider than that of the main lens 31.

  FIG. 2 shows a display example of a captured image captured by the digital camera 11.

  As shown in FIG. 2, the digital camera 11 performs correction processing such as distortion correction on an image (sub-image) captured through the sub-lens 32, and performs predetermined image processing (such as blurring, luminance, Etc.). In addition, the digital camera 11 performs predetermined image processing (such as increasing luminance and saturation) on the image (main image) captured through the main lens 31. The digital camera 11 adjusts the position and angle of view of the main image and the sub image, and combines the main image and the sub image (synthesized image) provided on the back of the digital camera 11 (not shown). For example, the display is displayed on the display unit as shown in FIG. The user can recognize the angle of view to be captured by confirming the display content of the display unit (not shown).

  The synthesized image shown in FIG. 2 is synthesized such that the sub image (wide-angle image) that has been subjected to predetermined correction and image processing and the main image (standard image) match the position of the subject in each image. ing.

[Functional configuration example of digital camera]
Next, a functional configuration example of the digital camera 11 will be described with reference to the block diagram of FIG.

  3 includes a main lens 31, a sub lens 32, an imaging unit 51, a distortion correction unit 52, an optical axis correction unit 53, an image processing unit 54, an imaging unit 55, an image processing unit 56, a combining unit 57, and The display unit 58 is configured.

  The main lens 31 and the sub lens 32 are the same as those described with reference to FIG.

  The imaging unit 51 is configured to include an imaging device and an A / D (Analog / Digital) conversion unit. The imaging unit 51 receives the light from the sub-lens 32 and performs photoelectric conversion to image a subject, and A / D converts the obtained analog image signal. The imaging unit 51 supplies digital image data (wide-angle image) obtained as a result of A / D conversion to the distortion correction unit 52.

  The distortion correction unit 52 corrects the distortion of the sub lens 32 in the wide-angle image (sub image) from the imaging unit 51 and supplies the corrected sub image to the optical axis correction unit 53.

  The optical axis correction unit 53 aligns the optical axis of the sub lens 32 with the optical axis of the main lens 31, and determines the position of the main image to be combined with the sub image in the sub image from the distortion correction unit 52. The sub-image is supplied to the image processing unit 54 together with the position information indicating.

  The image processing unit 54 performs predetermined image processing (adjustment of picture, brightness, color density, tint, sharpness, etc.) on the sub image from the optical axis correction unit 53 so as to lower the image quality from the main image. Applied to the combining unit 57 together with the position information.

  Here, configuration examples of the three types of image processing units 54 will be described with reference to the block diagrams of FIGS. 4 to 6.

  FIG. 4 is a first configuration example and shows a configuration example of an image processing unit 54 that performs blurring processing on a sub-image.

  The image processing unit 54 in FIG. 4 includes an RGB / YUV conversion unit 61, an LPF (Low Pass Filter) 62, and a YUV / RGB conversion unit 63.

  The RGB / YUV conversion unit 61 converts the RGB signal as the sub image supplied from the optical axis correction unit 53 into a YUV signal based on the following equation (1).

  The RGB / YUV converter 61 supplies a Y signal (luminance signal) of the converted YUV signals to the LPF 62 and supplies U and V signals (color difference signals) to the YUV / RGB converter 63.

  The LPF 62 smoothes the Y signal supplied from the RGB / YUV conversion unit 61 to remove a high frequency component and supplies the Y signal to the YUV / RGB conversion unit 63.

  The YUV / RGB conversion unit 63 converts the Y signal (Y ′) from the LPF 62 and the U and V signals from the RGB / YUV conversion unit 61 into RGB signals based on the following equation (2).

  The YUV / RGB conversion unit 63 supplies the converted RGB signals (R ′, G ′, B ′) to the synthesis unit 57 as a sub image.

  With the above configuration, the image processing unit 54 can perform blurring processing on the sub-image.

  FIG. 5 is a second configuration example and shows a configuration example of the image processing unit 54 that lowers the saturation of the sub-image.

  The image processing unit 54 in FIG. 5 includes an RGB / HSV conversion unit 71, a saturation adjustment unit (S adjustment unit) 72, and an HSV / RGB conversion unit 73.

  The RGB / HSV conversion unit 71 converts the RGB signal as the sub image supplied from the optical axis correction unit 53 into an HSV signal based on the following equation (3).

  In Expression (3), MAX represents the maximum value of R, G, and B, and MIN represents the minimum value of R, G, and B, respectively. The RGB / HSV conversion unit 71 supplies the S signal (saturation signal) of the converted HSV signal to the S adjustment unit 72, and converts the H signal (hue signal) and the V signal (brightness signal) into HSV / RGB conversion. To the unit 73.

  The S adjustment unit 72 multiplies the S signal supplied from the RGB / HSV conversion unit 71 by a predetermined coefficient α (0 <α <1) and supplies the product to the HSV / RGB conversion unit 73.

  The HSV / RGB conversion unit 73 converts the S signal (S ′ = αS) from the S adjustment unit 72 and the H and V signals from the RGB / HSV conversion unit 71 into RGB signals.

That is, H _i = [H / 60] mod 6 ([x]: maximum integer less than or equal to x, A mod B: remainder of A / B), f = (H / 60) −H _i , p = V ( 1-S), q = V (1-fS), t = V (1- (1-f) S), HSV signal is converted to RGB signal based on the following equation (4) Is done.

  The HSV / RGB conversion unit 73 supplies the converted RGB signals (R ′, G ′, B ′) to the synthesis unit 57 as a sub image.

  With the above configuration, the image processing unit 54 can reduce the saturation of the sub-image.

  FIG. 6 is a third configuration example and shows a configuration example of the image processing unit 54 that reduces the brightness of the sub-image.

  The image processing unit 54 in FIG. 6 includes an RGB / HSV conversion unit 81, a brightness adjustment unit (V adjustment unit) 82, and an HSV / RGB conversion unit 83.

  The RGB / HSV conversion unit 81 converts the RGB signal as the sub-image supplied from the optical axis correction unit 53 into an HSV signal based on the above-described equation (3), and converts the V signal in the converted HSV signal. In addition to supplying to the V adjustment unit 82, the H signal and the S signal are supplied to the HSV / RGB conversion unit 83.

  The V adjustment unit 82 multiplies the V signal supplied from the RGB / HSV conversion unit 81 by a predetermined coefficient β (0 <β <1) and supplies the product to the HSV / RGB conversion unit 83.

  The HSV / RGB conversion unit 83 converts the V signal (V ′ = βV) from the V adjustment unit 82 and the H and S signals from the RGB / HSV conversion unit 81 into RGB signals based on the above equation (4). The converted RGB signals (R ′, G ′, B ′) are supplied to the synthesizing unit 57 as sub-images.

  With the above configuration, the image processing unit 54 can reduce the brightness of the sub-image.

  In this way, any of the three types of image processing units 54 described with reference to FIGS. 4 to 6 can lower the image quality of the sub image from the image quality of the main image.

  Returning to the description of FIG. 3, the imaging unit 55 is configured to include an imaging device and an A / D conversion unit. The imaging unit 55 receives the light from the main lens 31 and photoelectrically converts the image of the subject, and A / D converts the obtained analog image signal. The imaging unit 55 supplies digital image data (standard image) obtained as a result of A / D conversion to the image processing unit 56.

  The image processing unit 56 performs predetermined image processing (adjustment of picture, brightness, color density, hue, sharpness, etc.) so as to improve the image quality of the main image (standard image) from the imaging unit 55 compared to the sub image. ) And supplied to the combining unit 57.

  Here, configuration examples of the three types of image processing units 56 will be described with reference to the block diagrams of FIGS.

  FIG. 7 is a first configuration example and shows a configuration example of the image processing unit 56 that performs enhancement processing for enhancing the shape and contour of the main image.

  The image processing unit 56 in FIG. 7 includes an RGB / YUV conversion unit 91, an HPF (High Pass Filter) 92, an amplifier 93, an adder 94, and a YUV / RGB conversion unit 95.

  The RGB / YUV conversion unit 91 converts the RGB signal as the main image from the imaging unit 55 into a YUV signal based on the above-described equation (1), and the Y signal of the converted YUV signal is converted into an HPF 92 and an adder. In addition, the U and V signals are supplied to the YUV / RGB converter 95.

  The HPF 92 extracts the high frequency component of the Y signal supplied from the RGB / YUV conversion unit 91 and supplies it to the amplifier 93. The amplifier 93 amplifies the high frequency component of the Y signal from the HPF 92 by A (A> 1) times and supplies the amplified signal to the adder 94. The adder 94 adds the high frequency component of the amplified Y signal from the amplifier 93 to the Y signal from the RGB / YUV conversion unit 91 and supplies the result to the YUV / RGB conversion unit 95.

  The YUV / RGB conversion unit 95 converts the Y signal (Y ′) from the adder 94 and the U and V signals from the RGB / YUV conversion unit 91 into RGB signals based on the above equation (2). The converted RGB signals (R ′, G ′, B ′) are supplied to the synthesis unit 57 as the main image.

  With the above configuration, the image processing unit 56 can perform enhancement processing on the main image.

  FIG. 8 is a second configuration example and shows a configuration example of the image processing unit 56 that increases the saturation of the main image.

  The image processing unit 56 in FIG. 8 includes an RGB / HSV conversion unit 101, a saturation adjustment unit (S adjustment unit) 102, and an HSV / RGB conversion unit 103. The RGB / HSV conversion unit 101 and the HSV / RGB conversion unit 103 have the same functions as the RGB / HSV conversion unit 71 and the HSV / RGB conversion unit 73 provided in the image processing unit 54 of FIG. Description is omitted.

  That is, the S adjustment unit 102 multiplies the S signal supplied from the RGB / HSV conversion unit 101 by a predetermined coefficient α (α ≧ 1) and supplies the product to the HSV / RGB conversion unit 103.

  With the above configuration, the image processing unit 56 can increase the saturation of the main image.

  FIG. 9 is a third configuration example and illustrates a configuration example of the image processing unit 56 that increases the brightness of the sub-image.

  The image processing unit 56 in FIG. 9 includes an RGB / HSV conversion unit 111, a brightness adjustment unit (V adjustment unit) 112, and an HSV / RGB conversion unit 113. The RGB / HSV conversion unit 111 and the HSV / RGB conversion unit 113 have the same functions as the RGB / HSV conversion unit 81 and the HSV / RGB conversion unit 83 provided in the image processing unit 54 of FIG. Description is omitted.

  That is, the V adjustment unit 112 multiplies the V signal supplied from the RGB / HSV conversion unit 111 by a predetermined coefficient β (β ≧ 1) and supplies the product to the HSV / RGB conversion unit 113.

  With the above configuration, the image processing unit 56 can increase the brightness of the main image.

  In this way, any of the three types of image processing units 56 described with reference to FIGS. 7 to 9 can improve the image quality of the main image over the image quality of the sub image.

  Returning to the description of FIG. 3, the combining unit 57 combines the main image from the image processing unit 56 and the sub image from the image processing unit 54 based on the position information from the image processing unit 54, and combines the combined image. Is supplied to the display unit 58.

  The display unit 58 displays the combined image from the combining unit 57.

[Digital camera image display processing]
Next, image display processing of the digital camera 11 in FIG. 3 will be described with reference to the flowchart in FIG.

  In step S <b> 11, the distortion correction unit 52 corrects the distortion of the sub lens 32 in the wide-angle image (sub image) from the imaging unit 51, and supplies the corrected sub image to the optical axis correction unit 53.

  Here, the distortion correction processing of the distortion correction unit 52 will be described with reference to FIGS. 11 and 12.

  FIG. 11 shows pixel positions (X, Y) of pixels in a corrected image having a predetermined size obtained by distortion correction, and pixel positions (x of pixels) in a sub-image (circular fisheye image) before distortion correction. , y) is a diagram for explaining the correspondence with y).

  As shown in FIG. 11, consider the upper hemisphere of radius R with the center of the circumferential fish-eye image as the origin of the xyz coordinates and the circumferential fish-eye image as a cut surface in diameter. Note that the corrected image is in contact with the upper hemisphere at the point (0, 0, R).

  The intersection of the xyz coordinate origin and point (X, Y, R) and the upper hemisphere sphere is (x, y, z), and the xyz coordinate origin and intersection (x, y, z) When the distance between and is R, the following equation (5) holds.

  That is, the distortion correction unit 52 sets the pixel value of the point (X, Y) in the corrected image after correction as the pixel value of the point (x, y) in the corresponding circumferential fisheye image before correction, A corrected image is generated.

  In addition, as shown in FIG. 12, the point (x, y) in the circumferential fisheye image before correction corresponding to the point (X, Y) in the corrected image after correction is positioned at the grid point in the pixel arrangement. If not, the surrounding pixels (pixel values) a to d may be interpolated to obtain the pixel value of the point (x, y). As an interpolation method, bilinear interpolation, bicubic interpolation, or the like is used. In FIG. 12, when bilinear interpolation is used, the pixel value p at the point (x, y) is p = (1-t) {(1-s) a + sb} + t {(1-s) c + sd}.

  Returning to the flowchart of FIG. 10, in step S <b> 12, the optical axis correction unit 53 matches the optical axis of the sub lens 32 with the optical axis of the main lens 31, and is synthesized with the sub image in the sub image from the distortion correction unit 52. The position of the main image to be determined is determined. The optical axis correction unit 53 supplies position information indicating the position and the sub image to the image processing unit 54.

  Here, the optical axis correction processing of the optical axis correction unit 53 will be described with reference to FIG.

  FIG. 13 is a diagram for explaining the relationship between the coordinate system of the optical axis of the main lens 31 (main lens coordinate system) and the coordinate system of the optical axis of the sub lens 32 (sub lens coordinate system).

  In FIG. 13, a vector h represents a physical arrangement difference in the xy plane between the main lens 31 and the sub lens 32 arranged in the digital camera 11, and the rotation amount R represents the light of the main lens 31. This represents a rotational deviation amount with reference to the z axis between the axis and the optical axis of the sub lens 32. That is, the vector v ′ = (x ′, y ′, z ′) in the sub-lens coordinate system is expressed as v ′ = Rv + h using the vector v = (x, y, z) in the main lens coordinate system. Is done. That is, the vector v ′ in the sub-lens coordinate system is a vector obtained by rotating the vector v by the rotation amount R and translating it by the vector h. The optical axis correction unit 53 can match the optical axis (coordinates) of the sub lens 32 with the optical axis (coordinates) of the main lens 31 using this relationship.

  Further, the optical axis correction unit 53 performs matching between the image in the region near the center of the sub image from the distortion correction unit 52 and the main image from the imaging unit 55 corresponding to the region, thereby sub-image. The position of the main image to be synthesized is determined.

  Returning to the flowchart of FIG. 10, in step S <b> 13, the image processing unit 54 performs predetermined image processing on the sub-image from the optical axis correction unit 53 so that the image quality is lower than that of the main image. More specifically, the image processing unit 54 reduces the luminance, saturation, or brightness of the sub image. The image processing unit 54 supplies the sub-image with reduced image quality to the combining unit 57 together with the position information from the optical axis correction unit 53. Here, the image processing unit 54 can extract only the luminance signal of the sub-image and supply the monochrome image to the synthesizing unit 57.

  In step S <b> 14, the image processing unit 56 performs predetermined image processing on the main image from the imaging unit 55 so that the image quality is higher than that of the sub image. More specifically, the image processing unit 56 increases the luminance, saturation, or brightness of the main image. The image processing unit 56 supplies the main image with improved image quality to the combining unit 57. Here, the image processing unit 56 may add a frame of a predetermined color or line type around the main image and supply the frame to the combining unit 57.

  In step S15, the combining unit 57 combines the main image from the image processing unit 56 and the sub image from the image processing unit 54 based on the position information from the image processing unit 54, and displays the combined image as a display unit. 58.

  At this time, the size of the main image needs to be smaller than that of the sub image, and this depends on the size of the image sensor provided in each of the imaging unit 51 and the imaging unit 55.

  For example, when the image pickup devices provided in the image pickup unit 51 and the image pickup unit 55 have the same size, images in different ranges are generated as images of the same size. Therefore, in order to synthesize the main image and the sub image, it is necessary to adjust the size of one of the images.

  Here, with reference to FIG. 14, adjustment of the image size of the main image and the sub image in the composition processing of the composition unit 57 will be described.

As shown in Figure 14A, when the origin of the xyz coordinate the position of the main lens 31, and the origin, the distance between the imaging element of the imaging unit 55 for Projecting main image I _m a (z axis direction) f _m .

Further, as shown in FIG. 14B, when the origin of the xyz coordinate the position of the sub-lens 32, the distance of the origin, the image sensor of the imaging unit 51 for copy sub image I _s (z-axis direction) Let f _s .

Then, when considering a certain object distance Z, as shown in FIG. 14C, the captured area on the imaging element of the imaging unit 55 of the main image I _m and l _m, the captured area to the imaging unit 51 of the sub-image I _s Let l _s . In addition, the size of each image sensor of the imaging unit 51 and the imaging unit 55 is assumed to be l.

  At this time, the following equation (6) is obtained from the relationship of FIG. 14C.

Further, the relationship of Equation (6), the ratio l _m / l _s of the magnitude of the size and the sub-image I _s of the main image I _m is determined by the following equation (7).

In this case, the combining unit 57, by multiplying the size of the main image I _m f _s / f _m, and generates the main image I _'m, synthesizes the sub-image I _s.

In this way, as shown in FIG. 14D, the main image I _'m composite image in size than the sub-image I _s is small is produced.

The size of the imaging device of the imaging unit 51 and the imaging unit 55 is not limited to the case were identical, the relationship of the size of the main image I _m and the sub-image I _s is, if not as shown in FIG. 14D is, by predetermined multiple of the size of the main image I _m, it is possible to adjust the size of the main image.

  Returning to the flowchart of FIG. 10, in step S <b> 16, the display unit 58 displays the combined image from the combining unit 57.

  According to the above processing, as shown in FIG. 2, a composite image in which a standard image with a desired angle of view is combined is displayed at the center of the wide-angle image with reduced image quality. You can easily check the images at the same time.

  In addition, since the user can confirm the composition other than the angle of view to be captured in the wide-angle image, the user can also search for a better composition.

  In the above description, the image quality of the image other than the angle of view to be photographed has been described so that only the composition is confirmed by reducing the image quality. However, the object included in the image other than the angle of view to be photographed is emphasized and displayed. Also good.

<2. Second Embodiment>
[Example of digital camera configuration]
FIG. 15 shows a configuration example of a digital camera that displays an object in an image other than the angle of view to be photographed. In addition, in the digital camera 121 of FIG. 15, about the structure provided with the function similar to what was provided in the digital camera 11 of FIG. 3, the same name and the same code | symbol shall be attached | subjected, and the description is abbreviate | omitted suitably. And

  15 differs from the digital camera 11 in FIG. 3 in that an object detection unit 151 and an image processing unit 152 are newly provided, and a combining unit 153 is provided in place of the combining unit 57. is there.

  The optical axis correction unit 53 in FIG. 15 supplies the image processing unit 54 with positional information indicating the optical image corrected sub-image and the position where the main image in the sub-image is combined, and also the optical image corrected sub-image. Is supplied to the object detection unit 151.

  The object detection unit 151 detects an object in the sub image from the optical axis correction unit 53, and displays an image of the region including the detected object (detected object image) together with position information indicating the position in the sub image, as an image processing unit. 152.

  Here, a configuration example of the two types of object detection units 151 will be described with reference to FIGS. 16 and 17.

  FIG. 16 shows a first configuration example of the object detection unit 151.

  The object detection unit 151 in FIG. 16 includes a frame buffer 161, an inter-frame difference calculation unit 162, a threshold processing unit 163, and a labeling unit 164.

  The frame buffer 161 stores the sub image from the optical axis correction unit 53 for each frame.

The inter-frame difference calculation unit 162 reads the sub-image I _t-1 of the previous frame stored in the frame buffer 161, and the sub-image I _t-1 and the sub-image of the current frame from the optical axis correction unit 53. and a I _t, for each pixel, and calculates an inter-frame difference value is a difference between the pixel values. The interframe difference calculation unit 162 supplies the calculation result to the threshold processing unit 163.

  Based on the inter-frame difference value from the inter-frame difference calculation unit 162, the threshold processing unit 163 sets the pixel having a difference value equal to or greater than the predetermined threshold to 1, and sets the pixel having the difference value not satisfying the predetermined threshold to 0. Convert to value. Further, the threshold processing unit 163 divides the binarized image into predetermined blocks, counts the pixels having a pixel value of 1 in each block for each block, and counts the predetermined threshold or more based on the count number. A block having a number is detected as a motion block. Then, the threshold processing unit 163 supplies the detection result (motion block) to the labeling unit 164.

  The labeling unit 164 performs a labeling process based on the motion block supplied from the threshold processing unit 163. Further, the labeling unit 164 detects a rectangle (surrounding frame) surrounding each labeled block from the outside, and supplies the detection result to the image processing unit 152 as a detected object image.

  With the above configuration, the object detection unit 151 can detect a moving object in the sub-image.

  FIG. 17 illustrates a second configuration example of the object detection unit 151.

  The object detection unit 151 in FIG. 17 includes a face detection unit 171.

The face detection unit 171 detects a face from the sub-image I _t from the optical axis correcting unit 53 extracts a face image based on the position and size of the face detection area is an area to detect the face, detected object The image is supplied to the image processing unit 152 as an image. For example, the face detection unit 171 learns a face image of a face are oriented in various directions, the sub-image I _t, an image of a region of the same size as the face detection area, the learning face image The face is detected by comparing and evaluating whether or not it is a face.

  With the above configuration, the object detection unit 151 can detect a human face in the sub-image.

  In the above description, the object detection unit 151 is configured to detect a moving object or a person (face), but is not limited thereto, and detects a predetermined object (automobile, building, etc.). You may make it do.

  Returning to the description of FIG. 15, the image processing unit 152 performs predetermined image processing (picture, brightness, color density, hue) on the detected object image from the object detection unit 151 so as to improve the image quality from the sub-image. , Adjustment of sharpness, etc.) is performed and supplied to the combining unit 153 together with the position information.

  Here, a configuration example of the image processing unit 152 will be described with reference to the block diagram of FIG.

  FIG. 18 illustrates a configuration example of the image processing unit 152 that performs enhancement processing on the detected object image.

  The image processing unit 152 in FIG. 18 includes an RGB / YUV conversion unit 181, an HPF 182, an amplifier 183, an adder 184, and a YUV / RGB conversion unit 185. The RGB / YUV conversion unit 181 and HPF 182, the amplifier 183, the adder 184, and the YUV / RGB conversion unit 185 are the RGB / YUV conversion unit 91, HPF 92, the amplifier 93, provided in the image processing unit 56 of FIG. Since functions similar to those of the adder 94 and the YUV / RGB conversion unit 95 are provided, description thereof will be omitted.

  With the above configuration, the image processing unit 152 can perform enhancement processing on the detected object image.

  Returning to the description of FIG. 15, the synthesizing unit 153 synthesizes the main image from the image processing unit 56 and the sub-image from the image processing unit 54 based on the position information from the image processing unit 54. Further, the synthesis unit 153 synthesizes the detected object image from the object detection unit 151 in a region corresponding to the sub-image of the synthesized image based on the position information from the image processing unit 152 and displays the synthesized image. Supplied to the unit 58.

[Digital camera image display processing]
Next, image display processing of the digital camera 121 in FIG. 15 will be described with reference to the flowchart in FIG. Note that the processing of steps S31 to S33 and S36 in the flowchart of FIG. 19 is the same as the processing of steps S11 to S14 described with reference to the flowchart of FIG.

  That is, in step S34, the object detection unit 151 performs object detection processing for detecting an object in the sub-image from the optical axis correction unit 53. The object detection unit 151 supplies an image (detected object image) of a region including the detected object to the image processing unit 152 together with position information indicating the position in the sub image.

  Here, an example of object detection processing for detecting a moving object corresponding to the object detection processing in step S34 will be described with reference to the flowchart of FIG.

In step S 51, the inter-frame difference calculation unit 162 reads the sub-image I _t-1 one frame before stored in the frame buffer 161, and the sub-image I _t-1 and the current image from the optical axis correction unit 53. and a sub-image I _t of the frame, for each pixel, and calculates a difference value between frames. The interframe difference calculation unit 162 supplies the calculation result to the threshold processing unit 163.

  In step S52, the threshold processing unit 163 sets a pixel having a difference value equal to or greater than a predetermined threshold based on the inter-frame difference value from the inter-frame difference calculation unit 162 to a pixel having a difference value less than the predetermined threshold. Is binarized with 0 being 0. Further, the threshold processing unit 163 divides the binarized image into predetermined blocks, counts the pixels having a pixel value of 1 in each block for each block, and counts the predetermined threshold or more based on the count number. A block having a number is detected as a motion block. Then, the threshold processing unit 163 supplies the detection result (motion block) to the labeling unit 164.

  In step S <b> 53, the labeling unit 164 performs labeling processing based on the motion block supplied from the threshold processing unit 163. Further, the labeling unit 164 detects a rectangle (surrounding frame) surrounding each labeled block from the outside, and supplies the detection result to the image processing unit 152 as a detected object image.

  According to the above processing, a moving object can be detected in the sub-image.

  In the object detection process in step S34, a human face may be detected.

  Here, an example of object detection processing for detecting a human face will be described with reference to the flowchart of FIG.

In step S61, the face detection unit 171 detects a face from the sub-image I _t from the optical axis correcting unit 53, extracts a face image based on the position and size of the face detection area is an area for detecting a face The detected object image is supplied to the image processing unit 152.

  According to the above processing, a human face can be detected in the sub-image.

  The object detection processing described with reference to the flowcharts of FIGS. 20 and 21 is executed in parallel by having the object detection unit 151 of the digital camera 121 have the configuration described with reference to FIGS. Can be.

  Returning to the flowchart of FIG. 19, in step S <b> 35, the image processing unit 152 performs predetermined image processing on the detected object image from the object detection unit 151 so that the image quality is higher than that of the sub image, and combines it with the position information. To the unit 153. More specifically, the image processing unit 152 increases the brightness, saturation, or brightness of the detected object image. The image processing unit 152 supplies the detected object image with improved image quality to the synthesis unit 153. Here, the image processing unit 152 may add a frame of a predetermined color around the detected object image and supply the frame to the combining unit 153.

  In step S <b> 37, the combining unit 153 combines the main image from the image processing unit 56 and the sub image from the image processing unit 54 based on the position information from the image processing unit 54. Further, based on the position information from the image processing unit 152, the synthesizing unit 153 displays the detected object image from the object detecting unit 151 in an area corresponding to the sub image of the synthesized image that is synthesized, for example, as shown in FIG. Then, the combined image is supplied to the display unit 58.

  FIG. 22 shows an example of a composite image in which the detected object image is highlighted.

  In FIG. 22, in addition to the synthesized image shown in FIG. 2, a detected object image 201 and a detected object image 202 are displayed in a region corresponding to the sub image of the synthesized image.

  22, a detected object image 201 is an image of a dog as a moving object, the object detection unit 151 in FIG. 16 detects an object (dog), and the image processing unit 152 detects the detected object image 201. By increasing the image quality, the composite image is highlighted.

  Further, the detected object image 202 is a human image, and the object detection unit 151 in FIG. 17 detects a human face, and the image processing unit 152 increases the image quality of the detected object image 202, so that Highlighted.

  In this manner, an image in which an object included in an image other than the angle of view (main image) to be photographed is highlighted in the composite image, so that a still image is captured in a state where the focal position is fixed in advance. When the “place pin” is shot, even if an object other than the desired subject is framed in the main image, the object can be confirmed on the composite image, so that the shutter timing can be easily measured.

  According to the above processing, the standard image having the angle of view to be photographed is synthesized at the center of the wide-angle image with the lowered image quality, and the image in which the object included in the image other than the angle of view to be photographed is emphasized is synthesized. A composite image is displayed. Therefore, the user can easily check images of two compositions at the same time, and can also check an object included in an image other than the angle of view to be photographed.

  In the above description, the object is detected only in the sub-image, but the object may be detected in the main image. With this configuration, an object that is displayed across the sub image and the main image can be detected. For example, the object is emphasized in the sub image and the image quality is enhanced in the main image. Can be lowered. Thereby, even when an object other than the desired subject is framed in the main image, the display of the object can be made inconspicuous.

  In the above description, the configuration in which the composite image centered on the angle of view that the user wants to shoot has been described. However, the angle of view to be captured with a more appropriate composition may be displayed on the composite image.

<3. Third Embodiment>
[Example of digital camera configuration]
FIG. 23 shows a configuration example of a digital camera in which the angle of view that allows the user to capture an image with a more appropriate composition is displayed on the composite image. In addition, in the digital camera 221 in FIG. 23, components having the same functions as those provided in the digital camera 121 in FIG. 15 are denoted by the same names and the same reference numerals, and description thereof will be omitted as appropriate. And

  That is, the digital camera 221 in FIG. 23 differs from the digital camera 121 in FIG. 15 in that a composition analysis unit 251 and a recommended composition extraction unit 252 are newly provided, and image processing is performed instead of the image processing unit 152 and the composition unit 153. This is a point where a unit 253 and a synthesis unit 254 are provided.

  The object detection unit 151 in FIG. 23 detects an object in the sub image from the optical axis correction unit 53, and sends position information indicating the position of the detected object in the sub image and the sub image to the composition analysis unit 251. Supply.

  The composition analysis unit 251 analyzes and determines the composition corresponding to the arrangement of the objects in the sub image based on the sub image from the object detection unit 151 and the position information of the object in the sub image. For example, the composition analysis unit 251 stores a plurality of composition patterns in advance, and selects a composition pattern closest to the arrangement of the objects in the sub image from the composition patterns by pattern matching, and corresponds to the sub image. The composition (recommended composition). The composition analysis unit 251 supplies composition information indicating the determined composition (for example, the upper left position and the lower right position of the rectangle that is the outline of the composition) to the recommended composition extraction unit 252 together with the sub image.

  The recommended composition extraction unit 252 extracts a recommended composition image (recommended composition image) from the sub image based on the composition information from the composition analysis unit 251 and the sub image, and supplies the extracted image to the image processing unit 253 together with the composition information. .

  The image processing unit 253 has the same function as the image processing unit 56, and performs predetermined image processing (picture, brightness) on the recommended composition image from the recommended composition extraction unit 252 so that the image quality is higher than that of the sub image. , Adjustment of color density, hue, sharpness, etc.), and the composition information is supplied to the composition unit 254.

  The combining unit 254 combines the main image from the image processing unit 56 and the sub image from the image processing unit 54 based on the position information from the image processing unit 54. Further, based on the composition information from the image processing unit 253, the combining unit 254 combines the recommended composition image from the image processing unit 253 in the area corresponding to the sub-image of the combined image, and displays the combined image. Supplied to the unit 58.

[Digital camera image display processing]
Next, image display processing of the digital camera 221 in FIG. 23 will be described with reference to the flowchart in FIG. Note that the processing of steps S71 to S74 and S78 in the flowchart of FIG. 24 is the same as the processing of steps S31 to S34 and S36 described with reference to the flowchart of FIG.

  In step S75, the composition analysis unit 251 analyzes the composition corresponding to the arrangement of the objects in the sub image based on the sub image from the object detection unit 151 and the position information of the object in the sub image, and recommends the composition. To decide. For example, the composition analysis unit 251 stores a plurality of composition patterns in advance, and selects a composition pattern closest to the arrangement of the objects in the sub image from the composition patterns by pattern matching, and corresponds to the sub image. With the recommended composition. The composition analysis unit 251 supplies composition information indicating the determined composition to the recommended composition extraction unit 252 together with the sub-image.

  FIG. 25 shows an example of a recommended composition determined by the composition analysis unit 251.

  The composition shown in FIG. 25 is a composition called a three-part composition, and is balanced by placing an object (subject) at one of the intersections of the vertical and horizontal lines (filled circles in the figure). It is known to be an image (photo). Note that the composition (composition pattern) determined by the composition analysis unit 251 is not limited to the three-part composition, but is a horizontal line composition that is used when the left and right spreads are desired, and a vertical line composition that is used when the vertical direction is desired to be emphasized. It is assumed that a plurality of contrast compositions used when two identical or similar ones are arranged are prepared.

  Returning to the flowchart of FIG. 24, in step S76, the recommended composition extraction unit 252 extracts a recommended composition image from the sub image based on the composition information and the sub image from the composition analysis unit 251, and performs image processing together with the composition information. Supplied to the unit 253.

  In step S <b> 77, the image processing unit 253 performs predetermined image processing on the recommended composition image from the recommended composition extraction unit 252 so that the image quality is higher than that of the sub-image, and supplies the image to the synthesis unit 254 together with the composition information. Here, the image processing unit 253 may add a frame of a predetermined color or line type around the recommended composition image and supply the frame to the synthesis unit 254.

  In step S <b> 79, the combining unit 254 combines the main image from the image processing unit 56 and the sub image from the image processing unit 54 based on the position information from the image processing unit 54. Further, based on the composition information from the image processing unit 253, the combining unit 254 shows the recommended composition image from the image processing unit 253 in an area corresponding to the sub image of the combined image, for example, as shown in FIG. Then, the combined image is supplied to the display unit 58.

  FIG. 26 shows an example of a composite image in which a recommended composition is displayed.

  In FIG. 26, in addition to the composite image shown in FIG. 2, on the composite image, a recommended composition is shown by a broken line representing its outline.

  In the above description, the image quality of the image in the recommended composition is assumed to be higher than the image quality of the sub-image. However, when a part of the main image (current angle of view) and a part of the recommended composition overlap, There is a risk of being indistinguishable. Therefore, as shown in FIG. 26, the outline (view angle) may be simply displayed as the recommended composition.

  According to the above processing, the recommended composition can be displayed on the composite image.

  Since the recommended composition is displayed on the composite image in this way, the user can confirm a composition that is more appropriate than the composition to be photographed (recorded).

  In the above description, a digital camera having two imaging systems that displays a more appropriate composition has been described. However, a more appropriate composition can be displayed even in a digital camera that has only one imaging system. Can be.

  By the way, in recent years, with the widespread use of high-definition television receivers, it is desirable to capture and record images with an aspect ratio (aspect ratio) of 16: 9 using an imaging device such as a digital camera and view them on a high-definition television receiver. There is an increasing demand.

  Therefore, as shown in FIG. 27, a general imaging apparatus having a solid-state imaging device (hereinafter simply referred to as an imaging device) 301 having an aspect ratio of 4: 3 (or 3: 2) has an aspect ratio of 4. : 3 (or 3: 2) standard mode for capturing an image and panoramic mode for capturing an image with an aspect ratio of 16: 9. In the standard mode, an image with an aspect ratio of 4: 3 (or 3: 2) is captured, displayed, and recorded using the entire image sensor 301. On the other hand, in the panorama mode, the image pickup device of the rectangular region 313 having an aspect ratio of 16: 9 is used without using the image pickup devices of the upper region 311 and the lower region 312 in the image pickup device 301 of FIG. Thus, an image with an aspect ratio of 16: 9 is captured, displayed, and recorded.

  In the imaging apparatus as described above, the display area of the display unit for displaying a so-called through image is configured to display an image corresponding to the entire imaging element 301 of FIG. Therefore, when the user captures a subject in the panorama mode, the display area where the through image is displayed on the display unit is an area corresponding to the rectangular area 313 in the entire image sensor 301 in FIG.

  Therefore, hereinafter, a digital camera including only one imaging system that displays a composition more appropriate than the composition that the user intends to shoot in the panorama mode of the imaging apparatus as described above will be described.

<4. Fourth Embodiment>
[Functional configuration example of digital camera]
FIG. 28 illustrates a configuration example of a digital camera including only one imaging system.

  The digital camera 411 in FIG. 28 includes a lens 431, an imaging unit 432, an image processing unit 433, a cutout unit 434, an extraction unit 435, a synthesis unit 436, a display control unit 437, a display unit 438, an operation input unit 439, and a recording control unit 440. And a recording unit 441.

  The lens 431 is a so-called standard lens, like the main lens 31 of the digital camera shown in FIG.

  The imaging unit 432 is configured to include an imaging element such as a CCD (Charge Coupled Device) or an A / D conversion unit having an aspect ratio of 4: 3. The imaging unit 432 receives the light from the lens 431 and performs photoelectric conversion to image a subject, and A / D converts the obtained analog image signal. The imaging unit 432 supplies digital image data (hereinafter referred to as a captured image) obtained as a result of A / D conversion to the image processing unit 433.

  The image processing unit 433 performs predetermined image processing on the captured image from the imaging unit 432. When the image capturing mode is a standard mode for capturing an image with an aspect ratio of 4: 3, the image processing unit 433 supplies the captured image subjected to the image processing to the display control unit 437 and the recording control unit 440. In addition, when the imaging mode is a panorama mode in which an image with an aspect ratio of 16: 9 is captured, the image processing unit 433 supplies the captured image subjected to the image processing to the cutout unit 434 and the extraction unit 435.

  When the picked-up image is supplied from the image processing unit 433, the cutout unit 434 cuts out an image having an aspect ratio of 16: 9 having a size different from that of the picked-up image. The cutout unit 434 supplies the cut out image (cutout image) to the combining unit 436, the display control unit 437, or the recording control unit 440 as necessary.

  When the captured image is supplied from the image processing unit 433, the extracting unit 435 detects a subject with a high degree of attention in the captured image, includes the subject, and has a different aspect ratio from the captured image and has an aspect ratio of 16: 9. Extract the image of the region to be. The extraction unit 435 supplies the extracted image (extraction image) to the synthesis unit 436 or the recording control unit 440 as necessary.

  The combining unit 436 combines the cut image from the cut unit 434 and the extracted image from the extraction unit 435, and supplies the combined image to the display control unit 437.

  The display control unit 437 controls the display unit 438 to display various images on the display unit 438. For example, the display control unit 437 causes the display unit 438 to display the captured image supplied from the image processing unit 433 when the imaging mode is the standard mode. Further, when the imaging mode is the panorama mode, the display control unit 437 causes the display unit 438 to display the cut-out image supplied from the cut-out unit 434 or the composite image supplied from the combining unit 436.

  The display unit 438 appropriately displays various images under the control of the display control unit 437.

  The operation input unit 439 is operated by the user in order to input an instruction to the digital camera 411. The operation input unit 439 includes, for example, various operation buttons, a remote controller, a touch panel, a microphone, and the like, receives an operation from the user, and supplies a signal (information) indicating the operation content to each block of the digital camera 411. .

  The recording control unit 440 controls the recording unit 441. For example, the recording control unit 440 causes the recording unit 441 to record the captured image from the image processing unit 433 based on the signal from the operation input unit 439 when the imaging mode is the standard mode. In addition, when the imaging mode is the panorama mode, the recording control unit 440 records the extracted image from the extracting unit 434 and the extracted image from the extracting unit 435 in the recording unit 441 based on the signal from the operation input unit 439. Let Further, the recording control unit 440 reads an image from the recording unit 441 based on a signal from the operation input unit 439 and supplies the image to the display control unit 437.

  The recording unit 441 records various images as appropriate under the control of the recording control unit 440.

[Digital camera image display processing]
Next, image display processing of the digital camera 411 in FIG. 28 will be described with reference to the flowchart in FIG. 29 is started when the operation input unit 439 receives an operation for selecting the panorama mode as the imaging mode by the user.

  In step S111, the imaging unit 432 images the subject. More specifically, the imaging unit 432 receives the light from the lens 431 and performs photoelectric conversion to image a subject, A / D converts the obtained analog image signal, and converts the obtained captured image into an image. This is supplied to the image processing unit 433.

  In step S <b> 112, the image processing unit 433 performs image processing such as demosaic processing, white balance adjustment processing, and gamma correction processing on the captured image from the imaging unit 432, and supplies the processed image to the clipping unit 434 and the extraction unit 435.

  In step S <b> 113, the cutout unit 434 cuts out an image with an aspect ratio of 16: 9 from the captured image supplied from the image processing unit 433, and supplies the cutout image obtained as a result to the combining unit 436. More specifically, the cutout unit 434 cuts out an image of a portion corresponding to the rectangular region 313 in the image pickup element 301 described with reference to FIG. 27 in the picked-up image having an aspect ratio of 4: 3 from the image processing unit 433. put out.

  For example, as illustrated on the left side of FIG. 30, when a captured image 511 (aspect ratio 4: 3) in which the subject is a cat is obtained, the cutout unit 434 extracts the region indicated by the alternate long and short dash line in the captured image 511. The image is cut out to obtain a cut image 512 shown on the right side of FIG. As shown in FIG. 30, in the cut-out image 512 having an aspect ratio of 16: 9, the cat that is the subject is located generally upward, and the top of the head including the ears is missing. Yes. The cut image 512 cannot be said to be an image with a good composition as an image shot and displayed in the panoramic mode.

  As described above, in a conventional digital camera having an image sensor with an aspect ratio of 4: 3, when a user photographs a subject in the panorama mode, an image displayed on the display unit is captured by the entire image sensor. With respect to the captured image 511, an image of an area indicated by a one-dot chain line in FIG. When a user unfamiliar with the operation of such a digital camera is used in the panorama mode, an image such as the cutout image 512 in FIG. 30 may be displayed and recorded on the display unit.

  Returning to the flowchart of FIG. 29, in step S114, the extraction unit 435 detects a subject with a high degree of attention in the captured image supplied from the image processing unit 433. The extraction unit 435 extracts an image of an area including the subject and having an aspect ratio of 16: 9, and supplies the extracted image obtained as a result to the synthesis unit 436.

  More specifically, the extraction unit 435 indicates a luminance information map indicating information regarding luminance, a color information map indicating information regarding color, an edge information map indicating information regarding edge, and information regarding motion in each region of the captured image. A motion information map is generated.

  Here, when it is not necessary to individually distinguish each of the luminance information map or the motion information map, if the information map is simply referred to as an information map, the information included in these information maps is included more in the region including the subject. Information indicating the feature amount of the feature is used. An information map is formed by arranging the information in correspondence with each area of the captured image. That is, the information map can be said to be information indicating the feature amount in each region of the captured image.

  Therefore, in each information map, a region with a larger amount of information, that is, a region on the captured image corresponding to a region with a large amount of features is a region with a higher possibility of including a subject. Can be specified.

  Then, based on the luminance information map, the color information map, the edge information map, and the motion information map, the extraction unit 435 identifies an area that includes the subject in the captured image and has an aspect ratio of 16: 9, and the area Are extracted as extracted images.

  For example, as illustrated on the left side of FIG. 31, when a captured image 511 (aspect ratio 4: 3) is obtained, the extraction unit 435 detects a cat that is a subject as an object with a high degree of attention in the captured image 511. Then, an image of an area (aspect ratio 16: 9) indicated by a broken line is extracted so as to include the subject, and an extracted image 513 shown on the right side of FIG. 31 is obtained. As shown in FIG. 31, the cat, which is the subject, is located substantially at the center in the extracted image 513 having an aspect ratio of 16: 9. The extracted image 513 can be said to be an image with a good composition as an image shot and displayed in the panoramic mode.

  The method of extracting information such as luminance, color, and edge from the captured image is described in detail in, for example, “Laurent Itti, Christof Koch, and Ernst Niebur,“ A Model of Saliency-Based Visual Attention for Rapid Scene Analysis ”. Has been.

  Returning to the flowchart of FIG. 29, in step S115, the combining unit 436 combines the cut-out image from the cut-out unit 434 and the extracted image from the extraction unit 435 so that the positions of the subjects match, and combines the combined image. Is supplied to the display control unit 437.

  In step S116, the display control unit 437 causes the display unit 438 to display the combined image supplied from the combining unit 436.

  FIG. 32 shows an example of a composite image displayed on the display unit 438.

  In FIG. 32, the display area of the display unit 438 of the digital camera 411 has an aspect ratio of 4: 3. In the panorama mode, the display control unit 437 displays a black image (so-called black belt) in the upper region and the lower region of the display region of the display unit 438 so that a display region with an aspect ratio of 16: 9 is displayed. Form.

  As shown in FIG. 32, the display control unit 437 displays a portion corresponding to the cut-out image in the composite image in the display area of the display unit 438 having an aspect ratio of 16: 9. In addition, the display control unit 437 causes the display unit 438 to display a frame indicated by a broken line in the drawing so as to emphasize a portion corresponding to the extracted image in the composite image. At this time, of the portion corresponding to the extracted image in the composite image, the portion covering the black belt (upper side) of the display area of the display unit 438 is displayed as a composite image with low luminance. Note that the extracted image displayed on the display unit 438 may be displayed so that the user can confirm the display range of the extracted image in addition to being emphasized by a frame as shown in FIG.

  According to the above processing, in the panoramic mode of a digital camera having only one imaging system, the image of the composition that the user is going to shoot and the image of the composition in which the subject with high attention is located in the center are combined. Since it can be displayed, the user can confirm a composition more appropriate than the composition to be photographed (recorded).

  In the above-described processing, the cutout image and the extracted image are displayed together on the display unit 438. However, the display area of the display unit 438 is displayed so that the captured image and the extracted image are displayed together. The luminance of the portion covering the black belt may be lowered.

  In the above, the processing for displaying the image of the composition that the user is going to shoot in the panorama mode and the image of the composition in which the subject with high attention is located at the center has been described. You may make it record the image of the composition which it is going to image | photograph, and the image of the composition in which the high attention subject is located in the center.

[Digital camera image recording]
Here, the image recording process of the digital camera 411 in FIG. 28 will be described with reference to the flowchart in FIG. Note that the image recording process in FIG. 33 is started when the operation input unit 439 receives an operation for selecting the panorama mode as the imaging mode by the user.

  Note that the processing of steps S211 to S214 in the flowchart of FIG. 33 is performed in that the cut image is supplied to the display control unit 437 and the recording control unit 440 in step S213, and the extracted image is supplied to the recording control unit 440 in step S214. Except for the point supplied, it is basically the same as the processing of steps S111 to S114 in the flowchart of FIG.

  In step S215, the display control unit 437 causes the display unit 438 to display the cut-out image supplied from the cut-out unit 434. More specifically, the display control unit 437 displays the cut-out image in the display area with an aspect ratio of 16: 9 in the display unit 438 as described with reference to FIG. At this time, in the display area of the display unit 438, black bands are displayed in the upper area and the lower area.

  In step S216, the recording control unit 440 determines whether or not the shutter button as the operation input unit 439 has been pressed.

  If it is determined in step S216 that the shutter button has not been pressed, the process returns to step S211 and the subsequent processes are repeated.

  On the other hand, if it is determined in step S216 that the shutter button has been pressed, that is, if a signal indicating that the shutter button has been pressed is supplied from the operation input unit 440 to the recording control unit 440, the process proceeds to step S217. move on.

  In step S217, the recording control unit 440 causes the recording unit 441 to record the extracted image from the extracting unit 434 and the extracted image from the extracting unit 435 based on a signal from the operation input unit 440. For example, the cutout image 512 described in FIG. 30 and the extracted image 513 described in FIG. 31 are recorded in the recording unit 441.

  In a conventional digital camera, when an image such as the cut-out image 512 described in FIG. 30 is an image of a composition that a user unfamiliar with shooting is trying to shoot in the panorama mode, the recorded image is also shown in FIG. An image similar to the cut-out image 512 described in 30 is obtained. In this case, it is not possible to obtain an image with a better composition by generating a portion of the subject that is not included in the recorded image (has been missing).

  According to the above processing, since the cutout image and the extracted image can be recorded in the panorama mode, the image of the composition taken by the user unfamiliar with photography, and the image of the composition more appropriate for the composition taken by the user Can be recorded.

  In addition, since an image of a composition photographed by the user and an image of a composition more appropriate than the composition photographed by the user are recorded, the user can compare the two images, and help the user improve the photographing technique. It becomes.

  In the processing described above, each of the cut-out image and the extracted image is recorded in the recording unit 441. However, the composite image obtained by combining the cut-out image and the extracted image described in the flowchart of FIG. 29 is recorded in the recording unit 441. You may make it make it.

  FIG. 34 shows an example of a composite image recorded in the recording unit 441.

  In the composite image 601 in FIG. 34, the part indicated by the alternate long and short dash line corresponds to the cut-out image, and the part indicated by the broken line corresponds to the extracted image. Here, the composite image 601 is considered divided into three regions 611 to 613. At this time, the region 611 is a portion that does not exist in the cutout image but exists in the extracted image, the region 612 is a portion that exists in both the cutout image and the extracted image, and the region 613 exists in the cutout image. This is a portion that does not exist in the extracted image.

  Therefore, when a signal for instructing the reproduction (display) of the clipped image is supplied from the operation input unit 439 by the user's operation, the recording control unit 440 reads out the region 612 and the region 613 in the composite image 601. What is necessary is just to supply to the display control part 437. Further, when a signal for instructing the reproduction of the extracted image is supplied from the operation input unit 439 by the user's operation, the recording control unit 440 reads out the area 611 and the area 612 in the composite image 601 and displays them. 437 may be supplied.

  As described above, in consideration of reproducing (displaying) each of the cutout image and the extracted image, it is only necessary to record one composite image without recording the cutout image and the extracted image in the recording unit 441. Therefore, the recording capacity of the recording unit 441 can be reduced.

  When the composite image 601 is not generated, the image corresponding to the region 611 and the cutout image may be recorded together, or the extracted image and the image corresponding to the region 613 may be recorded together. It may be.

  In the processing described above, the extracted image is recorded when the shutter button as the operation input unit 439 is operated. For example, the composition of the extracted image is a three-part composition or a horizontal line prepared in advance. The extracted image may be recorded when matching with a composition pattern such as composition or radiation composition.

  Thus, an image with a better composition can be recorded without the user operating the shutter button.

  Further, in the above-described processing, the image processing unit 433 performs predetermined image processing on the image data obtained from the entire image sensor of the imaging unit 432, but the shutter button as the operation input unit 439 is used. Until the shutter button is operated so that image processing is performed on the image data obtained from the image sensor corresponding to the region 313 of the image sensor 301 in FIG. Image processing may be performed on the image data obtained from the above.

  Thereby, it is possible to reduce the processing load of the image processing unit 433 until the shutter button is operated.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 35 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are connected to each other by a bus 904.

  An input / output interface 905 is further connected to the bus 904. The input / output interface 905 includes an input unit 906 made up of a keyboard, mouse, microphone, etc., an output unit 907 made up of a display, a speaker, etc., a storage unit 908 made up of a hard disk, nonvolatile memory, etc., and a communication unit 909 made up of a network interface. A drive 910 for driving a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 901 loads the program stored in the storage unit 908 to the RAM 903 via the input / output interface 905 and the bus 904 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 901) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 911 which is a package medium including a memory or the like, or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 908 via the input / output interface 905 by attaching the removable medium 911 to the drive 910. The program can be received by the communication unit 909 via a wired or wireless transmission medium and installed in the storage unit 908. In addition, the program can be installed in the ROM 902 or the storage unit 908 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 11 Digital camera, 31 Main lens, 32 Sub lens, 51 Imaging part, 52 Distortion correction part, 53 Optical axis correction part, 54 Image processing part, 55 Imaging part, 56 Image processing part, 57 Composition part, 58 Display part, 121 Digital camera, 151 Object detection unit, 152 Image processing unit, 153 Composition unit, 221 Digital camera, 251 Composition analysis unit, 252 Recommended composition extraction unit, 253 Image processing unit, 254 Composition unit

Claims (9)

At least one of brightness, color strength, hue, and sharpness of the second image so that the image quality of the second image having a different angle of view from the first image is different from the image quality of the first image. First image quality adjusting means for adjusting any one of them;
Object detection means for detecting a moving object in the second image;
The subject of the first image and the second image is added to the second image in which at least one of brightness, color density, hue, and sharpness has been adjusted by the first image quality adjusting unit. An image processing apparatus comprising: a combining unit that combines the first image and an object image of the region of the object in the second image detected by the object detection unit so that the positions match. The image processing apparatus according to claim 1, further comprising display means for displaying a synthesized image synthesized by the synthesizing means. The image quality of a region of the object in the previous SL object detecting means by said second detection by image, so as to vary the quality of the second image, the brightness of regions of the object in the second image A second image quality adjusting means for adjusting at least one of the image quality, the color density, the color shade, and the sharpness;
The synthesizing unit adjusts brightness, color strength, hue, and sharpness by the first image and the second image quality adjusting unit so that the position of the subject coincides with the second image. the image processing apparatus according to claim 1 or 2 at least one is synthesizing the object image region of the object in the adjusted second image. Composition analysis means for analyzing the composition of the second image;
Composition extracting means for extracting, from the second image, a composition having an angle of view different from the angle of view of the first image based on the composition analyzed by the composition analyzing means;
The composition means synthesizes the first image and an extracted image of the composition extracted by the composition extraction means so that the position of the subject coincides with the second image. Image processing device. The first image quality adjusting unit lowers the luminance, saturation, or brightness of the second image so that the image quality of the second image is lower than the image quality of the first image. Image processing device. Adjusting at least one of brightness, color density, hue, and sharpness of the first image so that the image quality of the first image is different from the image quality of the second image; The image processing apparatus according to claim 1, further comprising: 2 image quality adjusting means. The second image quality adjusting means, the image quality of the first image so as to increase than the quality of the second image, according to claim 6 to improve the luminance of the first image, saturation or brightness, Image processing device. At least one of brightness, color strength, hue, and sharpness of the second image so that the image quality of the second image having a different angle of view from the first image is different from the image quality of the first image. An image quality adjustment step for adjusting any one of them,
An object detection step of detecting a moving object in the second image;
The position of the subject of the first image and the second image coincides with the second image in which at least one of brightness, color density, hue, and sharpness has been adjusted in the image quality adjustment step. And a synthesis step of synthesizing the first image and an object image of the region of the object in the second image detected in the object detection step . At least one of brightness, color strength, hue, and sharpness of the second image so that the image quality of the second image having a different angle of view from the first image is different from the image quality of the first image. An image quality adjustment step for adjusting any one of them,
An object detection step of detecting a moving object in the second image;
The position of the subject of the first image and the second image coincides with the second image in which at least one of brightness, color density, hue, and sharpness has been adjusted in the image quality adjustment step. As described above, a program that causes a computer to execute a process including the first image and a synthesis step of synthesizing an object image of the region of the object in the second image detected in the object detection step .

Images