JPH1169288A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To speedily generate a panoramic still image by providing a panorama generating mans for generating the panoramic still image by joining images selected by an image-selecting means for selecting at least tow images among plural images from outside obtained by an image obtaining means. SOLUTION: An image-obtaining means 10 obtains plural images which are equivalent to a single frame of animation in a form compressed by an inter- frame coding system or an in-frame coding system, or obtains the image of a still image. The image-selecting means 11 selecting plural images among these images selects the image compressed by the in-frame coding system or an image obtained at each lapse of a previously decided time. In addition, the panorama-generating means 12 generates a panoramic still image, by joining the images selected by the means 11. Thereby, the panoramic still image high in quality is speedily generated while improving processing efficiency at the time of expansion.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing apparatus for generating a panoramic still image by joining a plurality of images.

[0002]

2. Description of the Related Art Conventionally, an apparatus for generating a single panoramic still image from a moving image obtained by photographing a wide range of subjects by panning is disclosed in Japanese Patent Application Laid-Open Nos. Hei 6-284321 and Hei 8-307770. There is a panoramic still image creation device like this. JP-A-6-28432
Japanese Patent Laid-Open No. 1 discloses a technique for superimposing continuous images using a motion vector of an image corresponding to each frame of a moving image captured by a camera-integrated video tape recorder or the like.

[0003] Japanese Patent Application Laid-Open No. 8-307770 discloses a technique for generating a panoramic still image by correcting image distortion caused by a change in distance to a subject due to aberration or the like.

[0004]

In these prior arts, an operator can instruct a range for generating a panoramic still image from a captured moving image, but all the images within such a range can be specified. Is used to generate a panoramic still image. Therefore, when panning is performed at a low speed, there is a problem that the overlapping portion increases and processing efficiency is poor.

[0005] Further, since it is not assumed that photographing conditions such as exposure and focal length change during moving image photographing, joints may not be performed or joints may be conspicuous. Therefore, the invention according to claims 1 to 6 aims to provide an image processing apparatus for quickly generating a panoramic still image, and the inventions according to claims 7 to 9 provide an image processing apparatus for generating a panoramic still image. An object of the present invention is to provide an image processing apparatus that performs the processing with high accuracy.

[0006]

FIG. 1 is a block diagram showing the principle of the first to fourth aspects of the present invention. According to the first aspect of the present invention, there is provided an image acquiring unit for acquiring a plurality of externally provided images, and an image selecting unit for selecting at least two images among the plurality of images acquired by the image acquiring unit. And a panorama generation unit 12 that joins the images selected by the image selection unit 11 to generate a panoramic still image.

According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the image acquiring means 10 converts an image corresponding to one frame of a moving image by an inter-frame encoding method or an intra-frame encoding method. The image is obtained in a compressed format, and the image selecting unit 11 selects an image compressed by the intra-frame encoding method from among the plurality of images obtained by the image obtaining unit 10.

According to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the image obtaining means 10 obtains an image corresponding to one frame of a moving image or a still image.
The image selection unit 11 selects a still image from a plurality of images acquired by the image acquisition unit 10. According to a fourth aspect of the present invention, in the image processing apparatus according to the first aspect, the image selecting unit 11 selects an image acquired through the image acquiring unit 10 every time a predetermined time elapses. It is characterized by the following.

FIG. 2 is a block diagram showing the principle of the present invention. According to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect, a photographing direction acquisition in which a photographing direction when the image is generated is associated with each image acquired by the image acquiring unit 10. The image selecting means 11 is provided with means for selecting an image in which the photographing direction acquired by the photographing direction acquiring means 14 indicates a predetermined direction.

FIG. 3 is a block diagram showing the principle of the present invention. According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, the angle-of-view acquisition unit 16 that calculates or acquires the angle of view of the image in association with each image acquired by the image acquisition unit 10 is provided. Provided, image selection means 1
1 is characterized in that the direction at the time of selecting an image is determined based on the angle of view calculated or obtained by the angle-of-view obtaining means 16.

FIG. 4 is a block diagram showing the principle of the present invention. According to a seventh aspect of the present invention, a panoramic still image is obtained by splicing a plurality of images acquired through the image acquiring unit and acquiring a plurality of images supplied from the outside. And a panorama generation means 21 for generating the image data. The image processing apparatus includes: A panorama generation unit 21 is provided with a size conversion unit 22 for enlarging or reducing, and the panorama generation unit 21 generates a panoramic still image by joining images enlarged or reduced by the size conversion unit 22.

FIG. 5 is a block diagram showing the principle of the present invention. According to an eighth aspect of the present invention, in the image processing apparatus according to the seventh aspect, there is provided a magnification acquiring unit 24 for calculating or acquiring a photographing magnification of the image in association with each image acquired by the image acquiring unit 20. The size conversion unit 22 enlarges or reduces an image based on the photographing magnification calculated or acquired by the magnification acquisition unit 24.

FIG. 6 is a block diagram showing the principle of the ninth aspect of the present invention. According to a ninth aspect of the present invention, a panoramic still image is obtained by splicing a plurality of images acquired via the image acquiring unit 30 and a plurality of images acquired via the image acquiring unit 30. A panorama generating means 31 for generating a brightness, a brightness detecting means 32 for detecting a difference in brightness near an overlapping portion or a seam of an image joined by the panoramic generating means 31, Brightness conversion means 33 for converting the brightness of the image into a predetermined brightness based on the difference in brightness detected by the brightness detection means 32,
The panorama generation means 31 generates a panoramic still image by joining the images whose brightness has been converted by the brightness conversion means 33.

(Operation) In the image processing apparatus according to the first aspect of the present invention, the image obtaining means 10 obtains a plurality of externally applied images, and the image selecting means 11 obtains the plurality of images obtained by the image obtaining means 10. Of at least two images are selected. The panorama generating means 12
Is, for the image selected by the image selecting means 11,
The panorama still image is generated by performing the joining while allowing the overlap.

Therefore, the processing efficiency when generating a panoramic still image can be improved as compared with the case where all the images acquired through the image acquiring means 10 are joined. In the image processing apparatus according to the second aspect of the present invention, the image obtaining means 10 obtains an image corresponding to one frame of a moving image in a format compressed by an inter-frame coding method or an intra-frame coding method.

The image selecting means 11 selects an image compressed by the intra-frame coding method from among the plurality of images obtained by the image obtaining means 10. The panorama generation means 12 generates a panoramic still image by joining the images selected in this way while allowing overlap. Therefore, compared to the case where an inter-coded image is selected, deterioration of the image due to the motion compensation processing and the like is suppressed, so that a high-quality panoramic still image can be generated.

Also, since an inter-coded image needs to be expanded using the preceding and succeeding frames, an intra-coded image can be easily expanded without referring to other frames. When a panoramic still image is generated only with an intra-frame encoded image, processing efficiency at the time of decompression is improved, and a panoramic still image can be generated quickly.

In the image processing apparatus according to the third aspect of the present invention, the image obtaining means 10 obtains an image corresponding to one frame of a moving image or a still image. Further, the image selecting unit 11 selects a still image from the plurality of images acquired by the image acquiring unit 10. Therefore, since a still image taken at a predetermined interval during shooting of a moving image or a still image intentionally taken by an operator can be joined, a high-quality panoramic still image can be obtained. Can be generated.

In the image processing apparatus according to a fourth aspect of the present invention, the image selecting means 11 selects an image obtained through the image obtaining means 10 every time a predetermined time elapses. That is, since images to be joined can be selected at intervals based on a predetermined time, a panoramic still image can be generated quickly and reliably.

In the image processing apparatus according to the fifth aspect of the present invention, the photographing direction acquiring means 14 acquires the photographing direction when the image was generated, in association with each image acquired by the image acquiring means 10. Further, the image selecting means 11
An image in which the shooting direction obtained by the shooting direction obtaining means 14 indicates a predetermined direction is selected. In other words, from an image in which the photographer changes the shooting direction and shoots the surroundings,
By selecting an image indicating a predetermined shooting direction, a panoramic still image can be quickly generated.

In the image processing apparatus according to the sixth aspect of the present invention, the angle-of-view obtaining means 16 calculates or obtains the angle of view of the image in association with each image obtained by the image obtaining means 10. Further, the image selecting unit 11 determines a direction at the time of selecting an image based on the angle of view calculated or obtained by the angle of view obtaining unit 16. That is, since the photographer changes the shooting direction to associate the angle of view of the image of the surroundings with the shooting direction and selects the image to be joined, the joining accuracy is improved and the panoramic still image is reliably obtained. Can be generated.

In the image processing apparatus according to the present invention, the image obtaining means 20 obtains a plurality of externally supplied images, and the panorama generating means 21 obtains the plurality of images obtained through the image obtaining means 20. The panoramic still image is generated by performing the splicing with respect to the image of FIG. The size conversion unit 22 enlarges or reduces the image so that the size of the same target object shown in each of the images that are adjacently joined by the panorama generation unit 21 matches.

Therefore, even when the size of the object shown in the image to be joined is different, a still panoramic image can be reliably generated. Claim 8
In the image processing device according to the invention described in (1), the magnification obtaining unit 24 calculates or obtains the imaging magnification of the image in association with each image obtained by the image obtaining unit 20. The size conversion unit 22 enlarges or reduces the image based on the photographing magnification calculated or acquired by the magnification acquisition unit 24.

That is, by acquiring the photographing magnification, it is possible to easily match the size of the same object shown in a plurality of images to be joined. Therefore, even when the photographing magnification of the image to be joined changes, a panoramic still image can be reliably generated. In the image processing apparatus according to the ninth aspect, the image acquisition unit 30 acquires a plurality of images supplied from outside. The panorama generating unit 31 generates a panoramic still image by joining the plurality of images acquired through the image acquiring unit 30 while allowing overlap.

The brightness detecting means 32 detects a difference in brightness in the vicinity of the overlapping portion or the seam of the images joined by the panorama generating means 31, and the brightness converting means 33 controls the brightness detecting means 32. The brightness of the image is adjusted to a predetermined brightness based on the difference in the detected brightness. Therefore, even when the overlapping portions of the images to be joined or the brightness in the vicinity of the joint is different, a panoramic still image can be generated without making the joint stand out.

[0026]

Embodiments of the present invention will be described below in detail with reference to the drawings. Note that an embodiment corresponding to the first to ninth aspects of the present invention will be described using an electronic camera which is an embodiment of an image processing apparatus.

(First Embodiment) FIG. 7 is a view showing an appearance of an electronic camera, and FIG. 8 is a functional block diagram of an embodiment corresponding to the first and second aspects of the present invention. 8, the control unit 40 includes an imaging unit 44, a disk drive 46, a display processing unit 48,
Monitor 50, touch panel 52, memory 54, compression unit 5
6, the expansion unit 58, the motion vector detection unit 60, and the correlation detection unit 62 are connected. The disk drive 46 is loaded with a magnetic disk 64.

As for the correspondence between the principle block diagram shown in FIG. 1 and the present embodiment, the image acquisition means 10 corresponds to the disk drive 46 and the magnetic disk 64, and the image selection means 11 corresponds to the image data of the control section 40. , The panorama generating means 12 includes a memory 54, a motion vector detecting section 60, a correlation detecting section 62, and a control section 40.
It supports the function of joining images.

In this embodiment, for the sake of simplicity, moving picture data of 360 degrees around the picture taken by panning by the photographer is compressed in a format applicable to MPEG and stored in the magnetic disk 64 in advance as a moving picture file. It has been done. Here, the moving image file includes a header area including additional information such as a shooting date and time and a total number of frames, an image area including moving image data, and a thumbnail area including image data of a thumbnail image described later. And A still image file stored together with the moving image file on the magnetic disk 64 is composed of a header area and an image area.

FIG. 9 is an operation flowchart of an embodiment according to the first and second aspects of the present invention. Less than,
The operation of the embodiment according to the first and second aspects of the present invention will be described with reference to FIGS.

When the main power is turned on, the control unit 40 instructs the display processing unit 48 to display an operation screen including a thumbnail image as shown in FIG. 10 (S1 in FIG. 9). FIG.
0, rewind button 100, fast forward button 10
1, a stop button 102, a panorama button 103, a reverse play button 104, a play button 105, and the like indicate operation buttons. Thumbnail images such as "thumbnail A" and "thumbnail B" surrounded by rectangles are displayed. Indicates the status. A thumbnail image surrounded by double lines or highlights indicates that the image is a processing target such as reproduction, and a thick thumbnail display indicates a moving image.

The display processing unit 48 includes a disk drive 46
The image data of the thumbnail image is read out from the thumbnail area of each moving image file in the magnetic disk 64 via the, and stored in a predetermined position of an internal frame memory (not shown) together with the image data indicating the thickness. Further, the display processing unit 48 reads out still image data from the still image file in the magnetic disk 64 via the disk drive 46, performs pixel density conversion, and stores it as a thumbnail image in the frame memory.

Further, the display processing section 48 overlays the image data stored in the frame memory and the image data corresponding to the operation buttons as described above, and gives the overlay data to the monitor 50. On the monitor 50, the image data thus provided is displayed as an operation screen.

While the operation screen is being displayed, the control unit 40 determines whether or not an instruction to generate a panoramic still image has been given by an external operation received via the touch panel 52 (S2 in FIG. 9). That is, when a thumbnail image is selected (here, “thumbnail D” is selected) and the panorama button 103 is externally operated, it is recognized that generation of a panoramic still image has been instructed, and the disk drive A moving image file corresponding to the thumbnail image is opened via 46.

The control unit 40 also controls the disk drive 4
6, image data (intra-frame encoded image data) of a frame corresponding to an I picture is sequentially read from such a moving image file based on a predetermined GOP (Group Of Picture) structure (S3 in FIG. 9). ). The image data read in this way is
The data is expanded by the expansion unit 58 and sequentially stored in the memory 54.

The control unit 40 detects a motion vector of an image in a frame unit stored in the memory 54 via the motion vector detection unit 60 (S4 in FIG. 9). The control unit 40
Calculates the amount of shift between frames based on the motion vectors detected in this way, and obtains the overlapping portion of the image of each frame (S5 in FIG. 9). Further, the control unit 40 obtains a correlation for each vertical line in the overlapping portion of each frame via the correlation detection unit 62 (S6 in FIG. 9).

Here, of two adjacent frames,
The correlation between pixels of one line in the vertical direction at the center of the overlapping portion of one frame and pixels of each line near the center of the overlapping portion of the other frame is determined. The control unit 40 joins the images with the line showing the strongest correlation among the lines for which the correlation has been obtained in this manner (S7 in FIG. 9).

For example, as shown in FIG.
When the image data A and B in frame units are stored in the image data, the image data A is copied to another area, and the “line indicating the strongest correlation” of the image data B is added to the image data A thus copied. Are superimposed on the corresponding image data from to. The control unit 40 determines whether or not the splicing of the images of all the frames in the memory 54 is completed (S8 in FIG. 9), and continues until the recognizing that the generation of the panoramic still image has been completed by such a determination. Alignment (Fig. 9
Step S4 and subsequent steps) are repeated. Note that the first frame and the last frame read from the moving image file are also joined by a line showing the strongest correlation in the overlapping portion.

The panoramic still image generated in this way is compressed via the compression unit 56 and written as an image file on the magnetic disk 64 via the disk drive 46 (S9 in FIG. 9). That is, according to the present embodiment, a panoramic still image can be quickly and accurately generated by selecting only intra-frame coded frames without selecting predictively coded frames.

By the way, a thumbnail image indicating an image file corresponding to a panoramic still image may be displayed in a curved shape like a "thumbnail E" shown in FIG.
That is, a rectangular and thick thumbnail image indicates a still image, a rectangular and thick thumbnail image indicates a moving image, and a curved thumbnail image indicates a panoramic still image.

Therefore, by displaying in this manner, the operator can quickly recognize the shape (still image / moving image / panoramic still image) of the data of the image file stored in the magnetic disk 64. . Also,
When such a “thumbnail E” is selected and the panorama button 103 is operated externally, a part of the panorama still image is displayed in a double-lined frame as shown in FIG. When the fast-forward button 101 (or the reverse playback button 104 and the playback button 105) is externally operated, a panoramic still image may be scroll-displayed.

Here, by using the operation member of the existing electronic camera as the panorama button 103, a part of the panorama still image can be displayed and scrolled without changing the hardware. In the scroll display, the end of the image data and the leading end of the image data can be displayed continuously.

(Second Embodiment) FIG. 14 is an operation flowchart of an embodiment corresponding to the first and third aspects of the present invention. The feature of the present embodiment is that the control unit 40
In the procedure of the processing, the hardware configuration,
Since this is the same as the functional block diagram of the embodiment shown in FIG. 7, illustration is omitted here.

Further, it is assumed that the electronic camera to which the present embodiment is applied can capture a still image while capturing a moving image. When such shooting is performed, the still image data is compressed in a format applicable to JPEG and stored on the magnetic disk 64 as a still image file. on the other hand,
The moving image data is divided when a still image is captured, and stored on the magnetic disk 64 as a plurality of moving image files compressed in a format applicable to MPEG.

Here, these still image files and moving image files are provided to the operator as a single moving image file, and the header area of each file includes
The “start address of the preceding file” and the “start address of the following file” are stored, and the fact that these files have a bidirectional chain structure is stored.

Hereinafter, referring to FIGS. 7 and 14, claim 1 will be described.
The operation of the embodiment corresponding to the inventions described in 3 and 3 will be described. When the main power is turned on, the control unit 40 instructs the display processing unit 48 to display an operation screen including a thumbnail image as in the above-described embodiment (S1 in FIG. 14). The display processing unit 48, as in the embodiment described above,
4 are stored in an internal frame memory. The display processing unit 48 overlays the image data thus stored in the frame memory and the image data corresponding to the operation buttons shown in FIG.

On the monitor 50, the image data given in this way is displayed as an operation screen. Control unit 40
In the state where the operation screen is displayed, similarly to the above-described embodiment, it is determined whether or not the generation of the panoramic still image is instructed by an external operation received via the touch panel 52 (S2 in FIG. 14).

As in the above-described embodiment, the control section 40 selects the thumbnail image and
Is externally operated, it is recognized that generation of a panoramic still image has been instructed, and a moving image file corresponding to the selected thumbnail image is opened. In this embodiment, FIG.
As shown in FIG. 0, “thumbnail D” is selected, and as a moving image file corresponding to “thumbnail D”, the surroundings 3
It is assumed that image data corresponding to a moving image extending over 60 degrees and a still image shot during shooting of such a moving image is stored in the above-described format.

The control unit 40 refers to the header area of the moving image file corresponding to "thumbnail D" and determines whether or not the moving image file is composed of a plurality of files having a bidirectional chain structure. . That is, it is determined whether still image data is recorded together with the moving image data (S3 in FIG. 14). When the control unit 40 recognizes that the still image data is recorded by such a determination (YES side of S3 in FIG. 14), the “subsequent” stored in the header area of each file via the disk drive 46. The still image data is read from the still image file by sequentially referring to the "head address of the file" (S4 in FIG. 14).

The image data read in this manner is expanded by the expansion unit 58 and stored in the memory 54 in sequence. When the control unit 40 recognizes that the still image data is not recorded (NO in FIG. 14S3), the control unit 40 sequentially reads out the intra-frame encoded image data as in the above-described embodiment ( 9S3 and subsequent processes).

When the still image data is read from the still image file and stored in the memory 54, the control unit 40 detects a motion vector between the still images via the motion vector detecting unit 60 (see FIG. 4). 14S5). Further, the control unit 40 calculates the amount of deviation between the still images based on the motion vector detected in this way, and obtains the overlapping portion of each still image (S6 in FIG. 14).

Further, the control unit 40 obtains a correlation for each vertical line in the overlapping portion of each still image via the correlation detecting unit 62 (S7 in FIG. 14). Here, of two adjacent still images, one line of pixels in the vertical direction at the center of the overlapping portion of one still image, and pixels of each line near the center of the overlapping portion of the other still image, Is determined.

The control unit 40 joins the images on the line showing the strongest correlation among the lines for which the correlation has been obtained, in the same manner as in the above-described embodiment (S8 in FIG. 14).
Further, the control unit 40 determines whether or not the splicing of all the images in the memory 54 has been completed (S9 in FIG. 14). Until such a determination is made, it is determined that the panoramic still image generation has been completed. Repeat the alignment. Note that, for the first still image and the last still image, the image data is deleted at the line showing the strongest correlation.

The panoramic still image generated in this way is compressed via the compression section 56 and written to the magnetic disk 64 as an image file via the disk drive 46 (S10 in FIG. 14). That is, according to the present embodiment, it is possible to generate a high-quality panoramic still image by using a still image shot during shooting of a moving image.

In the present embodiment, an electronic camera in which a still image is shot while a moving image is shot is applied.
For example, when the photographer shoots a panorama button
An electronic camera that has a function of positively selecting a frame for generating a panoramic still image by, for example, externally operating the camera, and that records the selected frame as a still image may be applied.

When the panorama button 103 is externally operated during the shooting of a moving image, the shooting mode is automatically switched, and the shooting is performed while the shooting magnification, exposure level, lens aperture, and the like are kept constant. By doing so, joining can be performed smoothly. Furthermore, in the present embodiment, a plurality of moving image files and still image files have a bidirectional chain structure. However, the present invention is not limited to image data having such a structure. Image data stored in any format may be used as long as the data can be handled as continuous image data.

(Third Embodiment) FIG. 15 is an operation flowchart of an embodiment corresponding to the first and fourth aspects of the present invention. The feature of the present embodiment is that the control unit 40
In the procedure of the processing, the hardware configuration,
Since this is the same as the functional block diagram of the embodiment shown in FIG. 7, illustration is omitted here.

In this embodiment, the photographer performs panning while keeping the zoom constant, and the surroundings 3
It is assumed that moving image data over 60 degrees is stored on the magnetic disk 64 as a moving image file. It should be noted that information on the focal length corresponding to the zoom kept constant as described above is stored in the header area of the moving image file.

Further, the angle of view generally indicates the angle in the diagonal direction of the screen. In this embodiment and the later-described embodiments,
For simplicity, the description will be made with the horizontal angle of the screen as the angle of view. The operation of the embodiment according to the first and fourth aspects of the present invention will be described below with reference to FIGS.

When the main power is turned on, the control unit 40 instructs the display processing unit 48 to display an operation screen including a thumbnail image as in the above-described embodiments (S1 in FIG. 15).
The display processing unit 48 stores the thumbnail images of the moving image file and the still image file in the magnetic disk 64 in the internal frame memory, as in the above-described embodiments. The display processing unit 48 overlays the image data thus stored in the frame memory and the image data corresponding to the operation buttons shown in FIG.
Give to 0.

On the monitor 50, the image data given in this way is displayed as an operation screen. Control unit 40
In the state where the operation screen is displayed, it is determined whether or not generation of a panoramic still image has been instructed by an external operation received via the touch panel 52, similarly to the above-described embodiments (S2 in FIG. 15). .

When recognizing that the generation of the panoramic still image has been instructed, the control unit 40 opens the moving image file corresponding to the selected thumbnail image, as in the above-described embodiments. The control unit 40 obtains information on the focal length from the header area of the moving image file thus opened, obtains the angle of view of the moving image data in frame units (S3 in FIG. 15), and calculates the total recording time from the header area. Acquire (Fig. 1
5S4).

Here, the control unit 40 selects a frame to be used for generating a panoramic still image. In the present embodiment, the control unit 40 selects a frame at regular time intervals from the beginning of the moving image file. Note that such a time interval is referred to as a “frame extraction interval”.

The control unit 40 calculates a "frame extraction interval" based on the angle of view and the total recording time (S5 in FIG. 15).
For example, when the zoom is kept on the wide-angle side, the angle of view is 90 degrees, and the total recording time is 30 seconds, assuming that a panoramic still image is generated in 10 frames in consideration of the overlap of each frame, The “frame extraction interval” may be 3 seconds.

The control unit 40 reads out image data from the moving image file in frame units based on the calculated “frame extraction interval” (S6 in FIG. 15). The image data read in this way is expanded by the expansion unit 58 and stored in the memory 54 in sequence. Further, the control unit 40
Is stored in the memory 54 via the motion vector detection unit 60 in the same manner as in the first and second embodiments of the present invention.
Is detected (S7 in FIG. 15). The control unit 40 calculates the amount of deviation between frames based on the motion vector detected in this way, and obtains the overlapping portion of the image of each frame (FIG. 15S
8).

Further, the control unit 40 obtains a correlation for each vertical line in the overlapping portion of each frame via the correlation detection unit 62 (S9 in FIG. 15). Further, the control unit 40
Is the line showing the strongest correlation among the lines for which the correlation is obtained, and the images are joined in the same manner as in the above-described embodiments (S10 in FIG. 15). Further, the control unit 40 determines whether or not the splicing of all the images in the memory 54 has been completed (S11 in FIG. 15). Splicing is repeated in the same manner as in each embodiment described above.

The panoramic still image generated in this way is compressed via the compression section 56 and written as an image file on the magnetic disk 64 via the disk drive 46 (S12 in FIG. 15).

That is, according to the present embodiment, image data to be joined can be easily selected at a predetermined “frame extraction interval”, so that a panoramic still image can be quickly generated. . In each of the above-described embodiments, even when the electronic camera shakes up and down during shooting of a moving image, the shake can be detected by the motion vector, so that the joining can be performed smoothly. It is.

In each of the above embodiments, the surrounding area 36
Although a panoramic still image is generated from a moving image obtained by shooting through 0 degrees, a panoramic still image may be generated from a moving image obtained by shooting while the photographer moves. (Fourth Embodiment) FIG. 16 is a functional block diagram of an embodiment corresponding to the first, fifth, and ninth aspects of the present invention.

In the figure, the same functions as those in the functional block diagram shown in FIG. 7 are denoted by the same reference numerals.
The control unit 70 includes an imaging unit 72 via the system bus 42,
A disk drive 46, a display processing unit 48, a monitor 50,
Touch panel 52, memory 54, compression unit 56, expansion unit 5
8, connected to the correlation detection unit 62, the luminance level detection unit 74, and the image processing unit 76. The disk drive 46
Is mounted with a magnetic disk 64, and an imaging unit 72 is provided with an angular velocity sensor 78 and a focal length detection unit 80.

The angular velocity sensor 78 detects a change in the photographing direction of the electronic camera with reference to the azimuth at the start of photographing, and the focal length detecting unit 80 operates when a moving image is photographed by the photographer. The focal length is detected based on the zoom operation. The correspondence between the principle block diagrams shown in FIGS.
0 and 30 are the disk drive 46 and the magnetic disk 6
4, the image selection means 11 corresponds to the function of selecting image data of the control unit 70, and the panorama generation means 12, 2
Reference numerals 1 and 31 denote a memory 54, a correlation detector 62, and a controller 7
0 corresponding to the function of splicing images of the disk drive 46 and the magnetic disk 64
, And the angle-of-view obtaining means 16 includes a disk drive 46,
The size conversion unit 22 corresponds to the function of reducing the image data of the image processing unit 76, and the magnification obtaining unit 24 corresponds to the disk drive 46, the magnetic disk 64 and the magnetic disk 64. The brightness detection means 32 corresponds to the brightness level detection section 74, and the brightness conversion means 33 corresponds to the brightness level conversion function of the image processing section 76. .

FIG. 17 is an operation flowchart of an embodiment according to the first, fifth, and ninth aspects of the present invention. Hereinafter, the operation of the embodiment according to the first to fifth aspects of the present invention will be described with reference to FIGS. In addition,
In the present embodiment, for the sake of simplicity, it is assumed that moving image data of 360 degrees around the image captured by panning by the photographer is compressed in a format applicable to MPEG and stored in advance on the magnetic disk 64 as a moving image file. I do.

In the header area of the moving image file,
It is assumed that the amount of change in the imaging direction from the first frame (hereinafter, referred to as “imaging angle”) and information on the focal length are stored in association with each frame. When recognizing that the generation of the panoramic still image has been instructed in the same manner as in the above-described embodiments, the control unit 70 stores the moving image file corresponding to the selected thumbnail image in the disk drive 46.
Open through.

The control unit 70 reads out image data in frame units from the moving image file thus opened, decompresses it by the decompression unit 58, and sequentially stores it in the memory 54 (FIG. 17S).
1). Further, the control unit 70 reads out the information of the focal length from the header area of the moving image file, and calculates the photographing magnification of the frame having the shortest focal length (hereinafter, referred to as “wide-angle photographing magnification”) (S2 in FIG. 17).

Further, the control unit 70 instructs the image processing unit 76 to convert the photographing magnification of each frame into the "wide-angle photographing magnification" (S3 in FIG. 17). The image processing unit 76 calculates the photographing magnification by referring to the information of the focal length corresponding to each frame, and based on the ratio between the photographing magnification thus calculated and the “wide-angle photographing magnification”, the image data of the image data is frame-by-frame. Perform reduction. That is, by reducing the image data, the photographing magnification is converted to “wide-angle photographing magnification”.

Here, the control unit 70 selects a frame to be used when generating a panoramic still image. In the present embodiment, since such selection and splicing processes are repeatedly performed, the splicing reference is used. Is referred to as a “previous frame”. Also, a frame that is a candidate for joining with the “preceding frame” is referred to as a “subsequent frame”.

That is, in the initial state, the first frame is handled as a “preceding frame”. The “subsequent frame” for which the joining process has been completed is treated as a “preceding frame” in the subsequent joining process. The control unit 70 performs initialization (setting of initial values of β and K to be described later) related to the joining process (S4 in FIG. 17).

The control unit 70 calculates the angle of view θ1 of the “preceding frame” based on the information on the focal length (S 17 in FIG. 17).
Along with 5), a frame (corresponding to a “subsequent frame”) in which the difference in the “photographing angle” from the “preceding frame” indicates β (S6 in FIG. 17). Here, β is a value that defines the difference in “shooting angle”, and an arbitrary value (here, 60 degrees) is set at the time of initialization (S4 in FIG. 17).

The controller 70 calculates the angle of view θ2 of the “subsequent frame” selected in this way (S7 in FIG. 17), and calculates the angle of view θ.
1. It is determined whether or not the relational expression of (θ1 + θ2) / 2−β ≧ K holds between the angle of view θ2 and β (S8 in FIG. 17).
Note that K is a value equal to or greater than 0, and defines the angle of the overlapping portion between the “preceding frame” and the “subsequent frame”. here,
If “(θ1 + θ2) / 2−β” is less than 0, it indicates that the “preceding frame” and the “subsequent frame” do not overlap.

The control unit 70 corresponds to the case where the formula 1 does not hold (the case where the “preceding frame” and the “subsequent frame” do not overlap or the angle of the overlapping portion is less than K:
(NO side in FIG. 17S8), β-5 is substituted for β (FIG. 17S
9), a process of selecting a frame (a process corresponding to S5 in FIG. 17) is repeatedly performed. That is, the interval for selecting a frame used for generating a panoramic still image is narrowed.

On the other hand, the case where Expression 1 is satisfied (corresponding to the case where the angle of the overlapping portion is K or more: YES in FIG. 17 S8)
Side), the control unit 70 obtains an overlapping portion based on “(θ1 + θ2) / 2−β” (S10 in FIG. 17), and adjusts the brightness of the overlapping portion of each frame thus obtained by the brightness level detecting unit 74 and the image. Instruct processing unit 76 (S1 in FIG. 17)
1).

When instructed in this way, the luminance level detecting section 74 calculates the average luminance level of the overlapping portion in each frame. Further, the image processing unit 76 adjusts the luminance of the entire image data of the “subsequent frame” based on the difference in the average luminance level of the overlapping portion in each frame so as to maintain continuity with the “preceding frame”. . Control unit 70
When the brightness of the image data is adjusted in this way, the pixel of one vertical line at the center of the overlapping part of the “preceding frame” and the overlapping The correlation with the pixels on each line near the center of the portion is determined (S12 in FIG. 17).

The control unit 70 joins the images with the line showing the strongest correlation among the lines for which the correlation has been obtained in this way (S13 in FIG. 17). The control unit 70 determines whether or not the joining of the images over 360 degrees is completed (FIG. 1).
7S14), until such a determination is made that the generation of the panoramic still image has been completed, processing of frame selection and image joining including initialization (FIG. 17S
4) is repeated.

The panoramic still image generated in this way is compressed via the compression section 56 and written to the magnetic disk 64 as an image file via the disk drive 46 (S15 in FIG. 17). Therefore, according to the present embodiment, it is possible to reliably select the image to be joined while adjusting the angle of the overlapping portion between the frames, so that the panoramic still image can be quickly and accurately generated. it can.

In this embodiment, the angular velocity sensor 78 for detecting the “photographing angle” is provided. However, it is necessary to provide an angular velocity sensor for measuring the rotation angle with respect to the optical axis to measure the attitude of the electronic camera in detail. Thereby, blurring during panning can be corrected, and discontinuity of a panoramic still image can be reduced. In each of the above embodiments, the surroundings 3
Although the image data extending over 60 degrees is stored on the magnetic disk 64 as an image file, the present invention can be applied to an image file including image data of less than 360 degrees.

Further, in each of the above-described embodiments, the surrounding 3
A panoramic still image is generated by performing left and right splicing based on an image extending over 60 degrees. However, the present invention is not limited to such a panoramic still image, and a panoramic still image is formed by splicing images up and down. Can also be generated.

[0087]

As described above, according to the first aspect of the present invention, the processing efficiency when generating a panoramic still image is improved as compared with the case where all the images obtained through the image obtaining means are joined. can do.

According to the second aspect of the present invention, since the deterioration of the image due to the motion compensation processing or the like is suppressed as compared with the case where the inter-coded image is selected, a high-quality panoramic still image can be obtained. Can be generated. In addition, since the processing efficiency at the time of expansion is improved, a panoramic still image can be quickly generated. Further, claim 3
According to the invention described in (1), since a still image taken at a predetermined interval during the shooting of a moving image or a still image intentionally shot by an operator can be spliced, a high-quality A panoramic still image can be generated.

According to the fourth aspect of the present invention, an image to be joined can be selected at intervals based on a predetermined time, so that a panoramic still image can be quickly and reliably generated. Can be. Further, according to the fifth aspect of the present invention, the panoramic still image is quickly generated by the photographer changing the photographing direction and selecting an image indicating a predetermined photographing direction from the images photographed around the photographer. be able to.

According to the present invention, the photographer changes the photographing direction, associates the angle of view of the image photographed around it with the photographing direction, and selects the image to be spliced. The accuracy of alignment can be improved, and a panoramic still image can be reliably generated. Further, according to the seventh aspect of the present invention, a panoramic still image can be reliably generated even when the size of an object shown in an image to be joined is different.

According to the eighth aspect of the present invention, the size of the same object shown in a plurality of images to be joined can be easily matched by acquiring the photographing magnification. Even when the photographing magnification of the image to be spliced changes, a panoramic still image can be reliably generated. Further, according to the ninth aspect of the present invention, even when the brightness of the overlapping portion of the images to be joined or the brightness in the vicinity of the joint is different, a panoramic still image can be generated without making the joint stand out.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the principle of the present invention.

FIG. 2 is a principle block diagram of the invention according to claim 5;

FIG. 3 is a principle block diagram of the invention according to claim 6;

FIG. 4 is a principle block diagram of the invention according to claim 7;

FIG. 5 is a principle block diagram of the invention according to claim 8;

FIG. 6 is a principle block diagram of the invention according to claim 9;

FIG. 7 is a diagram illustrating an appearance of an electronic camera.

FIG. 8 is a functional block diagram of an embodiment according to the first and second aspects of the present invention.

FIG. 9 is an operation flowchart of an embodiment according to the first and second aspects of the present invention.

FIG. 10 is a diagram illustrating a display example of a monitor.

FIG. 11 is a diagram illustrating image joining.

FIG. 12 is a diagram illustrating a display example of a monitor.

FIG. 13 is a diagram illustrating a display example of a monitor.

FIG. 14 is an operation flowchart of an embodiment according to the first and third aspects of the present invention.

FIG. 15 is an operation flowchart of the embodiment according to the first and fourth aspects of the present invention.

FIG. 16 is a functional block diagram of an embodiment corresponding to the first, fifth, and ninth aspects of the present invention;

FIG. 17 is an operation flowchart of the embodiment corresponding to the first, fifth, and ninth aspects of the present invention;

[Explanation of symbols]

 10, 20, 30 Image acquisition means 11 Image selection means 12, 21, 31 Panorama generation means 14 Shooting direction acquisition means 16 Angle of view acquisition means 22 Size conversion means 24 Magnification acquisition means 32 Brightness detection means 33 Brightness conversion means 40 , 70 control unit 42 system bus 44, 72 imaging unit 46 disk drive 48 display processing unit 50 monitor 52 touch panel 54 memory 56 compression unit 58 decompression unit 60 motion vector detection unit 62 correlation detection unit 64 magnetic disk 74 brightness level detection unit 76 image Processing unit 78 Angular velocity sensor 80 Focal length detection unit 100 Rewind button 101 Fast forward button 102 Stop button 103 Panorama button 104 Reverse playback button 105 Play button

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification code FI H04N 5/262 G06F 15/66 470J 5/765 H04N 5/781 510F 5/781 7/13 Z 7/24

Claims (9)

[Claims] An image acquisition unit configured to acquire a plurality of images given from outside; and a plurality of images acquired by the image acquisition unit.
An image processing apparatus comprising: an image selection unit that selects at least two images; and a panorama generation unit that joins the images selected by the image selection unit to generate a panoramic still image. . 2. The image processing apparatus according to claim 1, wherein the image acquisition unit acquires an image corresponding to one frame of a moving image in a format compressed by an inter-frame encoding method or an intra-frame encoding method. The image selecting unit, among the plurality of images acquired by the image acquiring unit,
An image processing apparatus for selecting an image compressed by an intra-frame encoding method. 3. The image processing apparatus according to claim 1, wherein the image acquisition unit acquires an image corresponding to one frame of a moving image or an image of a still image, and the image selection unit acquires the image by the image acquisition unit. Of the multiple images acquired,
An image processing apparatus for selecting a still image. 4. The image processing apparatus according to claim 1, wherein said image selecting means selects an image obtained through said image obtaining means every time a predetermined time elapses. Image processing device. 5. The image processing apparatus according to claim 1, further comprising: a photographing direction acquiring unit that associates each image acquired by the image acquiring unit with a photographing direction when the image is generated, The image processing apparatus according to claim 1, wherein the image selecting unit selects an image in which the photographing direction acquired by the photographing direction acquiring unit indicates a predetermined direction. 6. The image processing apparatus according to claim 5, further comprising: an angle-of-view obtaining unit that calculates or obtains an angle of view of the image in association with each image obtained by the image obtaining unit; The image processing apparatus is characterized in that the means determines a direction at the time of selecting an image based on the angle of view calculated or obtained by the angle of view obtaining means. 7. An image acquisition unit for acquiring a plurality of images supplied from outside, and a panorama generation for joining a plurality of images acquired via the image acquisition unit to generate a panoramic still image And an image processing apparatus comprising:
An image processing apparatus comprising: a size conversion unit configured to enlarge or reduce the image; wherein the panorama generation unit generates a panoramic still image by joining images enlarged or reduced by the size conversion unit. . 8. The image processing apparatus according to claim 7, further comprising: a magnification acquiring unit that calculates or acquires a photographing magnification of the image in association with each image acquired by the image acquiring unit; The image processing apparatus is characterized in that the means enlarges or reduces the image based on the photographing magnification calculated or acquired by the magnification acquiring means. 9. An image acquisition unit for acquiring a plurality of images supplied from the outside, and a panorama generation for joining a plurality of images acquired via the image acquisition unit to generate a panoramic still image An image processing apparatus comprising: a brightness detection unit configured to detect a difference in brightness near an overlapping portion or a seam of an image to be joined by the panorama generation unit; Brightness conversion means for converting the brightness of the image to a predetermined brightness based on a difference in brightness, wherein the panorama generation means has a brightness converted by the brightness conversion means. An image processing apparatus for generating a panoramic still image by joining images.