JP2024098065A - Imaging apparatus - Google Patents

Abstract

To provide a technique capable of generating a panoramic image with high accuracy without recapturing an image from the beginning again.SOLUTION: An imaging apparatus includes an imaging unit that captures images of a subject scene in chronological order to generate a plurality of images, a display unit that displays the plurality of images, and an image processing unit that connects the plurality of images as a target image in chronological order to generate a composite image, and the image processing unit detects the amount of displacement of the target image relative to the composite image on the basis of image features of the target image and the composite image, connects the target image into the composite image according to the amount of displacement, and displays a warning indicating failure on the display unit along with the plurality of images when detection of the amount of displacement fails.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device.

Conventionally, a panoramic image is generated by capturing multiple images while panning a digital camera or the like, and stitching the images together in sequence. For example, Patent Document 1 discloses a technology for panoramic synthesis processing using frame images from a video.

JP 2000-244814 A

However, with conventional technology, if the amount of panning movement is too large to stitch the images together, the images must be re-captured from the beginning.

The object of the present invention is to provide a technology that can generate a panoramic image with high accuracy without having to recapture the image from the beginning.

In order to solve the above problems, the imaging device of the present invention includes an imaging unit that captures images of a subject scene in chronological order to generate multiple images, a display unit that displays the multiple images, and an image processing unit that connects each of the multiple images as a target image in chronological order to generate a composite image, the image processing unit detects the amount of displacement of the target image relative to the composite image based on image features of the target image and the composite image, connects the target image to the composite image according to the amount of displacement, and if detection of the amount of displacement fails, displays a warning indicating failure along with the multiple images on the display unit.

According to the present invention, it is possible to generate a panoramic image with high accuracy without recapturing the image from the beginning.

FIG. 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention; A flowchart showing the image processing operation of a digital camera according to an embodiment. FIG. 1 shows an example of a monitor display in panoramic mode. FIG. 2 shows an example of a monitor display in panoramic mode. FIG. 11 is a diagram showing another example of the display on the monitor in the panoramic mode. FIG. 11 is a diagram showing another example of the display on the monitor in the panoramic mode.

Figure 1 is a block diagram showing the configuration of a digital camera according to one embodiment of the present invention.

The digital camera of this embodiment is composed of an imaging optical system 11, an imaging element 12, an AFE 13, a CPU 14, a memory 15, a monitor 16, a recording interface (I/F) 17, an operation member 19, a release button 20, a touch panel 21, and a bus. The CPU 14, the memory 15, the monitor 16, and the recording I/F 17 are each connected via the bus so that information can be transmitted. The operation member 19, the release button 20, and the touch panel 21 are each connected to the CPU 14. Note that the digital camera of this embodiment has, as one function of the shooting mode, a panorama mode that generates a single panoramic image from multiple images obtained by dividing and capturing the subject scene.

The imaging optical system 11 is composed of multiple lens groups including a zoom lens and a focusing lens. The lens positions of the imaging optical system 11 are adjusted in the optical axis direction by a lens drive unit (not shown). For simplicity, the imaging optical system 11 is illustrated in FIG. 1 as a single lens.

The image sensor 12 is a device that captures an image of a scene formed by a light beam that has passed through the imaging optical system 11. The output of the image sensor 12 is input to the AFE 13. Note that the image sensor 12 in this embodiment may be a sequential scanning solid-state image sensor (CCD, etc.) or an XY address solid-state image sensor (CMOS, etc.).

In addition, multiple light receiving elements are arranged in a matrix on the light receiving surface of the image sensor 12. Red (R), green (G), and blue (B) color filters are arranged in the well-known Bayer array on each light receiving element of the image sensor 12. Therefore, each light receiving element of the image sensor 12 outputs an image signal corresponding to each color through color separation by the color filter. This allows the image sensor 12 to capture a color image.

Here, in the shooting mode of the digital camera, the image sensor 12 captures a recording image (main image) in response to the full press of the release button 20. In the shooting mode, the image sensor 12 also captures low-resolution images (through images) for composition confirmation at predetermined time intervals while waiting to shoot. This through image data is output by thinning readout from the image sensor 12. The through image data is used for image display on the monitor 16 and various calculation processes by the CPU 14. In addition, when the shooting mode is video mode or panorama mode, the image sensor 12 captures video at a predetermined frame rate (such as 30 fps or 60 fps) as the main image.

The AFE 13 is an analog front-end circuit that performs analog signal processing on the output of the image sensor 12. This AFE 13 performs correlated double sampling, adjustment of the gain of the image signal, and A/D conversion of the image signal. The output of the AFE 13 is connected to the CPU 14. In this embodiment, the image sensor 12 and the AFE 13 constitute the imaging unit.

The CPU 14 is a processor that controls the overall operation of the digital camera. By executing a control program, the CPU 14 performs known autofocus (AF) control using a phase difference detection method or a contrast detection method, auto exposure (AE) calculation, auto white balance calculation, and the like, based on the data of the through image. The CPU 14 also performs display processing of the setting screen, the captured main image, and the through image. Furthermore, the CPU 14 operates as an image processing unit 22 that performs image processing on the digital image signal output from the AFE 13 by executing an image processing program stored in the memory 15.

The image processing unit 22 performs various image processing (e.g., color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment, etc.) on the digital image signal output from the AFE 13. Furthermore, as described below, when the shooting mode is set to the panorama mode, the image processing unit 22 of this embodiment connects frames (target images) captured as a video in chronological order based on a known panorama synthesis process to synthesize the frames into a panorama, thereby generating a panorama image (synthetic image).

The memory 15 stores image data captured by the digital camera, control programs and image processing programs executed by the CPU 14, etc. A non-volatile semiconductor memory or the like can be used for the memory 15.

The monitor 16 is a monitor such as an LCD monitor, and displays various setting screens and images according to instructions from the CPU 14.

The recording I/F 17 is formed with a connector for connecting a storage medium 18. The recording I/F 17 then writes and reads data to the storage medium 18 connected to the connector. The storage medium 18 is composed of a hard disk, a memory card with built-in semiconductor memory, and the like. Note that FIG. 1 shows a memory card as an example of the storage medium 18.

The operation members 19 are composed of, for example, a command dial, a cross-shaped cursor key, a confirmation button, etc. The operation members 19 receive various inputs from the user to the digital camera. For example, the operation members 19 are used to switch the operation mode of the digital camera, input operations on a setting screen, etc.

The release button 20 accepts input from the user of a command to start AF operation before capturing an image by pressing it halfway, and a command to start image capturing operation by pressing it all the way.

The touch panel 21 detects the position of a stylus (or a fingertip, etc.) that touches the panel surface, and outputs the detected position information to the CPU 14 to accept instruction input from the user. The touch panel 21 may be configured as either a capacitive type or a pressure-sensitive type. The touch panel 21 in this embodiment is configured as a transparent panel of the same shape as the monitor 16, and is layered over the entire surface of the monitor 16. The user inputs instructions to the CPU 14 via the touch panel 21 while visually checking the GUI-style buttons displayed on the monitor 16.

Next, the image processing operation of the digital camera of this embodiment will be described with reference to the flowchart in FIG. 2. In the following, it is assumed that the shooting mode has been set in advance to the panorama mode by the user via the operation member 19.

When the user presses the power button of the operation member 19, the CPU 14 turns on the power to the digital camera and initializes each section. When the CPU 14 is in shooting mode and waiting to capture an image, it causes the image sensor 12 to start capturing a through image, and for example, displays the through image in the entire display range 30 of the monitor 16 (Figure 3a). In response to an image capture instruction by the user fully pressing the release button 20, the CPU 14 causes the image sensor 12 to start capturing a video of the subject scene in panorama mode. At the same time, the user starts a panning operation of the digital camera. Note that in this embodiment, the user will be described as panning the digital camera horizontally, but the same applies to the vertical direction and other cases, so description will be omitted.

When the CPU 14 starts video capture in panorama mode, it switches the display on the monitor 16 from the display shown in FIG. 3a to the display shown in FIG. 3b. That is, as shown in FIG. 3b, instead of a through image, the CPU 14 reduces and displays in real time the frames of the captured video in a display area 40 located to the right of the display range 30 of the monitor 16. Meanwhile, the CPU 14 displays in real time a portion of the panorama images that are sequentially generated by the panorama synthesis process described below in a display area 50 located to the left of the display range 30. However, as shown in FIG. 3b, at the start of video capture, there are no panorama images that have been subjected to the panorama synthesis process, so nothing is displayed in the display area 50.

Step S101: The CPU 14 reads the first frame output from the image sensor 12 via the AEF 13 and sets it as a composite image that will be the basis for the panoramic synthesis process. The CPU 14 then reads the second frame and sets it as a target image to be stitched into the composite image.

Step S102: The image processing unit 22 extracts image features such as edge amount and color information of the target image and the composite image by known image processing. Based on the extracted image features, the image processing unit 22 determines whether or not the amount of movement and pan direction of the target image relative to the composite image have been detected by the panning operation of the user's digital camera, for example, by correlation processing between the target image and the composite image. If the amount of movement and pan direction have been detected, the image processing unit 22 proceeds to step S103 (YES side), and if the amount of movement and pan direction have not been detected, the image processing unit 22 proceeds to step S106 (NO side).

Step S103: The image processing unit 22 performs a panoramic synthesis process to stitch together the image area newly captured by the target image into a composite image based on the amount of movement and pan direction detected in step S102.

Step S104: The image processing unit 22 displays the determined pan direction on the monitor 16 with an arrow 60, and at the same time displays the composite image panorama-composite-created in step S103 in real time while sliding it in the direction opposite to the pan direction in the display area 50 (Figure 4a). Here, Figure 4a shows, as an example, the display state of the monitor 16 immediately after the start of the panoramic composition process.

When the target images are images that are shifted up or down or captured at an angle relative to the composite image, the CPU 14 preferably trims and displays the composite image in which the target images are panoramic-combined. Alternatively, the CPU 14 may display a reduced version of the composite image so that the protruding image areas are also displayed.

Step S105: The CPU 14 determines whether or not an instruction to end image capture by the user pressing the release button 20 all the way again has been received. If the instruction to end image capture has been received, the CPU 14 ends video capture in panorama mode. The image processing unit 22 generates a composite image as a panorama image file and records it in the memory 15 or the storage medium 18. The CPU 14 ends the series of processes. Note that in this embodiment, it is preferable for the CPU 14 to generate a panorama image file by trimming the vertical size of the composite image to the vertical width of the image of the first frame, for example. Also, when the user pans the digital camera vertically, it is preferable for the CPU 14 to trim the horizontal size of the composite image to the horizontal width of the image of the first frame, for example.

On the other hand, if the CPU 14 has not received an instruction to end image capture, it proceeds to step S101 (NO side) and reads the next captured frame as the target image. The CPU 14 performs the processes of steps S101 to S104 on the target image.

Step S106: When the image processing unit 22 is unable to detect the amount of movement and the pan direction of the target image relative to the composite image, for example, because the speed of the panning operation of the digital camera by the user is too fast and the amount of movement is too large, or the attitude of the digital camera changes too much, the image processing unit 22 notifies the user of the failure to detect. That is, the image processing unit 22 notifies the user by displaying a warning display as shown in FIG. 4b on the monitor 16. For example, the image processing unit 22 instructs the user to display the direction in which the digital camera should be panned by using an arrow 70 instead of the arrow 60 in order to capture a target image whose amount of movement is detectable and which will be stitched into the composite image. The image processing unit 22 may also display a warning such as a symbol 71 or a sentence 72 on the monitor 16 together with the arrow 70. Alternatively, as another example of a warning display, the image processing unit 22 may display the area in which the target image should be stitched in the composite image as a shaded area 73.

It is also preferable that the CPU 14 display the above warning to the user if, for example, the user stops the digital camera for a certain period of time without panning it, or pans it in a direction different from that of the arrow 60.

Step S107: The image processing unit 22 temporarily stores the image features of the composite image extracted in step S102, for example, in an internal memory (not shown) of the CPU 14. Note that the stored image features of the composite image may be only the image features of an image area of the composite image that is the same size as the target image or about 70-80% of the size of the target image from the end of the composite image on the side where the target image is to be spliced. This makes it possible to reduce the memory capacity and miniaturize the circuit scale.

Step S108: The image processing unit 22, while having the user refer to the frame in the display area 40, pans, for example, in the direction of the arrow 70 based on the warning display, and reads in the next frame (for example, the third frame) captured by the image sensor 12 as a new target image.

Step S109: The image processing unit 22 extracts image features of the target image read in step S108. The image processing unit 22 determines whether or not the amount of movement and pan direction of the target image for the compositing process have been detected based on the image features of the extracted target image and the image features of the composite image stored in an internal memory (not shown). If the amount of movement and pan direction have been detected, the image processing unit 22 proceeds to step S110 (YES side). On the other hand, if the amount of movement and pan direction have not been detected, the image processing unit 22 proceeds to step S108 (NO side). The image processing unit 22 performs the processes of steps S108 to S109 until the amount of movement and pan direction of the target image for the compositing process are detected.

Step S110: The image processing unit 22 cancels the warning display shown in FIG. 4b and switches to the display shown in FIG. 4c. The image processing unit 22 proceeds to step S103 and resumes the panoramic synthesis process of stitching together the target images newly loaded in step S108 into a composite image.

In this manner, in this embodiment, even if the digital camera is panned significantly with respect to the composite image, by displaying the moving image and the composite image that has been panoramic-composed up to that point in real time, it is possible to easily determine how far the panoramic composition has been performed and where the panoramic composition has failed, and it is possible to easily return the imaging direction of the digital camera to the point where the panoramic composition has failed. This makes it possible to generate a panoramic image with high accuracy without having to perform imaging in the panoramic mode again from the beginning.
Supplementary Notes on the Embodiments
(1) In the above embodiment, an example was described in which the processing of the image processing unit 22 was realized in software by the CPU 14. However, the processing may also be realized in hardware by using an ASIC.

(2) In the above embodiment, the image processing unit 22 detects the amount of movement and pan direction of the target image based on the image feature values of the entire image area of the target image and the composite image, but the present invention is not limited to this. For example, a margin of a predetermined width may be set around the target image and the composite image, and the image processing unit 22 may detect the amount of movement and pan direction for the composite process of the target image based on the image feature values in the image area other than the margin.

The digital camera may also be equipped with an acceleration sensor, electronic gyro, etc., and the image processing unit 22 may stitch together target images into a composite image based on the amount of movement and pan direction detected by the image processing unit 22 and the amount of movement and pan direction detected by the acceleration sensor, etc. This allows for more accurate panoramic composition.

(3) In the above embodiment, when the user starts panning the digital camera, the CPU 14 displays video frames in real time in the right display area 40 of the display range 30 of the monitor 16, and slides a composite image created by the panoramic synthesis process in the left display area 50, but the present invention is not limited to this. For example, as shown in FIG. 5a, the CPU 14 may display video frames in real time in the display area 40' above the center of the monitor 16, and display a composite image in real time in the lower display area 50' according to the progress of the panoramic synthesis process. FIG. 5b shows an example of a warning display when the CPU 14 fails to detect the amount of movement and pan direction of the target image relative to the composite image.

Before starting imaging in panorama mode, the CPU 14 may accept a designation of the imaging range for panoramic imaging, such as 180 degrees or 360 degrees, from the user via the operation member 19 or the like, and set the size of the display area 50' according to that imaging range. This allows the user to easily recognize up to what range imaging has been completed. Furthermore, when imaging has been completed up to the set imaging range, the CPU 14 may automatically end imaging in panorama mode even if the release button 20 is not pressed all the way down again.

Furthermore, while the panoramic synthesis process is proceeding smoothly, the CPU 14 may display the video frames full screen in the display range 30 shown in FIG. 3a, and if detection of the amount of movement and pan direction of the target image for the synthesis process fails, switch to a warning display as shown in FIG. 4b or FIG. 5b.

Figure 6 also shows an example of the display on the monitor 16 when the user pans the digital camera vertically to capture an image of a tower, high-rise building, etc.

(4) In the above embodiment, in step S105, the CPU 14 ends image capture in the panoramic mode when the user presses the release button 20 all the way again. However, the present invention is not limited to this. For example, it is preferable that the user can press the release button 20 all the way at any time during image capture in the panoramic mode, and the CPU 14 can receive an interrupt for the end of image capture.

(5) In the above embodiment, video is captured and then synthesized into a panorama, but the present invention is not limited to this and can also be applied to cases where continuous shooting or still images are captured one by one and then synthesized into a panorama.

The features and advantages of the embodiments will become clear from the above detailed description. It is intended that the claims extend to the features and advantages of the above-described embodiments without departing from the spirit and scope of the claims. Furthermore, any improvements and modifications would be easily conceivable to a person of ordinary skill in the art, and there is no intention to limit the scope of the inventive embodiments to those described above, and appropriate improvements and equivalents within the scope of the disclosed embodiments are also possible.

REFERENCE SIGNS LIST 11: imaging optical system, 12: imaging element, 13: AFE, 14: CPU, 15: memory, 16: monitor, 17: recording I/F, 18: storage medium, 19: operation member, 20: release button, 21: touch panel, 22: image processing unit

Claims (1)

an imaging unit that captures an object scene continuously in time series to generate a plurality of images;
A display unit that displays the plurality of images;
an image processing unit that generates a composite image by connecting the plurality of images in chronological order as a target image,
the image processing unit detects an amount of displacement of the target image relative to the composite image based on image features of the target image and the composite image, joins the target image to the composite image according to the amount of displacement, and, if detection of the amount of displacement fails, displays a warning indicating the failure on the display unit together with the multiple images.

Images