JP7231598B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device, an imaging control method, and an imaging program, and more particularly to an imaging device, an imaging control method, and an imaging program for synthesizing images captured by changing exposure amounts of a plurality of cameras to generate an HDR (High Dynamic Range) composite image. .

2. Description of the Related Art A digital camera that converts an object image into an electric signal using an electronic device such as a CCD (Charge Coupled Device) and records the converted electric signal in a memory has become widespread. Recently, information terminal devices such as mobile phones, smart phones, and tablet terminals equipped with digital cameras are also widely used.

An imaging element used in a digital camera or the like has a very narrow dynamic range compared to a film, and therefore, depending on the imaging conditions, so-called blocked up shadows or blown out highlights may occur, resulting in significant deterioration in image quality. In order to eliminate such a drawback, multiple images are taken while changing the exposure, and when combining these images, dark areas are extracted from the long-exposure image, and bright areas are extracted from the short-exposure image. Therefore, an HDR synthesis function that obtains a wide dynamic range while suppressing blown-out highlights and blocked-up shadows is attracting attention.

For example, Japanese Patent Laid-Open No. 2002-200001 discloses that two image sensors are provided, one of which captures a high-resolution, low-exposure image, and the other captures a low-resolution, high-exposure image with a short exposure time to reduce camera shake. A video generation device that creates an HDR composite image with reduced noise due to, etc., is disclosed.

Japanese Unexamined Patent Application Publication No. 2007-336561

However, the technique described in Patent Document 1 aims to reduce noise such as camera shake by shortening the exposure time when capturing an image with a large amount of exposure. No consideration is given to an HDR synthesis method for obtaining more preferable image quality according to the imaging environment such as camera shake. Also, no mention is made of how to effectively use the two imaging elements.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus, an imaging control method, and a program that generate a high-quality HDR composite image adapted to an imaging environment.

A brief outline of typical inventions disclosed in the present application is as follows.

An imaging apparatus according to a representative embodiment of the present invention includes a video input unit that captures an image of a subject and generates an image signal of the subject; a video signal processing unit that generates a captured image of the subject based on the image signal; Based on the signal, motion information of the subject is detected, based on the motion information, the subject is imaged a plurality of times with different exposure amounts to the video input unit, and a plurality of images with different exposure amounts are sent to the video signal processing unit. a controller for generating an HDR composite image of the subject based on the signal.

Among the inventions disclosed in the present application, the effects obtained by representative ones are briefly described below.

That is, according to the representative embodiments of the present invention, it is possible to provide an imaging apparatus, an imaging control method, and a program that generate a high-quality HDR composite image adapted to the imaging environment.

BRIEF DESCRIPTION OF THE DRAWINGS It is an external view which shows an example of the imaging device 100 which concerns on Embodiment 1 of this invention. 1 is a block diagram showing an example of the configuration of an imaging device according to Embodiment 1 of the present invention; FIG. 1 is a block diagram showing an example of the configuration of an imaging device according to Embodiment 1 of the present invention; FIG. 3 is a block diagram showing an example of the configuration of a video signal processing section and a video input section according to Embodiment 1 of the present invention; FIG. FIG. 2 is a flow chart diagram relating to an imaging method according to Embodiment 1 of the present invention. FIG. 4 is a flow chart diagram showing an example of processing in a motion information detection step, an imaging step, a captured image generation step, etc., according to Embodiment 1 of the present invention. FIG. 4 is a flow chart diagram showing an example of processing in a motion information detection step, an imaging step, a captured image generation step, etc., according to Embodiment 1 of the present invention. FIG. 4 is a diagram showing a timing chart for imaging according to Embodiment 1 of the present invention; FIG. 4 is a diagram schematically showing HDR synthesis processing according to Embodiment 1 of the present invention; FIG. 4 is a diagram schematically showing HDR synthesis processing according to Embodiment 1 of the present invention; FIG. 4 is a diagram schematically showing HDR synthesis processing according to Embodiment 1 of the present invention; FIG. 4 is a block diagram showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention; FIG. 4 is a block diagram showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention; FIG. 9 is a flow chart diagram relating to an imaging method according to Embodiment 2 of the present invention. FIG. 9 is a flow chart diagram relating to an imaging method according to Embodiment 2 of the present invention. FIG. 9 is a diagram schematically showing HDR synthesis processing according to Embodiment 2 of the present invention; FIG. 9 is a diagram schematically showing HDR synthesis processing according to Embodiment 2 of the present invention; FIG. 9 is a diagram schematically showing HDR synthesis processing according to Embodiment 2 of the present invention;

Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to the following embodiments. Moreover, it may be configured by appropriately combining a part of each embodiment described later.
(Embodiment 1)
1A and 1B are diagrams showing an example of the appearance of an imaging device according to Embodiment 1 of the present invention. FIG. 1A is a plan view of the imaging device, and FIG. be. FIG. 2 is a block diagram showing an example of the configuration of the imaging device according to Embodiment 1 of the present invention. FIG. 3 is a diagram showing an example of the software configuration of the imaging device according to Embodiment 1 of the present invention.

Although FIG. 1 exemplifies an imaging device mounted on a smartphone, other imaging devices may also be used, for example, mobile phones, PDA (Personal Digital Assistants) such as tablet terminals, and laptop PCs (Personal Computers). It may be installed in an information processing device such as. Also, the imaging device may be configured by a digital still camera.

As shown in FIGS. 1 and 2, the imaging apparatus 100 includes a main control unit (control unit) 101, a system bus 102, a memory unit 104, a storage unit 110, a video processing unit 120, an audio processing unit 130, an operation unit 140, It includes a communication processing unit 150, a sensor unit 160, an extended interface unit 170, a display unit 121, and the like.

A system bus 102 is a data communication path that connects each component of the imaging device 100 . , data input/output is performed between the main control unit 101 and each unit in the imaging apparatus 100 via the system bus 102 .

The storage unit 110 is composed of a nonvolatile memory such as a flash ROM (Read Only Memory), SSD (Solid State Drive), HDD (Hard Disc Drive), or the like, and can store information even when power is not supplied to the imaging apparatus 100 . remembered.

For example, as shown in FIG. 3, the storage unit 110 includes a basic operation program storage area 110a for storing basic operation programs for executing basic operations of the imaging apparatus 100, and a camera function program storage area for storing camera function programs for realizing camera functions. A region 110b, a voice recognition program storage region 110c for storing a voice recognition program for realizing the voice recognition function, a timing notification program storage region 110d for storing a timing notification program for realizing the timing notification function, and other programs for storing other programs. It includes a storage area 110e, various information/data storage area 110f for storing various information such as operation setting values of the imaging device 100 and user information.

Further, the storage unit 110 stores captured images captured by the imaging device 100, moving images, HDR composite images described later, thumbnail images thereof, and the like in the various information/data storage area 110f, for example. The storage unit 110 also stores new application programs downloaded from application servers on the Internet, for example, in the other program storage area 110e.

The storage unit 110 also stores a first motion threshold that serves as a reference for determining the motion of a subject, which will be described later, and a second motion threshold that is greater than the first motion threshold. These motion thresholds are appropriately set through experiments or the like, for example.

In the memory unit 104, various programs stored in each unit of the storage unit 110 are developed. When various programs are executed by the main control unit 101, the memory unit 104 is configured with various execution units that implement the functions of the respective programs. For example, when the basic operation program stored in the basic operation program storage area 110 a is developed in the memory unit 104 , the memory unit 104 includes a basic operation execution unit 104 a that implements the basic operation of the imaging apparatus 100 . Further, when the camera function program stored in the camera function program storage area 110b is developed in the memory unit 104, the memory unit 104 is configured with a camera function executing unit 104b that implements the camera function. Further, when the speech recognition function program stored in the speech recognition function program storage area 110c is developed in the memory unit 104, the memory unit 104 includes a speech recognition function execution unit 104c that implements the speech recognition function. Further, when the timing notification function program stored in the timing notification function program storage area 110d is developed in the memory unit 104, the memory unit 104 includes a timing notification function execution unit 104d that implements the timing notification function. The memory unit 104 also has a temporary storage area 104e, etc., in which data is temporarily stored as needed.

Note that each unit such as the basic operation execution unit 104a, the camera function execution unit 104b, the voice recognition function execution unit 104c, the timing notification function execution unit 104d, etc. may be configured by hardware having the same function as these units. Moreover, the memory unit 104 may be configured integrally with the main control unit 101 .

The display unit 121 is provided on the surface 100a of the imaging device 100 on which the third video input unit 125 is provided, as shown in FIG. 1(A). The display unit 121 is, for example, a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel. display.

The audio processing unit 130 includes an audio output unit 131 , an audio signal processing unit 132 and an audio input unit 133 . The audio output unit 131 is a speaker, and is provided on the surface 100a of the imaging device 100 around the display unit 121, as shown in FIG. 1A, for example. The audio output unit 131 emits audio based on the audio signal processed by the audio signal processing unit 132 . The audio input unit 133 is a microphone, and is provided on the side of the imaging device 100 on the side opposite to the audio output unit 131 with respect to the display unit 121, as shown in FIGS. ing. The voice input unit 133 receives voice input from the user of the imaging device 100 or the like, and converts the input voice into a voice signal. The audio input unit 133 outputs the input audio signal to the audio signal processing unit 132 . Note that the audio input unit 133 may be configured separately from the imaging device 100, and in this case, the audio input unit 133 and the imaging device 100 may be connected by wired communication or wireless communication.

The operation unit 140 is an instruction input unit that inputs an operation instruction to the imaging device 100 . In this embodiment, for example, as shown in FIGS. 1A and 1B, a touch panel 140a is arranged over the display unit 121, operation keys 140b are provided on the side surface of the imaging device 100, and the imaging device 100 operation keys 140c provided in the vicinity of the display unit 121 on the surface 100a. However, it is not necessary to have all of these, and any one of them may be provided. Also, the touch panel 140a may be configured integrally with the display unit 121 . Further, the operation unit 140 may be configured with a keyboard (not shown) or the like connected to the extended interface unit 170 described later. Further, the operation unit 140 may be configured by a separate information terminal device connected via wired communication or wireless communication.

The communication processing unit 150 includes, for example, a LAN (Local Area Network) communication unit 151, a mobile phone network communication unit 152, and a close proximity wireless communication unit 153. The LAN communication unit 151 transmits and receives data via wireless communication connected via a wireless communication access point on the Internet, for example. The mobile phone network communication unit 152 is connected to a base station of the mobile phone network, and performs telephone communication (phone call) and data transmission/reception via wireless communication via the base station. The close proximity wireless communication unit 153 transmits and receives data to and from a reader/writer compatible with close proximity wireless. The LAN communication unit 151, the mobile telephone network communication unit 152, and the close proximity wireless communication unit 153 include various devices such as an encoding circuit, a decoding circuit, and an antenna (not shown). Further, the communication processing unit 150 may include an infrared communication unit or the like that performs infrared communication.

The sensor unit 160 is a sensor group that detects the state of the imaging device 100 . In this embodiment, the sensor unit 160 is composed of, for example, a GPS (Global Positioning System) receiver 161, an acceleration sensor 162, a gyro sensor 163, a geomagnetic sensor 164, a light amount sensor 165, and a proximity sensor 166. Note that the sensor unit 160 may include sensors other than these.

The GPS receiver 161 receives GPS signals transmitted from a plurality of satellites using GPS. A GPS signal received by the GPS receiving unit 161 is output to, for example, the main control unit 101, and the main control unit 101 detects the position of the imaging device 100 based on the GPS signal.

The acceleration sensor 162 measures the magnitude and direction of acceleration (for example, gravitational acceleration) applied to the imaging device 100 . The magnitude and direction of acceleration measured by the acceleration sensor unit 133 are output to the main control unit 101 as acceleration information, and the main control unit 101 detects the acceleration applied to the imaging device 100 based on the acceleration information. be done.

The gyro sensor 163 measures the angular velocity of the imaging device 100 generated when the imaging device 100 is moved by the user. The angular velocity measured by the gyro sensor unit 163 is output to the main control unit 101 as angular velocity information, for example. The main control unit 101 detects the angular velocity of the imaging device 100 based on the angular velocity information.

A geomagnetic sensor 164 measures the magnitude and direction of the geomagnetism applied to the imaging device 100 . The magnitude and direction of the geomagnetism measured by the geomagnetic sensor 164 are output to the main control unit 101 as geomagnetism information. The main control unit 101 detects the geomagnetism applied to the imaging device 100 based on the geomagnetism information.

A light sensor 165 measures the brightness around the imaging device 100 . For example, the light amount sensor 165 measures the amount of light around the subject when capturing an image of the subject. The light quantity measured by the light quantity sensor 165 is output to the main controller 101 as light quantity information. The main control unit 101 detects the amount of light around the imaging device 100 based on the information on the amount of light.

The proximity sensor 166 measures the proximity of the imaging device 100 to objects around it. The proximity sensor 166 measures, for example, the distance and direction to objects around the imaging device 100 . A proximity state measurement value detected by the proximity sensor 166 is output to the main control unit 101 as proximity state information. The main control unit 101 detects the proximity state of the imaging device 100 to surrounding objects based on the proximity state information.

The extended interface unit 170 is a group of interfaces for extending the functions of the imaging device 100 . The expansion interface unit 170 includes, for example, a video/audio interface 171 , a USB (Universal Serial Bus) interface 172 and a memory interface 173 .

The video/audio interface 171 is connected to an external video/audio output device and receives input of video/audio signals output from the video/audio output device. Also, the video/audio interface 171 outputs video/audio signals to video/audio input devices. The USB interface 172 is connected to a USB device such as a keyboard, and inputs and outputs information to and from the USB device. The memory interface 173 is connected to a memory medium such as a memory card, and inputs/outputs data to/from the memory medium.

Next, the video input section 120 and the video signal processing section 122 will be described. FIG. 4 is a block diagram showing an example of configurations of a video input unit and a video signal processing unit according to Embodiment 1 of the present invention. Note that the third video input unit 125 is omitted in FIG.

The video input unit 120 includes a first video input unit 123, a second video input unit 124, a third video input unit 125, and an exposure control unit 126, as shown in FIGS. 2 and 4, for example. The first video input unit 123 and the second video input unit 124 are arranged side by side on the rear surface 100b of the imaging device 100, as shown in FIG. 1B, for example. The third video input unit 125 is provided near the display unit 121 on the surface 100a of the imaging device 100, as shown in FIG. 1(A).

Note that in FIG. 1B, the first video input unit 123 and the second video input unit 124 are provided on the rear surface 100b of the imaging device 100, but are provided on the front surface 100a of the imaging device 100, for example. may Also, the first video input section 123 and the second video input section 124 may be configured as a single unit.

The first video input unit 123 includes a first imaging optical system 123a and a first imaging element 123b, as shown in FIG. 4, for example. The second video input unit 124 includes a second imaging optical system 124a and a second imaging element 124b, as shown in FIG. 4, for example.

The first imaging optical system 123a and the second imaging optical system 124a are composed of, for example, a plurality of lenses for condensing incident light from a subject, an aperture (such as an iris aperture), a mechanical or electronic shutter, and the like. ing. The first imaging element 123b and the second imaging element 124b are configured by, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like.

The first video input unit 123 and the second video input unit 124 capture an image of a subject and generate an image signal of the subject. Specifically, in each pixel of the first imaging element 123b, the light from the first imaging optical system 123a is stored as electric charge, thereby converting the input light into an electric signal. Also, in each pixel of the second imaging element 124b, the light from the second imaging optical system 124a is stored as electric charge, thereby converting the input light into an electric signal. Thus, the first image sensor 123 and the second image sensor 124 generate electric signals as image signals.

In this embodiment, the first imaging element 123b is an imaging element with a first resolution (high resolution), and the second imaging element 124b has a second resolution lower than that of the first imaging element 123b. (low resolution) imaging device.

Based on instructions from the main control unit 101, the exposure control unit 126 defines the aperture and shutter speed of the first imaging optical system 123a and the second imaging optical system 124a, controls the amount of exposure input to the image sensor 124b.

The video signal processing unit 122 generates a captured image of the subject based on the image signal. For example, as shown in FIG. 4, the video signal processing section 122 includes a first image processing section 122a, a second image processing section 122b, an image synthesizing section 122c, and an image output section 124d.

The first image processing unit 122a is connected to the first imaging element 123b, and converts an electric signal (image signal) output from the first imaging element 123b into digital image data having a predetermined bit gradation width. do. Then, the first image processing unit 122a performs image processing on the digital image data to generate intermediate image data suitable for HDR image synthesis. The first image processing unit 122a outputs the generated intermediate image data to the image synthesizing unit 122c. Also, the first image processing unit 122 a outputs the captured image before synthesis to the storage unit 110 .

The second image processing unit 122b is connected to the second image pickup device 124b, and converts the electrical signal (image signal) output from the second image pickup device 124b into digital image data having a predetermined bit gradation width. do. Then, the second image processing unit 122b performs image processing on the digital image data to generate intermediate image data suitable for HDR image synthesis. The second image processing unit 122b outputs the generated intermediate image data to the image synthesizing unit 122c. Also, the second image processing unit 122b outputs the captured image before synthesis to the storage unit 110 .

The image synthesizing unit 122c synthesizes the intermediate image data input from the first image processing unit 122a and the second image processing unit 122b to generate an HDR synthesized image as a captured image. The image composition unit 122c outputs the generated HDR composite image to the storage unit 110. FIG.

The image output unit 122d outputs the HDR synthesized image generated by the image synthesizing unit 122c, the captured image before synthesizing, the moving image, thumbnail images thereof, and the like to the display unit 121 for display.

The flash unit 129 is provided adjacent to the first image input unit 123 and the second image input unit 124 on the rear surface 100b of the imaging device 100, as shown in FIG. 1B, for example. The flash unit 129 irradiates a subject with flash light, for example, when the first image input unit 123 and the second image input unit 124 capture an image.

A main control unit (control unit) 101 is composed of a computer such as a microprocessor unit. The main control unit 101 executes each execution unit 100 a to 100 d configured in the memory unit 104 to realize the function of each program and operate each component of the imaging apparatus 100 .

The main control unit 101 detects subject motion information based on the image signal generated by the video input unit 120 . In addition, based on the motion information, the main control unit 101 causes the image input unit 120 to image the subject a plurality of times with different exposure amounts. Specifically, the main control unit 101 compares the motion information with a first motion threshold value and a second motion threshold value, and switches to each imaging mode to be described later based on the result to capture the subject. The main control unit 101 also generates exposure information such as the exposure amount, shutter speed, and aperture for the first video input unit 123 and the second video input unit 124 based on the image signal or the captured image. The main control section 101 outputs to the video input section 120 . Further, the main control unit 101 causes the video signal processing unit 122 to generate an HDR composite image of the subject based on a plurality of image signals with different exposure amounts.

Next, an imaging method according to this embodiment will be described. In this embodiment, for example, various programs such as the camera function program 110b shown in FIG. An operation related to imaging is performed.

FIG. 5 is a flowchart of the imaging method according to Embodiment 1 of the present invention. As shown in FIG. 5, when capturing an image, a video input step S10, a motion information detection step S20, an image capture step S30, and an HDR composite image generation step S40 are performed.

When an imaging instruction is given from the operation unit 140, an operation related to imaging of the subject is started. In image input step S10, the image input unit 120 captures an image of a subject and generates an image signal of the subject. The first image input unit 123 and the second image input unit 124 capture an image of a subject, and the first image sensor 123b and the second image sensor 124b generate electric signals as image signals. The first imaging device 123b outputs the generated electrical signal to the first image processing unit 122a of the video signal processing unit 122, and the second imaging device 124b outputs the generated electrical signal to the video signal processing unit 122. to the second image processing unit 122b. Here, the case where both the first video input unit 123 and the second video input unit 124 capture an image of the subject has been described. good too. Then, the process proceeds to motion information detection step S20.

6 and 7 are flowcharts showing an example of processing in a motion information detection step, an imaging step, a captured image generation step, etc., according to Embodiment 1 of the present invention. In motion information detection step S20, the main control unit 101 detects motion information of the subject based on the image signal. Specifically, the first image processing unit 122a of the video signal processing unit 122 outputs the electrical signal output from the first imaging element 123b to the main control unit 101. FIG. The second image processing section 122b of the video signal processing section 122 outputs to the main control section 101 the electrical signal output from the second imaging element 124b. Note that the first image processing unit 122 a and the second image processing unit 122 b may generate captured images based on the electrical signals and output the generated captured images to the main control unit 101 .

The main control unit 101 detects motion information (for example, motion vector) of the subject based on these input electrical signals (image signals). The main control unit 101 detects the motion vector of the subject by using a well-known block matching method or the like for these image signals. Note that the main control unit 101 may detect subject movement information based on the input captured image. Then, the process proceeds to imaging step S30.

In the imaging step S30, the image input unit 120 images the subject a plurality of times with different exposure amounts based on the motion information detected in the motion information detection step S20. In the imaging step S30, step S102 is first performed. In step S102, the motion of the subject is determined based on the motion information. Specifically, the main control unit 101 determines the motion of the subject by comparing the motion vector detected in the motion information detection step S20 with the first motion threshold and the second motion threshold. Specifically, the main control unit 101 reads the first motion threshold and the second motion threshold stored in the storage unit 110 to the memory 104, and compares the motion vector with the first motion threshold and the second motion threshold. .

[Still image capture mode]
If the main control unit 101 determines in step S102 that the motion vector is less than the first motion threshold, step S113 is performed. That is, the main control unit 101 determines that the subject is hardly moving, and switches to the still image capturing mode.

FIG. 8 is a timing chart for imaging according to Embodiment 1 of the present invention. FIG. 9 is a diagram schematically showing HDR synthesis processing according to Embodiment 1 of the present invention. In step S113, the first image input unit 123 successively captures images of the subject with different exposure amounts, as shown in FIG. 8A, for example. Specifically, the main control unit 101 controls the exposure amount (for example, shutter speed, aperture value, etc.). For example, as shown in FIG. 8(A), the main control unit 101 sets the exposure amount in the first imaging to an exposure amount L _A0 that does not cause the dark portion on the low illuminance side to be underexposed. is set to an exposure amount L _A1 smaller than L _A0 (L _A0 >L _A1 ), for example, so that bright portions on the high illuminance side are not overexposed. The main control unit 101 outputs information about the determined exposure amount to the image input unit 120 as exposure information. In the image input unit 120, based on the input exposure information, the first image input unit 123 images the subject with the exposure amount _LA0 for the first time and images the subject with the exposure amount _LA1 for the second time.

The first imaging element 123b generates an electric signal related to a captured image as shown in FIG. 9, for example, each time an image of a subject is captured. The first imaging element 123b generates an electric signal (A0) with an exposure amount L _A0 and an electric signal (A1) with an exposure amount L _A1 as shown in FIG. The signal (A0, A1) is output to the first image processing section 122a of the video signal processing section 122. FIG. Although the subject is imaged twice with different exposure amounts, the subject may be imaged three times or more, for example.

On the other hand, the second image input unit 124 captures a moving image (Bm) of the subject, for example. In addition, the second video image input unit 124 may change the aperture value to change the depth of field to capture an image of the subject. The second image sensor 124b outputs an image signal related to a moving image or the like to the second image processing section 122b of the video signal processing section 122 . Then, the process proceeds to HDR composite image generation step S40.

Step S114 is performed in the HDR composite image generation step S40. In step S114, an HDR composite image of the first resolution is generated based on the plurality of image signals (A0, A1) with different exposure amounts generated in step S113. Specifically, the first image processing unit 122a of the video signal processing unit 122 converts each of the electrical signals (A0, A1) input from the first imaging element 123b into a predetermined bit width of gradation. Convert to digital image data. Then, the first image processing unit 122a performs image processing on each piece of digital image data to generate each piece of intermediate image data suitable for HDR image synthesis. Then, the first image processing unit 122a outputs each intermediate image data to the image synthesizing unit 122c. In addition, the first image processing unit 122a generates each captured image before synthesis based on the electrical signals (A0, A1). The first image processing unit 122a also generates thumbnail images corresponding to each of the captured images before synthesis.

The second image processing unit 122b generates a moving image based on the image signal input from the second video input unit 124, or generates captured images with different depths of field. The second image processing unit 122b also generates thumbnail images corresponding to the generated moving images and captured images.

The image synthesizing unit 122c weights and synthesizes the respective intermediate image data input from the first image processing unit 122a, thereby producing, for example, a first resolution (high resolution) HDR synthetic image ( Captured image) (A0+A1) is generated. The HDR composite image (A0+A1) generated in this manner eliminates blocked-up shadows in dark areas and blown-out highlights in bright areas, and is an image close to what the user sees with the naked eye. Also, the image composition unit 122c generates a thumbnail image corresponding to the generated HDR composite image.

Note that even if the first image processing unit 122a and the second image processing unit 122b convert a high-resolution image signal into a low-resolution image signal, and the image synthesizing unit 122c generates a low-resolution HDR synthesized image, good. Then, the process proceeds to step S115.

In step S<b>115 , the HDR composite image generated in step S<b>114 , the captured image before composite, the moving image, the thumbnail image, and the like are stored in the storage unit 110 . Specifically, the image synthesizing unit 122c outputs the generated HDR synthesized image (A0+A1) and thumbnail image to the storage 110, and the storage 110 stores the input HDR image synthesized image (A0+A1) and thumbnail image. In addition, the first image processing unit 122a and the second image processing unit 122b output the captured image before synthesis, the moving image, and the thumbnail image to the storage 110, and the storage 110 stores the input captured image before synthesis, Stores moving images and thumbnail images. When these processes are performed, the process of HDR synthetic image generation step S40 is completed, and the process proceeds to step S104. Step S104 will be described later.

[Fine motion imaging mode]
10 and 11 are diagrams schematically showing HDR synthesis processing according to Embodiment 1 of the present invention. If the main control unit 101 determines in step S102 of the imaging step S30 that the motion vector is greater than or equal to the first motion threshold, step S123 or step S133 is performed. Specifically, when the main control unit 101 determines that the motion vector is greater than or equal to the first motion threshold and less than or equal to the second motion threshold, step S123 is performed. That is, the main control unit 101 determines that the movement of the subject is larger than in the still image imaging mode, but smaller than in the moving object imaging mode, which will be described later, and switches to the fine motion imaging mode.

In step S123, the main control unit 101 causes the first image input unit 123 and the second image input unit 124 to simultaneously image the subject with different exposure amounts, and then causes the second image input unit 124 to On the other hand, the subject is imaged with different exposure amounts. Specifically, as shown in FIG. 8B, for example, the main control unit 101 adjusts the exposure amount in imaging by the first image input unit 123 to an exposure amount that does not cause a dark subject on the low illuminance side to become blackened. Set to L _A0 . In other words, the first image input unit 123 images the subject with proper exposure. It should be noted that the proper exposure here means capturing an image of a dark subject on the low-illuminance side with a sufficient amount of exposure. In addition, the main control unit 101 sets the exposure amount in the first imaging by the second image input unit 124, for example, between the exposure amount that causes blackout on the low illuminance side and the exposure amount that causes overexposure on the high illuminance side. is set to the exposure amount L _B0 of , and the exposure amount in the second imaging by the second image input unit 124 is set to an exposure amount L _A1 smaller than L _B0 so as not to cause bright portions to be overexposed (L _A0 > L _B0 >L _B1 ).

In addition, since the resolution of the second imaging element 124b is lower than that of the first imaging element 123b, the area per pixel increases and the amount of exposure increases. The same amount of exposure can be obtained even with a shorter imaging time (faster shutter speed) than the unit 123 . Therefore, the imaging interval (time difference) in the second image input unit 124 can be made shorter than that in the first image input unit 123, so that the generation of noise due to slight movement of the subject during HDR synthesis processing can be suppressed.

The main control unit 101 outputs information about the determined exposure amount to the image input unit 120 as exposure information. In the video input unit 120, based on the input exposure information, the first video input unit 123 captures an image of the subject with an exposure amount of L _A0 , and the second video input unit 124 captures an image of the subject with an exposure amount of L _B0 for the first time. The subject is imaged at , and the subject is imaged at the exposure amount L _B1 for the second time.

The first image sensor 123b generates an electric signal (A0) with an exposure amount L _A0 as shown in FIG. Output to the processing unit 122a.

The second imaging element 124b generates an electric signal (B0) with an exposure amount L _B0 and an electric signal ( _B1 ) with an exposure amount L B1 as shown in FIG. The generated electrical signals (B0, B1) are output to the second image processing section 122b of the video signal processing section 122. FIG. Then, the process proceeds to HDR composite image generation step S40.

In HDR composite image generation step S40, step S124 is first performed. In step S124, the resolution of the captured image of the electrical signal generated in step S123 is converted. Specifically, the main control unit 101 causes the video signal processing unit 122 to convert the resolution of the image signal generated by the first video input unit 123 into the second resolution, and converts the resolution of the image signal generated by the first video input unit 123 into the second resolution. converts the resolution of the image signal generated in to a first resolution.

Specifically, the main control unit 101 causes the first image processing unit 122a to convert the resolution of the electrical signal (A0) generated by the first imaging device 123b from the first resolution to the second resolution. , causes the second image processing unit 122b to convert the resolution of the electric signal (B0) generated by the second imaging element 124b from the second resolution to the first resolution. More specifically, the first image processing unit 122a performs resolution conversion on the electric signal (A0) of the first resolution, and converts the electric signal (A0d) of the second resolution shown in FIG. to generate The second image processing unit 122b performs resolution conversion on the electric signal (B0) of the second resolution, and performs processing such as interpolation processing to obtain the first image shown in FIG. 10A. to generate an electrical signal (B0u) with a resolution of . Then, the process proceeds to step S125.

In step S125, a first resolution HDR composite image and a second resolution HDR composite image are generated based on the plurality of image signals (A0, B0, B1, A0d, B0u) generated in step S124. be. Specifically, the first image processing unit 122a converts each of the electric signal (A0) with the first resolution and the electric signal (A0d) with the second resolution into a digital image having a predetermined bit width of gradation. Convert to data. Then, the first image processing unit 122a performs image processing on each piece of digital image data to generate each piece of intermediate image data suitable for HDR image synthesis. Then, the first image processing unit 122a outputs each intermediate image data to the image synthesizing unit 122c. Also, the first image processing unit 122a generates each captured image before synthesis based on the electrical signals (A0, A0d). The first image processing unit 122a also generates thumbnail images corresponding to each of the captured images before synthesis.

The second image processing unit 122b converts each of the electrical signals (B0, B1) of the second resolution and the electrical signal (B0u) of the first resolution into digital image data having a predetermined bit width of gradation. do. Then, the second image processing unit 122b performs image processing on each piece of digital image data to generate each piece of intermediate image data suitable for HDR image synthesis. Then, the second image processing unit 122b outputs each intermediate image data to the image synthesizing unit 122c. In addition, the second image processing unit 122b generates each captured image before synthesis based on the electrical signals (B0, B1, B0u). In addition, the second image processing unit 122b generates thumbnail images corresponding to each of the generated captured images before synthesis.

The image synthesizing unit 122c weights and synthesizes each intermediate image data output from the first image processing unit 122a and each intermediate image data output from the second image processing unit 122b to obtain the first image data. and a second resolution HDR composite image.

Specifically, the image synthesizing unit 122c performs HDR synthesizing at the first resolution, for example, as shown in FIG. Generate an image (A0+B0u). Further, the image synthesizing unit 122c generates a second resolution HDR synthesized image (A0d+B0 ). Further, the image synthesizing unit 122c generates a second resolution HDR synthesized image (B0+B1 ). Further, the image synthesizing unit 122c generates an HDR synthesized image of the second resolution, for example, shown in FIG. Generate (A0d+B0+B1). Also, the image composition unit 122c generates a thumbnail image corresponding to the generated HDR composite image.

When the user browses the HDR composite images later, the user may select the image that is visually judged to be the best from these images and handle it as the image of the final processing result of the HDR composite processing. Then, the process proceeds to step S126.

In step S<b>126 , the HDR composite image generated in step S<b>125 , the captured image before composite, the thumbnail image, and the like are stored in the storage unit 110 . Specifically, the image composition unit 122c outputs the generated HDR composite images (A0+B0u), (A0d+B0), (B0+B1), (A0d+B0+B1) and thumbnail images to the storage 110, and the storage 110 stores the input HDR images Synthesis (A0+B0u), (A0d+B0), (B0+B1), (A0d+B0+B1), thumbnail images are stored. Further, the first image processing unit 122a and the second image processing unit 122b output the captured image before synthesis and the thumbnail image to the storage 110, and the storage 110 stores the input captured image before synthesis and the thumbnail image. Remember. When these processes are performed, the process of HDR synthetic image generation step S40 is completed, and the process proceeds to step S104. Step S104 will be described later.

[Moving object imaging mode]
If the main control unit 101 determines in step S102 of the imaging step S30 that the motion vector exceeds the second motion threshold, step S133 is performed. That is, the main control unit 101 determines that the subject moves more than in the fine motion imaging mode, and switches to the moving object imaging mode.

In step S133, the same processing as in step S123 is performed. That is, in step S123, the main control unit 101 causes the first image input unit 123 and the second image input unit 124 to simultaneously image the subject with different exposure amounts. The first video input unit 123 captures an image of the subject with an exposure amount L _A0 , and generates an electric signal (A0) for the captured image of the first resolution as shown in FIGS. 10A and 10B, for example. . The second video input unit 124 captures an image of the subject with an exposure amount L _B0 and generates an electrical signal (B0) for the captured image of the second resolution as shown in FIGS. 10A and 10B, for example. . Note that in the moving object imaging mode, unlike the fine motion imaging mode, the second image input unit 124 does not image the subject with a time lag. This is because the movement of the subject is large, and subject shake becomes noticeable in the generated HDR composite image if the images are captured with a time lag.

The first imaging element 123b outputs an electric signal (A0) of the exposure amount L _A0 to the first image processing section 122a. The second imaging element 124b outputs an electric signal (B0) of the exposure amount L _B0 to the second image processing section 122b. Then, the process proceeds to HDR composite image generation step S40.

In HDR composite image generation step S40, step S134 is first performed. In step S134, the same processing as in step S124 described above is performed. That is, in step S134, the first image processing unit 122a performs resolution conversion on the electrical signal (A0) of the first resolution, and converts the electrical signal (A0) of the second resolution to the electrical signal of the second resolution shown in FIG. 10B, for example. Generate (A0d). The second image processing unit 122b performs resolution conversion on the electric signal (B0) of the second resolution to generate the electric signal (B0u) of the first resolution shown in FIG. 10(A). Then, the process proceeds to step S135.

In step S135, substantially the same processing as in step S125 described above is performed. That is, in step S135, a first resolution HDR composite image and a second resolution HDR composite image are generated based on the plurality of image signals (A0, B0, A0d, B0u) generated in step S134. be. Specifically, the image synthesizing unit 122c generates the first resolution HDR synthesized image (A0+B0u) shown in FIG. 10B and the second resolution HDR synthesized image (A0d+B0) shown in FIG. 10A. do. Also, the first image processing unit 122a and the second image processing unit 122b generate captured images before synthesis and thumbnail images respectively corresponding to the captured images before synthesis.

When the user browses the HDR composite images later, the user may select the image that is visually judged to be the best from these images and handle it as the image of the final processing result of the HDR composite processing. Then, the process proceeds to step S126.

In step S<b>126 , the HDR composite image generated in step S<b>135 , the captured image before composite, the thumbnail image, and the like are stored in the storage unit 110 . Specifically, the image composition unit 122c outputs the generated HDR composite images (A0+B0u), (A0d+B0), and the thumbnail image to the storage 110, and the storage 110 stores the input HDR image composites (A0+B0u), (A0d+B0). , store the thumbnail image. Further, the first image processing unit 122a and the second image processing unit 122b output the captured image before synthesis and the thumbnail image to the storage 110, and the storage 110 stores the input captured image before synthesis and the thumbnail image. Remember. When these processes are performed, the process of HDR synthetic image generation step S40 is completed, and the process proceeds to step S104.

In step S104, an image or the like to be displayed on the display unit 121 is selected. Specifically, the images generated in steps S114, S125, and S135 of each imaging mode and stored in the storage unit 110 (HDR synthesized image, captured image before synthesis, moving image) are displayed on the display unit. An image or the like to be displayed on 121 is selected.

For example, the storage unit 110 outputs information (for example, thumbnail images) related to stored images and the like to the image output unit 122d. The image output unit 122 d outputs the input thumbnail image to the display unit 121 . The display unit 121 displays the input thumbnail images. The user selects an image or the like to be displayed on the display unit 121 from the displayed thumbnail images. Then, the process proceeds to step S105.

In step S105, an image or the like corresponding to the selected thumbnail image is displayed. Specifically, the image output unit 122d outputs from the storage unit 110, for example, an image corresponding to the thumbnail image selected by the user. The image output unit 122 d outputs the read image and the like to the display unit 121 . The display unit 121 displays an input image or the like.

Note that the user may arbitrarily select the images and the like to be displayed on the display unit 121, or register the priority order of display in the imaging apparatus 100 in advance so that the images and the like are sequentially displayed according to the order. good too.

Also, an image or the like displayed on the display unit 121 may be enlarged or reduced as appropriate according to the resolution of the display unit 121 . At this time, the plurality of images and the like may be displayed in the same size, or the plurality of images and the like may be displayed in different sizes according to the resolution of the selected image and the like.

Imaging device 100 according to the present embodiment is not limited to the configuration shown in FIG. For example, the imaging device 100 may not include the communication processing unit 150, the sensor unit 160, etc., or may include various functions such as a digital television broadcast reception function and an electronic money settlement function.

According to the present embodiment, based on the motion vector of the subject, the main control unit 101 causes the image input unit 120 to image the subject a plurality of times with different exposure amounts, and instructs the image signal processing unit to capture the exposure amount generates an HDR composite image of a subject based on a plurality of image signals with different values.

With this configuration, it is possible to select an appropriate imaging mode according to the movement of the subject, so it is possible to generate a high-quality HDR composite image corresponding to the imaging environment such as movement of the subject and camera shake.

Further, according to the present embodiment, when the main control unit 101 determines that the motion vector is less than the first motion threshold, the main control unit 101 switches to the still image capturing mode, and selects the high-resolution first resolution image. The first image input unit 123 having one image sensor 123b is caused to continuously image the subject with different exposure amounts, and the image signal processing unit 122 receives a plurality of image signals (A0, Based on A1), a first resolution HDR composite image is generated.

According to this configuration, since the subject hardly moves in the still image capturing mode, even if the subject is captured with a time difference, the occurrence of noise due to the movement of the subject in the HDR composite image can be suppressed. As a result, the subject can be continuously imaged using only the high-resolution imaging device, so a high-quality HDR composite image is generated.

Further, according to this configuration, the electric signal (A0) generated by imaging a dark subject on the low-illuminance side with a sufficient amount of exposure and the electric signal (A0) generated by imaging a bright subject on the high-illuminance side with a reduced amount of exposure. A high-quality HDR composite image with an expanded dynamic range is generated because the electric signal (A1) is combined with the electric signal (A1).

Further, according to the present embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the first motion threshold, the main control unit 101 switches to the fine motion imaging mode or the moving object imaging mode, and the first image input unit 123 and the second image input unit 124 are caused to simultaneously image the subject with different exposure amounts, and the image signal processing unit 122 is caused to convert the image signal of the first resolution into the image signal of the second resolution. , converts the second resolution image signal into a first resolution image signal to generate a first resolution HDR composite image and a second resolution HDR composite image.

According to this configuration, even if the subject is moving, the HDR composite image is generated based on a plurality of image signals captured at the same time. An enlarged high quality HDR composite image is generated.

In addition, according to this configuration, a plurality of HDR images with different resolutions are generated, so HDR images suitable for the user's application are provided, and the user-friendly imaging apparatus 100 is provided.

Further, according to the present embodiment, when the control unit 101 determines that the motion vector is equal to or greater than the first motion threshold and equal to or less than the second motion threshold, the control unit 101 switches to the fine motion imaging mode and captures the first image. After the input unit 123 and the second image input unit 124 are caused to simultaneously image the subject with different exposure amounts, the second image input unit 124 is caused to image the subject with different exposure amounts.

According to this configuration, HDR synthesis processing is performed based on at least three types of image signals with different exposure amounts, so an HDR synthesized image of higher quality is generated.

Moreover, according to the present embodiment, the resolution of the second imaging element 124b is lower than the resolution of the first imaging element 123b.

According to this configuration, the second video input unit 124 can obtain the same amount of exposure as the first video input unit 123 even if the imaging time is shorter. As a result, even if the second image input unit 124 takes images of the subject with a time lag, the occurrence of noise due to the motion of the subject can be suppressed, so the occurrence of noise due to the motion of the subject in the HDR composite image can be suppressed.

Moreover, according to this configuration, the cost of the second imaging device 124 can be suppressed, so the manufacturing cost of the imaging device 100 can be suppressed.

Further, according to the present embodiment, display unit 121 displays a thumbnail image such as an HDR composite image, and when a thumbnail image is selected, displays an HDR composite image corresponding to the selected thumbnail image.

According to this configuration, the user can easily identify the HDR composite image or the like stored in the storage unit 110 , thereby reducing the burden when selecting the HDR image or the like to be displayed on the display unit 121 .

Further, according to the present embodiment, the image input step S10 in which the image input unit 120 captures an image of the subject and generates an image signal of the subject, and the main control unit 101 detects movement information of the subject based on the image signal. a motion information detection step S20; an imaging step S30 in which the main control unit 101 causes the video input unit 120 to capture images of the subject a plurality of times with different exposure amounts based on the motion information; and an HDR composite image generation step S40 for generating an HDR composite image of the subject based on a plurality of image signals with different amounts.

According to this configuration, it is possible to select an appropriate imaging mode according to the movement of the subject to capture an image of the subject. can provide.

Further, according to the present embodiment, the video input step S10 causes the video input unit 120 to image the subject and generates an image signal of the subject, and the motion information detection step S20 detects the motion information of the subject based on the image signal. Then, based on the motion information, an imaging step S30 in which the image input unit is caused to image the subject a plurality of times with different exposure amounts, and the image signal processing unit 122 receives the subject based on a plurality of image signals with different exposure amounts. HDR synthetic image generation step S40 for generating an HDR synthetic image of .

According to this configuration, it is possible to pick up an image of the subject by selecting an appropriate image pickup mode according to the movement of the subject. can provide.

(Embodiment 2)
Next, Embodiment 2 will be described. In this embodiment, a case will be described in which the imaging mode is switched based on the motion of the subject and the amount of light. In addition, below, the detailed explanation about the part which overlaps with the above-mentioned Embodiment 1 may be abbreviate|omitted suitably.

12 and 13 are block diagrams showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention. The imaging device 200 includes a video input unit 220 and the like, as shown in FIG.

The video input unit 220 includes a second video input unit 224, a resolution conversion unit 228, etc., as shown in FIGS. The second video input section 224 has a second imaging element 224b. The second image pickup device 224b is an image pickup device with a first resolution (high resolution) like the first image pickup device 123b. The first imaging device 123b and the second imaging device 224b are configured such that the resolution is appropriately converted by the resolution conversion unit 228. FIG.

The light amount sensor 165 measures the amount of light around the imaging device 200 and around the subject. The light amount sensor 165 then outputs the measured light amount information to the exposure control section 126 and the main control section 101 .

The exposure control unit 126 adjusts the exposure when the first image input unit 123 and the second image input unit 224d capture images based on the light amount information output from the light amount sensor 165 and the instruction from the main control unit 101, for example. set the amount.

The main control unit 101 switches the imaging mode based on the amount of light measured by the light amount sensor 165 and the detection result of the motion vector of the subject. Further, the main control unit 101 causes the resolution conversion unit 228 to group a plurality of pixels so that the resolutions of the first image sensor 123b and the second image sensor 224b are set to the first resolution (high resolution). to a second resolution (lower resolution). Further, the main control unit 101 causes the resolution conversion unit 228 to cancel the grouping of a plurality of pixels, thereby changing the resolution of the first image sensor 123b and the second image sensor 224b to the second resolution (lower resolution). resolution) to a first resolution (high resolution).

For example, the main control unit 101 outputs resolution conversion information for converting the resolutions of the first imaging device 123b and the second imaging device 224b to the resolution conversion unit 228 according to the imaging mode. The resolution converter 228 converts the resolutions of the first imaging element 123b and the second imaging element 224b based on the resolution conversion information. In the image sensor converted to low resolution, a plurality of grouped pixels are treated as one pixel. As a result, the area of one pixel after grouping is larger than that of one pixel before grouping, and the amount of exposure increases, so the exposure time (shutter speed) for obtaining the same amount of exposure is shortened.

The light intensity sensor 165 measures, for example, the amount of light from a high light intensity area to a low altitude area of the object. 12 and 13 exemplify the case where the light amount sensor 165 is provided independently, but it is not limited to such a configuration. The imaging device 224b may also have the function of the light amount sensor 165. FIG.

Next, an imaging method according to this embodiment will be described. 14 and 15 are flowcharts relating to the imaging method according to Embodiment 2 of the present invention. Also in this embodiment, the video input step S10, the motion information detection step S20, the imaging step S30, and the HDR composite image generation step S40 shown in FIG. 5 are performed. The image input step S10 and the motion information detection step S20 are the same as in the first embodiment described above.

In the imaging step S30, step S202 is first performed. In step S202, the motion of the subject is determined based on the motion information. Specifically, the main control unit 101 determines the motion of the subject by comparing the motion vector detected in the motion information detection step S20 with the third motion threshold. Specifically, the main control unit 101 reads the third motion threshold stored in the storage unit 110 to the memory 104 and compares the motion vector with the third motion threshold. The third motion threshold is appropriately set by, for example, experiments.

[Still image capture mode]
If the main control unit 101 determines in step S202 that the motion vector is less than the third motion threshold, step S214 is performed. That is, the main control unit 101 determines that the subject is hardly moving, and switches to the still image capturing mode.

In step S214, the same processing as in step S113 in the first embodiment is performed. That is, the first image input unit 123 continuously images the subject with different exposure amounts. The first video input unit 123 captures an image of the subject, for example, with an exposure amount L _A0 in the first imaging, and captures an image with an exposure amount L _A1 smaller than L _A0 in the second imaging (L _A0 >L _A1 ). . The first image sensor 123b generates electrical signals (A0, A1) for a high-resolution captured image shown in FIG. 9, for example, and transmits the generated electrical signals (A0, A1) to the first image processing unit 122a. Output. Note that although a case where two types of image signals with different exposure amounts are synthesized is described here, for example, three or more types of image signals may be synthesized.

On the other hand, the second image input unit 224 may capture, for example, a moving image (Bm) of the subject, or may capture the subject with different depths of field. In addition, since the second imaging element 224b of the second image input unit 224 also has a high resolution, for example, the subject can be imaged at the same time as the first image input unit 123 by changing the exposure amount (for example, L _A1 ). may The second image sensor 224b generates an electric signal (A0) of the exposure amount L _A0 and an electric signal ( _{A1) of the exposure amount L A1} , and outputs the generated electric signals (A0, A1) to the first image processing unit 122a. output to The second image sensor 224b outputs an image signal related to a moving image or the like to the second image processing section 122b of the video signal processing section 122 . Then, the process proceeds to HDR composite image generation step S40.

Step S215 is performed in the HDR composite image generation step S40. In step S215, the same process as in step S114 in the first embodiment is performed. That is, based on the plurality of image signals (A0, A1) with different exposures generated in step S214, the image composition unit 122c generates the HDR composite image (A0+A1) of the first resolution. Also, the image synthesizing unit 122c may generate an HDR synthesized image based on the electrical signals generated by the first imaging element 123b and the second imaging element. Also, the image synthesizing unit 122c generates a thumbnail image corresponding to the HDR synthesized image. The first image processing unit 122a and the second image processing unit 122b generate captured images before synthesis, moving images, and thumbnail images corresponding to these images.

Note that even if the first image processing unit 122a and the second image processing unit 122b convert a high-resolution image signal into a low-resolution image signal, and the image synthesizing unit 122c generates a low-resolution HDR synthesized image, good. Then, the process proceeds to step S216.

In step S216, the same processing as in step S115 in the first embodiment is performed. That is, the storage unit 110 stores the HDR synthesized image, the captured image before synthesis, the moving image, the thumbnail image, and the like generated in step S215.

[Moving object imaging mode]
FIG. 16 is a diagram schematically showing HDR synthesis processing according to Embodiment 2 of the present invention. If the main control unit 101 determines in step S202 of the imaging step S30 that the motion vector is greater than or equal to the third motion threshold, it determines that the motion of the subject is large, and switches to the moving object imaging mode. After switching to the moving object imaging mode, the process proceeds to step S203.

In step S<b>203 , the imaging mode is switched based on the amount of light measured by the light amount sensor 165 . Specifically, the light amount sensor 165 measures the amount of light around the imaging device 200 and outputs information about the measured amount of light to the main control unit 101 as light amount information. The main control unit 101 compares the measured light intensity with a first light intensity threshold and a second light intensity threshold larger than the first light intensity threshold based on the input light intensity information. Specifically, the main control unit 101 reads the first light intensity threshold and the second light intensity threshold stored in the storage unit 110 into the memory 104, and compares the light intensity with the first light intensity threshold and the second light intensity threshold. The first light amount threshold and the second light amount threshold are appropriately set by, for example, experiments.

<High light intensity moving object imaging mode>
When the main control unit 101 determines that the light amount exceeds the second light amount threshold, the process proceeds to step S224. That is, the main control unit 101 determines that the amount of light is high, and switches to the high-light amount moving object imaging mode.

In step S224, the main control unit 101 causes the first image input unit 123 and the second image input unit 224 to simultaneously image the subject with different exposure amounts. At this time, the main control unit 101 sets the resolutions of the first imaging element 123b and the second imaging element 224b to the first high resolution.

For example, if the resolution of the first image pickup device 123b or the second image pickup device 224b is set to a low resolution, the main control unit 101 sends resolution conversion information for converting the resolution of the corresponding image pickup device to the resolution conversion unit 228. output to The resolution conversion unit 228 converts the resolution of the corresponding image sensor from low resolution to high resolution based on the input resolution conversion information.

When the resolution of the image pickup device is set in this way, the first image input unit 123 picks up an image of the subject with an exposure amount of L _A0 , and for example, an electric signal (A0 ). The second video input unit 124 captures an image of the subject with an exposure amount L _B0 less than L _A0 (L _A0 >L _B0 ), and outputs an electrical signal (B0) for a high-resolution captured image as shown in FIG. 16, for example. Generate.

The first imaging element 123b outputs an electric signal (A0) of the exposure amount L _A0 to the first image processing section 122a. The second imaging element 124b outputs an electric signal (B0) of the exposure amount L _B0 to the second image processing section 122b. Then, the process proceeds to HDR composite image generation step S40.

In the HDR composite image generation step S40, step S225 is performed. In step S225, based on the plurality of image signals (A0, B0) generated in step S224, the image composition unit 122c generates a high-resolution HDR composite image (A0+B0) shown in FIG. 16, for example. Also, the image synthesizing unit 122c generates a thumbnail image corresponding to the HDR synthesized image. The first image processing unit 122a and the second image processing unit 122b generate captured images before synthesis, moving images, and thumbnail images corresponding to these images.

Note that even if the first image processing unit 122a and the second image processing unit 122b convert a high-resolution image signal into a low-resolution image signal, and the image synthesizing unit 122c generates a low-resolution HDR synthesized image, good. Then, the process proceeds to step S237.

In step S237, the same processing as step S126 in the first embodiment is performed. That is, the storage unit 110 stores the HDR composite image generated in step S<b>225 , the captured image before composite, the moving image, the thumbnail image, and the like. Then, the process proceeds to step S204. Step S204 will be described later.

<Medium light amount moving object imaging mode>
FIG. 17 is a diagram schematically showing HDR synthesis processing according to Embodiment 2 of the present invention. When the main control unit 101 determines that the light amount is equal to or greater than the first light amount threshold and equal to or less than the second light amount threshold, the process proceeds to step S234. That is, the main control unit 101 determines that the amount of light is less than in the high light amount moving object imaging mode, and switches to the medium light amount moving object imaging mode.

In step S234, the main control unit 101 causes the first image input unit 123 and the second image input unit 224 to simultaneously image the subject with different exposure amounts. At this time, the main control unit 101 causes the resolution conversion unit 228 to convert the resolution of the first image sensor 123b into the first resolution and convert the resolution of the second image sensor 224b into the second resolution.

For example, if the resolution of the first imaging device 123b is set to a low resolution, the main control unit 101 outputs resolution conversion information for converting the resolution of the first imaging device 123b to the resolution conversion unit 228. The resolution conversion unit 228 converts the resolution of the first imaging element 123b from low resolution to high resolution based on the input resolution conversion information. Also, if the resolution of the second imaging element 224b is set to a high resolution, the main control section 101 outputs resolution conversion information for converting the resolution of the second imaging element 224b to the resolution conversion section 228 . The resolution conversion unit 228 converts the resolution of the second imaging element 224b from high resolution to low resolution based on the input resolution conversion information.

When the resolution of the image pickup device is set in this way, the first image input unit 123 picks up an image of the subject with an exposure amount of L _A0 , for example, as shown in FIGS. 17A and 17B. Generate an electrical signal (A0) for the image. The second video input unit 124 captures an image of the subject with an exposure amount L _b0 that is less than L _A0 (L _A0 >L _b0 ), and for low-resolution captured images such as those shown in FIGS. to generate an electric signal (b0) of . Since the resolution of the second video input unit 224 is lower than that of the first video input unit 123, it is possible to appropriately set the exposure amount for expanding the dynamic range in the HDR composite image.

The first imaging element 123b outputs an electric signal (A0) of the exposure amount L _A0 to the first image processing section 122a. The second image sensor 124b outputs an electrical signal (b0) of the exposure amount _Lb0 to the second image processing section 122b. Then, the process proceeds to HDR composite image generation step S40.

In HDR composite image generation step S40, step S235 is first performed. In step S235, for example, the same process as in step S124 in the first embodiment is performed. That is, in step S235, the resolution of the electrical signal generated in step S234 is converted. Specifically, the first image processing unit 122a performs resolution conversion on the high-resolution electrical signal (A0) to generate a low-resolution electrical signal (A0d) shown in FIG. 17B, for example. do. The second image processing unit 122b performs resolution conversion on the low-resolution electrical signal (b0) to generate a high-resolution electrical signal (b0u) shown in FIG. 17A, for example. Then, the process proceeds to step S236.

In step S236, the image synthesizing unit 122c generates a high-resolution HDR synthetic image (A0+b0u ), for example, to generate a low-resolution HDR composite image (A0d+b0) shown in FIG. 17B. Also, the image synthesizing unit 122c generates a thumbnail image corresponding to the HDR synthesized image. The first image processing unit 122a and the second image processing unit 122b generate captured images before synthesis, moving images, and thumbnail images corresponding to these images. Then, the process proceeds to step S237.

In step S<b>237 , the HDR composite image generated in step S<b>236 , the captured image before composite, the moving image, the thumbnail image, and the like are stored in the storage unit 110 . Then, the process proceeds to step S204. Step S204 will be described later.

<Low-light moving object imaging mode>
FIG. 18 is a diagram schematically showing HDR synthesis processing according to Embodiment 2 of the present invention. When the main control unit 101 determines that the light amount is less than the first light amount threshold, the process proceeds to step S244. That is, the main control unit 101 determines that the amount of light is lower than that in the middle light amount imaging mode, and switches to the low light amount moving object imaging mode.

In step S224, the main control unit 101 causes the first image input unit 123 and the second image input unit 224 to simultaneously image the subject with different exposure amounts. At this time, the main control unit 101 causes the resolution conversion unit 228 to convert the resolutions of the first imaging device 123b and the second imaging device 224b to a second resolution, which is a low resolution.

For example, if the resolution of the first image pickup device 123b or the second image pickup device 224b is set to high resolution, the main control unit 101 sends resolution conversion information for converting the resolution of the corresponding image pickup device to the resolution conversion unit 228. output to The resolution conversion unit 228 converts the resolution of the corresponding image sensor from high resolution to low resolution based on the input resolution conversion information.

When the resolution of the image pickup device is set in this way, the first image input unit 123 picks up an image of the subject with an exposure amount L _a0 , and for example, the electric signal (a0) for the low-resolution picked-up image shown in FIG. to generate The second video input unit 124 captures an image of the subject with an exposure amount _Lb0 less than _La0 , and generates an electrical signal (b0) for the low-resolution captured image shown in FIG. 18, for example. Since the resolutions of the first video input unit 123 and the second video input unit 224 are set to low resolution, it is possible to appropriately set the exposure amount for expanding the dynamic range in the HDR composite image. can.

The first imaging element 123b outputs an electric signal (a0) of the exposure amount _La0 to the first image processing section 122a. The second image sensor 124b outputs an electrical signal (b0) of the exposure amount _Lb0 to the second image processing section 122b. Then, the process proceeds to HDR composite image generation step S40.

In the HDR composite image generation step S40, step S225 is performed. In step S225, based on the plurality of image signals (a0, b0) generated in step S224, the image composition unit 122c generates a low-resolution HDR composite image (a0+b0) shown in FIG. 18, for example. Also, the image synthesizing unit 122c generates a thumbnail image corresponding to the HDR synthesized image. The first image processing unit 122a and the second image processing unit 122b generate captured images before synthesis, moving images, and thumbnail images corresponding to these images. Then, the process proceeds to step S237.

In step S<b>237 , the HDR composite image generated in step S<b>225 , the captured image before composite, the moving image, the thumbnail image, and the like are stored in storage unit 110 .

Note that even if the first image processing unit 122a and the second image processing unit 122b convert a low-resolution image signal into a high-resolution image signal, and the image synthesizing unit 122c generates a high-resolution HDR synthesized image, good. Then, the process proceeds to step S204.

In step S204, the same processing as in step S104 in Embodiment 1 is performed, and an image or the like to be displayed on display unit 121 is selected. Then, the process proceeds to step S205.

In step S205, the same process as step S105 in Embodiment 1 is performed, and the image selected by the user in step S204 is displayed on display unit 121. FIG.

According to this embodiment, the light amount sensor 165 for measuring the light amount is provided, and the main control unit 101 causes the image input unit 220 to image the subject a plurality of times with different exposure amounts based on the motion vector and the light amount. .

According to this configuration, a more appropriate imaging mode can be selected according to the movement of the subject and the amount of light in the surroundings, so a high-quality HDR composite image corresponding to the imaging environment is generated.

Further, according to the present embodiment, the main control unit 101 compares the motion vector with the third motion threshold, and when it is determined that the motion information is less than the third motion threshold, the still image capturing mode is selected. , the first image input unit 123 is caused to continuously image the subject with different exposure amounts, and the image signal processing unit 122 is caused to generate an HDR composite image of the first resolution.

According to this configuration, since the subject hardly moves, even if the subject is imaged with a time difference, it is possible to suppress the occurrence of noise due to the movement of the subject. As a result, using only the first video input unit 123, a high-quality HDR composite image corresponding to the imaging environment is generated. Further, this makes it possible to use the second image input unit 224 for capturing a moving image, etc., and thus provides the image capturing apparatus 200 that is easy to use.

Further, according to the present embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the third motion threshold and the light amount exceeds the second light amount threshold, the main control unit 101 switches to the high light amount moving object imaging mode. By switching, the first video input unit 123 and the second video input unit 224 are caused to simultaneously image the subject with different exposure amounts, and the video signal processing unit 122 is caused to generate a high-resolution HDR composite image.

With this configuration, it is possible to suppress the occurrence of noise due to the movement of the subject, so that a high-quality, high-resolution HDR composite image (A0+B0) is generated in which the movement of the subject is large.

Further, according to the present embodiment, when the main control unit 101 determines that the motion vector is greater than or equal to the third motion threshold and the light intensity is greater than or equal to the first light intensity threshold and less than or equal to the second light intensity threshold, Switching to the medium light amount moving object imaging mode, causing the resolution conversion unit 228 to convert the resolution of the second imaging element 224 to the second resolution, and for the first image input unit 123 and the second image input unit 224, To photograph subjects at the same time by varying the amount of exposure. Then, the main control unit 101 converts the resolution of the image signal generated by the first video input unit 123 into the second resolution, and converts the resolution of the image signal generated by the second video input unit 224 into the first resolution. and causes the video signal processing unit 122 to generate an HDR composite image of the first resolution and an HDR composite image of the second resolution.

According to this configuration, the resolution of the second image sensor 224b is converted to a low resolution, and the area of the pixels after resolution conversion is increased, so the sensitivity of each pixel is improved. As a result, the imaging time (exposure time) of the second image input unit 224 is shortened, so even in situations where the amount of light is low, the occurrence of noise due to the movement of the subject can be suppressed, and the high-definition image corresponding to the imaging environment can be obtained. A quality HDR composite image (A0+b0u, A0d+b0) is generated.

In addition, according to this configuration, a plurality of HDR composite images having different resolutions are generated, so an HDR image suitable for the user's application is provided, and the user-friendly imaging device 200 is provided.

Further, according to the present embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the third motion threshold and the light amount is less than the first light amount threshold, the main control unit 101 switches to the low light amount moving object imaging mode. , causing the resolution conversion unit 228 to convert the resolutions of the first image pickup device 123b and the second image pickup device 224b to the second resolution, and for the first image input unit 123 and the second image input unit 224, The subjects are simultaneously imaged with different exposure amounts, and the video signal processing unit 122 is caused to generate an HDR composite image of the second resolution.

According to this configuration, in the first imaging element 123b and the second imaging element 224b, the resolution is converted to a lower resolution, and the area of the pixels after the resolution conversion increases, so the sensitivity of each pixel increases. improves. As a result, the imaging time of both the first video input unit 123 and the second video input unit 224 is shortened, so even in a situation where the amount of light is even lower, the occurrence of noise due to the movement of the subject can be suppressed. A high-quality HDR composite image (a0+b0) corresponding to the imaging environment is generated.

Further, according to the present embodiment, the main control unit 101 causes the resolution conversion unit 228 to group a plurality of pixels so that the resolutions of the first imaging element 123b and the second imaging element 224b are changed to The resolution of the first imaging element 123b and the second imaging element 224b is changed from the low resolution to the high resolution by converting the high resolution to the low resolution and causing the resolution conversion unit 228 to cancel the grouping of the plurality of pixels. convert to

According to this configuration, it is not necessary to prepare an imaging device for each resolution, so the imaging device can be downsized. In addition, this can suppress an increase in the manufacturing cost of the imaging device.

(Other embodiments)
In Embodiments 1 and 2 described so far, the main control unit 101 controls the motion information (for example, motion vector). However, in addition to this, for example, motion information of the subject may be detected based on the focal length of the image input unit 120 . Specifically, when the focal length of the image input unit 120 is short, the effect of camera shake is small, and when the focal length is long, the effect of camera shake is large. That is, when the focal length is short, the main control unit 101 detects motion information indicating that the motion of the subject is small, and detects motion information indicating that the motion of the subject is large when the focal length is long. Therefore, the main control unit 101 compares the focal length of the image input unit 120 with a first focal length threshold and a second focal length threshold larger than the first focal length threshold. When the main control unit 101 determines that the focal length of the image input unit 120 is less than the first focal length threshold value, the main control unit 101 switches to the still image capturing mode. Further, when the main control unit 101 determines that the focal length of the image input unit 120 is equal to or greater than the first focal length threshold value and equal to or less than the second focal length threshold, the main control unit 101 switches to the fine motion imaging mode. Further, when the main control unit 101 determines that the focal length of the image input unit 120 exceeds the second focal length threshold value, the main control unit 101 switches to the moving object imaging mode. Thus, even if the imaging mode is switched based on the focal length of the video input unit 120, the above-described effects can be obtained.

Furthermore, in Embodiments 1 and 2, the case where two cameras, the first video input unit 123 and the second video input unit 124 (224), capture an image of a subject and generate an HDR composite image has been described. For example, three or more cameras may be used to image the subject and generate an HDR composite image.

Although the embodiments of the present invention have been described above, it goes without saying that the configuration for realizing the technology of the present invention is not limited to these embodiments. Numerical values appearing in the text and drawings are merely examples, and the effect of the present invention is not impaired even if different values are used.

Some or all of the functions of the present invention described above may be realized by hardware, for example, by designing an integrated circuit, or a computer such as a microprocessor unit may implement a program for realizing each function. It may be realized by software by interpreting and executing, or may be realized by using both hardware and software.

Also, the control lines and information lines shown in the drawings are those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In practice, it may be considered that almost all configurations are interconnected.

DESCRIPTION OF SYMBOLS 100... Imaging apparatus 101... Main control part 104a... Basic operation execution part 104b... Camera function execution part 110a... Basic operation program storage area 110b... Camera function program storage area 120... Video input part 123... 3rd 1 image input unit 123b... first image sensor 124... second image input unit 124b... second image sensor 165... light intensity sensor 224... second image input unit 224b... second image sensor image sensor, 228... resolution converter

Claims (7)

a video input unit that captures an image of a subject and generates an image signal of the subject;
a video signal processing unit that generates a captured image of the subject based on the image signal;
Control for causing the image input unit to image the subject a plurality of times with different exposure amounts, and for causing the image signal processing unit to generate an HDR composite image of the subject based on the plurality of captured images with different exposure amounts. Department and
with
The video input unit includes a first video input unit that generates an image signal with a first resolution ; a second video input unit that generates an image signal with a second resolution different from the first video input unit; with
The control unit
The subject is continuously imaged with different exposure amounts for the first video input unit , and the video signal processing unit is processed based on the image signal generated by the first video input unit. a first imaging mode for generating the captured image of the first resolution using the captured image of the first resolution , and generating the HDR composite image using the captured image of the first resolution ;
The first image input unit and the second image input unit are caused to simultaneously image the subject with different exposure amounts , and the image signal processing unit is generated by the first image input unit. generate a captured image of the first resolution from the image signal of the first resolution generated by the second video input unit; generate a captured image of the second resolution from the image signal of the second resolution generated by the second video input unit; Further, converting the captured image of the second resolution to the first resolution, or converting the captured image of the first resolution to the second resolution, and converting the captured image of the first resolution and the first resolution. or generate the HDR composite image using the captured image converted to the second resolution and the captured image at the second resolution. , a second imaging mode;
comprising
Imaging device. The imaging device according to claim 1,
The control unit
detecting motion information of the subject based on the image signal;
comparing the motion information to a motion threshold;
When it is determined that the motion information is less than the motion threshold, the first imaging mode,
When it is determined that the motion information is equal to or greater than the motion threshold, the second imaging mode is selected,
An imaging device that switches control. The imaging device according to claim 1,
An image pickup device, wherein the resolution of a first image pickup element of the first image input section is different from the resolution of a second image pickup element of the second image input section. In the imaging device according to claim 3,
The imaging device, wherein the resolution of the second imaging element is lower than the resolution of the first imaging element. The imaging device according to claim 1,
A display unit that displays the HDR composite image,
The control unit
An imaging device that displays the HDR composite image on the display unit in different sizes based on the resolution of the HDR composite image. a video input unit that captures an image of a subject and generates an image signal of the subject;
a video signal processing unit that generates a captured image of the subject based on the image signal;
Control for causing the image input unit to image the subject a plurality of times with different exposure amounts, and for causing the image signal processing unit to generate an HDR composite image of the subject based on the plurality of captured images with different exposure amounts. Department and
with
The image input unit includes a first image input unit having an image sensor with a first resolution and a second image input unit having an image sensor with a second resolution different from the first resolution ,
The control unit
The subject is continuously imaged with different exposure amounts for the first video input unit , and the video signal processing unit is processed based on the image signal generated by the first video input unit. a first imaging mode for generating the captured image of the first resolution using the captured image of the first resolution , and generating the HDR composite image using the captured image of the first resolution ;
causing the video signal processing unit to generate a captured image of the first resolution based on the image signal generated by the first video input unit, and converting the captured image of the first resolution to the second resolution ; using the captured image converted to the second resolution and the captured image of the second resolution generated based on the image signal generated by the second video input unit; a third imaging mode to generate the HDR composite image;
comprising
Imaging device. In the imaging device according to claim 6 ,
A display unit that displays the HDR composite image,
The control unit
An imaging device that displays the HDR composite image on the display unit in different sizes based on the resolution of the HDR composite image.

Images