JP4335728B2 - Image processing apparatus and method - Google Patents

Description

  The present invention relates to an image processing apparatus that processes an imaging signal generated by an imaging device, and relates to, for example, an image processing apparatus and method that process imaging signals having different sensitivities to generate processed image data with a wide dynamic range. Is.

  In recent years, digital cameras equipped with a CCD solid-state image sensor or a CMOS type image sensor have become widespread. In such a digital camera, the output signal of the image sensor is converted into a digital signal, and the pixel signal generated at each pixel of the image sensor is processed as a digital image signal, whereby a desired image signal is converted into a semiconductor memory or the like. Can be recorded on the information recording medium.

  In the light receiving portion of the solid-state imaging device, a plurality of photodiodes are arranged in the vertical and horizontal scanning directions to form pixels that perform photoelectric conversion. Recently, due to the reduction in the light receiving area of the image sensor and the increase in the number of pixels, the pixel density has increased and the pixel area has decreased. For this reason, the dynamic range of the image sensor is relatively narrow, and the dynamic range is expanded or the exposure is changed so that a desired gradation can be reproduced.

For example, Patent Document 1 discloses a television camera device that combines two images with different exposure conditions. Patent Document 2 discloses an electronic still camera that performs bracket shooting in which a plurality of frames are continuously shot while changing the exposure amount during shooting.
JP 2001-94870 A Japanese Patent Laid-Open No. 11-4380

  However, with regard to the field of view taken by the photographer of the camera, it is possible to check the approximate image by displaying the result of the photographing with the digital camera on the monitor, but for example, photographing with the image displayed on the monitor on the liquid crystal display panel When checking the state, there is a problem that it is difficult to determine the details of whether the photographed image is the best or not. That is, it is difficult for the operator to determine how much the dynamic range of the subject is.

  Also, when capturing images of multiple frames while changing the exposure value by bracketing shooting, the subject cannot be shot at the same timing, so the same image can be obtained with moving objects. Can not. Also, with bracketing shooting, it is not possible to shoot for better reproduction including the highlight part. To reproduce the gradation of the highlight part, It is necessary to give an exposure that gives a state, and in this case, there is a possibility that the entire image becomes dark.

  In addition, since whether to reproduce the highlight area as “soft” or “hard” depends on the preference, there is a demand to increase the degree of freedom. In addition, even when performing a wider dynamic range process, if the dynamic range is increased more than necessary in a shooting scene in the scene with a small dynamic range, the image will be darkened as a whole. appear. Therefore, there is a problem that it is difficult for the photographer to appropriately determine the necessary dynamic range at the time of photographing.

  The present invention eliminates the disadvantages of the prior art, and an image processing apparatus that generates a plurality of wide dynamic range images having different dynamic ranges by performing wide dynamic range processing on low-sensitivity imaging signals and high-sensitivity imaging signals. And to provide a method.

  In order to solve the above-described problem, the present invention processes a first imaging signal from a high-sensitivity pixel and a second imaging signal from a low-sensitivity pixel that are disposed in an imaging unit that images an object scene. In the image processing apparatus, the apparatus pre-processes the first and second imaging signals by the same exposure, converts the first and second imaging signals into digital values, and performs the pre-processing and the converted first image. Preprocessing means for outputting data and second image data, and first and second having a plurality of conversion characteristics for gradation-converting the first and second image data processed by the preprocessing means, respectively. Gradation conversion means, calculation means for combining the gradation conversion outputs of the first and second gradation conversion means, and gradation conversion characteristics in the gradation conversion means are automatically selected, and a plurality of dynamic ranges can be selected. Composite image data of multiple frames Characterized in that it comprises an automatic gradation processing means for made.

  In this case, the apparatus includes detection means for detecting the maximum dynamic range of the object scene, and the automatic gradation processing means may select the gradation conversion characteristics based on the detection result by the detection means, and the detection The means may detect the maximum dynamic range based on the second image data generated by pre-processing, and the automatic gradation processing means may synthesize a plurality of frames of a plurality of dynamic ranges according to the maximum dynamic range. The processing for generating image data may be limited.

  The apparatus may be an imaging apparatus including an imaging unit, and may further include a recording unit that records the composite processed image data. The apparatus may include a control unit that controls the imaging unit, and the control unit may have an exposure bracket shooting mode for capturing an image of the object scene in a stepwise exposure, and the control unit may further include an exposure bracket shooting mode. The first and second image data may be generated for each photographing frame from the first and second imaging signals of the plurality of frames generated in step S2.

  In order to solve the above-described problem, the present invention provides a first imaging signal from a high-sensitivity pixel and a second imaging signal from a low-sensitivity pixel disposed in an imaging unit that images the object scene. In the image processing method to be processed, this method pre-processes the first and second imaging signals by the same exposure, converts the first and second imaging signals into digital values, and performs the pre-processing and the converted first first. A preprocessing step for outputting the image data and the second image data, and first and second for converting the gradation of the first and second image data processed in the preprocessing step with a plurality of conversion characteristics, respectively. Automatically select the gradation conversion process, the calculation process for synthesizing the values converted in the first and second gradation conversion processes, and the gradation conversion characteristic to be processed in the gradation conversion process. Multiple frames of multiple dynamic ranges Characterized in that it comprises an automatic gradation processing step of generating the data.

  In this case, this method includes a detection step of detecting the maximum dynamic range of the object scene, and the automatic gradation processing step may select the gradation conversion characteristic based on the detection result of the detection step, and the detection. The process may detect the maximum dynamic range based on the second image data generated by pre-processing, and the automatic gradation processing process may be a combination process of a plurality of frames of a plurality of dynamic ranges according to the maximum dynamic range. The processing for generating image data may be limited.

  In addition, this method may include a recording step of recording the composite processed image data, and also includes a control step of controlling the imaging means, and the exposure is performed in an exposure amount bracket shooting mode in which the object scene is captured at a stepwise exposure. The scene may be imaged. In this case, the control process takes the first and second image data from the first and second imaging signals of the plurality of frames generated in the exposure amount bracket shooting mode for each shooting frame. It should be generated.

  According to the present invention, when performing wide dynamic range processing using respective imaging signals obtained from low-sensitivity sub-pixels and high-sensitivity main pixels arranged in the imaging device, gradation conversion having different conversion characteristics is performed. By processing and synthesizing with a table, a plurality of wide dynamic range images having different dynamic ranges can be automatically generated, and the processed image data can be output and recorded on an information recording medium or the like.

  In addition, for example, the dynamic range of the object scene such as the block maximum value is measured from the photometric result such as the block integrated value, and the wide dynamic range processing up to the stage where the gradation up to the dynamic range can be reproduced is automatically performed. It is possible to automatically limit the wide dynamic range processing beyond that level, so appropriate processing according to the shooting scene is performed to reduce the processing load and the number of recorded sheets is unnecessary Can be prevented.

  Furthermore, an image preferred by the user can be selected from a plurality of high quality processed images that have been automatically processed with a wide dynamic range within such an appropriate range, and can be used as a final image. In addition, it is possible to perform a plurality of wide dynamic range processes by using stepwise photography with exposure correction. In this case, the range of selection of a user's favorite image is widened. The degree of freedom for selecting an image can be expanded.

  Next, embodiments of an image processing apparatus and method according to the present invention will be described in detail with reference to the accompanying drawings.

  Referring to FIG. 2, there is shown an embodiment in which the image processing apparatus and method according to the present invention are applied to a digital camera. As shown in the figure, the digital camera 20 processes the image signal of the object scene imaged by the imaging unit 22 with an analog front end circuit (AFE) 24, a signal processing unit 26, and a compression encoding unit 28, The imaging apparatus records the processing data on an information recording medium loaded in the recording apparatus 30, and is an imaging apparatus capable of displaying the captured image on the display unit 32 on a monitor.

  The imaging unit 22 is an imaging unit including an imaging lens (not shown), a diaphragm, a mechanical shutter, and a drive circuit that drives these, and a solid-state imaging device that photoelectrically converts an optical image formed through the imaging lens. An optical image of the object scene is formed on the imaging surface of the solid-state imaging device and exposed for a predetermined time, thereby outputting a color imaging signal corresponding to the formed image.

  The solid-state image sensor is a two-dimensional image sensor in which an imaging region 310 based on effective pixels is arranged in 1416 pixels in the horizontal scanning direction and 2128 pixels in the vertical scanning direction, as in the imaging device 300 shown in FIG. In the embodiment, a CCD solid-state imaging device in which each vertical charge transfer path VCCD and horizontal charge transfer path is formed by a charge imaging element (CCD) is used. Note that the image sensor 300 may be configured with a CMOS image sensor. Around these imaging areas 310, optical black (OB) portions that shield the light-sensitive areas and generate optical black level signals are arranged with a predetermined width around the imaging area 310. Omitted.

  FIG. 4 partially shows a configuration around a photodiode that constitutes a pixel of the photoelectric conversion unit arranged on the imaging surface of the imaging device 300. As shown in the figure, the photodiode PD is, for example, an octagonal polygon. The photosensitive region of the photodiode PD is made relatively to the main pixel PDL having a relatively large light receiving area by the light shielding member 310 formed on the upper surface and bent obliquely and horizontally and the separation region below the light shielding member 310. This is a double photosensitive region configuration divided into sub-pixels PDS having a small light receiving area. The main pixel PDL has a larger amount of signal charge generation and higher light receiving sensitivity depending on the area ratio than the sub-pixel PDS. Therefore, the pixel PDL saturates faster than the pixel PDS when receiving the same amount of light. This is shown in FIG.

  In the figure, as an example, it is a graph showing the image sensor output with respect to the input luminance when the area ratio of the main pixel PDL and the sub-pixel PDS is 4: 1. The main pixel output 500 is saturated at the 100% luminance level, The subpixel output 510 is not saturated until the 400% luminance level, and the sensitivity ratio is 16: 1. When the value of 100% to 400% of the sub-pixel output 510 is combined with the main pixel output 500, the dynamic range can be expanded so that the gradation of the highlight part can be reproduced as shown in the characteristic 520. it can.

  Returning to FIG. 4, the signal charges generated in the pixels PDL and PDS are respectively transferred to the corresponding vertical charge transfer path VCCD in response to a read pulse for one area of the adjacent vertical charge transfer path VCCD. Signal charges generated in the PDL and the pixel PDS can be independently shifted and read out. It is also possible to read out the signal charges generated in the pixels PDL and PDS to the vertical charge transfer path VCCD, mix them, and transfer them as a set of pixels.

  Each photodiode PD is arranged in a plane with a ½ pixel interval shifted in the horizontal and vertical scanning directions, and each vertical power transfer path VCCD is arranged to sew between each photodiode PD in a zigzag manner. ing. A transfer electrode (not shown) is connected to each region of each VCCD to form a potential well corresponding to the applied potential, and a vertical transfer pulse that changes the position of the potential well is supplied, thereby providing a photodiode. The signal charge shifted from the PD is transferred in the direction of a horizontal charge transfer path (not shown).

  In addition, on the upper surface of these photodiodes PD, for example, RGB primary color filters or complementary color filters, and microlenses for condensing light on each main pixel PDL and subpixel PDS, for example, each photodiode PD unit Are not shown in the drawing. The signal charge is vertically transferred through the vertical charge transfer path (VCCD) and further horizontally transferred through the horizontal charge transfer path by a horizontal transfer pulse. The horizontally transferred signal charge is detected by an output amplifier (not shown) connected to the end of the horizontal charge transfer path, and the output amplifier picks up an electric signal corresponding to the signal charge amount as a pixel signal representing the pixel value of each pixel. Output from unit 22.

  Returning to FIG. 2, the image pickup device 300, the image pickup lens, the diaphragm, and the mechanical shutter of the image pickup unit 22 are provided with a release signal (S 1, S 2) and an electronic shutter control signal (ES) supplied from a timing signal generation circuit (TG) 40. And a timing signal such as a mechanical shutter opening / closing signal, a drive pulse such as a transfer pulse, and various control signals.

  The output of the imaging unit 22 is connected to an analog front end circuit (AFE) 24, and the AEF 24 is a correlated double sampling circuit (CDS) that performs correlated double sampling of the output of the imaging unit 22, an amplification circuit, and an analog / An analog signal processing circuit including a digital (AD) conversion circuit. The AD conversion circuit in this embodiment converts an input signal into, for example, a 14-bit digital signal and outputs it. The AFE 24 outputs a digital image signal generated by converting the imaging signal to digital to the signal processing unit 26. The AFE 24 processes each pixel value by a timing signal such as a pixel clock supplied from the TG 40.

  The timing signal generation circuit (TG) 40 generates these timing signals in accordance with control signals given from the control circuit (CPU) 50. In addition, when a light emission control signal is output from the control circuit 50 to the speedlight 52 during the exposure period by the timing control circuit 40, the TG 40 outputs a signal for controlling the timing for causing the speedlight 52 to emit light. The irradiation time can be controlled by adjusting the timing of the flash time according to the instruction.

  The signal processing circuit 26 connected to the output of the AFE 24 generates various information for imaging by processing the input digital pixel signal, and creates a desired image signal from the input pixel signal. An image forming processing unit. The signal processing circuit 26 stores an image signal in a memory (DRAM) 54 connected to the bus 60 and sequentially processes each data of the main pixel PDL and the sub-pixel PDS with respect to the stored data. It has a processing function for generating image data with a wide dynamic range by synthesizing image data from the main pixel PDL and sub-pixel PDS. The memory 54 has a storage area for storing at least two frames of image data of one frame of the main pixel PDL and image data of the sub-pixel PDS. Note that the storage capacity of the memory 54 may further include a storage area for storing image data for a plurality of frames in accordance with, for example, a shooting interval when performing exposure amount bracket shooting (step exposure). Further, the storage capacity of the memory 54 may include a storage area for temporarily storing image data of a plurality of frames after processing according to the number of frames of the processed image generated in the dynamic range / bracket processing mode described later. .

  The signal processing circuit 26 performs various processes under the control of the control circuit 50 via the bus 60, and performs a plurality of processes by changing the dynamic range width by a predetermined amount by the functions of the signal processing circuit 26 and the control circuit 50, etc. It has a dynamic range bracket processing mode for automatically generating a frame processed image. This dynamic range bracket processing mode can be selected in advance so as to function effectively when the mode is set by the operator. The detailed configuration of the signal processing circuit 26 will be described later.

  A compression encoding unit 28, a recording device 30, and a display unit 32 are further connected to the bus 60. The compression encoding processing unit 30 is a processing unit that performs information amount suppression for recording the image data processed by the signal processing circuit 26 on an information recording medium or transmitting the data to the outside. Then, compression encoding processing corresponding to the moving image is performed. The compressed and encoded data is sent to, for example, the recording device 30 and recorded on an information recording medium such as a semiconductor memory or an optical disk. When display data is created by the signal processing circuit 26, or when display data is created by decoding the encoded data by the compression encoding unit 28, the display data is supplied to the display unit 32. Then, the display unit 32 displays an image and video corresponding to the input display data on a display device such as a liquid crystal display panel.

  A control circuit (CPU) 50 connected to the bus 60 is a computer circuit that controls imaging recording and monitor reproduction and its peripheral circuits, and is operated and set by the operation unit 70, which is operated by firmware that controls each part. A control signal for controlling each unit is supplied to each functional unit according to the operation information.

  A detailed configuration diagram of the signal processing circuit 26 is shown in FIG. As shown in the figure, the signal processing circuit 26 calculates the black reference by processing the optical black (OB) portion of the main pixel data (high) generated and input by the main pixel PDL, and also the data of the OB portion. OB processing unit 100 that deletes and outputs, linear matrix processing unit 102 that calculates each pixel value and corrects the spectral characteristics, and calculation processing that corrects the spectral sensitivity of each color component and adjusts the white balance (WB) And a WB gain processing unit 104 that performs a preprocessing circuit 106 that performs preprocessing on main pixel data (high) from the main pixel PDL.

  Further, the signal processing circuit 26 calculates the black reference level by processing the optical black (OB) portion of the subpixel data (low) generated and input by the subpixel PDS and deletes the data of the OB portion. An OB processing unit 108, a linear matrix processing unit 110 that performs a matrix operation on each pixel value, and a WB gain processing unit 112 that performs an operation process for adjusting white balance (WB), which are subpixels from the subpixel PDS. A pre-processing circuit 114 that performs pre-processing for data (low) is configured. The linear matrix processing unit 110 performs matrix processing on the image data obtained by generating the three primary colors R, G, B pixel data and the luminance data YH, and generates processed image data Y, Cr, Cb. The output of the linear matrix processing unit 110 is connected to the integration block circuit 116 together with the WB gain processing unit 112.

The integration block circuit 116 is an arithmetic circuit that detects the average luminance and the maximum value of each block in the photometry area set in the imaging region 310 based on the image data Y generated by the linear matrix processing unit 110. Specifically, as shown in FIG. 3, a 16 × 16 divided integration area 320 divided into 16 blocks in the horizontal direction and 16 × 16 blocks in the vertical direction is set at the center of the imaging region 310, and the luminance level of these 256 blocks is set. An average value of the values integrated for each block is calculated as a block integrated value, and a block maximum value Gmax having the maximum block integrated value is detected. Further, the integration block circuit 116 has a TTL metering function that outputs the calculated block integration value to the control circuit 50 and causes the control circuit 50 to determine the exposure amount (EV) at the time of shooting. The integration block circuit 116 outputs the detected block maximum value Gmax to the automatic gradation processing circuit 118.

  Based on the block maximum value Gmax supplied from the integration block circuit 116, the automatic gradation processing circuit 118 generates a plurality of D range processed images having different dynamic ranges (hereinafter, sometimes abbreviated as “D range”). When performing dynamic range bracket processing to be generated, an automatic gradation processing function for sequentially determining a dynamic range to be processed and automatically selecting a synthesis table of gradation conversion characteristics corresponding to the determined dynamic range is provided. The automatic gradation processing circuit 118 has a function of generating a composite image data of a plurality of images having different dynamic ranges by determining a dynamic range when combining captured images and selecting a gradation conversion characteristic. The automatic gradation processing circuit 118 in this embodiment divides the dynamic range into, for example, six stages, and selects gradation conversion characteristics corresponding to these plural stages. Specifically, a γ correction circuit 122 (to be described later) and a conversion table stored in the lookup table 130 are sequentially selected according to each dynamic range.

  Further, the automatic gradation processing circuit 118 has a function of automatically limiting the number of frames created with processed images having different dynamic ranges based on the block maximum value Gmax supplied from the integration block circuit 116. Specifically, the automatic gradation processing circuit 118 determines how many times the dynamic range of the object scene can be expressed by the image on the high sensitivity side of the main pixel PDL, for example, the state of the shooting scene, for example, the block maximum Judgment is made based on the value Gmax, and when dynamic range bracket mode is set, processing up to the dynamic range including the block maximum value Gmax is performed, and dynamic range image generation processing exceeding the dynamic range is automatically performed. Control not to do.

  The calculation process of the dynamic range of the object scene will be described. The automatic gradation processing circuit 118 indicates that the maximum value of the main pixel PDL is represented by 14 [bit] and the saturation level of the sub-pixel PDS as shown in FIG. Is 1/4 of the main pixel, so the value of the sub-pixel PDS is represented by 12 [bit] of "0 to 4095", and the maximum value "4095" is set to a dynamic range of 400% according to the calculation formula (1) Determine the dynamic range Dx of the object scene.

Dx = block maximum value Gmax x 400 [%] / (4095) [%] (1)
Note that the automatic gradation processing circuit 118 may limit the width of the dynamic range to be processed in consideration of, for example, the actual exposure amount and exposure correction amount controlled by the control circuit 50 during automatic exposure.

  Returning to FIG. 1, the outputs of the WB gain processing units 104 and 112 constitute the outputs of the preprocessing circuits 106 and 114, and are connected to the γ correction circuits 120 and 122, respectively. The γ correction circuits 120 and 122 are circuits for correcting gamma (γ) of main pixel data (high) of 14-bit data and sub-pixel data (low) of maximum 12-bit data, respectively. The main pixel data (high) represented by, for example, 8 bits is output after being corrected to the gradation corresponding to the processing purpose such as the output form. The other γ correction circuit 122 includes a plurality of conversion tables for correcting the subpixel data (low), and uses the conversion table of the conversion characteristics selected by the selection signal supplied from the automatic gradation processing circuit 118 to use the subpixels. Unlike the γ correction circuit 120 in that the data (low) is subjected to gradation conversion, the converted sub-pixel data (low) is output. The output of the γ correction circuit 120 that processes the main pixel PDL is connected to a lookup table (LUT) 130.

  The look-up table (LUT) 130 is a gain table in which gradation characteristics corresponding to a plurality of dynamic range parameters for adjusting the gain are respectively set in advance for the main pixel image data (high) from the main pixel PDL. h_gain) is a gradation conversion circuit including 132. Furthermore, the look-up table (LUT) 130 is a gain table in which gradation characteristics corresponding to a plurality of dynamic range parameters for adjusting gain are set in advance for sub-pixel image data (low) from the sub-pixel PDS. (l_gain) 134 is included. Each of the gain table (h_gain) 132 and the gain table (l_gain) 134 stores a plurality of conversion tables corresponding to six stages of dynamic ranges. Each conversion table converts input data into 8-bit data. Convert and output.

  The output of the gain table (h_gain) 132 and the value before the table conversion, that is, the output of the γ correction circuit 120 are connected to a multiplier 140 that multiplies the input data. The output of the gain table (l_gain) 134 and the output of the γ correction circuit 122 generated from the sub-pixel PDS and processed are connected to a multiplier 142 that multiplies the input data. Furthermore, the outputs of the multipliers 140 and 142 are connected to an adder 144 that adds and calculates input data, and the output of the adder 144 is connected to a limiter circuit 146. The limiter circuit 146 is a circuit that limits the output data of the adder 144 to, for example, 8-bit data, and outputs a fixed value of “255” when data of a value “256” or more is input. The output of the limiter circuit 146 forms the output of the signal processing unit 26 and is connected to the bus 60 via the bus interface. As described above, the gain table (h_gain) 132 is connected to the output of the γ correction circuit 120 and is set with a table value for determining a weight for gradation conversion of the value of the main pixel PDL. The gain table (l_gain) 134 is connected to the output of the γ correction circuit 122 and is set with a table value for determining a weight for gradation conversion of the value of the subpixel PDS.

  Here, the gradation conversion table stored in the lookup table (LUT) 130 and the wide dynamic range synthesis process will be described. When a synthesis formula for the wide dynamic range synthesis processing is shown, the synthesized value “data” output to the limiter circuit 146 is derived by the following formula (2).

“data” = (high + MIN (high / th, 1) × low) × MAX (1-lg (high / th), p) (2)
= MAX (1-lg (high / th), p) × high + (MAX (1-lg (high / th), p) × MIN (high / th, 1)) × low (3)
here,
h_gain = MAX (1-lg (high / th), p) (4)
l_gain = MAX (1-lg (high / th), p) × MIN (high / th, 1)) = h_gain × wl (5)
Then,
data = h_gain × high + l_gain × low (6)
It is.

  As shown in Equation (6), the operation result obtained by multiplying the main pixel data (high) and the gain table (h_gain) 132 by the multiplier 140, the sub-pixel data (low), the gain table (l_gain) 134, The operation result obtained by multiplying the multiplication result by the multiplier 142 and the addition result by the adder 144 is output to the limiter circuit 146.

  The gain table (h_gain) 132 of the LUT 130 has an input / output characteristic 600 as shown in FIG. 6, for example, and in the embodiment, a plurality of types (in accordance with parameters “th”, “p”, “lg”) ( 6 patterns) conversion table. The gain table (l_gain) 134 is obtained by multiplying the input / output characteristics shown in FIG. 6 by, for example, the input / output characteristics 700 shown in FIG. 7, and in the embodiment, the parameters “th”, “p”, “lg” A plurality of types (six patterns) of conversion tables set according to "are provided.

  As shown in FIG. 8, the automatic gradation processing circuit 118 changes the dynamic range up to 400 [%], for example, in order to switch the dynamic range processing to the processing patterns of a plurality of steps (a) to (f). The processing range is divided into 6 stages of range thresholds D12, D23, D34, D54, D56 and D56 or more, and the sub-pixel γ table (BLUT 0-5) and synthesis table (MIXLUT 0-5) corresponding to each range. Are selected based on the correspondence table 800. The automatic gradation processing circuit 118 outputs a selection signal for sequentially selecting one of the sub-pixel γ tables (BLUT: 0 to 5) for each processing stage to the γ correction circuit 122, and the synthesis table (MIXLUT: 0 to 5). ) Is output to the look-up table (LUT) 130. This selection signal may be, for example, address information such as a table number for directly designating each address of the sub-pixel γ table (BLUT 0 to 5) and the synthesis table (MIXLUT 0 to 5). It should be noted that the automatic gradation processing circuit 118 can perform a wide dynamic range combining process up to a stage including the block maximum value Gmax and limit the dynamic range process beyond that stage to be not performed.

  Returning to FIG. 2, the control circuit (CPU) 50 is a system control unit that controls each unit according to operation information operated and set by the operation unit 70, and stores a microcomputer circuit, its firmware, and data. Including elements and computer peripheral circuits.

  When the control circuit 50 recognizes the shutter release half-press (S1) of the release operation detected by the operation unit 70, the control circuit 50 preliminarily shoots the subject field, and performs photometry and distance measurement to obtain shooting control information. Based on the control information, the exposure value and the focus position are controlled, and control for causing the display unit 32 to display the captured image on the monitor is performed.

  Further, when recognizing the shutter release full press (S2), the control circuit 50 causes the subject to be photographed based on, for example, photographing control information. When an exposure amount bracket shooting mode is instructed from the operation unit 70, the control circuit 50 performs exposure exposure bracketing for stepwise exposure using the determined exposure value and an exposure value obtained by positively correcting and / or negatively correcting the exposure value. Control shooting. In the exposure bracket shooting mode, for example, continuous shooting is performed with stepwise exposure with correction amounts of −1 / 3EV, 0EV, + 1 / 3EV, + 2 / 3EV, + 1EV,.

  Further, when the dynamic range / bracket processing mode is instructed from the operation unit 70, the control circuit 50 controls the signal processing unit 26 so as to obtain a wide dynamic range synthesis process for the captured image data so as to obtain a plurality of different dynamic range images. It has the function to do. In this embodiment, at the time of exposure bracket shooting, a plurality of wide dynamic range processed images can be generated when shooting on the plus correction side in a state where the dynamic range bracket processing mode is set. In this case, the processing for generating the wide dynamic range processed image can be limited to one frame at the time of shooting on the minus correction side and at the time of non-correction (± 0 EV) shooting.

  With reference to FIG. 9, the operation of the digital camera 20 in the present embodiment having the above-described configuration will be described. FIG. 9 shows a basic imaging sequence of the digital camera 20. First, in the period S1 after the shutter release is half-pressed, the mechanical shutter is released, and the imaging control for capturing the monitor image is performed based on the imaging signal, and the exposure amount and the focus adjustment are automatically adjusted. The picked-up image is thinned out, and a moving image of 15 to 30 frames per second is displayed on the display unit 32, for example.

  Therefore, when the shutter release full-press is detected, the electronic shutter pulse ES is given to the image sensor 300 of the imaging unit 22 in the period S2. The image pickup device 300 stops the sweeping at a timing corresponding to the electronic shutter pulse ES after the high-speed sweeping of unnecessary charges, and the mechanical shutter is continuously released. The image sensor 300 generates signal charges according to the amount of received light, and when the mechanical shutter is closed and the exposure time ends, the signal charge generated in the main pixel PDL is responsive to the field shift pulse FS1 for the main pixel PDL. Then, it is read out and transferred to the vertical and horizontal charge transfer paths, and a pixel signal corresponding to the signal charge is output. Pixel signals from these main pixels PDL are processed by the AFE 24 and the signal processing unit 26 and stored in the memory 54.

  Next, when the field shift pulse FS2 for the sub-pixel PDS is supplied from the TG 40 to the image sensor 300, the signal charge generated in the sub-pixel PDS is read out to the vertical and horizontal charge transfer paths, and according to the signal charge A pixel signal is output. Pixel signals from these sub-pixels PDS are processed by the AFE 24 and the signal processing unit 26 and stored in the memory 54. While the sub-pixel capture is performed, the signal processing unit 26 performs automatic white balance adjustment, pixel defect correction, OB processing, linear matrix correction, and white balance gain processing on the main pixel signal already stored in the memory 54 by main pixel capture. Etc. are pre-processed.

  When the sub-pixel capture is completed, the sub-pixel signal stored in the memory 54 is pre-processed by the signal processing unit 26, and then the pre-processed main pixel PDL and sub-pixel PDS image data are stored in the sub-pixel signal. Tone synthesis is performed and automatic gradation processing is performed to widen the dynamic range, and the processed composite image data having a wide dynamic range is output from the signal processing unit 26 to, for example, the compression coding processing unit 28 and compressed and encoded. Then, the encoded data is recorded on the information recording medium by the recording device 30, and thus a series of imaging sequences is completed.

  Here, the operation of the digital camera 20 when the dynamic range bracket shooting mode and the exposure amount bracket shooting mode are set will be described with reference to FIG. 10. First, in step S1000, operation information to the operation unit 70 Thus, when the D range bracket shooting mode and the exposure amount bracket shooting mode are instructed, each mode is set to the on state. Next, when it is detected in S1010 that the release switch S1 is turned on, the process proceeds to step S1020 to shift to the preliminary photographing operation state. In this operation, as in the S1 period shown in FIG. 9, the mechanical shutter is released and moving image shooting is started, and imaging control information for automatic exposure and automatic focus adjustment is generated from the moving image signal. The captured moving image is displayed on the monitor.

  Next, when an on operation of the release switch S2 is detected in S1030, the process proceeds to step S1040 and main imaging is performed. In this case, since the exposure bracket shooting mode is set, the exposure amount corresponding to the exposure correction step is selected from among a plurality of steps.

  In the main imaging operation, as in the S2 period shown in FIG. 9, one frame of exposure is started at the exposure timing corresponding to the electronic shutter pulse ES while the mechanical shutter is continuously released. The exposure is completed by the electronic shutter operation (charge sweeping operation) or the closing of the mechanical shutter at the stage exposure shooting. In step S1040, similarly to the period S2 shown in FIG. 9, the pixel signal corresponding to the signal charge generated in each of the main pixel PDL and the sub-pixel PDS is preprocessed by the signal processing unit 26 and stored in the memory 54. Stored in The signal processing unit 26 processes the pixel signals fetched into the memory 54, and further performs a wide dynamic range process in a range corresponding to the subject. Specifically, in step S1050, the sub-pixel input data is subjected to matrix processing in the pre-processing circuit 114 of the signal processing unit 26 and converted into image data Y, Cr, Cb, and in step S1050, image data representing a luminance signal Y is input to the integrating block circuit 116.

  In step S1050, the luminance levels of 256 blocks in the imaging region 310 are integrated for each block, and the maximum block value Gmax having the maximum value among them is detected (step S1060). In this embodiment, the block maximum value Gmax is calculated from the image data Y generated from the sub-pixel PDS. However, the present invention is not limited to this. For example, the generated signal charge of the main pixel PDL and the generated signal charge of the sub-pixel PDS are each a vertical charge. The integrated value of each block and the block maximum value Gmax may be obtained from image signal data that is mixed with the transfer path VCCD, read from the image sensor 300, and preprocessed.

  The automatic gradation processing circuit 118 calculates the above equation (1) based on the block maximum value Gmax to recognize the dynamic range Dx of the object scene, and includes the dynamic range Dx from the D range 100 [%], for example. Select a dynamic range in the range up to the minimum D range xx [%]. In this embodiment, for example, as shown in FIG. 8 and FIG. 11, these D ranges have six stages (a) for each of the dynamic range threshold values “less than D12”, “D23”, “D34”, “D45”, and “D56 or higher”. ) To (f), the automatic gradation processing circuit 118, for example, if the recognized dynamic range Dx is 200 [%], the steps (a), (b), (c) And the conversion table corresponding to (d) is selected sequentially.

  In this routine, first, the sub-pixel γ table BLUT0 and the synthesis table MIXLT0 corresponding to the range of the step (a) are selected. Selection signals for selecting these tables are supplied to the γ correction circuit 122 and the LUT 130, respectively, and each circuit performs gradation conversion on the input image data using the selected table. The table-converted image data is subjected to a synthesis operation by multipliers 140 and 142 and an adder 144, and is output via a limiter circuit 146. The output data of the signal processing unit 26 is encoded by, for example, the compression encoding unit 28, and then the processed image data is supplied to the recording device 30 and recorded on the information recording medium (S1090).

  Next, when proceeding to step S1100, the presence / absence of the next D range process is confirmed. If the process is present (Yes), the process returns to step S1080 and the subpixel γ table corresponding to the stage (b) of the next range. (BLUT1) and the synthesis table (MIXLT1) are selected, and the subsequent processing is continued. Conversely, if it is determined in step S1100 that there is no D range processing at the next stage (No), the process proceeds to step S1110.

  For example, when the dynamic range Dx is 200 [%], wide dynamic range processing using the sub-pixel γ table (BLUT3) and synthesis table (MIXLUT3) corresponding to the stage (d) up to the range shown in FIG. Is done. Proceeding to step S1110, it is determined whether or not there is shooting with the next stage exposure, and if it is determined that there is shooting (Yes), the process returns to step S1040 and the step exposure according to the exposure amount bracket shooting is performed. Processing after the main imaging is performed for one frame is continued. When the stepwise exposure shooting of the set correction range is completed and the set D range bracket processing is completed in each shooting, and the processed image data is recorded on, for example, an information recording medium, the digital camera 20 performs shooting. Return to standby state.

  Referring now to FIG. 12, the combined gradation conversion characteristics when the dynamic range processing is 100%, 130%, 230% and 400% are shown. In the figure, the gradation conversion characteristics when the dynamic range processing is 130% and 300% are omitted to avoid complication of the drawing. As shown in the figure, in the output level (0 to 255 level) of the final signal processing image according to the subject luminance, the dynamic range processing 100% is indicated by the characteristic 1200, the 130% case is indicated by the characteristic 1210, The case of 230% is indicated by characteristic 1220, and the case of 400% is indicated by characteristic 1230.

  For example, when the block maximum value Gmax detects that the dynamic range of the field luminance is 350%, it is determined that the processing from FIG. 11 to step (f) is performed, and step (a) A wide dynamic range processed image of 100 to 400% from step (f) to step (f) can be created and recorded for six exposures for one exposure.

  In this embodiment, for each frame of the exposure bracket shooting of the step exposure, as shown by the characteristics 1200 to 1230, a plurality of wide dynamic range processes within the range limited according to the block maximum value Gmax are performed. A plurality of images made are synthesized and created, and recorded on an information recording medium. This is because, for example, when shooting with a positive correction in the overexposure direction than the appropriate exposure value obtained by photometry, it is possible to perform composition processing that applies high-intensity compression, so the gradation in the highlight area is reproduced. Possible composite image data can be recorded and saved. When the block maximum value is 100% or less, it is preferable to record a processed image by performing one dynamic range process at each step exposure. In addition, because the processing range is limited, for example, for image data with a dynamic range of 100% or less with a low block maximum value, it is limited to perform a wide dynamic range processing of 400%, for example, preventing excessive processing. In addition, the recording area of the information recording medium can be prevented from being compressed.

It is a block diagram which shows the detailed structural example of the signal processing part in the digital camera in the Example shown in FIG. 1 is a block diagram when an image processing apparatus to which the present invention is applied is applied to a digital camera. It is the schematic which shows the 16x16 division | segmentation integrated area set to the imaging area | region of an imaging device with which an imaging part is provided, and an imaging area | region. It is a figure which shows the structural example of the light-receiving part of an image pick-up element. It is a figure which shows the image pick-up element output state with respect to input luminance in case the area ratio of the main pixel of an image pick-up element and a subpixel is 4: 1. It is a figure which shows an example of the input / output property of the gain table (h_gain) in a look-up table (LUT). It is a figure which shows an example of the input / output property of the gain table (l_gain) in a look-up table (LUT). It is a figure which shows the correspondence of the subpixel (gamma) table (BLUT0-5) corresponding to the process pattern of several steps (a)-(f), and a synthetic | combination table (MIXLUT0-5). It is a figure which shows the basic imaging sequence of the digital camera in an Example. It is a flowchart showing operation | movement of the digital camera by which dynamic range bracket imaging mode and exposure amount bracket imaging mode were set. The dynamic range is divided into six stages (a) to (f) by the dynamic range thresholds “D12”, “D23”, “D34”, “D45”, “D56”, and sub-pixel γ tables ( It is a figure for demonstrating the operation | movement which selects a BLUT 0-5 and a synthetic | combination table (MIXLUT 0-5). It is a figure which shows the example of a synthetic | combination gradation conversion characteristic in case dynamic range processing is 100%, 130%, 230%, and 400%.

Explanation of symbols

20 Digital camera
22 Imaging unit
26 Signal processor
50 Control circuit (CPU)
54 Memory (DRAM)
106,114 Pre-processing circuit
116 Integration block circuit
118 Automatic gradation processing circuit
120,122 γ correction circuit
130 Look-up table (LUT)
132 Gain table (h_gain)
134 Gain table (l_gain)
140,142 multiplier
144 Adder
146 Limiter circuit

Claims (15)

A second imaging signal obtained in the first image pickup signal and the low-sensitivity obtained with high sensitivity which is disposed in the signal indicating the object scene image thus obtained imaging means for imaging an object scene In the image processing apparatus for processing
First and second imaging signals by the same exposure are preprocessed, the first and second imaging signals are converted into digital values, and the preprocessed and converted first image data and second image data Preprocessing means for outputting
First and second gradation conversion means each having a plurality of conversion characteristics for gradation conversion of the first and second image data processed by the preprocessing means;
Arithmetic means for combining the gradation conversion outputs of the first and second gradation conversion means;
The processing range of the dynamic range is divided into predetermined threshold values, and the gradation conversion characteristics in the gradation converting means are automatically selected for each of the divided processing ranges, and a composite processed image of a plurality of frames in a plurality of dynamic ranges An image processing apparatus comprising: automatic gradation processing means for generating data. The image processing apparatus according to claim 1, wherein the apparatus includes detection means for detecting a maximum dynamic range of the object scene,
The image processing apparatus, wherein the automatic gradation processing means selects the gradation conversion characteristics based on a detection result by the detection means.   3. The image processing apparatus according to claim 2, wherein the detection unit detects the maximum dynamic range based on the second image data generated by the preprocessing.   4. The image processing apparatus according to claim 3, wherein the automatic gradation processing means limits processing for generating composite image data of a plurality of frames having a plurality of dynamic ranges according to the maximum dynamic range. Image processing device.   The image processing apparatus according to claim 1, wherein the apparatus is an imaging apparatus including the imaging unit.   6. The image processing apparatus according to claim 5, wherein the apparatus includes recording means for recording the composite processed image data.   6. The image processing apparatus according to claim 5, wherein the apparatus includes a control unit that controls the imaging unit, and the control unit has an exposure amount bracket shooting mode that images the scene with stepwise exposure. An image processing apparatus.   The image processing apparatus according to claim 7, wherein the control unit obtains the first and second image data from first and second imaging signals of a plurality of frames generated in the exposure bracket imaging mode. An image processing apparatus characterized by generating for each photographing frame. A second imaging signal obtained in the first image pickup signal and the low-sensitivity obtained with high sensitivity which is disposed in the signal indicating the object scene image thus obtained imaging means for imaging an object scene In an image processing method for processing
First and second imaging signals by the same exposure are preprocessed, the first and second imaging signals are converted into digital values, and the preprocessed and converted first image data and second image data A preprocessing step for outputting,
First and second gradation conversion steps for gradation-converting the first and second image data processed in the preprocessing step with a plurality of conversion characteristics, respectively;
A calculation step of synthesizing the values obtained by gradation conversion in the first and second gradation conversion steps;
The processing range of the dynamic range is divided into predetermined threshold values, and the gradation conversion characteristics processed in the gradation conversion step are automatically selected for each of the divided processing ranges, and a plurality of frames in a plurality of dynamic ranges are selected. And an automatic gradation processing step of generating composite processed image data. The image processing method according to claim 9, wherein the method includes a detecting step of detecting a maximum dynamic range of the object scene,
In the automatic gradation processing step, the gradation conversion characteristic is selected based on a detection result in the detection step.   The image processing method according to claim 10, wherein the detecting step detects the maximum dynamic range based on the second image data generated by the preprocessing.   12. The image processing method according to claim 11, wherein the automatic gradation processing step limits a process of generating composite image data of a plurality of frames having a plurality of dynamic ranges according to the maximum dynamic range. Image processing method.   The image processing method according to claim 9, wherein the method includes a recording step of recording the composite processed image data.   The image processing method according to claim 9, wherein the method includes a control step of controlling the imaging unit, and images the object field in an exposure amount bracket photographing mode in which the object field is imaged with stepwise exposure. An image processing method.   15. The image processing method according to claim 14, wherein the control step converts the first and second image data from first and second imaging signals of a plurality of frames generated in the exposure bracket imaging mode, respectively. An image processing method characterized in that the image processing frames are generated.

Images