JP2023064501A - Image processing device and image processing method - Google Patents

Abstract

An object of the present invention is to provide an image processing apparatus and an image processing method capable of performing image synthesis in which pixel values outside an effective pixel area are appropriately used. Kind Code: A1 An image processing apparatus aligns and synthesizes image data of a plurality of frames. The image data includes first data of pixels in the effective pixel area of the image sensor, which is composed of signals corresponding to an arrangement of predetermined color components, and second data of pixels outside the effective pixel area. data is included. The image processor can align the first data independently of the second data. [Selection drawing] Fig. 3

Description

The present invention relates to an image processing device and an image processing method, and more particularly to image synthesizing technology.

There is known a technique for suppressing image blur by synthesizing a plurality of n images that are continuously shot with an exposure time that is 1/n of the required exposure time (Patent Document 1).

JP-A-2007-243775

When merging a plurality of images, it is necessary to align the images. However, if the images are simply aligned, the pixels outside the effective pixel area will be synthesized with their positions shifted. Therefore, a problem arises when it is desired to use the values of pixels outside the effective pixel area. Patent document 1 does not recognize such a problem.

The present invention has been made in view of such problems in the prior art, and one aspect of the present invention provides an image processing apparatus and an image processing method capable of synthesizing an image in which pixel values outside the effective pixel region can be appropriately used. offer.

The above object has acquisition means for acquiring multiple frames of image data, alignment means for aligning the multiple frames of image data, and synthesizing means for synthesizing the aligned multiple frames of image data. , the image data includes first data of pixels in the effective pixel area of the image pickup device, which is composed of signals corresponding to an arrangement of predetermined color components, and data of pixels outside the effective pixel area. Accomplished by an image processing apparatus characterized in that second data is included and the registration means are capable of registering the first data independently of the second data.

According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of performing image synthesis in which pixel values outside the effective pixel area are appropriately used.

1 is a block diagram showing a functional configuration example of an imaging device as an image processing device according to an embodiment; FIG. Schematic diagram of the pixel area of the image sensor Schematic diagram of RAW data synthesis processing in the embodiment Flowchart relating to operation of composite shooting mode in the embodiment

The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. In addition, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

In addition, below, the form which implements this invention by imaging devices, such as a digital camera, is demonstrated. However, the imaging function is not essential to the present invention, and can be implemented in any electronic device capable of handling image data. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smart phones, game consoles, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented in other electronic devices.

FIG. 1 is a block diagram showing an example functional configuration of an imaging device 10 as an example of an image processing device according to an embodiment of the present invention.

The control unit 113 has one or more processors capable of executing programs, a RAM, and a ROM. The control unit 113 can load a program stored in the ROM into the RAM and execute it by the processor. By executing a program, the control unit 113 controls the components of the imaging apparatus 10 including the functional blocks shown in FIG. 1 and/or the operations of external devices communicably connected. The functions of the imaging device 10 can be implemented by the control unit 113 executing a program. It should be noted that functions that can be realized by the control unit 113 executing a program may be implemented using hardware (for example, ASIC, FPGA, etc.). The ROM is electrically rewritable, for example, and stores programs executable by the processor of the control unit 113, setting values of the imaging device 10, GUI data, and the like. The control unit 113 can have the form of a system-on-chip (SoC) or a system-on-package (SiP), for example.

A photographing lens 100 has a plurality of lenses and a diaphragm, and generates an optical image of a subject. The imaging element 102 is, for example, a CMOS image sensor, and has a plurality of photoelectric conversion units (pixels) arranged two-dimensionally.

FIG. 2 is a diagram schematically showing the pixel arrangement of the image sensor 102. As shown in FIG. The image sensor 102 has an effective pixel area 200 and an optical black (OB) area 201 outside the effective pixel area as areas in which pixels are arranged. The effective pixel area 202 is an area used (exposed) for photographing a subject image. On the other hand, an OB area 201 is an area that is not used (unexposed) for photographing a subject image. Pixels in the OB region 201 may be provided with, for example, a light shielding film to prevent light from entering.

The pixels arranged in the OB region 201 have the same structure as the pixels arranged in the effective pixel region 200 except for the light shielding film. When the pixels in the effective pixel region 200 are provided with color filters, the pixels in the OB region 201 are also provided with color filters in the same manner as the pixels in the effective pixel region 200 .

Although the effective pixel area 200 and the OB area 201 are areas of the image sensor 102, the same expression is used in this specification for an image obtained by the image sensor 102 for convenience. For example, an image or an effective pixel area means an image area obtained by the pixels of the effective pixel area 200 of the image sensor 102 in the image of one frame obtained by shooting.

The color filter has a configuration in which unit filters of different colors are arranged, and one unit filter is provided for each pixel. For example, a primary-color Bayer array color filter has a configuration in which four types of unit filters, R (red), G1 (green), G2 (green), and B (blue), are arranged in a repeating unit of 2×2 pixels. . Below, the pixel provided with the unit filter G1 is called a G1 pixel. The same applies to pixels provided with other types of unit filters.

Pixels arranged in the OB region 201 are used, for example, to detect noise components of photoelectric conversion elements (for example, photodiodes) included in the pixels. For example, by subtracting the signal value of the pixel in the OB area 201 from the signal value of the pixel in the effective pixel area 200, the offset due to the noise component can be removed and the black level of the image can be corrected.

A pixel included in the image sensor 102 generates a pixel signal having a value corresponding to the amount of charge generated during the charge accumulation period. Note that in shooting with the mechanical shutter 101 opened and closed, the charge accumulation period corresponds to the exposure period. In photographing in which the mechanical shutter 101 is kept open, the charge accumulation period corresponds to the period from the resetting of the accumulated charges to the elapse of the exposure period. Generally, the former corresponds to still image shooting and the latter corresponds to moving image shooting, but the latter can also correspond to still image shooting.

When one charge accumulation period ends, a pixel signal group (analog image signal) for one frame is read out from the image sensor 102 . One frame of analog image signal includes pixel signals of an effective pixel area 200 and an OB area 201 . The analog image signal is converted into a digital image signal (digital format pixel signal group) by the A/D converter 103 . Note that the A/D conversion unit 103 may be omitted if the image sensor 102 can output a group of digital pixel signals.

At this stage, each pixel signal has only one color component corresponding to the color of the unit filter that the corresponding pixel has. In this specification, a digital image signal composed of pixel signals corresponding to a predetermined color component arrangement (a pixel signal having only one color component corresponding to the color of a unit filter) is referred to as RAW data. call.

RAW data output from the A/D conversion unit 103 or the image sensor 102 is temporarily stored in the memory 114 . The memory 114 is used to temporarily store RAW data, image data processed by the signal processing unit 111, and the like.

The OB integration unit 104 calculates an average pixel value for each type of pixel for the pixel values in the OB area 201 in the RAW data. Here, since the image sensor 102 has a color filter of primary color Bayer array, the OB integration unit 104 calculates the average pixel value of the R, G1, G2, and B pixels in the OB area 201 . This average pixel value is used as the black level.

The OB clamping unit 105 applies OB clamping processing for correcting the black level to the pixel values of the effective pixel region 200 based on the pixel values of the OB region 201 . Specifically, the OB clamping process may be a process of subtracting the average pixel value calculated by the OB integration unit 104 . The OB clamping process can suppress black floating, color shift, and the like in an image obtained from pixel signals in the effective pixel area 200 . Note that the OB clamp processing uses a black level corresponding to the type of pixel. For example, for the pixel values of the R pixels in the effective pixel area 200, the black level obtained from the R pixels in the OB area is subtracted. The OB clamping unit 105 applies OB clamping processing to each of the original images to be combined and to the combined image generated by the combining unit 108 .

A shading correction unit 106 applies shading correction to the pixel values of the effective pixel area 200 . The shading correction corrects luminance reduction according to the pixel position due to the optical characteristics of the imaging lens 100 and the microlenses of the pixels. Therefore, in shading correction, a gain is applied according to the pixel position.

A white balance (WB) processing unit 107 applies white balance adjustment processing to the image after shading correction. The white balance adjustment process is a process of applying a gain corresponding to the pixel type (R, G1, G2, B) to the shading-corrected pixel value.

The misregistration detection unit 109 detects the amount of misregistration for each of the multiple frames of images to be synthesized. The amount of positional deviation may be an absolute amount of deviation in which the amount of deviation in the first frame photographed first is set to 0, or may be an amount of relative deviation with respect to the immediately preceding frame. When detecting an absolute shift amount, the image of the first frame is the reference image.

The positional deviation detection unit 109 can detect the amount of deviation as a motion vector of the image of the effective pixel area 200 between frames. The positional deviation detection unit 109 can detect the amount of deviation using any known technique, such as a method using template matching between frames or a method using the output of a gyro sensor or the like included in the imaging device 10 . The positional deviation detection unit 109 outputs the detected positional deviation amount to the positional deviation correction unit 110 or stores it in the memory 114 .

The misregistration correction unit 110 aligns the images of the frames to be synthesized based on the amount of misregistration detected by the misregistration detection unit 109 . For example, the positional deviation correction unit 110 can perform positional alignment by changing the coordinate value of each pixel of the frame to be synthesized according to the amount of positional deviation.

Note that the misregistration correction unit 110 has a mode for aligning only the image of the effective pixel area 200 and a mode for aligning the image of an area including the effective pixel area 200 and the OB area 201 (for example, the entire frame). . In this manner, the positional deviation correction unit 110 can align the data of the effective pixel area 200 (first data) independently of the data of the OB area 201 (second data). For example, the control unit 113 can set the mode of the positional deviation correction unit 110 .

Even if a mode for aligning the entire frame is set, the control unit 113 is set to a mode for aligning only the image of the effective pixel region 200 when the condition that the compositing accuracy of the OB region 201 is lowered by alignment is satisfied. can be changed to For example, when the number of frames in which the OB area 201 overlaps is equal to or less than a predetermined ratio of the total number of frames N, the control unit 113 can change to a mode in which only the images of the effective pixel area 200 are aligned. The number of frames in which the OB region 201 overlaps can be grasped based on the positional displacement amount of each frame detected by the positional displacement detection unit 109 .

Combining unit 108 combines a plurality of images in one of a plurality of selectable combining modes to generate a combined image. It is not necessary to apply the synthesizing process to the entire frame, and it is sufficient if the synthesizing process is performed on the effective pixel area and the OB area in the reference image.

Synthesis unit 108 stores the generated synthetic image in memory 114 . In the present embodiment, the imaging apparatus 10 can select an addition mode, an average addition mode, or a comparatively bright mode as a synthesis mode. Note that these combining modes are examples, and other combining modes may be selectable, or there may be only one combining mode.

An image synthesizing method in each synthesizing mode will be described. Assume here that images of N frames (N is an integer equal to or greater than 2) are synthesized. A pixel constituting an image of each frame has coordinates (x, y) in an xy orthogonal coordinate system, and the luminance value of the pixel at the coordinates (x, y) is I_i (x, y) (i=1 to N). , and let I(x, y) be the luminance value of the pixel at the coordinates (x, y) of the synthesized image. The synthesizing unit 108 calculates the luminance value I(x, y) of each pixel of the synthesized image as follows, according to the synthesis mode.

・Addition mode I(x, y)=I_1(x, y)+I_2(x, y)+...+I_N(x, y)
In addition mode, the synthesizing unit 108 adds luminance values of pixels at the same coordinates in each frame to generate a synthetic image. The addition mode is used, for example, when N frames of images shot with an exposure amount that is 1/N of the proper exposure amount are combined to generate a proper exposure image.

・Averaging mode I(x, y)=(I_1(x, y)+I_2(x, y)+...+I_N(x, y))/N
In the averaging mode, the synthesizing unit 108 divides the brightness value obtained in the same manner as in the addition mode by the number of frames N, thereby generating a synthesized image in which the brightness value of each pixel is the average value of N frames. The averaging mode is used, for example, to reduce noise in images shot with high sensitivity.

・Light mode I(x, y)=max(I_1(x, y), I_2(x, y), . . . , I_N(x, y))
Here, max() is a function that extracts the maximum value of the elements in (). A composite image is obtained which consists of the maximum luminance value of the N pixels with the same coordinates in each frame. The comparatively bright mode is effective, for example, when synthesizing an image of fireworks or a starry sky.

In the addition mode and the averaging mode, the combination unit 108 converts the pixel value of the coordinate in another predetermined frame (for example, the first frame) to the coordinates for which there is no pixel value to be combined due to the change in the coordinates due to the alignment. can be used instead. In addition, the synthesizing unit 108 can perform synthesis using only existing pixel values in the averaging mode or the relatively bright mode (in the averaging mode, the divisor is set to the number of frames used for addition (<N)). change). Note that these are only possible examples, and synthesis may be performed using other methods.

Also, the synthesizing unit 108 prevents an image outside the effective pixel area 200 from being synthesized with the effective pixel area of the reference image. The synthesizing unit 108 can handle pixel values outside the effective pixel region 200 whose coordinates have been changed to overlap with the effective pixel region of the reference image by alignment in the same manner as when pixel values do not exist.

The signal processing unit 111 applies development processing to the (uncombined) RAW data and data of a combined image generated by the combining unit 108 (combined RAW data). It should be noted that the image area corresponding to the effective pixel area 200 is subjected to the development process. Development processing is a general term for a plurality of image processing such as color interpolation processing and tone correction processing (gamma processing). Color interpolation processing is processing that interpolates values of color components that cannot be obtained at the time of shooting, and is also called demosaicing processing. By applying color interpolation processing, each pixel comes to have a plurality of color components (for example, RGB and YCbCr) necessary for a color image, and is no longer RAW data.

The signal processing unit 111 performs detection processing such as detection of a feature region (for example, a face region or a human body region) and its movement, detection processing such as person recognition processing, synthesis processing, scaling processing, encoding, and decoding of image data after development. Treatment can be applied. The signal processing unit 111 also performs data processing such as header information generation processing, generation of signals and evaluation values used for automatic focus detection (AF), and evaluation value calculation such as calculation of evaluation values used for automatic exposure control (AE). Various image processing can be applied, such as processing. Note that these are examples of image processing to which the signal processing unit 111 can apply, and the image processing to which the signal processing unit 111 applies is not limited.

The recording unit 112 records RAW data, composite RAW data, developed image data, audio data accompanying these data, etc. on a recording medium such as a memory card, according to the shooting mode and recording settings.

The operation unit 115 is a general term for input devices (switches, keys, buttons, dials, touch panels, etc.) for the user to give various instructions to the imaging apparatus 10 .
The display unit 116 is, for example, a touch display, and is used to display live view images, reproduced images, GUI, setting values and information of the imaging device 10, and the like.

The operation of the imaging device 10 in the composite shooting mode will be described with reference to the flowchart of FIG. 4 . The composite photographing mode is a photographing mode for recording a composite image obtained by synthesizing a plurality of frames of images captured over time. In the composite photographing mode, one of the plurality of composite modes described above is set. Note that although still image shooting will be described here, it is also possible to perform generation processing for one frame of moving image shooting in which each frame is a composite image.

In S<b>101 , the control unit 113 receives setting of the number of frames N to be combined from the user through the operation unit 115 . Note that the number of frames N may be set in advance, similar to the setting of the synthesis mode.

In S<b>102 , the control unit 113 detects that the user has input a photographing instruction through the operation unit 115 . The shooting instruction may be, for example, a full-press operation of the shutter button included in the operation unit 115 . Upon detecting the input of the shooting instruction, the control unit 113 executes S103.

In S<b>103 , the control unit 113 executes imaging processing for one frame. The exposure conditions (shutter speed, aperture value, sensitivity) at the time of shooting are obtained by executing AE processing using, for example, a live view image before S102 (for example, when half-pressing the shutter button is detected). It can be an appropriate exposure amount. Note that when the synthesis mode is the addition mode, the control unit 113 sets an exposure condition that is 1/N of the appropriate exposure amount based on the number of frames N set in S101.

The control unit 113 stores RAW data obtained by shooting in the memory 114 . The control unit 113 skips S104 and executes S105 when photographing the first frame, and executes S104 when photographing the second and subsequent frames.

In S<b>104 , the positional deviation detection unit 109 detects the deviation amount of the image (RAW data) obtained in the most recent shooting. Here, for all images from the second frame onwards, the amount of deviation from the first frame image (reference image) is detected. Further, the displacement amount is detected using template matching using a part of the reference image as a template.

Here, in order to increase the detection accuracy of the amount of displacement, the positional displacement detection unit 109 generates a plurality of templates from the reference image and detects the amount of positional displacement for each template. Then, the positional deviation detection unit 109 determines the final positional deviation amount based on the plurality of detected positional deviation amounts. The positional deviation detection unit 109 can determine, for example, the most frequent positional deviation amount or the average value of the positional deviation amounts as the final positional deviation amount, but other methods may be used.

In S105, the OB integration unit 104 calculates the black level for each type of pixel using the pixel values of the OB area 201 in the image (RAW data) obtained by the most recent imaging.

In S106, the OB clamp unit 105 uses the black level calculated in S105 to apply OB clamp processing to the pixels in the effective pixel area of the image (RAW data) obtained in the most recent shooting.

In S107, the positional deviation correction unit 110 uses the positional deviation amount detected in S104 to align the image (RAW data) obtained in the latest photographing with the reference image. This alignment processing will be described with reference to FIG.

FIG. 3 schematically shows a configuration for aligning a second frame image (RAW data) 302 with respect to a first frame image (RAW data) 301, which is a reference image, and synthesizing the images. An image 301 has an effective pixel area 301a and an OB area 301b. Similarly, image 302 has an effective pixel area 302a and an OB area 302b.

Here, it is assumed that the mode of the positional deviation correction unit 110 is set to a mode in which only the images in the effective pixel area are aligned. Therefore, in the image 302' after alignment, only the effective pixel area 302a' has moved, and the OB area 302b has not moved. As a result, an image 302' after alignment has a region 302c where no pixels exist. When the mode of the positional deviation correction unit 110 is set to the mode of aligning the entire frame, the coordinate information is internally changed, but the appearance of the image 302 does not change before and after the alignment.

In S108, the synthesizing unit 108 synthesizes the image aligned in S107 (image 302' in FIG. 3) with the reference image or the synthesized image (synthesized RAW data) up to the previous frame. The synthesizing unit 108 performs the synthesizing process as described above according to the set mode.

FIG. 3 shows an example of compositing processing in the averaging mode or the comparatively bright mode in which a pixel-free region 302c generated by alignment is excluded from compositing targets. As a result, the effective pixel area 310a of the synthesized image 310 includes an area 310c where the image of the second frame is not synthesized. As for the OB area 310 of the synthesized image 310, the OB area 301b of the first frame and the OB area 302b of the second frame are synthesized by a method according to the mode.

Note that if the pixel signal is affected by noise in the negative direction, the pixel value may become less than 0 when the black level is reduced by the OB clamping process in S106. To prevent this, the composite image may be given a predetermined positive offset before being stored in memory 114 . If an offset is given to the composite image, the same amount of offset as that given to the composite image is given to the composite image and then the OB clamping process of S105 is applied. After applying the OB clamping process, the offset is removed from both images before applying the compositing process at S108.

In addition, in the flowchart of FIG. 4, synthesis is sequentially performed from the second frame on the reference image, but it is also possible to generate synthesized images from the second frame to the N-th frame and finally combine them with the reference image. In this case, synthesis from the 2nd frame to the Nth frame may be performed only in areas where all the frames overlap. In the effective pixel area of the reference image, the method according to the above-described synthesis mode is applied assuming that there is no pixel value to be synthesized for the portion where the synthesized image from the 2nd frame to the Nth frame is not generated. can be done.

In S109, the control unit 113 determines whether or not the shooting of the set number of frames N has been completed. The control unit 113 executes S110 if it is determined that the photographing has been completed, and executes the photographing of the next frame in S103 if it is not determined. Note that the control unit 113 is assumed to continuously perform shooting while the shooting instruction is continuously input, for example. When the input of the shooting instruction stops before the shooting of the set number of frames N is completed, for example, the control unit 113 may discard the processing results performed so far and return to the shooting standby state.

In S<b>110 , the OB integrator 104 calculates the black level based on the pixel values of the OB area of the synthesized image stored in the memory 114 . Calculation of the black level is the same as in S105, except that the pixel values in the OB area are pixel values after the compositing process.

In S111, the OB integrator 104 calculates a value (for example, a variance value) indicating variation in pixel values in the OB region of the synthesized image. The variance value may be calculated for each pixel type (color), or one variance value may be calculated for all pixels. The OB integrator 104 may execute the processes of S110 and S111 in parallel.

In S112, the OB clamp unit 105 uses the black level based on the composite image calculated in S110 and applies OB clamp processing to pixel values within the effective pixel area of the composite image. Note that the reason why the OB clamping process is applied after composition even though OB clamping is applied to the image before composition in S106 is to reduce the black floating due to composition.

For example, when synthesizing a plurality of frames of images shot with high sensitivity in the comparatively bright mode, the images of each frame contain a lot of noise. Lighter mode compositing selects the highest luminance value at each coordinate. Therefore, there is a high possibility that a luminance value that is higher than the original luminance value due to noise will be selected for each coordinate. Therefore, by applying the OB clamping process to the composite image using the black level based on the pixels of the OB region 201 (composite OB region) composited in the comparatively bright mode similarly to the effective pixel region 200, the black floating can be suppressed. Here, the composition in the comparatively bright mode has been described as a typical example of black floating due to composition. However, since black floating due to composition also occurs in other composition modes, the black level based on the pixels in the composite OB area is used for the composite image. OB clamping is applied.

In S113, the WB processing unit 107 applies white balance adjustment processing to the image (RAW data) of the effective pixel area of the combined image that has undergone OB clamp processing.

In S114, the signal processing unit 111 applies development processing to the RAW data to which the white balance adjustment processing has been applied. The signal processing unit 111 changes parameters of noise reduction processing applied in the process of development processing according to the variance value calculated in S111. Specifically, the signal processing unit 111 changes the parameters of the noise reduction processing so that stronger noise reduction processing is applied when the variance value is greater than a predetermined threshold value. This is because the amount of noise is greater when the variance value is greater than the threshold value than otherwise. Note that if there are three or more levels of noise reduction processing, two or more thresholds may be used to adjust the strength of the noise reduction processing. The signal processing unit 111 generates an image data file storing the image data from the image data after the development processing.

In S115, the recording unit 112 records the image data file generated in S114 in a recording medium such as a memory card or an external device. In addition to or instead of the developed image data, composite RAW data of the effective pixel area before the OB clamping process is applied in S112 may be recorded. Further, information necessary for the external device to execute the processes of S112 to S114 (the black level calculated in S110 and the dispersion amount calculated in S111) may be recorded in the combined RAW data in addition to the combined RAW data. In addition, the RAW data of N frames of the composition source may be recorded so that the processes of S104 to S114 can be executed by an external device.

When the recording by the recording unit 112 ends, the control unit 113 ends the operation shown in FIG. 4 and returns to, for example, the shooting standby state.

As described above, according to the present embodiment, when combining RAW data of a plurality of frames, alignment of RAW data can be applied separately to data in the effective pixel area and data in the OB area. . By making it possible to align the data of the effective pixel area independently, it is possible to prevent the data of the OB area from being synthesized with the data of the effective pixel area, thereby suppressing deterioration in the quality of the synthesized image. In addition, since it is possible to prevent the OB area from being shifted between frames to be synthesized, the black level can be calculated with high accuracy from the OB area after synthesis, and black floating in the image caused by synthesis can be effectively suppressed. can.

Further, when the imaging device 102 has a laminated structure, in addition to the A/D conversion unit 103 and the memory 114, the positional deviation detecting unit 109, the positional deviation correcting unit 110, and the synthesizing unit 108 are also mounted in the imaging device 102. be able to. Therefore, the imaging device 102 can be used for shooting, alignment, and synthesis.

(Other embodiments)
In the above-described embodiment, the case where a composite image is generated at the time of photographing by the imaging device has been described. However, as described above, the shooting function is not essential in the present invention. Therefore, instead of shooting in S103, the present invention can also be implemented in an image processing apparatus that acquires one frame of RAW data shot and recorded over time and executes the processes from S104 onward. In this case, the determination processing in S109 determines whether or not acquisition of the set number of frames has been completed.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The present invention is not limited to the content of the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 10... Imaging device, 102... Image sensor, 104... OB integrating part, 105... OB clamping part, 108... Synthesizing part, 109... Positional deviation detection part, 110... Positional deviation correction|amendment part, 113... Control part

Claims (11)

an acquisition means for acquiring image data of multiple frames;
alignment means for aligning the image data of the plurality of frames;
synthesizing means for synthesizing the aligned image data of the plurality of frames;
The image data includes first data of pixels in the effective pixel area of the imaging device, which is composed of signals corresponding to an arrangement of predetermined color components, and data of pixels outside the effective pixel area. includes second data,
wherein said alignment means is capable of aligning said first data independently of said second data;
An image processing apparatus characterized by: 2. The image processing apparatus according to claim 1, wherein said alignment means aligns said first data and does not align said second data. The synthesizing means synthesizes the first data of the plurality of frames, synthesizes the second data of the plurality of frames, synthesizes the first data, and synthesizes the second data. 3. The image processing apparatus according to claim 1, wherein the method for performing is the same. 4. The image processing apparatus according to any one of claims 1 to 3, wherein said second data is data of non-exposed pixels. 5. The image processing apparatus according to claim 4, wherein said second data is used for adjusting the black level of said first data. 6. The image processing apparatus according to claim 5, wherein said black level adjustment is performed on each of said plurality of frames of image data and said image data synthesized by said synthesizing means. further comprising calculating means for calculating a value representing variation of the second data of the image data synthesized by the synthesizing means,
the strength of noise reduction processing applied in the development processing of the image data synthesized by the synthesizing means is determined based on the value indicating the variation;
7. The image processing apparatus according to any one of claims 1 to 6, characterized by: 8. The image processing apparatus according to any one of claims 1 to 7, wherein said synthesizing means prevents said aligned second data from being synthesized with said first data. an imaging device;
an image processing device according to any one of claims 1 to 8;
has
An imaging apparatus, wherein the image data of the plurality of frames are captured over time using the imaging element. An image processing method executed by an image processing device,
obtaining multiple frames of image data;
aligning the frames of image data;
combining the aligned frames of image data;
The image data includes first data of pixels in the effective pixel area of the image sensor, which is composed of signals corresponding to an arrangement of predetermined color components, and second data of pixels outside the effective pixel area. 2 data included,
the first data can be aligned independently of the second data;
An image processing method characterized by: A program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 8.

Images