JP2011055337A - Imaging apparatus - Google Patents

Abstract

An imaging apparatus capable of performing stereoscopic viewing satisfactorily even when a phenomenon in which a plurality of images constituting a stereoscopic image are different occurs.
A digital camera having solid-state imaging devices 5R and 5L arranged at a predetermined distance, and a pair of photoelectric conversion elements included in each of the solid-state imaging devices 5R and 5L that images the same subject portion. An abnormal output pixel data detection unit for detecting abnormal output pixel data included in the imaging data of the solid-state imaging devices 5R and 5L, and abnormal output only for pixel data obtained from the photoelectric conversion element included in one of the pair When the pixel data is detected and the correction accuracy of the abnormal output pixel data determined based on the noise component included in the abnormal output pixel data is less than the threshold, it is obtained from the photoelectric conversion element included in the other of the pair. A digital signal processing unit 17 that functions as a data adjustment unit that adjusts the obtained pixel data so as to be at the same level as the abnormal output pixel data obtained from one side. Obtain.
[Selection] Figure 5

Description

  The present invention relates to an imaging apparatus capable of stereoscopic imaging.

  In a CCD type image sensor, smear may occur during moving image shooting. This smear is normally corrected by detecting a smear occurrence position and a smear level using a smear level detecting element, and subtracting the smear level from the pixel at the detected smear occurrence position. However, when the smear level has reached the saturation level of the photoelectric conversion element, if the smear level is subtracted from the pixel, the level of the subtracted portion of the pixel becomes zero and blackout occurs. For this reason, when the smear level reaches the saturation level, the image quality cannot be improved even if the smear is corrected.

  In recent years, there has been proposed a digital camera capable of stereoscopic imaging by mounting two imaging systems arranged with parallax and synthesizing two image data obtained by photographing with these two imaging systems. (See Patent Document 1). Even in such a digital camera capable of three-dimensional imaging, smear may occur when a CCD type imaging device is used. However, since the two imaging systems are arranged with parallax, even if the subject has a high-luminance part, smear due to the high-luminance part does not always occur in each of the two imaging systems.

  Let us consider a case where smear occurs only in one imaging system. In this case, if the smear level does not reach the saturation level, the stereoscopic image can be displayed without a sense of incongruity by performing smear correction. However, if the smear level has reached the saturation level and smear correction cannot be performed, or if black sink occurs even after smear correction, out of two image data obtained from the two imaging systems Only one of these will have smears or black sun. In this way, if the two pieces of image data are different from each other, there is a problem that, when the two pieces of image data are stereoscopically combined, they interfere with stereoscopic vision or become tired if they are stereoscopically viewed for a long time. Occurs.

  Such a problem occurs when two images obtained by imaging are different in a camera capable of stereoscopic imaging. For example, such a problem also occurs when white scratches are generated only in the solid-state imaging device included in one of the two imaging systems. This is because a plurality of white scratches are connected to each other, and when the pixel interpolation from the surrounding area cannot be corrected, a white scratch that cannot be corrected remains in only one of the two images. .

  In Patent Document 2, in an imaging apparatus having two imaging elements, a pixel in which smear is generated in an image obtained by one imaging element is referred to as a pixel in which smearing is not obtained by the other imaging element. A method for removing smears by replacing is disclosed.

JP 2008-167066 A Japanese Patent Laid-Open No. 5-344427

  The present invention has been made in view of the above circumstances, and provides an imaging apparatus capable of performing stereoscopic viewing satisfactorily even when a phenomenon occurs in which a plurality of images constituting a stereoscopic image differ. With the goal.

  The imaging device of the present invention is an imaging device having a first solid-state imaging device and a second solid-state imaging device arranged at a predetermined distance, and the first solid-state imaging that images the same subject portion A first photoelectric conversion element included in the element and a second photoelectric conversion element included in the second solid-state imaging element are defined as a pair, and imaging data of the first solid-state imaging element and the second solid-state imaging Abnormal output pixel data detecting means for detecting abnormal output pixel data indicating abnormal output included in imaging data of the element, and the abnormal output pixel only in the pixel data obtained from the first photoelectric conversion element of the pair When data is detected and the correction accuracy of the abnormal output pixel data is less than a threshold, pixel data obtained from the second photoelectric conversion element of the pair is obtained from the first photoelectric conversion element. Obtained And and a data adjusting means for adjusting said abnormal output pixel data and to the same level.

  According to the present invention, it is possible to provide an imaging apparatus that can perform stereoscopic viewing satisfactorily even when a phenomenon occurs in which a plurality of images constituting a stereoscopic image differ.

The figure which shows schematic structure of the digital camera which is an example of the imaging device for describing one Embodiment of this invention The figure for demonstrating the mechanism in which the stereoscopic display is attained by the display part in the digital camera shown in FIG. The figure which showed schematic structure of the solid-state image sensor in the digital camera shown in FIG. The figure which showed the example of the image acquired by each of the two solid-state image sensors in the digital camera shown in FIG. The flowchart for demonstrating the flow of the image correction process which the digital signal processing part in the digital camera shown in FIG. 1 performs. The figure for demonstrating that the position of the horizontal direction of the pixel data obtained by imaging the same to-be-photographed object site | part with two solid-state image sensors of the digital camera shown in FIG. 1 differs. The figure which showed schematic structure of the solid-state image sensor in the 1st modification of the digital camera shown in FIG. The flowchart for demonstrating the flow of the image correction process which the digital signal process part in the 1st modification of the digital camera shown in FIG. 1 performs. The flowchart for demonstrating the operation | movement at the time of imaging of the 2nd modification of the digital camera shown in FIG.

  FIG. 1 is a diagram showing a schematic configuration of a digital camera which is an example of an imaging apparatus for explaining an embodiment of the present invention. Here, a digital camera is taken as an example of the imaging device, but a video camera, a mobile phone with a camera, or the like may be used.

  The imaging system of the digital camera shown in the figure includes a right eye imaging system and a left eye imaging system that are arranged at a distance in the horizontal direction. The right-eye imaging system includes a lens group 1R, a CCD (Charge Coupled Device) type solid-state imaging device 5R, a diaphragm 2R provided therebetween, an infrared cut filter 3R, and an optical low-pass filter 4R. It is. The left-eye imaging system includes a lens group 1L, a CCD solid-state imaging device 5L, a diaphragm 2L provided therebetween, an infrared cut filter 3L, and an optical low-pass filter 4L. The digital camera shown in FIG. 1 has such an imaging system, so that the same subject can be photographed from a plurality of viewpoints (two left and right viewpoints are illustrated in FIG. 1).

  Each of the lens groups 1R and 1L includes a zoom lens for adjusting the zoom position, a focus lens for adjusting the focus position, and the like. The zoom lens and the focus lens are moved synchronously between the right eye imaging system and the left eye imaging system.

  The system control unit 11 that performs overall control of the entire electric control system of the digital camera is mainly configured by a processor that operates according to a predetermined program. The system control unit 11 controls the lens driving unit 8 to adjust the focus lens position and zoom lens position of the lens groups 1R and 1L, and controls the aperture amounts of the apertures 2R and 2L via the aperture driving unit 9. Adjust the exposure amount.

  Further, the system control unit 11 drives the solid-state imaging devices 5R and 5L via the imaging device driving unit 10, and outputs subject images captured through the lens groups 1R and 1L as imaging signals. An instruction signal from the user is input to the system control unit 11 through the operation unit 14.

  The electric control system of the digital camera further includes an analog signal processing unit 6R that performs analog signal processing such as correlated double sampling processing connected to the output of the solid-state imaging device 5R, and an image output output from the analog signal processing unit 6R. An A / D conversion circuit 7R that converts a signal into a digital signal, an analog signal processing unit 6L that performs analog signal processing such as correlated double sampling processing connected to the output of the solid-state imaging device 5L, and the analog signal processing unit 6L And an A / D conversion circuit 7L that converts the imaging signal output from the digital signal into a digital signal, and these are controlled by the system control unit 11.

  Further, the electric control system of the digital camera includes a main memory 16, a memory control unit 15 connected to the main memory 16, and imaging signals output from the A / D conversion circuits 7R and 7L. Digital signal processing unit 17 that generates image data by performing (interpolation calculation, gamma correction calculation, RGB / YC conversion processing, etc.), and the image data generated by digital signal processing unit 17 is compressed into a JPEG format or a compressed image A display control unit 22 to which a compression / decompression processing unit 18 for expanding data, an external memory control unit 20 to which a detachable recording medium 21 is connected, and a display unit 23 for displaying image data in a stereoscopic manner are connected. These are connected to each other by a control bus 24 and a data bus 25, and are controlled by commands from the system control unit 11.

  The display unit 23 is used as an image display unit for displaying a captured image, and is used as a GUI at various settings. At the time of shooting, images captured by the solid-state imaging devices 5R and 5L are continuously displayed as through images and used as an electronic viewfinder or the like.

  FIG. 2 is a diagram for explaining a mechanism that enables stereoscopic display on the display unit 23. The display area of the stereoscopic image displayed on the display unit 23 includes a strip image display area 24R for the right eye and a strip image display area 24L for the left eye. The right-eye strip image display area 24R and the left-eye strip image display area 24L each have an elongated strip shape in the y-axis direction in FIG. 2, and are alternately arranged in the x-axis direction in FIG.

  The display unit 23 is a liquid crystal panel, for example, and enables stereoscopic viewing by controlling the light emission direction in different directions in the strip image display area 24R for the right eye and the strip image display area 24L for the left eye. Yes.

  In FIG. 2, the light emission direction is controlled so that the right eye strip image displayed in the right eye strip image display area 24R of the display unit 23 is incident on the viewer's right eye ER. The light emission direction is controlled so that the left eye strip image displayed in the left eye strip image display area 24L of the display unit 23 is incident on the left eye EL of the viewer. Therefore, the viewer's right eye sees only the right-eye strip image, and the viewer's left eye sees only the left-eye strip image, and the right-eye image and the left eye are a set of these right-eye strip images. Stereoscopic viewing is possible by the left-right parallax of the left-eye image that is a set of ophthalmic strip images. With such a configuration, the display unit 23 can display a stereoscopic image from the right-eye image and the left-eye image.

  In addition to this configuration, a normal liquid crystal panel may be used as the display unit 23 and a lenticular lens may be attached to the display surface to enable stereoscopic viewing.

  FIG. 3 is a diagram showing a schematic configuration of the solid-state imaging device 5R in the digital camera shown in FIG. The configuration of the solid-state image sensor 5L is the same as that of the solid-state image sensor 5R.

  The solid-state imaging device 5R illustrated in FIG. 3 includes a pixel region 50, a horizontal charge transfer path 53, and an output unit 54 formed on a semiconductor substrate 500 such as silicon. The pixel region 50 includes an effective pixel part 51 and an OB part 52.

  In the pixel region 50, a plurality of lines in which a plurality of photoelectric conversion elements such as photodiodes are arranged in the horizontal direction are arranged in the vertical direction orthogonal to the horizontal direction.

  The charge generated in each photoelectric conversion element in the pixel region 50 is read out to a vertical charge transfer path (not shown) in the pixel region 50 and transferred here in the vertical direction. The charge for one line transferred through the vertical charge transfer path is transferred in the horizontal direction by the horizontal charge transfer path 53. At the end of the horizontal charge transfer path 53, there is provided an output unit 54 such as a floating diffusion amplifier (FDA) that converts the charge into a voltage signal (hereinafter also referred to as an imaging signal) proportional to the amount of charge. The charges transferred in the horizontal direction are converted into voltage signals by the output unit 54 and output to the outside.

  The OB portion 52 is a region in which several lines at the end of the pixel region 50 on the horizontal charge transfer path 53 side are shielded from light. Each photoelectric conversion element formed in the OB portion 52 forms a smear detection region for detecting smear according to the smear charge overflowing in the vertical charge transfer path. Since smear is a signal corresponding to the charge overflowing the vertical charge transfer path, the OB unit 52 has a configuration in which the photoelectric conversion element is omitted, that is, only the vertical charge transfer path exists. May be.

  The effective pixel portion 51 is a region where photoelectric conversion elements other than the photoelectric conversion elements in the OB portion 52 are formed. An opening is formed in the light-shielding film above each photoelectric conversion element of the effective pixel portion 51, and subject light enters from here.

  FIG. 4 is a diagram illustrating an example of an image obtained by imaging with each of the solid-state imaging device 5R and the solid-state imaging device 5L in the digital camera shown in FIG. In FIG. 4, reference numeral 40R indicates an image obtained by imaging with the solid-state imaging element 5R, and reference numeral 40L indicates an image obtained by imaging with the solid-state imaging element 5L.

  In the solid-state image sensor 5R and the solid-state image sensor 5L, for the same part of the subject, the photoelectric conversion element of the solid-state image sensor 5R that receives light from the part, and the photoelectric conversion element of the solid-state image sensor 5L that receives light from the part And the incident direction of light is different. This is because the angle formed between the line connecting one center of the two photoelectric conversion elements and the same part of the subject and the line connecting the other center of the two photoelectric conversion elements and the same part of the subject is determined by both humans. This is because the solid-state imaging device 5R and the solid-state imaging device 5L are arranged at a distance so as to have an angle corresponding to the eye parallax. Hereinafter, a combination of a photoelectric conversion element of the solid-state image sensor 5R and a photoelectric conversion element of the solid-state image sensor 5L that images the same part of the subject (receives light from the part) is defined as a pair.

  As described above, since the solid-state imaging device 5R and the solid-state imaging device 5L are arranged with parallax, even when a high-luminance part is present in the subject, a situation occurs in which smear occurs only in one of the solid-state imaging devices. obtain. As shown in FIG. 4, it is assumed that the sun 41 and the vehicle headlight 40 exist as a high-luminance part in the subject. In this case, since light is uniformly incident on both photoelectric conversion elements of the pair, the smear 41a is generated in both the image 40R and the image 40L. On the other hand, in the headlight 40, since a large amount of light is incident only on one of the photoelectric conversion elements in the pair (the solid-state imaging element 5L side), the smear 40a is generated only in the image 40L. Thus, if the smear 40a occurs only in one of the images, it is difficult to perform stereoscopic viewing satisfactorily when the images 40R and 40L are stereoscopically displayed.

  Therefore, in this digital camera, even when there is a smear occurring in only one image, such as the smear 40a shown in FIG. 4, the image 40R on the side where the smear 40a is not generated is corrected, The digital signal processing unit 17 performs image correction processing that can improve stereoscopic viewing. Details of the image correction process will be described below.

  FIG. 5 is a flowchart for explaining the flow of image correction processing performed by the digital signal processing unit 17 in the digital camera shown in FIG.

  When imaging is performed by the solid-state imaging devices 5 </ b> R and 5 </ b> L, imaging data obtained by imaging is stored in the main memory 16. This imaging data is data composed of signals (pixel data) obtained from the photoelectric conversion elements of the solid-state imaging elements 5R and 5L. Hereinafter, imaging data obtained from the solid-state imaging device 5R is referred to as imaging data R, and imaging data obtained from the solid-state imaging device 5L is referred to as imaging data L.

  After the imaging data R and L are stored in the main memory 16, the digital signal processing unit 17 detects smear from the imaging data R and L (step S41).

  The digital signal processing unit 17 extracts pixel data obtained from one line in the OB unit 52 from the pixel data included in the imaging data R, and pixel data whose level exceeds the threshold value among the extracted pixel data. Smear is detected by determining whether or not there is. When there is pixel data whose level exceeds the threshold value, the smear occurrence position and the smear level are detected based on the horizontal position (x coordinate position) of the pixel data and the level of the pixel data. Can do. If there is no pixel data whose level exceeds the threshold, it is determined that smear has not occurred. When it is determined that smear has not occurred, the digital signal processing unit 17 ends the image correction process, performs predetermined signal processing on the imaging data R and L, respectively, generates stereoscopic image data, and records this. .

  In each of the imaging data R and L, the pixel data obtained from the effective pixel unit 51 at the x-coordinate position of the smear detected in step S41 includes smear data that is a noise component in a signal corresponding to the subject light. It is superimposed. For this reason, this pixel data has an abnormal output level compared to other pixel data. Accordingly, pixel data including smear data among the pixel data obtained from the effective pixel unit 51 can be referred to as abnormal output pixel data. The digital signal processing unit 17 detects abnormal output pixel data of the imaging data R and L by detecting smear from the imaging data R and L.

  After detecting the smear, the digital signal processing unit 17 compares the occurrence position of the smear occurring in the imaging data R with the occurrence position of the smear occurring in the imaging data L (step S42).

  FIG. 6 is a diagram for explaining that the horizontal position of pixel data obtained by imaging the same subject region is different between the imaging data R and the imaging data L. FIG. For example, let A be the horizontal position of the pixel data from the photoelectric conversion element of the solid-state imaging element 5L that receives light from the headlight 40 shown in FIG. In this case, the horizontal position of the pixel data obtained from the photoelectric conversion element of the solid-state imaging device 5R that receives light from the headlight 40 is (A + α). This α is a value corresponding to the distance between the solid-state image sensor 5R and the solid-state image sensor 5L, and is a known value.

  Therefore, the digital signal processing unit 17 compares the imaging data R and the imaging by comparing the smear generation position detected from the imaging data L with the position obtained by subtracting α from the smear generation position detected from the imaging data R. The smear detected from the data L is generated in both the imaging data R and the imaging data L for the same part of the subject, or only in either the imaging data R or the imaging data L for the same part of the subject. It is possible to determine whether or not

  As a result of the comparison in step S42, the digital signal processing unit 17 determines that the smear detected in step S41 is the smear (1) generated in both the imaging data R and the imaging data L for the same part of the subject and the subject. The smear (2) generated only in either the imaging data R or the imaging data L for the same part is classified.

  The digital signal processing unit 17 performs a first process on the smear (1) and performs a second process on the smear (2). In the following description, an example in which the image based on the imaging data R is the image 40R illustrated in FIG. 4 and the image based on the imaging data L is the image 40L illustrated in FIG. In this case, the smear 41a shown in FIG. 4 becomes the smear (1), and the smear 40a shown in FIG. 4 becomes the smear (2).

<First treatment>
The digital signal processing unit 17 determines whether the smear (1) can be sufficiently corrected, that is, whether the correction accuracy is a threshold value. If the correction accuracy is a threshold value, the digital signal processing unit 17 corrects the smear (1). If the correction accuracy is less than the threshold, a determination is made not to correct smear (1) (step S44).

  For example, the digital signal processing unit 17 refers to the smear level of the smear (1) and determines whether or not the smear level has reached a threshold value (saturation level of the photoelectric conversion element). When the smear level has reached the threshold value, if the smear level is subtracted from the abnormal output pixel data at the smear occurrence position, the level of the abnormal output pixel data becomes zero. For this reason, even if the smear is corrected, the image quality cannot be improved. Therefore, the digital signal processing unit 17 determines that the smear correction accuracy is 0 when the smear level has reached the threshold value.

  On the other hand, if the smear level is less than the threshold value, only the smear can be removed from the abnormal output pixel data by subtracting the smear level from the abnormal output pixel data at the smear occurrence position. For this reason, smear can be corrected and image quality can be improved. Therefore, the digital signal processing unit 17 determines that the smear correction accuracy is 1 when the smear level is less than the threshold value. If the smear correction accuracy is 1, it is determined that the smear (1) is corrected. If the smear correction accuracy is less than 1 (= zero), it is determined that the smear (1) is not corrected.

  If it is determined in step S44 that smear correction is to be performed, the digital signal processing unit 17 performs smear (1) correction (step S45).

  In the example of FIG. 4, the digital signal processing unit 17 subtracts the level of the smear 41 a from the abnormal output pixel data of the imaging data R at the generation position of the smear 41 a detected from the imaging data R. Further, the digital signal processing unit 17 subtracts the level of the smear 41a from the abnormal output pixel data of the imaging data L at the generation position of the smear 41a detected from the imaging data L. Thereby, the smear 41a can be removed from the image 40R and the image 40L.

  If it is determined in step S44 that smear correction is not performed, the digital signal processing unit 17 ends the image correction process. In this case, the smear 41a remains in both the images 40R and 40L. However, when the images 40R and 40L are displayed in 3D, the smear 41a is also displayed in 3D. Therefore, it does not hinder stereoscopic viewing.

<Second processing>
The digital signal processing unit 17 determines whether the smear (2) can be sufficiently corrected, that is, whether the correction accuracy is a threshold value. If the correction accuracy is a threshold value, the digital signal processing unit 17 corrects the smear (2). If the correction accuracy is less than the threshold, a determination is made not to correct smear (2) (step S46).

  For example, the digital signal processing unit 17 refers to the smear level of the smear (2) and determines whether or not the smear level has reached a threshold value (saturation level of the photoelectric conversion element). If the smear level has reached the threshold value, the digital signal processing unit 17 determines that the smear correction accuracy is zero. On the other hand, when the smear level is less than the threshold, the digital signal processing unit 17 determines that the smear correction accuracy is 1. Then, the digital signal processing unit 17 determines that the smear (2) correction is performed if the smear correction accuracy is 1, and the smear (2) correction is performed if the smear correction accuracy is less than 1 (= zero). Is determined not to be performed.

  If it is determined in step S46 that smear (2) correction is to be performed, the digital signal processing unit 17 proceeds to step S45 and performs smear (2) correction.

  In the example of FIG. 4, the digital signal processing unit 17 subtracts the level of the smear 40 a from the abnormal output pixel data of the imaging data L at the generation position of the smear 40 a detected by the imaging data L. Thereby, the smear 40a is removed. For this reason, even when the images 40R and 40L are stereoscopically displayed, the smear 40a is not displayed on one side and the stereoscopic vision is not hindered.

  If it is determined in step S46 that the smear (2) correction is not performed, the digital signal processing unit 17 does not perform the smear (2) correction, and abnormal output pixel data of the imaging data L at the position where the smear (2) is generated. The pixel data obtained from the photoelectric conversion element (photoelectric conversion element of the solid-state image pickup device 5R) that forms a pair with the photoelectric conversion element of the output source (photoelectric conversion device of the solid-state image pickup device 5L) is the same level as the abnormal output pixel data (Step S47).

  If the coordinate position of the abnormal output pixel data of the imaging data L at the smear (2) occurrence position is C, the digital signal processing unit 17 is obtained from the effective pixel unit 51 at the coordinate position (C + α) in the imaging data R. The level of smear 40a detected in the imaging data L is added to the pixel data. Alternatively, the digital signal processing unit 17 replaces the pixel data obtained from the effective pixel unit 51 at the coordinate position (C + α) in the imaging data R with the abnormal output pixel data at the coordinate position C of the imaging data L.

  Thereby, the smear 40a is also added to the image 40R. For this reason, even when the images 40R and 40L are stereoscopically displayed, the smear 40a is not displayed on one side and the stereoscopic vision is not hindered.

  After the end of steps S45 and S47, the digital signal processing unit 17 performs predetermined signal processing on the imaging data R and L to generate stereoscopic image data, and records this.

  As described above, according to this digital camera, only the pixel data obtained from one of the pair is abnormal output pixel data, and the correction accuracy of the abnormal output pixel data obtained from one of the pair is a threshold value. Even if the abnormal output pixel data cannot be corrected at less than the value, the pixel data obtained from the other of the pair is corrected so as to have the same level as the abnormal output pixel data. For this reason, abnormal output pixel data is included in each of the two pixel data obtained from the pair, and even when the imaging data R and the imaging data L are stereoscopically combined, it is possible to perform stereoscopic viewing satisfactorily. Become.

  In the above description, it is determined whether or not the smear (1) is corrected in step S44. However, the smear (1) may be always corrected without performing this determination. Good.

  In the example of FIG. 4, when the correction accuracy of the smear 41a is 0, the digital signal processing unit 17 may correct the smear 41a. In this case, the smear 41a in the image 40R and the image 40L shown in FIG. 4 is darkened. However, when the image 40R and the image 40L are stereoscopically displayed, the black sunken portion is also stereoscopically displayed. Therefore, there is no such thing as obstructing stereoscopic vision.

  In the above description, it is determined whether or not the smear (2) correction is performed in step S46, but the smear (2) correction is always performed without performing this determination. Good.

  In the example of FIG. 4, when the correction accuracy of the smear 40a is 0, the digital signal processing unit 17 may correct the smear 40a. In this case, the portion of the smear 40a shown in FIG. 4 is darkened. If the image 40R and the image 40L are stereoscopically displayed, the stereoscopic view is hindered.

  Therefore, in this case, in step S47, the digital signal processing unit 17 causes the solid-state imaging element 5R that forms a pair with the photoelectric conversion element of the solid-state imaging element 5L that is the output source of the abnormal output pixel data at the position where the smear 40a is generated. The pixel data obtained from the photoelectric conversion element may be adjusted to have the same level as the abnormal output pixel data after smear correction. That is, the pixel data obtained from the effective pixel portion 51 of the imaging data R at the coordinate position obtained by adding α to the occurrence position of the smear 40a is used as the level (zero) after the smear correction of the abnormal output pixel data at the occurrence position of the smear 40a. Level).

  By doing so, black sunspots also exist in the image 40R shown in FIG. 4, and when the images 40R and 40L are displayed in three dimensions, the black sunken parts can also be displayed in three dimensions. it can. Therefore, it is possible to eliminate the possibility of obstructing stereoscopic vision.

  Next, a modification of the digital camera shown in FIG. 1 will be described.

(First modification)
The digital camera of this modification is obtained by changing the solid-state imaging device of the digital camera shown in FIG. 1 to the one shown in FIG. 7 and partially changing the function of the digital signal processing unit 17.

  FIG. 7 is a diagram showing a schematic configuration of a solid-state imaging device 5R in a modification of the digital camera shown in FIG. The configuration of the solid-state imaging element 5L is the same as that shown in FIG. The solid-state imaging device 5R shown in FIG. 7 is different from the solid-state imaging device 5R shown in FIG. 3 in that an OB portion 55 is added to the pixel region 50.

  The OB portion 55 is a region in which several lines at the end of the pixel region 50 opposite to the horizontal charge transfer path 53 side are shielded from light. Each photoelectric conversion element formed in the OB portion 55 also forms a smear detection region for detecting smear according to the smear charge overflowing in the vertical charge transfer path. The OB unit 55 may also have a configuration in which the photoelectric conversion element is omitted, that is, a configuration in which only the vertical charge transfer path exists.

  In the digital camera of this modification, the digital signal processing unit 17 detects smear using the pixel data obtained from the OB unit 55 as well. For example, the digital signal processing unit 17 extracts pixel data obtained from one line in the OB unit 55 from the pixel data included in the imaging data R, and pixel data that exceeds the threshold value among the extracted pixel data. Smear is detected by determining whether or not there is. If there is pixel data that exceeds the threshold, the smear occurrence position and smear level can be detected from the horizontal coordinate position of the pixel data and the level of the pixel data.

  The digital signal processing unit 17 uses the smear information detected by the OB unit 52 and the smear information detected by the OB unit 55 to determine whether the detected smear extends straight in the vertical direction. It has the function of judging whether it is inclined and extends diagonally.

  FIG. 8 is a flowchart for explaining the flow of image correction processing performed by the digital signal processing unit 17 in the first modification of the digital camera shown in FIG. In FIG. 8, the same processes as those in FIG. In the flowchart shown in FIG. 8, the processing after step S46 is different from that shown in FIG.

  If the digital signal processing unit 17 determines in step S46 that the smear (2) correction is not performed, the digital signal processing unit 17 determines that the smear (2) is in the vertical direction based on the pixel data obtained from the OB unit 55. It is determined whether or not it is bent obliquely (step S71). This determination is made by comparing the occurrence position of the smear (2) detected by the OB unit 52 with the position of pixel data at the same level as the smear (2) in the pixel data obtained from the OB unit 55. .

  For example, the digital signal processing unit 17 detects the pixel at the same level as the smear (2) in the coordinates of the right end of the smear (2) generation position detected by the OB unit 52 and the pixel data obtained from the OB unit 55. When the difference from the right end coordinate of the data is within a predetermined range, it is determined that the smear (2) extends substantially straight in the vertical direction. The digital signal processing unit 17 detects the right end coordinates of the smear (2) generation position detected by the OB unit 52 and the pixel data of the pixel data obtained from the OB unit 55 at the same level as the smear (2). When the difference from the right end coordinate is outside the predetermined range, it is determined that the smear (2) is bent.

  If the digital signal processing unit 17 determines that the smear (2) is not bent (step S71: NO), in this case, the digital signal processing unit 17 performs the process of step S47 shown in FIG.

  If the process of step S47 shown in FIG. 5 is performed when the smear (2) is bent, an oblique smear 40a is displayed in the image 40L and the image 40R is displayed in the example shown in FIG. The straight smear 40a is displayed, and stereoscopic vision is hindered. For this reason, if the digital signal processing unit 17 determines that the smear (2) is bent (step S71: YES), the digital correction processing unit 17 ends the image correction process without performing the process of step S47.

  As described above, according to the digital camera of this modification, when the smear (2) is bent, the level of the smear (2) is added to the imaging data in which the smear (2) is not originally generated. The processing is not performed. For this reason, it is possible to prevent the three-dimensional image from becoming three-dimensionally synthesized from the oblique smear and the straight smear and becoming difficult to see. Note that the predetermined range described in paragraph 0073 may be set to a range that allows the appearance of the stereoscopic image.

(Second modification)
So far, the image correction processing in the case where abnormal output pixel data including smear data as a noise component has occurred only in one solid-state imaging device has been described. This image correction process can be similarly applied even when abnormal output pixel data including white defect data as noise components is generated in the solid-state imaging device.

  The configuration of the digital camera of this modified example is different only in the function of the digital signal processing unit 17 of the digital camera shown in FIG.

  FIG. 9 is a flowchart for explaining the operation at the time of imaging of the second modification of the digital camera shown in FIG.

  In accordance with the imaging instruction given by the user, the system control unit 11 performs still image imaging with the solid-state imaging devices 5R and 5L (step S91). Imaging data (hereinafter referred to as imaging data R and L) obtained from the solid-state imaging devices 5R and 5L by still image imaging is stored in the main memory 16.

  Immediately after taking a still image, the system control unit 11 performs light-shielded imaging with the solid-state imaging devices 5R and 5L with the mechanical shutter of the digital camera closed (step S92). Imaging data obtained from the solid-state imaging devices 5R and 5L by light-shielding imaging (hereinafter referred to as light-shielding imaging data r and l) is stored in the main memory 16.

  Next, the digital signal processing unit 17 detects pixel data whose level exceeds a predetermined value in the light-shielded imaging data r (l) as white scratch data of the solid-state imaging device 5R (5L) (step S93).

  Among the imaging data R (L), the pixel data at the same address as the white spot data detected from the light-shielded imaging data r (l) has the white spot data, which is a noise component, superimposed on a signal corresponding to the subject light. It has become a thing. Since the pixel data of the imaging data R (L) at the same address as the white spot data detected from the shading imaging data r (l) has an abnormal output level compared to other pixel data, an abnormal output It can be called pixel data. In this way, the digital signal processing unit 17 detects abnormal output pixel data included in the still image imaging data by detecting white scratch data from the light-shielded image data.

  Next, the digital signal processing unit 17 performs correction processing of abnormal output pixel data independently for the imaging data R and the imaging data L, respectively. Hereinafter, a method for correcting abnormal output pixel data included in the imaging data L will be described.

  First, the digital signal processing unit 17 has two types: one in which the level of abnormal output pixel data of the imaging data L has reached the threshold (saturation level of the photoelectric conversion element) (b) and one that is less than the threshold (a). (Step S94). For the abnormal output pixel data (a) whose level is less than the threshold value, the digital signal processing unit 17 uses the abnormal output pixel data (a) to output a pixel having the same address as the abnormal output pixel data (a) of the light-shielded imaging data l Data is subtracted (step S95).

  For the abnormal output pixel data (b) whose level has reached the threshold value, the digital signal processing unit 17 determines whether or not to correct the abnormal output pixel data (b) (step S96).

  Specifically, the digital signal processing unit 17 determines whether or not a predetermined number or more of abnormal output pixel data (b) is continuously arranged. It is determined that the correction accuracy of b) is 0, and determination is made not to correct the abnormal output pixel data (b).

  When a large number of abnormal output pixel data (b) is continuous and the pixel data around the abnormal output pixel data (b) is also abnormal output pixel data, the abnormal output pixel data (b) Even if data interpolation is performed using data, the abnormal output pixel data cannot be corrected well. Therefore, here, when the abnormal output pixel data (b) is continuous for a predetermined number or more (when the correction accuracy is less than 1 (= zero)), the abnormal output pixel data (b) is not corrected. Judgment is made.

  On the other hand, the digital signal processing unit 17 determines that the correction accuracy of the abnormal output pixel data (b) is 1 when abnormal output pixel data (b) is not continuously arranged in a predetermined number or more. A determination is made to correct the output pixel data (b). When there is normal pixel data around the abnormal output pixel data (b), the abnormal output pixel data (b) is interpolated using the peripheral pixel data to obtain the abnormal output pixel data ( b) can be corrected well. Therefore, here, when a predetermined number or more of abnormal output pixel data (b) is not continuous (when the correction accuracy is 1), a determination is made to correct the abnormal output pixel data (b). .

  If it is determined in step S96 that the abnormal output pixel data (b) is to be corrected, the digital signal processing unit 17 sets the pixel data of the surrounding imaging data L at the address of the abnormal output pixel data (b) of the imaging data L. The abnormal output pixel data (b) is corrected by interpolating the pixel data from (step S97).

  When it is determined in step S96 that the abnormal output pixel data (b) is not corrected, the digital signal processing unit 17 sets a pair with the photoelectric conversion element of the solid-state imaging device 5L that is the output source of the abnormal output pixel data (b). The pixel data obtained from the photoelectric conversion element of the solid-state imaging element 5R to be configured is adjusted to be at the same level as the abnormal output pixel data (b) (step S98).

  Specifically, the digital signal processing unit 17 shields the pixel data of the imaging data R having an address obtained by adding (α, 0) to the address (x, y) of the abnormal output pixel data (b) of the imaging data L. Pixel data having the same address as the abnormal output pixel data (b) of the imaging data l is added.

  Alternatively, the digital signal processing unit 17 converts the pixel data of the imaging data R at the address obtained by adding (α, 0) to the address (x, y) of the abnormal output pixel data (a) of the imaging data L to the abnormal output pixel. Replace with data (b).

  As a result, white defect data that could not be corrected from the imaging data L is added to the imaging data R. For this reason, even when the imaging data R and the imaging data L are displayed in 3D, white scratches are not displayed on one side and stereoscopic viewing is not hindered.

  After steps S95, S97, and S98, the digital signal processing unit 17 performs the same processing on the imaging data R. Thereafter, the imaging data R and the imaging data L are each subjected to predetermined signal processing to generate stereoscopic image data, which is recorded.

  As described above, according to this digital camera, the pixel data obtained from one of the pairs is abnormal output pixel data including white defect data, and even if this abnormal output pixel data cannot be corrected well, The pixel data obtained from the other is corrected so as to be at the same level as the abnormal output pixel data. For this reason, white defect data is included in each of the two pieces of pixel data obtained from the pair, and even when the imaging data R and the imaging data L are three-dimensionally synthesized, it is possible to perform stereoscopic viewing satisfactorily. .

  In the above description, whether or not the abnormal output pixel data (b) is to be corrected is determined in step S96. However, the correction of the abnormal output pixel data (b) is always performed without performing this determination. You may carry out.

  That is, in FIG. 9, steps S96 and 98 may be deleted, and step S94: YES may be performed after step S97. In this case, when the abnormal output pixel data (b) is continuously arranged in a predetermined number or more, the abnormal output pixel data (b) cannot be completely corrected. When L is stereoscopically displayed, stereoscopic vision is hindered.

  Therefore, in this case, after step S97, the digital signal processing unit 17 performs photoelectric conversion of the solid-state imaging element 5R that forms a pair with the photoelectric conversion element of the solid-state imaging element 5L that is the output source of the abnormal output pixel data (b). The pixel data obtained from the element may be adjusted to the same level as the abnormal output pixel data (b) after correction in step S97. That is, the pixel data of the imaging data R at an address obtained by adding (α, 0) to the (x, y) address of the abnormal output pixel data (b) of the imaging data L is corrected after the abnormal output pixel data (b) is corrected. The pixel data may be replaced.

  By doing so, the imaging data R also has white scratches that could not be corrected by the imaging data L. For this reason, when the imaging data R and the imaging data L are stereoscopically displayed, the white scratch portion can also be stereoscopically displayed. Therefore, it is possible to eliminate the possibility of obstructing stereoscopic vision.

  In the description of FIG. 9, all abnormal output pixel data is classified in step S94. However, abnormalities that exist only in either the imaging data R or the imaging data L for the same part of the subject. The classification in step S94 may be performed on the output pixel data. In this case, the abnormal output pixel data existing in both the imaging data R and the imaging data L for the same part of the subject is corrected if it can be corrected without any problem, and is corrected if it cannot be corrected without any problem. Don't do it.

  Note that white scratches on the solid-state imaging devices 5R and 5L are normally corrected at the time of imaging by checking the white scratches at the time of shipment from the factory and storing data that can be corrected in the solid-state imaging device. Yes. For this reason, white scratches during normal use can be dealt with by such correction. However, the occurrence position of white scratches also changes depending on imaging conditions such as aging of the solid-state imaging device and long exposure. In particular, white scratches caused by aging of the solid-state imaging device cannot be grasped at the time of factory shipment. Therefore, the stereoscopic view can be favorably performed by employing the technique described in the second modification.

  In the above description, the number of imaging systems is two. However, the number of imaging systems is not limited to two.

  As described above, the following items are disclosed in this specification.

  The disclosed imaging apparatus is an imaging apparatus having a first solid-state imaging device and a second solid-state imaging device arranged at a predetermined distance, and the first solid-state imaging that images the same subject portion A first photoelectric conversion element included in the element and a second photoelectric conversion element included in the second solid-state imaging element are defined as a pair, and imaging data of the first solid-state imaging element and the second solid-state imaging Abnormal output pixel data detecting means for detecting abnormal output pixel data indicating abnormal output included in imaging data of the element, and the abnormal output pixel only in the pixel data obtained from the first photoelectric conversion element of the pair When data is detected and the correction accuracy of the abnormal output pixel data is less than a threshold, pixel data obtained from the second photoelectric conversion element of the pair is obtained from the first photoelectric conversion element. Gain The and a data adjusting means for adjusting the abnormal output pixel data and to the same level.

  With this configuration, when the abnormal output pixel data correction accuracy obtained from one of the pair is less than the threshold and the abnormal output pixel data cannot be corrected, or even if the abnormal output pixel data is corrected, the first imaging data Even when abnormal output pixel data remains, the pixel data obtained from the other of the pair is corrected so as to be at the same level as the abnormal output pixel data. For this reason, abnormal output pixel data is included in each of the two pieces of pixel data obtained from the pair, and even when the first imaging data and the second imaging data are three-dimensionally combined, the stereoscopic vision is favorably performed. It becomes possible.

  In the disclosed imaging device, the abnormal output pixel data is pixel data on which smear data is superimposed as the noise component.

  The disclosed imaging apparatus includes a smear level detection unit that detects a level of smear data included in the abnormal output pixel data, and an abnormal output pixel data correction unit that corrects the abnormal output pixel data, and the correction accuracy. Is less than the threshold value is a case where the level of the smear data has reached the threshold value, and the abnormal output pixel data correcting means is configured to detect the abnormal output pixel data when the level of the smear data has reached the threshold value. Do not make corrections.

  In the disclosed imaging apparatus, the data adjustment unit adds the smear level detected by the smear level detection unit to the pixel data obtained from the second photoelectric conversion element of the pair, so that the pixel data Is adjusted to the same level as the abnormal output pixel data.

  The disclosed imaging apparatus includes a smear determining unit that determines whether a smear image based on the smear data is bent when the abnormal output pixel data correcting unit does not perform correction, and the data adjusting unit includes When the smear determination means determines that the smear image is bent, the adjustment is not performed.

  In the disclosed imaging device, the abnormal output pixel data is pixel data in which white defect data is superimposed as the noise component.

  The disclosed imaging apparatus includes white defect data level detection means for detecting a level of white defect data included in the abnormal output pixel data, and abnormal output pixel data correction means for correcting the abnormal output pixel data, The case where the correction accuracy is less than a threshold is a case where the abnormal output pixel data is continuous and the number of continuous data exceeds the threshold, and the abnormal output pixel data correcting unit is configured to continue the abnormal output pixel data. When the number exceeds the threshold value, the abnormal output pixel data is not corrected.

  The disclosed imaging apparatus includes white defect data level detection means for detecting a level of white defect data included in the abnormal output pixel data, and abnormal output pixel data correction means for correcting the abnormal output pixel data, The case where the correction accuracy is less than the threshold value is a case where the abnormal output pixel data is continuous and the continuous number exceeds the threshold value, and the data adjustment unit is configured such that the continuous number of the abnormal output pixel data is a threshold value. Is exceeded, the pixel data obtained from the second photoelectric conversion element of the pair is adjusted to have the same level as the abnormal output pixel data corrected by the abnormal output pixel data correction means.

  In the disclosed imaging device, the data adjustment unit adds the white spot level detected by the white spot data level detection unit to the pixel data obtained from the second photoelectric conversion element of the pair, The pixel data is adjusted to have the same level as the abnormal output pixel data.

  In the disclosed imaging apparatus, the data adjustment unit replaces pixel data obtained from the second photoelectric conversion element of the pair with the abnormal output pixel data corrected by the abnormal output pixel data correction unit. The pixel data is adjusted to the same level as the abnormal output pixel data.

5R, 5L solid-state image sensor 17 digital signal processor

Claims (10)

An imaging apparatus having a first solid-state imaging device and a second solid-state imaging device arranged at a predetermined distance,
A first photoelectric conversion element included in the first solid-state image sensor that images the same subject portion and a second photoelectric conversion element included in the second solid-state image sensor are defined as a pair,
Abnormal output pixel data detection means for detecting abnormal output pixel data indicating abnormal output included in the imaging data of the first solid-state imaging device and the imaging data of the second solid-state imaging device;
Of the pair, when the abnormal output pixel data is detected only in the pixel data obtained from the first photoelectric conversion element, and when the correction accuracy of the abnormal output pixel data is less than a threshold, An image pickup apparatus comprising: a data adjustment unit configured to adjust pixel data obtained from the pair of second photoelectric conversion elements so as to have the same level as the abnormal output pixel data obtained from the first photoelectric conversion elements. The imaging apparatus according to claim 1,
The imaging apparatus, wherein the abnormal output pixel data is pixel data on which smear data is superimposed as the noise component. The imaging apparatus according to claim 2,
Smear level detection means for detecting the level of smear data included in the abnormal output pixel data;
An abnormal output pixel data correcting means for correcting the abnormal output pixel data;
The case where the correction accuracy is less than the threshold value is a case where the level of the smear data has reached the threshold value,
The abnormal output pixel data correcting unit is an imaging apparatus that does not correct the abnormal output pixel data when the level of the smear data reaches a threshold value. The imaging apparatus according to claim 3,
The data adjustment means adds the smear level detected by the smear level detection means to the pixel data obtained from the second photoelectric conversion element of the pair, thereby making the pixel data the abnormal output pixel data. An imaging device that adjusts to the same level. The imaging apparatus according to claim 3 or 4,
Smear determining means for determining whether a smear image based on the smear data is bent when correction is not performed by the abnormal output pixel data correcting means;
The data adjustment unit is an imaging apparatus that does not perform the adjustment when the smear determination unit determines that the smear image is bent. The imaging apparatus according to claim 1,
The imaging apparatus, wherein the abnormal output pixel data is pixel data in which white defect data is superimposed as the noise component. The imaging apparatus according to claim 6,
White spot data level detection means for detecting the level of white spot data included in the abnormal output pixel data;
An abnormal output pixel data correcting means for correcting the abnormal output pixel data;
The case where the correction accuracy is less than a threshold is a case where the abnormal output pixel data is continuous and the number of continuous values exceeds the threshold,
The abnormal output pixel data correction means is an imaging apparatus that does not correct the abnormal output pixel data when the number of consecutive abnormal output pixel data exceeds a threshold value. The imaging apparatus according to claim 6,
White spot data level detection means for detecting the level of white spot data included in the abnormal output pixel data;
An abnormal output pixel data correcting means for correcting the abnormal output pixel data;
The case where the correction accuracy is less than a threshold value is a case where the abnormal output pixel data is continuous and the number of continuous values exceeds the threshold value.
The data adjustment unit corrects the pixel data obtained from the second photoelectric conversion element of the pair after the abnormal output pixel data correction unit corrects the pixel data when the continuous number of the abnormal output pixel data exceeds a threshold value. An imaging device that adjusts the level so as to be the same level as the abnormal output pixel data. The imaging apparatus according to claim 7,
The data adjustment means adds the white spot level detected by the white spot data level detection means to the pixel data obtained from the second photoelectric conversion element of the pair, thereby outputting the abnormal output of the pixel data. An imaging device that adjusts to the same level as the pixel data. The imaging apparatus according to claim 8, wherein
The data adjustment unit replaces the pixel data obtained from the second photoelectric conversion element of the pair with the abnormal output pixel data after correction by the abnormal output pixel data correction unit, whereby the pixel data is converted to the abnormal data. An imaging device that adjusts to the same level as the output pixel data.

Images