JP4276817B2 - Imaging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging device such as a digital camera (electronic still camera) having a pixel defect correction function. In place It is related.
[0002]
[Prior art]
In general, it is known that a solid-state imaging device such as a CCD formed of a semiconductor is deteriorated by a local crystal defect or the like.
[0003]
As a method for performing pixel defect correction for such pixel degradation, for example, as disclosed in Japanese Patent Laid-Open No. 2000-59799, attention is paid to four adjacent pixels in the upper, lower, left, and right sides around the defective pixel. It is considered that the defective pixel is compensated by interpolating the defective pixel to suppress the pixel deterioration.
[0004]
According to such a pixel defect correction method, since adjacent pixels usually have the same information, resolution degradation due to interpolation using these adjacent pixels is not so much, and an effective result is obtained. .
[0005]
By the way, in recent years, an imaging apparatus such as a digital camera (electronic still camera) using a two-dimensional solid-state imaging element has been widely used. However, in such an imaging apparatus, an image with higher resolution is required. It has become.
[0006]
Therefore, as means for obtaining a high resolution, for example, as disclosed in Japanese Patent Laid-Open No. 8-251604, a pixel shifting method is adopted in which an image pickup device is displaced in a two-dimensional direction to pick up an image and combine them. Thus, a method for obtaining a high-definition image is used.
[0007]
However, for such a pixel shifting method, if a method of correcting defective pixels by interpolating defective pixels with the above-mentioned four adjacent pixels around the defective pixel is adopted, the defective pixels are interpolated by the adjacent defective pixels themselves. Therefore, it becomes difficult to suppress the deterioration of the image.
[0008]
To deal with such a problem, for example, as disclosed in Japanese Patent Laid-Open No. 7-322151, defective pixels are adjacent to the combined image by performing pixel shifting at a large interval separated by one pixel or more. A defective pixel correction method is also considered in which, when performing defective pixel compensation, the adjacent defective pixels themselves are not interpolated, and image degradation can always be suppressed by interpolation with normal pixels. .
[0009]
[Problems to be solved by the invention]
However, increasing the amount of displacement of the image sensor at the time of imaging can enhance the compensation effect of the defective pixel, but the image sensor is moved far away by one pixel or more, so the movement error of the image sensor increases. May cause image degradation. In addition, a piezoelectric element such as a piezo element is used to displace the image pickup element. However, in order to obtain a large amount of displacement, it is necessary to increase the drive voltage, which causes a problem that the drive circuit becomes large.
[0010]
The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide an imaging apparatus capable of correcting defective pixels by adjacent pixels more accurately, reducing errors due to displacement of the imaging element, and suppressing image deterioration. .
[0011]
[Means for Solving the Problems]
According to the first aspect of the present invention, an image sensor provided with a color filter for capturing a subject image in color, and the image sensor are periodically displaced at 3 × 3 positions at intervals of 2/3 pixels, and the respective displacement positions. A displacement means for outputting an image signal of the subject image from the imaging element, and an image signal of the imaging element at each displacement position by the displacement means, The defective pixel of the image sensor is identified based on the image signal, and the gradation of the pixel data of the defective pixel is replaced with 0. When the image processing means for rearranging the image signals and synthesizing the image and the image shift position of the image signal output by the image processing means is configured to be 3 × 3 pixels, it is diagonally up and down with respect to the middle pixel. For the defective pixel located, a correction value obtained by multiplying the sum of the signal levels of the four pixels of the same color in the vertical and horizontal directions by 1/2 is obtained. For the defective pixel located in the vertical and horizontal directions with respect to the middle pixel, Then, a correction value obtained by adding the signal levels of the four pixels of the same color in the upper, lower, left and right directions is obtained. A correction value obtained by multiplying by 1/2 is obtained, and defective pixel correction means for correcting each corresponding defect image using these three correction values is provided.
[0014]
Claim 2 The invention according to claim 1 is the invention according to claim 1. The above The defective pixel correction unit is configured to detect defective pixels in the image synthesized by the image processing unit. , Different correction processing is performed according to the color discrimination information and the displacement position by the displacement means.
[0015]
Claim 3 The described invention is claimed. 2 In the invention described above, when the defective pixel correcting means determines that the defective pixel is G (green) based on the color discrimination information, 1 / A correction value obtained by multiplying 4 is obtained, and correction processing is performed using the correction value.
[0016]
Claim 4 In the invention described in claim 1, in the invention described in claim 1, the defective pixel correction unit further includes a position information conversion unit, and the position information conversion unit converts the position information of the image signal after the pixel shift and outputs the image signal. Is divided into a plurality of pieces, and the divided image signals are processed in order.
[0017]
Claim 5 According to the invention described in claim 1, in the invention described in claim 1, a subject image is formed on the image sensor. Project An image output from the imaging element in a state where the microscope, the light shielding means for selecting whether or not to project the subject image onto the imaging element, and the projection of the subject image onto the imaging element by the light shielding means are shielded from light Each image of the signal Raw Defective pixel detection means for detecting defects And Furthermore, it is characterized by having.
[0019]
As a result, according to the present invention, it is possible to perform more accurate defective pixel correction by changing the defective pixel correction processing according to the pixel shift position, and the pixel shift width is as small as one pixel or less. Therefore, image deterioration due to an error due to the displacement of the image sensor can be suppressed at the same time.
[0020]
In addition, according to the present invention, a G (green) defective pixel can be corrected with a normal adjacent pixel of the same color, so that an accurate G (green) defective pixel that greatly affects the resolution of an image is obtained. Correction can be realized.
[0021]
Further, according to the present invention, since one line width can be reduced by dividing the image signal, an equivalent process can be performed using a small capacity.
[0022]
Furthermore, according to the present invention, correction processing can be performed on pixels where dark noise is conspicuous, so that image noise can be reduced in the case of photographing a dark sample such as fluorescence observation with a microscope.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0024]
(First embodiment)
FIG. 1 shows a schematic configuration of an imaging apparatus to which the present invention is applied. In FIG. 1, reference numeral 1 denotes an image pickup lens, and a subject image passing through the image pickup lens is formed on an image pickup surface of an image pickup element 2 made of a CCD. The imaging device 2 photoelectrically converts the subject image formed on the imaging surface, and outputs it as an analog image signal to the image processing unit 4 of the image processing means.
[0025]
Displacement means 3 is connected to the image sensor 2. The displacement means 3 has a piezoelectric element such as a piezoelectric element. The image sensor 2 is periodically displaced by the operation of the piezoelectric element, and outputs an image signal of a subject image for each displacement position. Accordingly, an image signal for each displacement position of the image sensor 2 is input to the image processing unit 4.
[0026]
The image processing unit 4 reads out and rearranges image signals for each displacement position of the subject, and synthesizes them into one image. The image processing unit 4 is connected with a defective pixel correction unit 5 as defective pixel correction means. The defective pixel correction unit 5 corrects defective pixels according to the displacement position by the displacement means 3 for the defective pixels in the image synthesized by the image processing unit 4.
[0027]
FIG. 2 is a diagram for explaining in more detail the imaging apparatus configured as described above, and the same parts as those in FIG. 1 are denoted by the same reference numerals.
[0028]
In this case, the displacement means 3 includes a piezoelectric element 31 such as a piezo element and a piezoelectric element driver 32 that drives the piezoelectric element 31, and the imaging element 2 is periodically moved by the piezoelectric element 31 that is driven by the piezoelectric element driver 32. The image signal of the subject image at each displacement position is output.
[0029]
The imaging device 2 here is a piezoelectric element 31 driven by a piezoelectric element driver 32, as shown in FIG. 3, at intervals of 2/3 pixels from the reference pixel position (1) by (1) → (2). Pixel shift is performed in the order of ▼ →… → 8/8 → 99, 3 places each in the vertical and horizontal directions, and 3 × 3 in total, and an image signal of the subject image is output for each of these displacement positions. It is supposed to be. 4A and 4B show the color arrangement of the pixels before and after the pixel shift, and the color arrangement of the R, G, and B pixels before the pixel shift shown in FIG. The image after shifting the pixels to 9 places at 2/3 pixel intervals according to the procedure shown in FIG. 3 is as shown in FIG. 5B, and the color arrangement of R, G, B remains unchanged, The number of pixels is about 3 times in the vertical direction.
[0030]
Returning to FIG. 2, the image processing unit 4 includes a solid-state image sensor driver 41, an A / D converter 42, a memory 43, a defect ROM 44, and a CPU 45.
[0031]
The solid-state image sensor driver 41 drives the image sensor 2 with an exposure time based on a predetermined timing signal, and outputs a subject image as an analog image signal. The A / D converter 42 converts an analog image signal output from the image sensor 2 into a digital image signal. The memory 43 stores the image signal digitized through the A / D converter 42 by the control signal from the CPU 45 for each pixel shift position. In the defect ROM 44, an address of a defective pixel position unique to the image sensor 2 is stored in advance. Further, the CPU 45 reads out and rearranges the image signal stored in the memory 43 for each pixel shift position for each pixel and composes it as one image, and then the image signal, the displacement position signal indicating the pixel shift position, and the pixel The address (position information) is output to the defective pixel correction unit 5.
[0032]
The defective pixel correction unit 5 includes a RAM 51, a defective address detection unit 52, and a defect compensation circuit 53.
[0033]
The RAM 51 writes the address of the defective pixel position unique to the image pickup device 2 stored in advance in the defective ROM 44 of the image processing unit 4, that is, the defective pixel address through the CPU 45. The defect address detection unit 52 compares the defective pixel address stored in the RAM 51 with the address (positional information) of each pixel of the image signal input through the CPU 45 of the image processing unit 4. It is determined that the pixel has been detected, and a defect detection signal is output to the defect compensation circuit 53. When the defect compensation circuit 53 receives the image signal through the CPU 45 of the image processing unit 4 and the displacement position signal indicating the pixel shift position of the image signal, and does not detect the defect detection signal from the defect address detection unit 52. The image signal is output without any processing, and when a defect detection signal is detected, the defective pixel is corrected by processing according to the displacement position signal.
[0034]
Next, a defective pixel correction process performed by the defect compensation circuit 53 will be described.
[0035]
5 and 6 show a schematic configuration of the defect compensation circuit 53. FIG. The defective pixel mask unit 55 in FIG. 5 replaces the gray level of the pixel data with 0 when the input image data is determined to be a defective pixel based on the defect detection signal Y input from the defect address detection unit 52. Output. Therefore, all of the defective pixels in the subsequent circuit have gradation 0. In FIG. 5, (H) is a line buffer, and (D) is a D-FF. These line buffers (H) and D-FF (D) are each provided with a CLK terminal (not shown) and transfer an image signal every 1 CLK.
[0036]
The pixel signals B1, C1, D1, which are delayed by 1 to 4 lines with respect to the pixel signal A1 by passing the line buffer (H) 1 to 4 times for the image signal input pixel by pixel in synchronization with CLK. E1 is generated. Further, the pixel signals A1, B1, C1, D1, and E1 are passed through D-FF (D) a plurality of times so that A1 to A9, B1 to B9, and C1 to C9 delayed to 4 lines + 8 CLK with respect to the image signal. , D1 to D9, and E1 to E9 pixel signals are generated. By doing this, it is possible to refer to an arbitrary image signal in a 5 × 9 pixel range in the same time zone, and the level of each pixel signal for Ai, Bi, Ci, Di, Ei (i = 1 to 9). Observable.
[0037]
FIG. 6 is a block diagram of a circuit that corrects a defective pixel in the defect compensation circuit 53. Reference numerals C5, A5, C3, C7, E5, C1, and C9 in the drawing represent pixel signals in the 5 × 9 pixel range shown in FIG. Among these, the pixel signal C <b> 5 is directly input to the output selection unit 531. The pixel signals A5, C3, C7, and E5 are input to the adder 532, and the sum output of the adder 532 is input to the output selection unit 531 as the correction signal A through the 1/2 multiplier 533. The output of the adder 532 is directly input to the output selection unit 531 as the correction signal B. Further, the pixel signals C 1 and C 9 are input to the adder 534, and the sum output of the adder 534 is input to the output selection unit 531 as the correction signal C through the 1/2 multiplier 535. As a result, assuming that the pixel signal C5 corresponds to the center pixel, the correction signal A has the signal levels of the four pixel signals A5, C3, C7, and E5 of the same color in the vertical and horizontal directions as shown in FIG. The correction signal B is equal to a value obtained by adding the signal levels of the four pixel signals A5, C3, C7, and E5 of the same color, as shown in FIG. 7B. Further, as shown in FIG. 7C, the correction signal C is equal to a value obtained by multiplying the sum of the signal levels of the pixel signals C1 and C9 of two pixels of the same color at a distance of four pixels left and right by ½, as shown in FIG. Will be equal.
[0038]
The output selection unit 531 is given a displacement position signal X input through the CPU 45 of the image processing unit 4 and a defect detection signal Y input from the defect address detection unit 52. When the defect detection signal Y is not given, the output selection unit 531 outputs the pixel signal C5 as it is. Further, when the defect detection signal Y is given, pixel correction processing is performed by one of the correction signals A, B, and C according to the type of the displacement position signal X input at this time. That is, the output selection unit 531 has the table shown in FIG. 8, and the displacement position signal X corresponds to the pixel shift positions (1), (3), (7), and (9) shown in FIG. Is equivalent to the correction signal A and the pixel shift position (2), (4), (6), (8), and the correction signal B is equivalent to the pixel shift position (5). In this case, the correction signal C is selected, and defective pixels are corrected using these correction signals A, B, and C.
[0039]
Next, the defective pixel correction by the defect compensation circuit 53 will be described in more detail. 9A, 9B, and 9C are obtained by extracting the pixels after the pixel shift described in FIG. 4 for each of R, B, and G colors. Here, paying attention to the B (blue) pixel shown in FIG. 9B, if there is a defect in a certain pixel, the nine pixels indicated by the hatched portion in FIG. A defect is included in the pixel B (blue).
[0040]
FIG. 11 shows a conventionally used defective pixel correction method. According to this method, as shown in FIG. 11 (a), the defective pixel B0 is vertically and horizontally controlled regardless of its position. Is corrected by the four pixels B1 to B4 of the same color adjacent to each other. Accordingly, for example, as shown in FIG. 4B, when the pixel B0 of interest is at the upper left of nine defects, the pixels B3 and B4 among the four pixels B1 to B4 of the same color adjacent vertically and horizontally are defective pixels ( In the case where the pixel B0 of interest is in the middle of nine defects as shown in FIG. 5C, all four pixels B1 to B4 of the same color that are adjacent vertically and horizontally are defective. It becomes a pixel (the shaded portion in the figure). For this reason, when defective pixel correction processing is executed by these methods, the processing results differ depending on the defect position, and it is difficult to expect a uniform effect.
[0041]
On the other hand, in the present invention, when the displacement position signal X corresponds to the pixel shift positions (1), (3), (7), (9) shown in FIG. 3, the same color as shown in FIG. Since two of the four adjacent pixels include a defective pixel (indicated by the hatched portion) (output gradation is 0), the correction signal A (correction value) is equal to the average value of two normal adjacent pixels of the same color. For example, at the pixel shift position {circle around (1)}, the pixel signals C7 and E5 out of the four pixel signals A5, C3, C7, and E5 of the same color shown in FIG. Since it becomes 0, the correction signal A (correction value) is obtained as the average value of the pixel signals A5 and C3 of two normal adjacent pixels of the same color. The same applies to the pixel shift positions {circle around (3)}, {circle around (7)}, {circle around (9)}, and the correction signal A (correction value) is obtained as an average value of two normal adjacent pixels of the same color.
[0042]
Further, when the displacement position signal X corresponds to the pixel shift positions (2), (4), (6), and (8) shown in FIG. 3, three of the four adjacent pixels of the same color are used as shown in FIG. Since a defective pixel (indicated by a hatched portion) (with an output gradation of 0) is included, the correction signal B (correction value) becomes equal to the value of a normal pixel adjacent to the same color. For example, at the pixel shift position {circle around (2)}, the pixel signals C3, C7, and E5 of the four pixel signals A5, C3, C7, and E5 of the same color in the vertical and horizontal directions shown in FIG. Since the tone is 0, the correction signal B (correction value) is obtained as the value of the pixel signal A5 of the normal pixel adjacent to the same color. The same applies to the pixel shift positions {circle around (4)}, {circle around (6)}, {circle around (8)}, and the correction signal B (correction value) is obtained as the value of one normal pixel adjacent to the same color.
[0043]
Further, when the displacement position signal X corresponds to the pixel shift position (5) shown in FIG. 3, all the four adjacent pixels of the same color as shown in FIG. Therefore, the correction signal C (correction value) is equal to the average value of the nearest two normal pixels of the same color. For example, at the pixel shift position {circle over (5)}, all four pixel signals A5, C3, C7, and E5 of the same color in the upper, lower, left, and right shown in FIG. The signal C (correction value) is obtained as an average value of the pixel signals C1 and C9 of the nearest two normal pixels of the same color.
[0044]
After that, if correction of defective pixels is performed using correction signals A, B, and C that differ according to these pixel shift positions, each defective pixel is uniformly represented by the same color pixel signal at any displacement position. Can be corrected.
[0045]
Therefore, in this way, it is possible to perform more accurate defective pixel correction by changing the correction processing of the defective pixel depending on the pixel shift position. Further, since the pixel shift width is as small as one pixel or less, an error due to the displacement of the image sensor can be reduced, and image degradation due to the displacement error can be suppressed at the same time.
[0046]
In the present embodiment, when all the pixel signals of the same color in the vertical and horizontal directions are defective pixels, the correction value is obtained from the normal color pixels near the left and right. However, when the number of line buffers is increased, It is also possible to obtain the same color pixel. Further, by making the number of delay elements (D-FF) variable at the same time as the line buffer, a correction value based on an image signal at an arbitrary position can be obtained.
[0047]
(Second Embodiment)
Next, a second embodiment of the present invention will be described.
[0048]
The feature of the second embodiment is that a color filter is provided and the defective pixel correction method is changed for each color.
[0049]
FIG. 12 shows a schematic configuration of the second embodiment of the present invention, and the same parts as those in FIG.
[0050]
In this case, the image sensor 2 is provided with a color filter 6 for obtaining color discrimination information in an array. Others are the same as in FIG.
[0051]
In such a configuration, the subject image formed on the image sensor 2 through the color filter 6 is photoelectrically converted by the image sensor 2 and then input to the image processing unit 4. In this case, the image pickup device 2 is pixel-shifted in nine places in total, three in the vertical and horizontal directions by the piezoelectric element 31 driven by the piezoelectric element driver 32, and the image signal at each displacement position is color. Is input to the image processing unit 4 together with the discrimination information.
[0052]
In the image processing unit 4, the image signal input from the image sensor 2 is combined as a single image, and the image signal, the displacement position signal of the image signal, the address (position information) of each pixel, and the color discrimination information are defective. Output to the compensation circuit 53.
[0053]
The defect compensation circuit 53 generates pixel signals A1 to A9, B1 to B9, C1 to C9, D1 to D9, and E1 to E9 that are delayed to 4 lines + 8 CLK with respect to the image signal by the circuit configuration described in FIG. To do.
[0054]
FIG. 13 is a block diagram of a circuit that corrects a defective pixel in the defect compensation circuit 53. Reference numerals C5, A5, C3, C7, E5, C1, C9, B4, B6, D4, and D6 in the drawing represent pixel signals in the 5 × 9 pixel range shown in FIG. For these pixel signals C5, A5, C3, C7, E5, C1, and C9, a circuit for generating correction signals A, B, and C similar to those described in FIG. 6 is configured. The pixel signals B4, B6, D4, and D6 are input to the adder 536, and the sum output of the adder 536 is input to the output selection unit 531 as the correction signal D through the 1/4 multiplier 537. As a result, assuming that the pixel signal C5 corresponds to the center pixel, the correction signal D is the sum of the signal levels of the four pixel signals B4, B6, D4, and D6 of the same diagonal color left and right as shown in FIG. It is equal to the value multiplied by 1/4.
[0055]
In addition to the displacement position signal X input from the image processing unit 4 and the defect detection signal Y input from the defect address detection unit 52, the output selection unit 531 further receives a color determination signal input from the image processing unit 4. Z is given. When the defect detection signal Y is not given, the output selection unit 531 outputs the pixel signal C5 as it is. Further, when the defect detection signal Y is given, pixel correction processing is performed by one of the correction signals A, B, C, and D according to the types of the displacement position signal X and the color discrimination signal Z input at this time. That is, the output selection unit 531 has the table shown in FIG. 15, and when the color discrimination signal Z is R (red) or B (blue), the displacement position signal X is the pixel shift position shown in FIG. In the case corresponding to (1), (3), (7), (9), it corresponds to the correction signal A and the pixel shift position (2), (4), (6), (8). If there is a correction signal B, the correction signal C is selected if it corresponds to the pixel shift position (5), and if the color discrimination signal Z is G (green), the correction signal D is selected. The defective pixels are corrected by these correction signals A, B, C, and D.
[0056]
In this case, the defective pixel correction processing method when the color discrimination signal Z is R (red) or B (blue) is the same as that in the first embodiment. That is, when the displacement position signal X corresponds to the pixel shift positions (1), (3), (7), and (9) shown in FIG. 3, two of the four adjacent pixels of the same color are used as shown in FIG. Since a defective pixel (indicated by a hatched portion) (with an output gradation of 0) is included, the correction signal A (correction value) is obtained as an average value of two normal adjacent pixels of the same color. Further, when the displacement position signal X corresponds to the pixel shift positions (2), (4), (6), and (8) shown in FIG. 3, three of the four adjacent pixels of the same color are used as shown in FIG. Since a defective pixel (indicated by a hatched portion) (with an output gradation of 0) is included, the correction signal B (correction value) is obtained as the value of a normal pixel adjacent to the same color. Further, when the displacement position signal X corresponds to the pixel shift position (5) shown in FIG. 3, all the four adjacent pixels of the same color as shown in FIG. Therefore, the correction signal C (correction value) is obtained as an average value of the nearest two normal pixels of the same color.
[0057]
On the other hand, the defective pixel correction method when the color discrimination signal Z is G (green) is different from that of the first embodiment. FIG. 9 shows the pixel-shifted pixels described in FIG. 4 extracted for each of R, B, and G colors. When attention is paid to the G pixel shown in FIG. 9C, FIG. (A) The number of pixels is larger than that of R and B pixels shown in (b), and pixels of the same color of G after pixel shifting are present at diagonally adjacent positions.
[0058]
Therefore, when the color determination signal Z is G (green), the correction signal D is supplied to the pixel signals B4, B6, D4, four pixels of the same color, as shown in FIG. A value obtained by multiplying the sum of the signal levels of D6 by 1/4 is used, and the correction processing of the defective pixel is performed using the correction signal D.
[0059]
Therefore, in this way, in the pixel shifting method in which displacement is performed at 9 places at 2/3 pixel intervals, G (green) defective pixels can be corrected with normal neighboring pixels of the same color. Accurate correction can be realized for G (green) defective pixels that greatly affect the resolution, and a reduction in image resolution can be suppressed.
[0060]
(Third embodiment)
Next, a third embodiment of the present invention will be described.
[0061]
The feature of the third embodiment is that image data is divided by converting the address of the pixel-shifted image output from the image processing unit, and defective pixels are sequentially corrected for each divided image signal. It is in.
[0062]
FIG. 16 shows a schematic configuration of the third embodiment of the present invention, and the same parts as those in FIG.
[0063]
In this case, the defective pixel correction unit 5 is provided with an address conversion unit 54 as position information conversion means. The address converting unit 54 divides the image signal into a plurality of parts by converting the image signal address after pixel shift and the defective pixel address based on the table stored in the defective ROM 44 while transmitting / receiving data to / from the CPU 45. The divided image signal and address are sequentially output to the defect compensation circuit 53. Others are the same as in FIG.
[0064]
In such a configuration, the subject image formed on the image sensor 2 through the lens 1 is photoelectrically converted by the image sensor 2, and further, three pixels in the vertical and horizontal directions are respectively provided by the piezoelectric element 31, and a total of nine pixels. After shifting, the image is input to the image processing unit 4. Then, the image processing unit 4 combines one image in the same procedure as described in the first embodiment, and then converts the image signal, the displacement position signal of the image signal, and the address (position information) of each pixel into an address. To the unit 54.
[0065]
The address conversion unit 54 changes the image signal address after pixel shift and the defective pixel address according to various tables stored in the defective ROM 44 while transmitting / receiving data to / from the CPU 45, and divides the image signal into a plurality of image signals. The divided image signal and address are sequentially output to the defect compensation circuit 53. The defect compensation circuit 53 corrects the defective pixel in the same procedure as described in the first embodiment based on the divided image signal input from the address conversion unit 54 and its address, and outputs the result. .
[0066]
Here, the address conversion procedure in the address conversion unit 54 will be described in detail. FIG. 17 shows an image having the number of pixels of 2Ya in the vertical direction and 2Xa in the parallel direction after pixel shifting, and addresses are assigned to (0 to 2Xa, 0 to 2Ya).
[0067]
First, the case where the image signal is divided into four images having a height Xa and a width Ya as shown in FIG. 18 will be described.
[0068]
FIG. 20 is a flowchart showing the address conversion processing procedure. First, in step 201, it is shown in which of the four divided regions the position (x, y) of the input pixel belongs. Judgment is made based on the table shown in FIG. Here, for example, if x <Xa and y <Ya, it is determined that the pixel position exists in the region (1). Next, in step 202, the image signal is converted for each region based on the table shown in FIG. That is, when it is determined that the pixel is in the region (1), the address is output as it is without being converted. If it is determined that the pixel is in the area (2), the horizontal address x is not changed, and the vertical address y is converted to y ′ = y−Ya. In the case of the area (3) (4), the conversion is performed in the same manner.
[0069]
For example, when (Xa, Ya) = (1000, 1000), the address of the pixel (x, y) = (1200, 1500) is determined to be in the region (4) because Xa <1200 and Ya <1500. The address is converted to (1200−1000, 1500−1000) = (200,500).
[0070]
In this way, the address of the image signal is obtained by executing the above-described processing in the area having the horizontal pixel number Xa and the vertical pixel number Ya, with the upper left corner of each divided area being (0, 0) as shown in FIG. Converted to an address.
[0071]
Next, a procedure for converting a defective pixel address in order to perform defect correction on the divided image signal will be described.
[0072]
In this case, the defective pixel exists uniquely for each image sensor 2, and the defective pixel address is represented as position information before the pixel shift. Therefore, in order to perform the defective pixel correction process on the image after the pixel shift, it is also necessary to convert the defective pixel address.
[0073]
Here, it is assumed that the number of pixels of the image sensor 2 is 2Xa ′ and 2Ya ′ in the horizontal and vertical directions, respectively.
[0074]
FIG. 21 is a flowchart showing a processing procedure for converting the defective pixel address stored in the defective ROM 44. First, in step 211, the defective pixel (x, y) is present in any divided region of the entire image. Judge whether to do. Next, in step 212, address conversion is performed for each area based on the table. The processing procedure until the address conversion is performed for each area is the same as that in the above-described image signal address conversion by replacing Xa → Xa ′ and Ya → Ya ′.
[0075]
Next, in step 213, processing corresponding to pixel shift is performed. In this case, when the pixel shift is performed at 9 points as in the above-described embodiment, the resolution (number of pixels) is tripled in the horizontal and vertical directions, respectively. The addresses in the areas {circle around (1)} to {circle around (4)} are set to a value tripled.
[0076]
In step 214, an address corresponding to the pixel shift position is added. In this case, as shown in FIG. 3, when the 2/3 pixel is displaced to 9 points as shown in FIG. 3, all of these positions become defective pixels. Therefore, the address conversion formula of the table shown in FIG. Substituting Δ and Φ at the pixel shift positions {circle around (2)} to {circle around (9)} in the table shown above, these eight calculated addresses are newly added as defective pixel addresses and stored in the RAM 51 of the defective pixel correction unit 5.
[0077]
Through such a series of processes, the image after pixel shifting is also divided into four regions of width Xa and height Ya, and both the image signal and the defective pixel address are converted to the range of 0 to Xa and 0 to Ya. That is, the width of one line and one frame is replaced with Xa and Ya, respectively.
[0078]
The image signal and the defective pixel address that have undergone address conversion are sequentially input to the defect compensation circuit 53. In this case, the image signal having the width 2Xa and the height 2Ya is input to the defect compensation circuit 53 at a time. Instead, the converted image signals of addresses 0 to Xa and 0 to Ya are sequentially input in four steps.
[0079]
Therefore, in this case, the line buffer (H) in the defect compensation circuit 53 shown in FIG. 5 can reduce the width of one line by dividing the image signal. Equivalent processing can be performed. In other words, according to the present embodiment, the high-resolution image signal after pixel shifting is divided, and image defect compensation processing is sequentially performed for each divided image, so that after pixel shifting from the defect information before pixel shifting. This pixel defect correction processing can be realized with a low-capacity memory.
[0080]
In this embodiment, an example in which the 9-point pixel shift of 2/3 pixel displacement is applied has been described. However, the present invention can be applied to an imaging apparatus having a pixel shift mechanism with different pixel shift amounts and frequency. is there. Also, the number of divisions of the image signal can be arbitrarily changed.
[0081]
(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described.
[0082]
The feature of the fourth embodiment resides in that an image pickup apparatus is attached to a microscope, and a pixel having a large dark current is detected and corrected.
[0083]
FIG. 27 shows a schematic configuration of the fourth embodiment of the present invention, and the same parts as those in FIG.
[0084]
In this case, a microscope 200 is connected to the imaging apparatus 100 of the present invention. In the microscope 200, a sample 11 as a subject is placed on a stage 12. The sample 11 is irradiated with light emitted from a light source 13 such as a halogen lamp disposed below the stage 12 through an illumination optical system 14 and a condenser lens 15.
[0085]
An image of light transmitted and scattered through the sample 11 is enlarged by the objective lens 16 and projected onto the imaging surface of the imaging device 2 by the projection lens 17.
[0086]
A shutter 18 serving as a light shielding unit is disposed in front of the imaging surface of the imaging device 2. A shutter driver 19 is connected to the shutter 18. The shutter driver 19 is driven according to a signal transmitted from the CPU 45 to open and close the shutter 18.
[0087]
The image pickup device 2 periodically shifts pixels in nine places in the vertical and horizontal directions by a piezoelectric element 31 driven by a piezoelectric element driver 32, and outputs an image signal of a subject image at these displacement positions. To do.
[0088]
The image signal output from the image sensor 2 is converted into a digital image signal by the A / D converter 42 and input to the memory 43. The output signal (digital image signal) of the A / D converter 42 is also input to a defective pixel detection circuit 20 as defective pixel detection means.
[0089]
When the luminance of a predetermined level (TH) is set in advance and the dark pixel of the level exceeding the predetermined level (TH) is detected, the defective pixel detection circuit 20 determines the pixel position (address) via the CPU 45 as a defective pixel. The data is stored in the RAM 51 of the correction unit 5.
[0090]
Others are the same as in FIG.
[0091]
In such a configuration, when the observer turns on the power supply of the imaging apparatus 100, the CPU 45 transmits a signal to the solid-state imaging element driver 41 and the piezoelectric element driver 32 with the shutter 18 closed, and the imaging element 2 Is shifted to three pixels in the vertical and horizontal directions, for a total of nine pixels, and the image signal of the subject image at these displacement positions is input to the A / D converter 42.
[0092]
The A / D converter 42 converts the input image signal into a digital signal, and as a luminance signal for each pixel position (1), (2),... In all pixel regions of the image sensor 2 as shown in FIG. Input to the defective pixel detection circuit 20. The output signal of the A / D converter 42 at this time is output from the image sensor 2 when the shutter 18 is in the closed state, and is the dark current level of the image sensor 2.
[0093]
As shown in FIG. 28, the defective pixel detection circuit 20 is set in advance with a predetermined level (TH) of luminance, and when detecting a dark current with a luminance level exceeding the predetermined level (TH), the pixel position is detected. The (address) is stored in the RAM 51 of the defective pixel correction unit 5 via the CPU 45. In this case, in FIG. 28, the luminance level of the dark current at the pixel positions (2), (3), (7), (10)... Exceeds a predetermined level (TH), and these pixel positions (addresses) And stored in the RAM 51 of the defective pixel correction unit 5.
[0094]
In this way, the defective pixel detection circuit 20 detects the luminance level of the image signal (dark current) of all the pixel regions of the image sensor 2. Thereafter, the CPU 45 also stores the defective pixel address stored in advance in the defective ROM 44 in the RAM 51.
[0095]
Further, the CPU 45 sends a signal to the shutter driver 19 to open the shutter 18. When the shutter 18 is opened, the image of the sample 11 from the microscope 200 is projected onto the image sensor 2 immediately after this.
[0096]
Further, the CPU 45 transmits a signal to the solid-state image sensor driver 41 and the piezoelectric element driver 32, and displaces the image sensor 2 to shift the pixels in nine places in total, three in the vertical and horizontal directions. The image signal is digitized via the A / D converter 42 and then stored in the memory 43.
[0097]
In this state, the CPU 45 does not operate the defective pixel detection circuit 20.
[0098]
The defect address detection unit 52 compares the address of the image signal output from the memory 43 with the address stored in the RAM 51, and transmits a detection signal to the defect compensation circuit 53 when they match. When the defect compensation circuit 53 receives the detection signal of the defect address detection unit 52, the defect compensation circuit 53 outputs the image signal subjected to the correction processing, and when the detection signal is not received, the defect compensation circuit 53 outputs the image signal input from the memory 43 as it is. . The correction processing in the defect compensation circuit 53 here is the same as in the first embodiment described above.
[0099]
In this case, the RAM 51 stores not only the defective pixels stored in the defective ROM 44 in advance, but also the addresses of the pixels whose dark current is determined to be greater than a predetermined level by the defective pixel detection circuit 20.
[0100]
As a result, the correction processing in the defect compensation circuit 53 is performed not only on the defective pixels but also on the pixels where dark noise is conspicuous, so that the microscope 200 performs imaging on the dark sample 11 such as fluorescence observation. In this case, image noise can be greatly reduced.
[0101]
In this embodiment, a pixel with a large dark current is detected when the power is turned on, but the detection condition is variable, and the correction target pixel is detected every predetermined time even during imaging, and is stored in the RAM 51. Can be corrected not only for time-varying dark current but also for moving image noise such as dust. The light shielding means for the image pickup device 2 is not limited to the shutter 18, and can be realized by on / off control of the light source 13 itself and insertion of an optical filter or the like. Furthermore, it is possible to detect a defective pixel during imaging by giving a defective pixel detection condition to the defective pixel detection circuit 20 without storing the position information in the defective ROM 44 in advance. At this time, it is not necessary to change the contents of the defective ROM 44 with respect to defective pixels unique to the image pickup device 2, and it becomes possible to correctly correct defective pixels that change over time.
[0102]
In addition, this invention is not limited to the said embodiment, In the implementation stage, it can change variously in the range which does not change the summary.
[0103]
Furthermore, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and is described in the column of the effect of the invention. If the above effect is obtained, a configuration from which this configuration requirement is deleted can be extracted as an invention.
[0104]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an imaging apparatus and method that can correct defective pixels by adjacent pixels more accurately, reduce errors due to displacement of the imaging element, and suppress image degradation.
[Brief description of the drawings]
FIG. 1 is a diagram showing a schematic configuration of an imaging apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram showing a schematic configuration for explaining the imaging device of the first embodiment in more detail.
FIG. 3 is a diagram illustrating a pixel displacement state according to the pixel shifting method of the first embodiment;
FIG. 4 is a diagram showing a color arrangement of pixels before and after pixel shifting in the first embodiment.
FIG. 5 is a diagram showing a schematic configuration of a defective pixel compensation circuit according to the first embodiment.
FIG. 6 is a block diagram of a circuit that performs defective pixel interpolation in the defect compensation circuit according to the first embodiment;
7 is a diagram for explaining a defect correction method for each pixel shift position of a defective pixel according to the first embodiment; FIG.
FIG. 8 is a diagram illustrating a table prepared in the output selection unit according to the first embodiment.
FIG. 9 is a diagram showing a pixel arrangement when pixels after pixel shifting according to the first embodiment are extracted for R, B, and G colors, respectively.
FIG. 10 is a diagram showing an example of the arrangement of defective pixels after pixel shifting according to the first embodiment.
FIG. 11 is a view for explaining a conventionally used defective pixel correction method as a comparative example of the first embodiment;
FIG. 12 is a diagram showing a schematic configuration of a second embodiment of the present invention.
FIG. 13 is a block diagram of a circuit that corrects a defective pixel in the defect compensation circuit according to the second embodiment.
FIG. 14 is a diagram for explaining a defective pixel correction method according to the second embodiment;
FIG. 15 is a diagram illustrating a table prepared in the output selection unit according to the second embodiment.
FIG. 16 is a diagram showing a schematic configuration of a third embodiment of the present invention.
FIG. 17 is a diagram for explaining an address conversion procedure in an address conversion unit according to the third embodiment;
FIG. 18 is a diagram for explaining an address conversion procedure in an address conversion unit according to the third embodiment;
FIG. 19 is a diagram for explaining an address conversion procedure in an address conversion unit according to the third embodiment;
FIG. 20 is a flowchart for explaining an address conversion procedure in the address conversion unit of the third embodiment;
FIG. 21 is a flowchart for explaining an address conversion procedure in the address conversion unit according to the third embodiment;
FIG. 22 is a table showing an example of a table stored in a defect ROM according to the third embodiment.
FIG. 23 is a table showing an example of a table stored in a defect ROM according to the third embodiment.
FIG. 24 is a table showing an example of a table stored in a defect ROM according to the third embodiment.
FIG. 25 is a diagram showing an example of a table stored in a defect ROM according to the third embodiment.
FIG. 26 is a table showing an example of a table stored in a defect ROM according to the third embodiment.
FIG. 27 is a diagram showing a schematic configuration of a fourth embodiment of the present invention.
FIG. 28 is a diagram for explaining a relationship between a predetermined level (TH) set in the defective pixel detection circuit according to the fourth embodiment and a luminance level of an image signal.
[Explanation of symbols]
1 ... Lens
2 ... Image sensor
3. Displacement means
31 ... Piezoelectric element
32. Piezoelectric element driver
4. Image processing unit
41 ... Solid-state image sensor driver
42 ... A / D converter
43 ... Memory
44 ... Defect ROM
45 ... CPU
5 ... defective pixel correction unit
6. Color filter
11 ... Sample
12 ... Stage
13 ... Light source
14 ... Illumination optical system
15. Condenser lens
16 ... Objective lens
17. Projection lens
18 ... Shutter
19 ... Shutter driver
20 ... defective pixel detection circuit
51 ... RAM
52. Defect address detection unit
53. Defect compensation circuit
531: Output selection unit
532, 534, 536 ... adders
533, 535, 537 ... multipliers
54 ... Address converter
100: Imaging device
200 ... Microscope

Claims (5)

An image sensor having a color filter for color-taking a subject image;
Displacement means for periodically displacing the image sensor at 3 × 3 positions at 2/3 pixel intervals and outputting an image signal of the subject image from the image sensor at each displacement position;
The image signal of the image sensor at each displacement position by the displacement means is input , the defective pixel of the image sensor is specified based on the image signal, and the gradation of the pixel data of the defective pixel is replaced with 0. Image processing means for rearranging image signals and synthesizing images;
When the image shift position of the image signal output by the image processing unit is configured to be 3 × 3 pixels, four pixels of the same color in the upper, lower, left, and right colors are provided for the defective pixel located in the diagonally up and down direction with respect to the middle pixel. A correction value obtained by multiplying the sum of the signal levels by 1/2 and adding the signal level of four pixels of the same color in the upper, lower, left, and right directions to the defective pixel located in the upper, lower, left, and right directions with respect to the middle pixel. For the defective pixel located in the middle, a correction value obtained by multiplying the sum of the signal levels of two pixels of the same color at a distance of four pixels left and right by 1/2 is obtained. An image pickup apparatus comprising: defective pixel correction means that performs correction processing of a corresponding defective image using each value.   2. The defective pixel correcting unit performs different correction processing on the defective pixel of the image synthesized by the image processing unit according to color discrimination information and a displacement position by the displacement unit. Imaging device.   When the defective pixel correction unit determines that the defective pixel is G (green) based on the color determination information, the defective pixel correction unit multiplies the signal level of four pixels of the same color in the upper, lower, left, and right colors by 1/4. The imaging apparatus according to claim 2, wherein a correction value is obtained and correction processing is performed using the correction value.   The defective pixel correction unit further includes a position information conversion unit, converts the position information of the image signal after pixel shift by the position information conversion unit, divides the image signal into a plurality of images, and converts the divided image signals into The imaging apparatus according to claim 1, wherein the processing is sequentially performed. A microscope for projecting a subject image on the image sensor;
Light shielding means for selecting whether or not to project the subject image onto the image sensor;
In a state where the projection of the subject image onto the image sensor is shielded by the light shield means, defective pixel detection means for detecting a defect of each pixel of the image signal output from the image sensor;
The imaging apparatus according to claim 1, further comprising:

Images