JP2012253490A - Solid state image pickup device, imaging apparatus, and image generating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a solid state image pickup device, an imaging apparatus, and an image generating method, capable of multiplexing addition units for pixel addition.SOLUTION: An imaging apparatus includes a pixel array part, a row scanning part 4, an A/D conversion part, a horizontal calculation part 8, and a vertical calculation part 9. A column range of addition units which is the range of pixels to be added is taken as a horizontal addition range, and a row range as a vertical addition range. In this case, the horizontal calculation part 8 weighs a pixel value contained in the horizontal addition range among row pixel values to each column, and outputs the added pixel values as a horizontal addition pixel value. The vertical calculation part 9 adds a horizontal addition pixel value contained in the vertical addition range among the horizontal addition pixel values, and outputs the added horizontal addition pixel values as an addition pixel value. The horizontal calculation part 8 outputs a horizontal addition pixel value, with the first horizontal addition range and the second horizontal addition range containing the pixels shared by the first horizontal addition range, as a horizontal addition range.

Description

  The present invention relates to a solid-state imaging device, an imaging device, an image generation method, and the like.

  An imaging device that converts an optical image into an electrical signal is mounted on an imaging device such as a digital camera or a digital video camera. In recent years, replacement of a CCD (Charge Coupled Device) type to a CMOS (Complementary Metal Oxide Semiconductor) type is accelerating in this imaging element.

  In recent years, CMOS image pickup devices are designed to sequentially read out the charges of a large number of pixels arranged two-dimensionally on the image pickup surface, and have an A / D (Analog to Digital) conversion function for each column of the arranged pixels. Some column parallel A / D converters are provided on elements (for example, Patent Documents 1 and 2).

  This CMOS image sensor can read out various types of pixel signals in addition to full scan by taking advantage of its features. For example, it is possible to read out the number of pixels corresponding to various resolutions by movie scanning for extracting a signal obtained by adding a plurality of pixels in the horizontal and vertical directions.

  When the resolution is lowered as compared with the full scan by the movie scan operation, the frame rate is improved, the accumulation time per pixel is reduced, and the signal amount to be read is simultaneously reduced. One of the purposes of pixel addition is to suppress degradation of the signal-to-noise ratio caused by a decrease in signal amount. That is, the pseudo accumulation time is increased by increasing the amount of signal per pixel by addition as compared with the movie scan in which thinning readout driving is simply performed horizontally and vertically.

  For example, in Patent Document 1, a solid-state imaging device includes an A / D conversion unit for each column, a column unit addition unit, an input / output selection unit provided between the A / D conversion unit and the column unit addition circuit, And a method of performing the column addition operation without increasing the column scanning frequency is disclosed.

  Further, in Patent Document 2, a solid-state imaging device having an A / D conversion unit for each column performs pixel addition by changing a weighting coefficient between lines to be added in the same color pixel of a pixel array, and by equal addition A technique for eliminating the non-uniform spatial center of gravity position after addition is disclosed.

JP 2008-294913 A JP 2009-212621 A

  Now, an image obtained by pixel addition has a lower resolution than when pixels are not added. Therefore, in order to obtain a high-definition still image during moving image shooting, for example, a method of performing still image shooting only when the shutter is pressed during moving image shooting can be considered.

  However, with this method, it is difficult to acquire a still image by pressing the shutter at a decisive moment. Therefore, it is conceivable to shoot a moving image by pixel addition and generate a high-definition still image at an arbitrary timing later from the moving image.

  As such a method, an addition unit that is a range of pixels to be added is set to be superimposed, a pixel value included in the addition unit is added to obtain a low resolution image, and a high resolution image is obtained from the low resolution video. A method for restoring a fine still image can be considered.

  However, in the conventional solid-state imaging device, since pixel addition is performed without superimposing the range of pixels to be added (for example, Patent Document 1 above), there is a problem that it is not possible to realize a method of performing pixel addition by superimposing addition units. is there.

  In the method disclosed in Patent Document 2, weighting calculation in pixel addition is performed by modulating an A / D conversion clock signal. For this reason, control is possible with a relatively low speed operation in units of rows, but it is unsuitable for controlling pixel addition in superimposed units of addition with high speed operation in pixel addition in column units.

  According to some aspects of the present invention, it is possible to provide a solid-state imaging device, an imaging device, an image generation method, and the like that can perform pixel addition by superimposing addition units.

  One embodiment of the present invention includes a pixel array unit in which a plurality of pixels that photoelectrically convert incident light are arranged, a row scanning unit that selects a row of the pixel array unit, performs vertical scanning, and A / D converter that converts an analog signal from a pixel to a pixel value of a digital signal, and a column range of an addition unit that is a range of pixels to be added is a horizontal addition range and a row range is a vertical addition range A horizontal calculation unit that weights and adds the pixel values included in the horizontal addition range among the pixel values of the row to each column, and outputs the added pixel values as horizontal addition pixel values; A vertical calculation unit that adds the horizontal addition pixel values included in the vertical addition range among the horizontal addition pixel values and outputs the added horizontal addition pixel value as an addition pixel value, and the horizontal calculation unit Is the first horizontal addition range and A second horizontal addition range including the common pixel and the first horizontal addition range, related to the solid-state imaging device that outputs the horizontal addition pixel value as the horizontal addition range.

  According to one embodiment of the present invention, a row of the pixel array unit is selected, and an analog signal from a pixel in the selected row is converted into a pixel value of a digital signal. Among the pixel values in the row, the pixel values included in the horizontal addition range are weighted and added to each column, and the addition value is output as a horizontal addition pixel value. At this time, the first horizontal addition range and the second horizontal addition range include common pixels. Then, of the horizontal addition pixel values, the horizontal addition pixel values included in the vertical addition range are added, and the addition value is output as the addition pixel value. This makes it possible to perform pixel addition by superimposing addition units.

  In the aspect of the invention, the horizontal calculation unit may weight the common pixel in the first horizontal addition range and the common pixel in the second horizontal addition range with different weighting factors. May be performed.

  In this way, independent weighting can be performed on common pixels in each addition range. Thereby, even when the addition units are overlapped, weighting addition can be performed in each addition unit.

  In one embodiment of the present invention, the first horizontal addition range includes a first pixel and a second pixel, and the second horizontal addition range includes the second pixel and a third pixel. The horizontal calculation unit includes a line memory for storing first to third pixel values, which are pixel values of the first to third pixels, and weighting with respect to the first pixel value by a first weighting factor. , A weight calculator that performs weighting with respect to the second pixel value by the second and third weighting factors and weighting with respect to the third pixel value by the fourth weighting factor, and the first and second weighting units, A horizontal adder that performs addition of the first and second pixel values weighted by a weighting factor and addition of the second and third pixel values weighted by the third and fourth weighting factors You may have.

  In this way, it is possible to configure a horizontal arithmetic unit that performs weighting with a different weighting coefficient in each horizontal addition range for pixels common to the first horizontal addition range and the second horizontal addition range.

  In the aspect of the invention, the weight calculation unit includes first to fourth multipliers that perform multiplication of the first to fourth weighting factors, and the first to fourth multipliers are The pixel values stored in the line memory may be multiplied by the first to fourth weighting coefficients at the same timing.

  In this way, it is possible to perform the weight calculation of the addition unit to be superimposed at one timing for one row. Thereby, since it is not necessary to perform the weight calculation with respect to the common pixel at two timings, it is not necessary to increase the operation speed as compared with the pixel addition which does not overlap.

  In the aspect of the invention, the vertical calculation unit may use the first vertical addition range and the second vertical addition range including a row common to the first vertical addition range as the vertical addition range. The added pixel value may be output.

  In this way, since the horizontal addition pixel values in the common row of the first and second vertical addition ranges can be added for each vertical addition range, pixel addition in the overlapping vertical addition range can be realized.

  In the aspect of the invention, the vertical calculation unit may include first to third line memories that store the horizontal addition pixel values of the first to third rows, and the first and second rows. You may have a vertical addition part which performs addition of a horizontal addition pixel value, and addition of the said 2nd, 3rd said horizontal addition pixel value.

  In this way, it is possible to configure a vertical calculation unit that adds the horizontal addition pixel values in the common row of the first and second vertical addition ranges in each vertical addition range.

  In one aspect of the present invention, the control unit includes: a control unit that sets the addition unit; and a selection unit that selects a column and a row based on the set addition unit. The pixel values of the columns may be weighted and added, and the vertical calculation unit may add the horizontal addition pixel values of the selected row.

  In this way, by selecting a column and a row based on the set addition unit, the pixel values of the pixels included in the addition unit can be added to obtain the added pixel value.

  In the aspect of the invention, the A / D conversion unit may be a column parallel A / D conversion unit that converts the analog signal of each column of the row selected by the row scanning unit into the digital signal in parallel. There may be.

  In this way, analog signals from one row of pixels can be converted into digital signals in parallel. This eliminates the need for high-speed A / D conversion, so that the operation speed of the solid-state imaging device can be reduced.

  According to another aspect of the present invention, the solid-state imaging device according to any one of the above, and an estimation calculation unit that estimates a pixel value of a pixel included in the addition unit based on the addition pixel value, When the first addition unit set at the first position and the second addition unit set at the second position where the first position is shifted overlap, the vertical arithmetic unit is The first and second addition pixel values which are the addition pixel values of the first and second addition units are output, and the estimation calculation unit is configured to output the first addition pixel value and the second addition. A difference value of pixel values is obtained, and a first intermediate pixel value that is an addition pixel value of the first area obtained by removing the overlap area from the first addition unit, and the overlap area is removed from the second addition unit. The relational expression with the second intermediate pixel value that is the added pixel value of the second area is expressed as the difference. The first intermediate pixel value is estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. Related to the imaging device.

  According to another aspect of the present invention, it is possible to estimate an intermediate pixel value from an addition pixel value of an addition unit that has been pixel shifted while being superimposed, and obtain a final estimated pixel value from the estimated intermediate pixel value. Thereby, a captured image can be estimated by simple processing.

  In another aspect of the present invention, the estimation calculation unit is included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern. A relational expression between intermediate pixel values is represented using the addition pixel value, the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the addition pixel value, and similarity is evaluated, Based on the similarity evaluation result, an intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity becomes the highest.

  In this way, an intermediate pixel value can be obtained by estimation based on a plurality of added pixel values acquired by an addition unit that is pixel-shifted while being superimposed.

  Still another aspect of the present invention provides a plurality of pixels that photoelectrically convert incident light when a column range of an addition unit that is a range of pixels to be added is a horizontal addition range and a row range is a vertical addition range. Are arranged, a row of the pixel array portion is selected, vertical scanning is performed, an analog signal from a pixel of the selected row is converted into a pixel value of a digital signal, The horizontal addition range and a second horizontal addition range including pixels common to the first horizontal addition range are set as the horizontal addition range, and the pixels included in the horizontal addition range among the pixel values of the row A value is added by weighting each column, and the added pixel value is output as a horizontal addition pixel value, and the horizontal addition pixel value included in the vertical addition range among the horizontal addition pixel values is added, The horizontal added pixels added It relates to an image generation method for outputting as an addition pixel value.

1 is a configuration example of a solid-state imaging device of the present embodiment. FIG. 2A to FIG. 2D are explanatory diagrams of superposition shift pixel addition sampling processing. The detailed structural example of a horizontal calculating part. The detailed structural example of a perpendicular | vertical calculating part. FIG. 5A shows an example of setting an addition range when performing the same color four-pixel addition. FIG. 5B shows an example of an output signal when performing the same color four-pixel addition. Operation | movement explanatory drawing in the case of performing the same color 4 pixel addition. FIG. 7A shows an example of setting an addition range in the case of performing four-color addition of different colors. FIG. 7B shows an example of an output signal in the case of performing different color 4-pixel addition. Operation | movement explanatory drawing in the case of performing different color 4 pixel addition. FIG. 9A shows an example of setting the addition range when performing normal four-pixel addition. FIG. 9B shows an example of an output signal when performing normal four-pixel addition. Operation | movement explanatory drawing in the case of performing normal 4-pixel addition. 11A and 11B are explanatory diagrams of an estimated pixel value and an intermediate pixel value. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process. 2 is a configuration example of an imaging device.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Outline of the present embodiment In this embodiment, as will be described later with reference to FIG. 2 (A) and the like, a low-resolution moving image is generated by setting a superposition-shifted addition unit and adding pixel values included in the addition unit. get. Then, as will be described later with reference to FIGS. 11A to 14, a high-resolution still image and a high-resolution moving image corresponding to the resolution of the image sensor are restored from the low-resolution moving image.

  Since a high-resolution image can be obtained after the fact, the user can specify a desired timing, and an image at a decisive moment can be easily obtained. Further, since the addition unit is shifted in a superimposed manner, it is possible to perform restoration by a simple process.

2. Solid-State Imaging Device FIG. 1 shows a configuration example of a solid-state imaging device according to this embodiment that acquires a low-resolution moving image by superposition shift sampling. The solid-state imaging device 1 (image sensor in a broad sense) includes a pixel circuit array 2 (a pixel array portion in a broad sense), a row scanning portion 4 (a row scanning circuit, a row selection portion in a broad sense), and a column parallel A / D conversion portion. 7 (column parallel A / D conversion circuit, A / D conversion unit in a broad sense), horizontal calculation unit 8 (horizontal calculation circuit), vertical calculation unit 9 (vertical calculation circuit), line memory 10, column scanning unit 11 (column Scanning circuit), timing control unit 12 (timing control circuit, control unit in a broad sense), and selection unit 13 (selection circuit).

  Note that the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of the components or adding other components are possible. For example, some of these components may be provided outside the solid-state imaging device 1.

  The pixel circuit array 2 includes unit pixel circuits 3, row selection lines 5, and column signal lines 6. The unit pixel circuits 3 are arranged in a two-dimensional shape (matrix shape). The unit pixel circuit 3 is a unit pixel including a photoelectric conversion element, and performs photoelectric conversion that converts optical information into electrical information. The pixel is composed of, for example, a photodiode or a phototransistor. The row selection line 5 is a selection line for scanning the pixel circuit array 2 in the vertical direction by the row scanning unit 4. The column signal line 6 is a signal line for transmitting the output signal of each pixel circuit in the selected row to the column parallel A / D conversion unit 7.

  When the imaging operation is started, the row scanning unit 4 performs vertical scanning after a predetermined accumulation time (charge accumulation time). Specifically, the row scanning unit 4 simultaneously selects the unit pixel circuits 3 arranged along the horizontal direction by the row selection line 5 and sequentially selects the row selection line 5 along the vertical direction to perform scanning. Do.

  A signal from each unit pixel circuit 3 in the selected row is input to the column parallel A / D converter 7 via the column signal line 6. The column parallel A / D converter 7 is configured to have an A / D conversion function for each column. That is, the column parallel A / D converter 7 has a plurality of A / D converters corresponding to the column signal lines 6. The column parallel A / D converter 7 simultaneously performs A / D conversion processing on signals from the row unit pixels (unit pixel circuit 3 in one row).

  The signal of each unit pixel (pixel value data of each pixel) converted into digital data by the column parallel A / D conversion unit 7 is input to the horizontal calculation unit 8. The horizontal calculation unit 8 performs addition processing of the input pixel signals for one row. Specifically, the horizontal calculation unit 8 includes a line memory 32, a weight calculation unit 33 (weighting calculation unit), and a horizontal addition unit 34.

  The line memory 32 temporarily stores row unit pixel signals. The line memory 32 includes a memory having a capacity corresponding to the number of pixels for one row of the pixel circuit array 2.

  The weight calculation unit 33 performs a process of assigning a weight (weighting coefficient) to each pixel signal. This process is a process for determining the amplification factor of each pixel signal. For example, the weight calculation unit 33 performs weight addition processing through addition / subtraction / division / calculation calculation processing, gradation shift of a digital signal, so-called bit shift processing, and the like.

  The horizontal adder 34 performs a process of adding pixel signals in the horizontal direction. Specifically, the column selection unit 15 of the selection unit 13 selects a column to be added, and the horizontal addition unit 34 adds the pixel signals of the selected column. The column selection unit 15 selects a column signal to be added based on an instruction from the timing control unit 12.

  The row signal added by the horizontal calculation unit 8 is input to the vertical calculation unit 9. The vertical calculation unit 9 performs a process of adding pixel signals of a plurality of rows added in the horizontal direction. Specifically, the vertical calculation unit 9 includes a line memory 42 and a vertical addition unit 43.

  The line memory 42 temporarily stores the row unit pixel signal that has been subjected to the arithmetic processing by the horizontal arithmetic unit 8. The number of lines stored in the line memory 42 (the number of line memories) is determined by the number of lines to be added, and is determined when the solid-state imaging device 1 is designed. For example, when the pixel circuit array 2 is a Bayer array and 4-pixel addition is performed by the superposition shift of this embodiment, three lines (three lines) of line memory are required. In addition, when adding four pixels of the same color without superimposing, two lines (two lines) of line memory are required. The capacity of the line memory in each row is determined according to the contents of the pixel addition processing in the horizontal direction by the horizontal calculation unit 8.

  The vertical adder 43 performs a process of adding pixel signals in the vertical direction. Specifically, the row selection unit 16 of the selection unit 13 selects a line memory in which the pixel signal of the addition target row is stored, and the vertical addition unit 43 reads and reads the pixel signal from the selected line memory. The obtained pixel signals are added. The row selection unit 16 selects a row signal to be added based on an instruction from the timing control unit 12.

  The signal calculated by the vertical calculation unit 9 is temporarily stored in the line memory 10. The column scanning unit 11 sequentially reads out the addition pixel signals stored in the line memory 10 in time series and outputs them to the output line 14.

3. Superposition Shift Pixel Addition Sampling Processing Next, the checkered superposition shift pixel addition sampling processing performed by the present embodiment will be described. Hereinafter, a case where pixel values of 2 rows and 2 columns are added will be described as an example. In the following description, a pixel of 3 rows and 3 columns, which is a part of the pixel array, will be described as an example. However, as will be described later with reference to FIG.

  As shown in FIG. 2A, in the first frame, pixels of group GP1 (pixels surrounded by a solid line) and pixels of group GP2 (pixels surrounded by a broken line) are to be added. The first frame is, for example, an odd frame. This first frame and a second frame described later are alternately repeated in the low-resolution moving image shooting.

  Here, the frame is, for example, a timing at which an image is captured by an image sensor or a timing at which one captured image is processed in image processing. Alternatively, one image in the image data is also referred to as a frame as appropriate.

FIG. 2B shows an example of weights given to pixel values in the first frame. In this example, the added pixel values of GP1 and GP2 are obtained by the following equation (1). When attention is paid to the pixel E, the pixel E belongs to the group GP1 and the group GP2, and the pixel E is given a weight in each group.
GP1 = A + (1/2) * B + (1/2) * D + (1/4) * E,
GP2 = E + (1/2) * F + (1/2) * H + (1/4) * I (1)

  As shown in FIG. 2C, in the second frame, pixels of the group GP3 (pixels surrounded by a solid line) and pixels of the group GP4 (pixels surrounded by a broken line) are to be added. The second frame is, for example, an even frame.

FIG. 2D shows an example of weights given to pixel values in the second frame. In this example, the added pixel values of GP3 and GP4 are obtained by the following equation (2).
GP3 = B + (1/2) * C + (1/2) * E + (1/4) * F,
GP4 = D + (1/2) * E + (1/2) * G + (1/4) * H (2)

  As described above, in the checkered superposition shift pixel addition sampling, a specific pixel is superimposed while shifting the pixel addition target in a checkered pattern, and a plurality of weights are given to the specific pixel to perform pixel addition processing.

The weight given to the pixel values A to I is not limited to the above. For example, if GP1 is taken as an example, the added pixel value may be obtained by the following equation (3). Here, r is a real number of r ≧ 1. The example of FIG. 2C and FIG. 2D is a case where r = 2 in the following formula (3).
GP1 = A + (1 / r) * B + (1 / r) * C + (1 / r ² ) * D (3)

4). Horizontal Calculation Unit Next, the horizontal calculation unit 8 will be described in detail.

  FIG. 3 shows a detailed configuration example of the horizontal calculation unit 8. The horizontal calculation unit 8 (horizontal calculation circuit in a narrow sense) includes a line memory 32, a weight calculation unit 33 (weight calculation circuit), and a horizontal addition unit 34 (horizontal addition circuit). Hereinafter, an example of the process for the pixels in the 3rd row and the 3rd column in the first frame described above with reference to FIG. 2A will be described.

  The A / D conversion output 30 from the column parallel A / D conversion unit 7 is input to the horizontal calculation unit 31. Specifically, the row signals (A, B, C), (D, E, F), (G, H, I) are temporarily stored in the line memory 32 sequentially. In the following, a case where (D, E, F) row signals are stored will be described as an example.

  The weight calculator 33 selects a column signal from the row signals (D, E, F) based on the timing control signal from the timing controller 12, and gives a predetermined weight to the selected column signal. . Specifically, the weight calculation unit 33 includes first to fourth multiplication units MP1 to MP4.

  The multipliers MP1 to MP4 multiply the pixel values D, E, E, and F by weights 1/2, 1/4, 1, and 1/2, respectively. In this example, two weights are given to the pixel value E, but which of the pixel values is given two weights is selected by the timing control signal. The multipliers MP1 to MP4 perform this multiplication at the same timing. The same timing is, for example, a timing for reading a pixel value from the line memory 32 or a same edge timing of a clock synchronized with a horizontal synchronization signal.

  The horizontal adder 34 adds pixel values to which weights are given, and outputs the added value as a horizontal pixel addition output 35. Specifically, the horizontal adder 34 includes first and second adders AH1 and AH2.

  Adder AH1 adds the outputs from multipliers MP1 and MP2, and outputs an added value ((1/2) * D + (1/4) * E). The adder AH2 adds the outputs from the multipliers MP3 and MP4, and outputs an added value (E + (1/2) * F).

5. Vertical Calculation Unit Next, the vertical calculation unit 9 will be described in detail.
FIG. 4 shows a detailed configuration example of the vertical calculation unit 9. The vertical calculation unit 9 (vertical calculation unit in a narrow sense) includes a line memory 42 and a vertical addition unit 43 (vertical addition circuit). In the following, an explanation will be given by taking an example of the processing for the pixels of 3 rows and 3 columns in the first frame described above with reference to FIGS.

  The vertical operation unit 9 receives the horizontal pixel addition output 35 from the horizontal operation unit 8. Specifically, the line memory 42 includes first to third line memories LM1 to LM3. A horizontal addition signal (A + (1/2) * B, B + (1/2) * C) is input to the line memory LM1 and temporarily stored. A horizontal addition signal ((1/2) * D + (1/4) * E, E + (1/2) * F) is input to the line memory LM2 and temporarily stored. A horizontal addition signal (G + (1/2) * H, (1/2) * H + (1/4) * I) is input to the line memory LM3 and temporarily stored.

  The vertical addition unit 43 receives the horizontal addition signal of the row selected based on the timing control unit. The line is selected by selecting a line memory corresponding to the line from the line memories LM1 to LM3. The vertical adder 43 adds the input horizontal addition signals and outputs the added value as the vertical pixel addition output 44. Specifically, the vertical adder 43 includes first and second adders AV1 and AV2.

  The adders AV1 and AV2 add the horizontal addition signals stored in the line memories LM1 and LM2, and add signals (A + (1/2) * B + (1/2) * D + (1/4) * E), respectively. , (B + (1/2) * C + E + (1/2) * F). The adders AV1 and AV2 add the horizontal addition signals stored in the line memories LM2 and LM3, respectively, and add signals ((1/2) * D + (1/4) * E + G + (1/2) * H ), (E + (1/2) * F + (1/2) * H + (1/4) * I).

  FIG. 4 shows an example of connection when signals from the line memories LM1 and LM2 are input to the vertical adder 43, but a signal from any line memory is input to the vertical adder 43. This is selected by the selector 13 based on the timing control signal.

  According to the above-described embodiment, as shown in FIG. 1, the solid-state imaging device 1 (solid-state imaging device, image sensor in a broad sense) includes a pixel array unit (pixel circuit array 2), a row scanning unit 4, and an A / D. A conversion unit (column parallel A / D conversion unit 7), a horizontal calculation unit 8, and a vertical calculation unit 9 are included.

  In the pixel array section, a plurality of pixels 3 that photoelectrically convert incident light are arranged. The row scanning unit 4 selects a row in the pixel array unit and performs vertical scanning. The A / D converter converts an analog signal from a pixel in the selected row into a pixel value (pixel signal) of a digital signal.

  As shown in FIG. 2A, the addition unit (for example, group GP1) is a range of pixels to be added. The horizontal addition range is a column range of addition units (for example, in GP1, the first column to the second column of the pixel array). The vertical addition range is a row range of addition units (for example, in GP1, the first row to the second row of the pixel array). As shown in FIG. 3, the horizontal calculation unit 8 adds the pixel values (for example, D and E in the second row) included in the horizontal addition range among the pixel values in the row, weighting each column, and the addition is performed. The obtained pixel value is output as a horizontal addition pixel value ((1/2) * D + (1/4) * E). As illustrated in FIG. 4, the vertical calculation unit 9 adds the horizontal addition pixel values included in the vertical addition range among the horizontal addition pixel values, and outputs the added horizontal addition pixel value as the addition pixel value.

  In this case, as shown in FIG. 2, the horizontal calculation unit 8 includes a first horizontal addition range (for example, D and E in the second row) and a pixel (E) common to the first horizontal addition range. The horizontal addition pixel value is output using the second horizontal addition range (E, F) as the horizontal addition range.

  Thereby, since the pixel addition of the 1st and 2nd horizontal addition range including a common pixel can be performed, pixel addition can be performed while superimposing the addition unit. In addition, since each column is weighted and added, each pixel value included in the horizontal addition range can be weighted and added. Further, by performing pixel addition while superimposing the addition units, it becomes possible to restore a high-resolution image by simple processing, as will be described later with reference to FIGS. Further, since the high-resolution image at an arbitrary timing can be extracted from the moving image by the restoration process, the operability for the user can be improved without missing a photo opportunity.

  In the present embodiment, as shown in FIG. 3, the horizontal calculation unit 8 includes the common pixel (E) in the first horizontal addition range (for example, D, E) and the second horizontal addition range (E, F). The common pixel (E) in () is weighted with different weighting factors (1/4, 1).

  In this way, independent weighting can be performed on common pixels in each addition range. Thereby, even when the addition units are overlapped, weighting addition can be performed in each addition unit.

  Specifically, the first horizontal addition range includes a first pixel (for example, D) and a second pixel (E), and the second horizontal addition range includes a second pixel (E) and a third pixel. (F) is included. The horizontal calculation unit 8 includes a line memory 32, a weight calculation unit 33, and a horizontal addition unit 34. The line memory 32 stores first to third pixel values (D, E, F) that are pixel values of the first to third pixels. The weight calculation unit 33 weights the first pixel value (D) with the first weighting factor (1/2), and the second and third weighting factors (1/4) for the second pixel value (E). 1) and weighting by the fourth weighting factor (1) for the third pixel value (F). The horizontal adder 34 adds the first and second pixel values weighted by the first and second weighting factors, and the second and third pixel values weighted by the third and fourth weighting factors. Is added.

  In this way, it is possible to configure a horizontal arithmetic unit that assigns different weighting coefficients in the respective horizontal addition ranges to the common pixels of the first horizontal addition range and the second horizontal addition range.

  In the present embodiment, the weight calculation unit 33 includes first to fourth multiplication units MP1 to MP4 that perform multiplication of the first to fourth weighting factors. The first to fourth multipliers MP1 to MP4 multiply the pixel values stored in the line memory 32 by the first to fourth weighting coefficients at the same timing.

  In this way, it is possible to perform the weight calculation of the addition unit to be superimposed at one timing for one row. Thereby, since it is not necessary to perform the weight calculation with respect to the common pixel at two times, the solid-state imaging device can be configured without increasing the operation speed as compared with the conventional pixel addition without superimposing. In addition, since the operation speed does not change, superposition shift sampling can be performed only by changing the horizontal arithmetic unit and the vertical arithmetic unit of the conventional solid-state imaging device, and the design cost can be reduced.

  In the present embodiment, as shown in FIG. 4, the vertical calculation unit 9 includes a first vertical addition range (for example, the first row and the second row) and a row common to the first vertical addition range (second row). The added pixel value is output using the second vertical addition range (second row and third row) including the row) as the vertical addition range.

  In this way, since the horizontal addition pixel values in the common row of the first and second vertical addition ranges can be added for each vertical addition range, pixel addition in the overlapping vertical addition range can be realized.

  Specifically, the vertical calculation unit 9 includes first to third line memories LM1 to LM3 and a vertical addition unit 43. The first to third line memories LM1 to LM3 store the horizontal addition pixel values of the first to third rows, respectively. The vertical addition unit 43 performs addition of the horizontal addition pixel values of the first and second rows and addition of the second and third horizontal addition pixel values.

  In this way, it is possible to configure a vertical calculation unit that adds the horizontal addition pixel values in the common row of the first vertical addition range and the second vertical addition range in each vertical addition range.

  In the present embodiment, as illustrated in FIG. 1, the solid-state imaging device 1 includes a control unit (timing control unit 12) and a selection unit 13. The control unit sets the addition unit. The selection unit 13 selects a column and a row based on the set addition unit. The horizontal calculation unit 8 adds the weighted pixel values of the selected column. The vertical calculation unit 9 adds the horizontal addition pixel values of the selected row.

  In this way, by selecting a column and a row based on the set addition unit, the pixel values of the pixels included in the addition unit can be added to obtain the added pixel value.

  In this embodiment, the A / D conversion unit is a column parallel A / D conversion unit 7 that converts the analog signals of the respective columns in the row selected by the row scanning unit 4 into digital signals in parallel.

  In this way, analog signals from one row of pixels can be converted into digital signals in parallel. This eliminates the need for high-speed A / D conversion, so that the operation speed of the solid-state imaging device 1 can be reduced.

6). Same Color Pixel Superposition Shift Addition Processing A case where the above-described superposition shift pixel addition is applied to an image sensor with a Bayer array will be described in detail. First, a case where four pixels of the same color are added will be described. In the following, a pixel array of 10 rows and 10 columns will be described as an example.

  FIG. 5A shows an example of setting the addition range when performing the same color four-pixel addition. In FIG. 5A, M represents a column number (column), and N represents a row number (line).

  As shown in FIG. 5A, addition ranges (addition units) GP00, GP40, GP22, GP62, GP04, GP44, GP26, and GP66 are set, and pixel signals of the same color 4 pixels included in each addition range are added. .

  FIG. 5B schematically shows an example of an output signal when performing the same color four-pixel addition. In FIG. 5B, weights given to the pixels are omitted, and the added pixels are represented by a group surrounded by a solid line. In addition, pixels in M columns and N rows are represented by a suffix “_MN”, and for example, R pixels in 3 rows and 2 columns are represented as R_32. HD represents a horizontal synchronizing signal, and 1H represents one horizontal scanning period. VD represents a vertical synchronizing signal, and 1V represents one vertical scanning period.

  As shown in FIG. 5B, the added pixel values are sequentially output as the rows are scanned. That is, as shown by A1, first, the addition pixel values of the Gr pixel and R pixel in the addition ranges GP00 and GP40 are output. Next, as shown in A2, the added pixel values of the B and Gb pixels in the addition ranges GP00 and GP40 are output. Thereafter, similarly, the addition pixel values of the addition ranges GP22 and GP62, the addition pixel values of the addition ranges GP04 and GP44, and the addition pixel values of the addition ranges GP26 and GP66 are sequentially output.

  In this way, an output signal of a total of 32 pixels of 4 columns and 8 rows per frame is obtained by the checkered superimposed shift pixel addition sampling of the Bayer array. When the first and second frames described above are combined in FIG. 2A or the like, a total of 64 pixel output signals are obtained, and pixel values are restored from the 64 pixel output signals.

  Next, the operation in the case of performing the above four-pixel addition of the same color will be described with reference to FIG. As shown in B1 of FIG. 6, the column parallel A / D converter 7 sequentially outputs the A / D conversion output signals of the respective lines. As shown in B2, the horizontal arithmetic unit 8 gives a weight to the A / D conversion output signal, adds the signal in the horizontal direction, and outputs a horizontal pixel addition output signal. In FIG. 6, weights are omitted.

  Next, as indicated by B3, the line memory LM1 of the vertical calculation unit 9 temporarily stores the horizontal pixel addition output signal of the R line (N = 0). As shown at B4, the line memory LM2 temporarily stores the horizontal pixel addition output signal of the B line (N = 1). As shown in B5, the line memory LM3 temporarily stores the horizontal pixel addition output signal of the R line (N = 2). Next, as shown in B6, the row selection unit 16 selects a row and a pixel to be added, and the vertical addition unit 43 adds the signals stored in the line memories LM1 and LM3 to obtain a 4-pixel addition signal. Output. In FIG. 6, the addition target is represented by marks (♦, ▲, ●), indicating that signals of the same mark are added. Further, the 4-pixel addition signal is abbreviated. For example, Gr_00_20_02_22 represents an addition value of the pixel values Gr_00, Gr_20, Gr_02, and Gr_22.

  Next, as indicated by B7, the line memory LM1 temporarily stores the horizontal pixel addition output signal of the B line (N = 3). Next, as indicated by B8, the vertical adder 43 adds the signals stored in the line memories LM2 and LM1, and outputs a 4-pixel addition signal.

  Next, as indicated by B9, the line memory LM2 temporarily stores the horizontal pixel addition output signal of the R line (N = 4). Next, as indicated by B10, the vertical adder 43 adds the signals stored in the line memories LM3 and LM2, and outputs a 4-pixel addition signal. Thereafter, the same operation is repeated.

  In this way, the 4-pixel addition value described with reference to FIG. 5B is obtained. As can be seen from FIG. 6, in the four-pixel addition sampling with the same color checkered superposition shift of the Bayer array, at least three line memories of the vertical arithmetic unit 9 are required. As described above, the required number of line memories depends on the number of pixels added in the vertical direction.

7). Different Color Pixel Superposition Shift Addition Processing Next, a case where four different color pixels are added by superposition shift pixel addition will be described. As in the example of 4 pixels of the same color, a 10 × 10 pixel array will be described as an example.

  FIG. 7A shows an example of setting the addition range in the case where four different color pixels are added. As shown in FIG. 7A, addition ranges (addition units) GP00, GP20, GP40, GP60, GP11, GP31, GP51, GP71,... Are set. Each addition range includes four adjacent pixels of different colors (one unit of Bayer array), and adds the pixel signals of the four pixels of different colors.

  FIG. 7B schematically shows an example of an output signal in the case where four different color pixels are added. As in FIG. 5B, weights given to the pixels are omitted. As indicated by D1 in FIG. 7B, first, the addition pixel values of the addition ranges GP00, GP20, GP40, and GP60 are output. Next, as shown in A2, the addition pixel values of the addition ranges GP11, GP31, GP51, and GP71 are output. Thereafter, the added pixel values are sequentially output as the rows are scanned.

  Next, the operation in the case of performing the above four-color addition of different colors will be described with reference to FIG. As in FIG. 6, weights are omitted, and addition targets are represented by marks (♦, ▲, ●). Further, the 4-pixel addition signal is abbreviated. For example, 00_10_01_11 represents an addition value of the pixel values Gr_00, R_10, B_01, and Gb_11.

  As indicated by E1 in FIG. 8, the column parallel A / D converter 7 sequentially outputs the A / D conversion output signals of the respective lines. As shown in E2, the horizontal arithmetic unit 8 gives weight to the A / D conversion output signal, adds the signal in the horizontal direction, and outputs a horizontal pixel addition output signal.

  Next, as indicated by E3, the line memory LM1 of the vertical arithmetic unit 9 temporarily stores the horizontal pixel addition output signal of the R line (N = 0). As indicated by E4, the line memory LM2 temporarily stores the horizontal pixel addition output signal of the B line (N = 1). Next, as indicated by E5, the row selection unit 16 selects a row and a pixel to be added, and the vertical addition unit 43 adds the signals stored in the line memories LM1 and LM2, and outputs a 4-pixel addition signal. Output.

  Next, as indicated by E6, the line memory LM3 temporarily stores the horizontal pixel addition output signal of the R line (N = 2). Next, as indicated by E7, the vertical adder 43 adds the signals stored in the line memories LM2 and LM3, and outputs a 4-pixel addition signal.

  Next, as indicated by E8, the line memory LM1 temporarily stores the horizontal pixel addition output signal of the B line (N = 3). Next, as indicated by E9, the vertical adder 43 adds the signals stored in the line memories LM3 and LM1, and outputs a 4-pixel addition signal. Thereafter, the same operation is repeated.

  According to the above embodiment, it is not necessary to complicate the processing and functions as compared with the conventional solid-state imaging device, and the time corresponding to the number of columns and the number of pixels to be read is not spent more than necessary, and the checkered superimposed shift pixels It is possible to perform additive sampling. As a result, imaging can be performed without being aware of the boundary between a moving image and a still image, which can greatly contribute to the versatility of the solid-state imaging device.

8). Addition process without superimposition In the present embodiment, it is also possible to perform normal four-pixel addition in which the addition range does not overlap. A case where the present embodiment is applied to a normal 4-pixel addition process will be described with reference to FIGS.

  FIG. 9A shows an example of setting the addition range when performing normal four-pixel addition. As shown in FIG. 9A, non-overlapping addition ranges (addition units) GP00, GP40, GP04, and GP44 are set, and pixel signals of four pixels of the same color included in each addition range are added.

  FIG. 9B schematically shows an example of an output signal when performing normal four-pixel addition. As in FIG. 5B, weights given to the pixels are omitted. As shown in FIG. 9B, first, the addition pixel values of the addition ranges GP00 and GP40 and the addition pixel values of the addition ranges GP04 and GP44 are sequentially output.

  Next, the operation when performing the above-described normal four-pixel addition will be described with reference to FIG. As in FIG. 10, weights are omitted, and addition targets are represented by marks (♦, ▲, ●). Further, the 4-pixel addition signal is abbreviated. For example, Gr_00_20_02_22 represents an addition value of the pixel values Gr_00, Gr_20, Gr_02, and Gr_22.

  As indicated by G1 in FIG. 10, the column parallel A / D converter 7 sequentially outputs the A / D conversion output signals of the respective lines. As indicated by G2, the horizontal calculation unit 8 gives weight to the A / D conversion output signal, adds the signal in the horizontal direction, and outputs a horizontal pixel addition output signal. At this time, superposition shift addition is not performed. That is, since the addition range does not overlap, one pixel is not added within the addition range of 2.

  Next, as indicated by G3, the line memory LM1 of the vertical arithmetic unit 9 temporarily stores the horizontal pixel addition output signal of the R line (N = 0). As indicated by G4, the line memory LM2 temporarily stores the horizontal pixel addition output signal of the B line (N = 1). Next, as indicated by G5, the horizontal pixel addition output signal of the R line (N = 2) is not temporarily stored, but the line memory is passed. Next, as indicated by G6, the row selection unit 16 selects a row and a pixel to be added, and the vertical addition unit 43 adds the signal stored in the line memory LM1 and the signal passed through to add 4 pixels. Output the addition signal.

  Next, as indicated by G7, the horizontal pixel addition output signal of the B line (N = 3) is not temporarily stored, but the line memory is passed. Next, as indicated by G8, the vertical adder 43 adds the signal stored in the line memory LM2 and the passed signal, and outputs a 4-pixel addition signal. Thereafter, the same operation is repeated.

  In this manner, normal pixel addition sampling is possible in this embodiment. As a matter of course, even when all the pixels are read out when the solid-state imaging device is fully scanned, the configuration of superposition shift addition does not become a restriction or condition.

  In the above embodiment, the case where 4-pixel addition is performed has been described as an example. However, the present embodiment is not limited to this, and functions and processing configurations are expanded to support other pixel addition numbers. Also good.

  According to the embodiments described above, an imaging apparatus and an imaging apparatus that realize signal readout in a low resolution mode necessary for increasing the resolution of image data without complicating functional blocks and without affecting the operation time. A control method can be provided.

9. Restoration Estimation Processing Next, processing for restoring a high resolution image by estimation from the added pixel values acquired by superposition shift sampling will be described in detail. In the following, the addition pixel values {a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ } will be described as an example (i and j are integers of 0 or more), but the same applies to other addition pixel values. Further, the case where the addition unit is set for every 2 × 2 pixels will be described as an example, but the present invention is not limited to this, and may be, for example, every 3 × 3 pixels.

FIGS. 11A and 11B are explanatory diagrams of the estimated pixel value and the intermediate pixel value. The added pixel values {a ₀₀ , a ₁₁ } illustrated in FIG. 11A correspond to the added pixel values of the groups GP1 and GP2 described with reference to FIG. The addition pixel values {a ₁₀ , a ₀₁ } correspond to the addition pixel values of the groups GP3 and GP4 described with reference to FIG. In the estimation process, final estimated pixel values v _{00 to} v ₂₂ are estimated using the added pixel values. The estimated pixel value v _ij corresponds to the pixel values of the pixels A to I described with reference to FIG.

As shown in FIG. 11B, first, intermediate pixel values b _{00 to} b ₂₁ (intermediate estimated pixel values) are estimated from the added pixel values a _{00 to} a ₁₁ . Intermediate pixel value corresponds to 2-pixel sum values, for example, _{b 00} corresponds to the sum of the pixel values _{v 00} and _{v 01.} The final pixel values v _{00 to} v ₂₂ are estimated from these intermediate pixel values b _{00 to} b ₂₁ .

First, a process for estimating the intermediate pixel value will be described. Hereinafter, a case where the intermediate pixel values b _{00 to} b ₂₀ of the first row in the horizontal direction are estimated will be described as an example. The intermediate pixel values b _{01 to} b _{21 in the} next row are estimated by the same method.

As shown in FIG. 12, the intermediate pixel values b _{00 to} b ₂₀ are estimated based on the added pixel values a ₀₀ and a ₁₀ in the first row in the horizontal direction. For the sake of simplicity, for example, assuming that the weighting factor r = 2, the addition pixel values a ₀₀ and a ₁₀ are expressed by the following expression (4).
a ₀₀ = v ₀₀ + (1/2) v ₀₁ + (1/2) v ₁₀ + (1/4) v ₁₁ ,
_{a 10 = v 10 + (1/2} ) v 11 + (1/2) v 20 + (1/4) v 21 (4)

B ₀₀ , b ₁₀ , and b ₂₀ are defined as shown in the following formula (5).
b ₀₀ = v ₀₀ + (1 / r) v ₀₁ = v ₀₀ + (1/2) v ₀₁ ,
b ₁₀ = v ₁₀ + (1 / r) v ₁₁ = v ₁₀ + (1/2) v ₁₁ ,
b ₂₀ = v ₂₀ + (1 / r) v ₂₁ = v ₂₀ + (1/2) v ₂₁ (5)

Next, when the above equation (4) is transformed using the above equation (5), the following equation (6) is established.
a ₀₀ = b ₀₀ + (1/2) b ₁₀ ,
a ₁₀ = b ₁₀ + (1/2) b ₂₀ (6)

In the above equation (6), when a ₀₀ and a ₁₀ are multiplied by a predetermined weighting factor to obtain the difference δi ₀ and rearranged, the following equation (7) is established.
δi ₀ = a ₁₀ -2a ₀₀
= (1/2) b ₂₀ -2b ₀₀ (7)

If b ₀₀ is an unknown (initial variable), intermediate pixel values b ₁₀ and b ₂₀ can be obtained as a function of b ₀₀ as shown in the following equation (8). In this way, a high-definition combination pattern of intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } is obtained with b ₀₀ as an unknown.
b ₀₀ = (unknown number),
b ₁₀ = 2 (a ₀₀ −b ₀₀ ),
b ₂₀ = 4b ₀₀ + 2δi ₀ = 4b ₀₀ +2 (a ₁₀ −2a ₀₀ ) (8)

Next, a description will be given of a method of obtaining the unknown _{b 00.} As shown in FIG. 13, the pattern {a ₀₀ , a ₁₀ } of the added pixel value is compared with the pattern {b ₀₀ , b ₁₀ , b ₂₀ } of the intermediate pixel value. Then, an unknown number b ₀₀ that minimizes the error E is derived and set as the intermediate pixel value b ₀₀ .

Specifically, the relationship of the following formula (9) is established between the added pixel value {a _ij } and the intermediate pixel value {b _ij , b _{(i + 1) j} }. Considering the weighting by the following equation (9), the evaluation function Ej shown in the following equation (10) is obtained. Then, with this evaluation function Ej, similarity evaluation between the pattern {a ₀₀ , a ₁₀ } and the pattern {b ₀₀ , b ₁₀ , b ₂₀ } is performed.
a _ij = b _ij + (1/2) b _{(i + 1) j} (9)

As shown in FIG. 14, an unknown b ₀₀ (= α) that minimizes Ej is obtained, and the value of b ₀₀ can be determined. Then, the estimated value of b ₀₀ is substituted into the above equation (8) to obtain b ₁₀ and b ₂₀ .

Next, a method for _{obtaining the} final estimated pixel value v _ij using the obtained intermediate pixel value b _ij will be described. In the following, description will be given taking the leftmost vertical column (i = 0 column) as an example. The final estimated pixel value v _ij is obtained in the same manner as the method for _obtaining the intermediate pixel value b _ij . That is, if the above equation (6) is replaced with the following equation (11), the subsequent processing is the same.
b ₀₀ = v ₀₀ + (1/2) v ₀₁ ,
b ₀₁ = v ₀₁ + (1/2) v ₀₂ (11)
9. Imaging device

  FIG. 15 shows a configuration example of an imaging apparatus that performs the above-described superposition shift sampling and restoration processing. The imaging device includes an imaging optical system 100 (lens), an optical low-pass filter 110, a solid-state imaging device 1, a data recording unit 140 (storage unit), a display processing unit 150, a monitor display unit 160, and a pixel value estimation calculation unit 210 (estimation calculation). Part) and the image output part 300.

  Note that the imaging apparatus according to the present embodiment is not limited to this configuration, and various modifications such as omitting some of the components or adding other components are possible. For example, the pixel value estimation calculation unit 210 and the image output unit 300 may be configured by an image processing device (for example, a PC) outside the imaging device.

  The imaging optical system 100 forms an image of a subject. For example, the optical low-pass filter 110 passes a band corresponding to the resolution of the solid-state imaging device 1. The solid-state imaging device 1 (for example, 12 megapixels) is configured by, for example, a CCD or a CMOS sensor. The solid-state imaging device 1 controls setting of addition units and addition reading, and acquires a fusion frame. A fusion frame is an image obtained by superposition shift sampling. The data recording unit 140 is realized by, for example, a memory card or the like, and records a moving image using a fusion frame. The monitor display unit 160 displays a live view of a moving image and a reproduced moving image.

  The pixel value estimation calculation unit 210 estimates a final estimated pixel value. The image output unit 300 outputs a still image or a moving image based on the final estimated pixel value. The image output unit 300 includes anti-aliasing filters 220 and 250, a low-pass filter 230, and an undersampling unit 240.

  The anti-aliasing filter 220 performs anti-aliasing processing on the final estimated pixel value and outputs a high-resolution still image (for example, 12 megapixels). The low-pass filter 230 limits the final estimated pixel value to a high-vision band. The undersampling unit 240 undersamples the band-limited final estimated pixel value to the number of high-definition pixels. The anti-aliasing filter 220 performs anti-aliasing processing on the undersampled image and outputs a high-definition moving image (for example, 2 megapixels). Note that a high-resolution moving image (for example, 12 megapixels) may be output without undersampling.

According to the above embodiment, as illustrated in FIG. 15, the imaging device (imaging system, for example, a digital camera) includes the solid-state imaging device 1 and an estimation calculation unit (pixel value estimation calculation unit 210). As described with reference to FIG. 11A, the estimation calculation unit, based on the addition pixel values {a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ }, the pixel values {v ₀₀ , v of pixels included in the addition unit. ₁₀ , v ₁₁ , v ₀₁ }.

As shown in FIG. 11A, the addition unit (eg, a ₀₀ ) set to the first position and the addition unit (eg, a ₁₀ ) set to the second position where the first position is shifted. Are superimposed. The vertical calculation unit 9 outputs first and second addition pixel values (a ₀₀ , a ₁₀ ) that are addition pixel values of the first and second addition units. As shown in the above equation (7), the estimation calculation unit obtains a difference value δi ₀ between the first and second addition pixel values a ₀₀ and a ₁₀ . As shown in FIG. 11B, the first intermediate pixel value b ₀₀ is an addition of the first area (v ₀₀ , v ₀₁ ) obtained by removing the overlapping area (v ₁₀ , v ₁₁ ) from the addition unit a _00. It is a pixel value. The second intermediate pixel value b ₂₀ is an addition pixel value of the second region (v ₂₀ , v ₂₁ ) obtained by removing the overlap region (v ₁₀ , v ₁₁ ) from the addition unit a ₁₀ . As shown in the above equation (8), a relational expression between the first and second intermediate pixel values b ₀₀ and b ₂₀ is expressed using a difference value δi ₀ . As shown in FIG. 13 and the like, the first and second intermediate pixel values b ₀₀ and b ₂₀ are estimated using the relational expression. Using the estimated first intermediate pixel value b ₀₀ , pixel values {v ₀₀ , v ₁₀ , v ₁₁ , v ₀₁ } of each pixel included in the addition unit are obtained.

  In this way, it is possible to simplify the estimation process of the high-resolution image by once estimating the intermediate pixel value from the addition pixel value subjected to the superposition shift and obtaining the estimation pixel value from the intermediate pixel value subjected to the superposition shift. . For example, complicated processing such as a repetitive calculation of a two-dimensional filter (Japanese Patent Laid-Open No. 2009-124621) and a search for a part suitable for setting an initial value (Japanese Patent Laid-Open No. 2008-243037) are not required.

Here, superimposing means having an area where the addition unit overlaps the addition unit. For example, as shown in FIG. 11A, the addition unit a ₀₀ and the addition unit a ₁₀ are two estimated pixels v _10. is to share the _{v 11.}

  Further, the position of the addition unit is the position and coordinates of the addition unit in the captured image, or the position and coordinates of the addition unit on the estimated pixel value data (image data) in the estimation process. The shifted position is a position shifted from the original position by a pixel, and is a position where the position and coordinates do not coincide with the original position.

In this embodiment, continuous intermediate pixel values including the first and second intermediate pixel values (for example, b ₀₀ , b ₂₀ ) are set as intermediate pixel value patterns {b ₀₀ , b ₁₀ , b ₂₀ }. As shown in the above equation (8), the estimation calculation unit represents the relational expression between the intermediate pixel values included in the intermediate pixel value pattern using the added pixel values a ₀₀ and a ₁₀ . As shown in FIG. 13, the similarity is evaluated by comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the added pixel value. Based on the similarity evaluation result, the intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ included in the intermediate pixel value pattern are determined so that the similarity is the highest.

  In this way, the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by the addition unit that is pixel-shifted while being superimposed.

  Here, the intermediate pixel value pattern is a data string (a set of data) of intermediate pixel values in a range used for estimation processing. The addition pixel value pattern is a data string of addition pixel values in a range used for the estimation process.

Further, in the present embodiment, as shown in the above equation (10), the estimation calculation unit includes the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } represented by the relational expression between the intermediate pixel values and the addition pixel. An evaluation function Ej representing an error from the values {a ₀₀ , a ₁₀ } is obtained. The intermediate pixel values b ₀₀ , b ₁₀ and b ₂₀ included in the intermediate pixel value pattern are determined so that the value of the evaluation function Ej is minimized.

  In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. For example, searching for an image portion suitable for initial value setting (Japanese Patent Laid-Open No. 2008-243037) is unnecessary.

In the present embodiment, as shown in the above equation (4), each pixel value (for example, v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ ) of the addition unit is weighted and added to the added pixel value (a ₀₀ ). get. Based on the obtained addition pixel value (a ₀₀ , a ₁₀ ) of the addition unit, the pixel value (v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ ) of each pixel of the addition unit is estimated.

  If it does in this way, each pixel value of an addition unit will carry out weighted addition, an addition image will be acquired, and the pixel value of a high-resolution image can be estimated from the acquired addition image. Thereby, in the estimation process, the reproducibility of the high-frequency component of the subject can be improved. That is, when the pixel values of the addition unit are simply added, a rectangular window function is convoluted for imaging. On the other hand, when the pixel value of the addition unit is weighted and added, a window function containing more high frequency components than the rectangle is convoluted for imaging. Therefore, it is possible to acquire an added image that includes more high-frequency components of the subject, and to improve the reproducibility of the high-frequency components in the estimated image.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the solid-state imaging device, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

1 solid-state imaging device, 2 pixel circuit array, 3 unit pixel circuit, 4 row scanning unit,
5 row selection line, 6 column signal line, 7 column parallel A / D converter, 8 horizontal arithmetic unit,
9 Vertical calculation unit, 10 line memory, 11 column scanning unit,
12 timing control unit, 13 selection unit, 14 output lines, 15 column selection unit,
16 row selection unit, 30 A / D conversion output, 31 horizontal calculation unit,
32 line memory, 33 weight calculator, 34 horizontal adder,
35 horizontal pixel addition output, 42 line memory, 43 vertical addition unit,
44 vertical pixel addition output, 100 imaging optical system,
110 optical low-pass filter, 140 data recording unit,
150 display processing unit, 160 monitor display unit, 210 pixel value estimation calculation unit,
220 anti-aliasing filter, 230 low-pass filter,
240 undersampling unit, 300 image output unit,
a _{00 to} a ₁₁ addition pixel value, b _{00 to} b ₂₀ intermediate pixel value, r weighting coefficient,
v _{00 to} v ₂₂ estimated pixel values, δi ₀ difference values, A to I pixels,
AH1, AH2 adder, AV1, AV2 adder, Ej evaluation function,
GP1-GP4 group, LM1-LM3 line memory, M line number,
MP1 to MP4 multiplier, N column number

Claims (11)

A pixel array unit in which a plurality of pixels that photoelectrically convert incident light are arranged;
A row scanning unit that selects a row of the pixel array unit and performs vertical scanning;
An A / D converter that converts an analog signal from the pixel in the selected row into a pixel value of a digital signal;
When the column range of the addition unit, which is the range of pixels to be added, is a horizontal addition range and the row range is a vertical addition range, the pixel values included in the horizontal addition range among the pixel values of the row are A horizontal calculation unit that weights and adds each column and outputs the added pixel value as a horizontal addition pixel value;
A vertical arithmetic unit that adds the horizontal addition pixel values included in the vertical addition range among the horizontal addition pixel values and outputs the added horizontal addition pixel values as addition pixel values;
Including
The horizontal calculation unit is
Solid-state imaging, wherein the horizontal addition pixel value is output using the first horizontal addition range and the second horizontal addition range including pixels in common with the first horizontal addition range as the horizontal addition range. apparatus. In claim 1,
The horizontal calculation unit is
The solid-state imaging device, wherein the weighting with different weighting coefficients is performed on the common pixel in the first horizontal addition range and the common pixel in the second horizontal addition range. In claim 2,
When the first horizontal addition range includes a first pixel and a second pixel, and the second horizontal addition range includes the second pixel and a third pixel,
The horizontal calculation unit is
A line memory for storing first to third pixel values which are pixel values of the first to third pixels;
Weighting the first pixel value with a first weighting factor, weighting the second pixel value with second and third weighting factors, and weighting the third pixel value with a fourth weighting factor A weight calculation unit for performing
Addition of the first and second pixel values weighted by the first and second weighting factors and addition of the second and third pixel values weighted by the third and fourth weighting factors A horizontal adder for performing
A solid-state imaging device. In claim 3,
The weight calculator is
Having first to fourth multipliers for multiplying the first to fourth weighting factors;
The first to fourth multipliers are:
A solid-state imaging device, wherein the pixel values stored in the line memory are multiplied by the first to fourth weighting coefficients at the same timing. In any one of Claims 1 thru | or 4,
The vertical arithmetic unit is
A solid-state imaging device that outputs the added pixel value using the first vertical addition range and the second vertical addition range including a row common to the first vertical addition range as the vertical addition range . In claim 5,
The vertical arithmetic unit is
First to third line memories for storing the horizontal addition pixel values of the first to third rows;
A vertical adder that performs the addition of the horizontal addition pixel values of the first and second rows and the addition of the second and third horizontal addition pixel values;
A solid-state imaging device. In any one of Claims 1 thru | or 6.
A control unit for setting the addition unit;
A selection unit for selecting a column and a row based on the set addition unit;
Including
The horizontal calculation unit includes:
Weighting and adding the pixel values of the selected column,
The vertical arithmetic unit is
A solid-state imaging device, wherein the horizontal addition pixel values of the selected row are added. In any one of Claims 1 thru | or 7,
The A / D converter is
A solid-state imaging device, comprising: a column parallel A / D conversion unit that converts the analog signal of each column of a row selected by the row scanning unit into the digital signal in parallel. A solid-state imaging device according to any one of claims 1 to 8,
An estimation calculator that estimates a pixel value of a pixel included in the addition unit based on the addition pixel value;
Including
When the first addition unit set in the first position and the second addition unit set in the second position where the first position is shifted overlap,
The vertical arithmetic unit is
Outputting first and second addition pixel values which are the addition pixel values of the first and second addition units;
The estimation calculation unit includes:
A difference value between the first addition pixel value and the second addition pixel value is obtained;
A first intermediate pixel value that is an addition pixel value of the first area obtained by removing the overlap area from the first addition unit, and an addition pixel of the second area obtained by removing the overlap area from the second addition unit. A relational expression with the second intermediate pixel value that is a value is expressed using the difference value,
The first and second intermediate pixel values are estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. Imaging device. In claim 9,
The estimation calculation unit includes:
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between intermediate pixel values included in the intermediate pixel value pattern is obtained using the added pixel value. Represent,
Comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the added pixel value to evaluate similarity;
An image pickup apparatus, comprising: determining an intermediate pixel value included in the intermediate pixel value pattern based on the similarity evaluation result so that the similarity becomes the highest. When the column range of the addition unit that is the range of pixels to be added is the horizontal addition range and the row range is the vertical addition range,
A pixel array unit in which a plurality of pixels that photoelectrically convert incident light are arranged is prepared,
Select a row of the pixel array unit, perform vertical scanning,
Converting the analog signal from the pixel of the selected row into a pixel value of a digital signal;
A first horizontal addition range and a second horizontal addition range including pixels in common with the first horizontal addition range as the horizontal addition range;
Among the pixel values of the row, the pixel values included in the horizontal addition range are weighted and added to each column,
The added pixel value is output as a horizontal addition pixel value,
Adding the horizontal addition pixel value included in the vertical addition range of the horizontal addition pixel value,
An image generation method, characterized in that the added horizontal added pixel value is output as an added pixel value.

Images