JP2007174266A - Imaging device - Google Patents

Abstract

An image pickup apparatus suitable for reducing image quality deterioration when a charge (pixel signal) is read from an image sensor (photoelectric conversion element) using a non-destructive reading method is provided.
An image sensor includes an image sensor configured by a photoelectric conversion element, an image processing unit that reads out a pixel signal from the image sensor by a nondestructive readout method, and outputs pixel data of the pixel signal; OB difference pixel data and minimum exposure pixel data are generated based on various pixel data such as pixel data of the OB area output from the imaging processing unit 10 and pixel data at the minimum exposure time, and based on the generated pixel data. An image processing unit 12 that generates differential pixel data obtained by removing noise components from pixel data during normal exposure time, various pixel data output from the imaging processing unit 10, OB differential pixel data, minimum exposure pixel data, and the differential pixel And a frame memory 14 for storing data.
[Selection] Figure 1

Description

  The present invention relates to an imaging apparatus capable of reading charges from a photoelectric conversion element in a nondestructive manner.

Conventionally, there are a destructive readout method and a non-destructive readout method as a method for reading out charges from an image sensor (photoelectric conversion element) of an imaging apparatus.
Here, the destructive readout method performs a reset process for emptying the charge accumulated in the photoelectric conversion element when the charge (pixel signal) is read from the photoelectric conversion element.
In the non-destructive readout method, when charge (pixel signal) is read from a photoelectric conversion element, the charge stored in the photoelectric conversion element is read without being emptied. That is, since the reset process is not performed at the time of reading the charge, the charge can be read as many times as necessary during the charge accumulation until the set exposure time is reached. Therefore, the nondestructive readout method has an advantage that multistage exposure can be easily realized.

  In addition, the charge (pixel signal) read out from the photoelectric conversion element includes noise generated due to variations in characteristics of each element constituting the pixel such as the photoelectric conversion element (spectral characteristics etc. coming from the pixel structure). There is a problem in that this noise significantly deteriorates the image quality of the captured image. In general, when a method of reading charges using destructive readout is used, the noise is removed by taking the difference between the readout charge (pixel signal) and the charge (pixel signal) immediately after the reset process after the readout. Is done. In other words, the noise is removed from the pixel signal by taking the difference between the pixel signal obtained in the exposure state at the exposure time at the time of imaging and the pixel signal obtained in the substantially light-shielded state immediately after reset (in the dark). The image quality has been improved.

On the other hand, as an imaging device that reads electric charges (pixel signals) from a photoelectric conversion element using nondestructive readout, for example, there is an imaging device described in Patent Document 1.
The imaging apparatus described in Patent Document 1 reads out a plurality of pixel signals having different exposure times in an imaging apparatus in which each of the plurality of pixels includes an amplification type photoelectric conversion element, and from among the one pixel signals having a long exposure time. When a saturated pixel signal is detected, the saturated pixel signal is replaced with the unsaturated pixel signal of the pixel having a shorter exposure time.
Japanese Patent Laid-Open No. 5-22670

  However, the conventional technique of Patent Document 1 uses a non-destructive readout method that does not perform resetting when reading pixel signals, and obtains a dark signal immediately after reset as in the destructive readout method described above. Therefore, there is a problem that the image quality of the captured image is deteriorated due to noise generated due to variations in characteristics of elements constituting the pixels such as the photoelectric conversion elements. .

  In the prior art disclosed in Patent Document 1, for example, a pixel signal in the dark (for example, at the minimum exposure time) is acquired before imaging and stored in advance, and the stored pixel signal in the dark is stored. It is possible to remove noise from the pixel signal at the time of imaging using this method. However, if such a method is used, the pixel signal at the time of darkness cannot be changed in real time. There has been a problem that the noise removal accuracy is lowered when the characteristics of each element changes due to environmental changes (for example, temperature changes) or the like, and the state of noise changes.

  Therefore, the present invention has been made paying attention to such an unsolved problem of the conventional technology, and reading out charges (pixel signals) from an image sensor (photoelectric conversion element) is nondestructive reading. An object of the present invention is to provide an image pickup apparatus suitable for reducing image quality degradation when the method is used.

[Mode 1] In order to achieve the above object, an imaging apparatus according to mode 1
A photoelectric conversion unit in which a plurality of photoelectric conversion elements for converting exposed light into electric charges and storing them are arranged in a matrix, and non-destructive reading of charges from pixels constituting the photoelectric conversion elements in the photoelectric conversion unit An imaging device comprising imaging means for performing,
Minimum exposure pixel data acquisition means for acquiring minimum exposure pixel data, which is pixel data corresponding to charges read at the time of exposure with a settable minimum exposure time;
The difference value between the pixel value of each pixel data corresponding to the electric charge read at the time of exposure with a predetermined exposure time and the pixel value of each acquired minimum exposure pixel data corresponding to each pixel data is calculated, and the calculated Difference pixel data generation means for generating difference pixel data composed of difference values.

  With such a configuration, the minimum exposure pixel data acquisition unit can acquire minimum exposure pixel data that is pixel data corresponding to the electric charge read out during exposure with a settable minimum exposure time. The pixel data generation means calculates a difference value between the pixel value of each pixel data corresponding to the charge read at the time of exposure with a predetermined exposure time and the pixel value of each acquired minimum exposure pixel data corresponding to each pixel data. It is possible to calculate and generate difference pixel data including the calculated difference value.

  Therefore, before capturing a moving image, for example, pixel data having a minimum exposure time that can be set by an electronic shutter function or the like is acquired as pixel data in the dark, and the minimum exposure pixel data is obtained by, for example, normal exposure at the time of image capturing. By subtracting from the pixel data read out over time, fixed pattern noise caused by variations in each element can be almost removed from the pixel data during normal exposure, so that the image quality of the captured image can be improved. The effect is obtained.

Here, the “minimum exposure time” is preferably the minimum exposure time that can be set, for example, by an electronic shutter function or the like, but is minimum if the image quality of the captured image composed of the difference pixel data does not deteriorate. It is not necessary.
The “imaging means” is configured by using, for example, CMOS technology, and examples of an image sensor using the CMOS technology include a threshold modulation image sensor (VMIS (Threshold Voltage Modulation Image Sensor)).

  Further, the “fixed pattern noise” includes dark current shading which is a problem during long exposure, and noise generated due to a difference in sensor sensitivity for each pixel. The image sensor can obtain the exposure amount on the contrary from the amount of the charge generated by the exposure, but there is a charge that is accumulated in proportion to the time without exposure, which is called dark current. The cause of dark current is mainly thermal excitation generated on the silicon surface. In addition, excitation of electrons due to the photoelectric effect is caused not only by light energy but also by thermal energy, and in general, the dark current doubles every time the temperature rises by 5 to 8 ° C. Also, the difference in sensor sensitivity for each pixel is caused by variations that occur during the manufacturing of the aperture ratio and the thickness of the photodiode.

[Embodiment 2] Furthermore, an imaging apparatus according to Embodiment 2 is the imaging apparatus according to Embodiment 1,
(N-1) The pixel data corresponding to the charge for each line of the pixel read at the time of exposure with the exposure time for each type corresponding to the exposure time of each type (N is a natural number of 2 or more). (N-1) first to (N-1) output channels for storing each of them independently and outputting the stored pixel data independently for each type;
In the exposure for the (N-1) types of exposure time, the charge for each line of the pixel at the time of exposure with the minimum exposure time is read, and pixel data corresponding to the read charge is stored and stored. An Nth output channel for outputting pixel data;
The minimum exposure pixel data acquisition means acquires the minimum exposure pixel data output via the Nth output channel;
The difference pixel data generation unit includes pixel values of pixel data corresponding to (N-1) types of exposure times output from the first to (N-1) th output channels for each line of the pixels, The difference value with the pixel value of the minimum exposure pixel data acquired by the minimum pixel data acquisition unit corresponding to each pixel data is calculated, and difference pixel data composed of the calculated difference value is generated. It is a feature.

  With such a configuration, the first to (N-1) th output channels correspond to (N-1) types (N is a natural number of 2 or more) of exposure time, and the exposure time for each type. The pixel data corresponding to the charges for each line of the pixels read during the exposure can be stored independently for each type, and the stored pixel data can be output independently for each type. Yes, the Nth output channel reads out the charges for each line of the pixels at the time of exposure with the minimum exposure time and stores the pixel data corresponding to the read charges in the exposure for the N-1 types of exposure times. In addition, the stored pixel data can be output.

  Further, the minimum exposure pixel data acquisition means can acquire the minimum exposure pixel data output via the Nth output channel, and the difference pixel data generation means can obtain each pixel line. , Pixel values of pixel data respectively corresponding to (N-1) types of exposure times output from the first to (N-1) th output channels, and acquisition of the minimum pixel data corresponding to each pixel data. It is possible to calculate a difference value from the pixel value of the minimum exposure pixel data acquired by the means, and generate difference pixel data including the calculated difference value.

  Therefore, (N-1) types of pixel data for each exposure time can be acquired via the independent first to (N-1) th channels, while the first to (N-1) th channels are the same. Since the pixel data at the minimum exposure time can be acquired via the independent Nth output channel, at the time of image capturing, in the same exposure section as the reading of the charge for each (N-1) type of exposure time, Since the charge can be read at the minimum exposure time, the pixel data at the minimum exposure time (in the dark) can be changed in real time and used for the generation of the difference pixel data. Even if the state of the noise changes due to a change in the noise, etc., it is possible to deal with the changed noise in real time.

[Mode 3] Furthermore, the imaging device of mode 3 is the imaging device of mode 2,
Pixel data corresponding to the charge during the exposure with the minimum exposure time, which is output through the first to (N-1) th output channels, is acquired and output through the Nth output channel. Pixel data corresponding to the charge during exposure with the minimum exposure time is acquired, and the minimum exposure acquired via each output channel for each output channel of the first to (N-1) th output channels. The difference value between the pixel value of the pixel data at the time and the pixel value of the pixel data at the minimum exposure obtained via the Nth output channel is calculated, and the minimum exposure difference pixel data including the difference value is generated. Exposure difference pixel data generating means;
Minimum exposure difference pixel data storage means for storing the minimum exposure difference pixel data,
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. A difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data is calculated, and the difference value of the pixel data and the pixel data are calculated. A difference value with respect to the value of the minimum exposure difference pixel data stored in the minimum exposure difference pixel data storage unit corresponding to is calculated, and difference pixel data composed of the difference value is generated. Yes.

  With such a configuration, the minimum exposure difference pixel data generation unit corresponds to the charge at the time of exposure with the minimum exposure time output through the first to (N-1) th output channels. Pixel data is obtained, and pixel data corresponding to the charge during exposure at the minimum exposure time output through the Nth output channel is obtained, and the first to (N-1) th output channels are obtained. For each output channel, the difference value between the pixel value of the pixel data at the minimum exposure obtained via each output channel and the pixel value of the pixel data at the minimum exposure obtained via the Nth output channel is calculated. It is possible to calculate and generate the minimum exposure difference pixel data including the difference value, and the minimum exposure difference pixel data can be stored by the minimum exposure difference pixel data storage means. It is.

  Further, the difference pixel data generation means is a pixel corresponding to each of (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. Calculating a difference value between the pixel value of the data and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data, and the difference value of the pixel data It is possible to calculate a difference value from the value of the minimum exposure difference pixel data stored in the minimum exposure difference pixel data storage unit corresponding to the pixel data, and generate difference pixel data including the difference value.

  Accordingly, in addition to the noise generated due to the variation in the characteristics of each pixel from the pixel data corresponding to each of the (N-1) types of exposure time, in the first to (N-1) output channels and the Nth output channel. Since it is possible to remove the noises generated, the effect of further improving the image quality of the captured image can be obtained.

[Mode 4] Furthermore, the imaging device of mode 4 is the imaging device of mode 3,
The minimum exposure difference pixel data for a predetermined one line of the pixel lines is stored in the minimum exposure difference pixel data storage means,
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. While calculating a difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data, the difference value of the pixel data and the minimum exposure difference A difference value with the value of the minimum exposure difference pixel data for one line stored in the pixel data storage means is calculated, and difference pixel data composed of the difference value is generated.

  With this configuration, it is possible to store the minimum exposure difference pixel data for a predetermined line of the pixel lines in the minimum exposure difference pixel data storage unit, and the difference pixel data generation unit , Pixel values of pixel data respectively corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels, and the respective pixels The difference value between the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the data is calculated, and the difference value of each pixel data is stored in the minimum exposure difference pixel data storage unit. It is possible to calculate a difference value with respect to the value of the minimum exposure difference pixel data for one line and generate difference pixel data including the difference value.

  Here, the present inventors have conducted experiments, and noises generated in the first to (N−1) th output channels and the Nth output channel are respectively detected in the images captured by the imaging apparatus having the configuration of the second aspect. It was confirmed that multiple vertical stripes were generated. That is, noise generated in the first to (N-1) th output channels and the Nth output channel is generated in a specific element constituting the line memory, and further vertical stripes are generated. Similar noise is generated for pixel data at the same pixel position in all lines. Accordingly, if the configuration is such that the minimum exposure difference pixel data for one line is stored as in the configuration of the present embodiment, it occurs in the first to (N-1) output channels and the Nth output channel for all lines. Noise can be reduced, and only the minimum exposure difference pixel data for one line needs to be stored. Therefore, in the first to (N-1) th output channels and the Nth output channel, The noise generated can be reduced, and the memory capacity necessary for storing the minimum exposure difference pixel data can be reduced.

[Embodiment 5] Furthermore, the imaging device of Embodiment 5 is the imaging device of Embodiment 3,
The minimum exposure difference pixel data for a plurality of lines smaller than the total number of lines of the pixel is stored in the minimum exposure difference pixel data storage means,
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. While calculating a difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data, the difference value of the pixel data and the minimum exposure difference A difference value with any value of the minimum exposure difference pixel data for a plurality of lines stored in the pixel data storage means is calculated, and difference pixel data including the difference value is generated. Yes.

  With such a configuration, the minimum exposure difference pixel data for a plurality of lines smaller than the total number of lines of the pixels can be stored in the minimum exposure difference pixel data storage unit, and the difference pixels The data generation means includes pixel values of pixel data respectively corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. Calculating a difference value between the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data and storing the difference value of the pixel data and the minimum exposure difference pixel data It is possible to calculate a difference value with any value of the minimum exposure difference pixel data for a plurality of lines stored in the means and generate difference pixel data composed of the difference value .

  Here, the inventors conducted an experiment, and the images captured by the imaging device having the configuration of the fourth aspect are the first to (N-1) output channels and the Nth output as in the configuration of the second aspect. Although noise generated in the channel can be reduced as compared with the case where no processing for reducing this noise is performed, the minimum exposure difference pixel data for all lines is used as in the configuration of the third embodiment. It was confirmed that it was not reduced more than when it was. Therefore, as in the configuration of the present embodiment, the minimum exposure difference pixel data for a plurality of lines is stored, and the minimum exposure difference pixel data for the plurality of lines is, for example, the same line as the minimum exposure difference pixel data and its vicinity. By using the minimum exposure difference pixel data for a plurality of lines, such as by applying to the line, noise can be reduced as compared with the configuration of the fourth aspect. Therefore, with the configuration of this embodiment, the memory capacity can be reduced as compared with storing the minimum exposure difference pixel data for all lines, and the noise can be reduced as compared with using the minimum exposure difference pixel data for one line. Is obtained.

[Mode 6] Furthermore, the imaging device of mode 6 is the imaging device of any one of modes 2 to 5,
Pixel data corresponding to the charges of the photoelectric conversion elements in the optical black region of the photoelectric conversion unit, which are output through the first to (N-1) th output channels, respectively, are acquired, and the Nth output Obtaining pixel data corresponding to the charges of the photoelectric conversion element in the optically black region output through the channel, and for each output channel of the first to (N-1) th output channels, A difference value between the pixel value of the pixel data of the optical black area acquired through each output channel and the pixel value of the pixel data of the optical black area acquired through the Nth output channel. OB difference pixel data generation means for generating OB difference pixel data consisting of values based on
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. The difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum pixel data acquisition unit corresponding to the pixel data is calculated, and the difference value of the pixel data is calculated as the difference value of the OB difference pixel data. It is characterized in that values are added and difference pixel data composed of the addition result is generated.

  If it is such a structure, the photoelectric conversion of the area | region used as the optical black of the said photoelectric conversion part each output via the said 1st-(N-1) output channel by the OB difference pixel data generation means Pixel data corresponding to the charge of the element is acquired, pixel data corresponding to the charge of the photoelectric conversion element in the optical black region, which is output through the Nth output channel, is acquired, For each output channel of the (N-1) th output channel, the pixel value of the pixel data of the optical black area acquired through each output channel and the optical acquired through the Nth output channel. OB difference pixel data having a value based on a difference value from the pixel value of the pixel data of a region that is a target black can be generated, and the difference pixel data generation unit can generate the difference pixel data for each line of the pixels. , Pixel values of pixel data respectively corresponding to (N-1) types of exposure times output through the first to (N-1) th output channels, and the minimum pixel corresponding to each pixel data. The difference value with the pixel value of the minimum exposure pixel data acquired by the data acquisition means is calculated, the value of the OB difference pixel data is added to the difference value of each pixel data, and the difference pixel data formed from the addition result is obtained. It is possible to generate.

  Here, the present inventors conducted an experiment, the pixel values of the pixel data respectively corresponding to the N-1 types of exposure times respectively output from the first to (N-1) output channels, and the respective pixels. When the difference value with the pixel value of the minimum exposure pixel data acquired by the minimum pixel data acquisition unit corresponding to the data was calculated, it was confirmed that the difference value might be a negative value. A negative pixel value is a value that cannot normally exist in image processing. Since such a negative value is fixed to “0” in advance, for example, this portion becomes a point. Appearing in the image, causing the image quality to deteriorate. Further, as one of the factors that become a negative value when the difference value is calculated, the pixel signal in the OB (optical black) area read through the first to (N-1) output channels and the Nth output channel is used. There is a difference in level. Therefore, as in the configuration of the present embodiment, each output of the pixel values of all the pixel signals in the OB area obtained through the first to (N-1) th output channels, for example, using the pixel signals in the OB area. The difference value between the average value for each channel and the average value of the pixel values of all the pixel signals in the OB area acquired via the Nth output channel, the OB area acquired via the first to (N-1) th output channels And the difference between the maximum value and the minimum value for each output channel of the pixel value of the pixel signal, and the difference between the maximum value and the minimum value of the pixel value of the pixel signal in the OB area acquired via the Nth output channel. OB difference pixel data is generated by calculating a value based on the difference value of the pixel value of the pixel signal in the OB area, such as calculating a difference, and this value is passed through the first to (N-1) output channels. N-1 types of exposure time output The difference pixel data is generated by adding the difference value between the pixel value of the corresponding pixel data and the pixel value of the minimum exposure pixel data acquired by the minimum pixel data acquisition unit corresponding to the pixel data, thereby generating OB There is an effect that the pixel value can be prevented from becoming negative due to the level difference of the pixel signals in the region.

Embodiments of an imaging apparatus according to the present invention will be described below with reference to the drawings. FIGS. 1-7 is a figure which shows embodiment of the imaging device 1 which concerns on this invention.
The imaging apparatus 1 includes an imaging device having an electronic shutter function capable of capturing an image with two types of exposure times (normal exposure time >> minimum exposure time) in one frame period (one exposure section). In addition, this image sensor sequentially outputs pixel signals that have been subjected to photoelectric conversion for each pixel column.

Hereinafter, a schematic configuration of the imaging apparatus 1 according to the present invention will be described with reference to FIG. Here, FIG. 1 is a block diagram showing a schematic configuration of the imaging apparatus 1 according to the present invention.
As shown in FIG. 1, the imaging device 1 reads an image sensor 10a (described later) composed of photoelectric conversion elements and the like and a pixel signal from the image sensor 10a by a nondestructive readout method, and obtains pixel data of the pixel signal. Based on the imaging processing unit 10 to be output and various pixel data output from the imaging processing unit 10, the OB difference pixel data and the minimum exposure pixel data are generated, and the difference pixel from which the noise component is removed based on the generated pixel data The image processing unit 12 generates data, and includes various pixel data output from the imaging processing unit 10, OB difference pixel data, minimum exposure pixel data, and a frame memory 14 that stores the difference pixel data.

The imaging apparatus 1 according to the present embodiment also performs various processes such as an OB difference pixel data generation process, a minimum exposure pixel data generation process, and a difference pixel data generation process based on various control signals from the host system 2 illustrated in FIG. Are controlled and executed.
Furthermore, the internal configuration of the imaging apparatus 1 will be described with reference to FIG. Here, FIG. 2 is a block diagram showing an internal configuration of the imaging apparatus 1 according to the present invention.

As shown in FIG. 2, the imaging processing unit 10 includes an image sensor 10a, an exposure time setting register 10b, a first AFE (Analog Front End) 10c, and a second AFE 10d.
The image sensor 10a condenses light from a subject on a sensor cell array (not shown) with an imaging lens (not shown), and accumulates charges corresponding to the amount of light collected in each pixel of the sensor cell array. Further, the image sensor 10a determines the charge group accumulated in each pixel column of the sensor cell array based on drive signals (pixel clock, horizontal synchronization signal, and vertical synchronization signal) output from the timing control unit 12f (described later). The voltage groups are converted into voltage groups sequentially, and the converted voltage groups are provided for each of the two different types of exposure times. The voltage group of one exposure time is a first horizontal transfer section (not shown). The first AFE 10c is sequentially output via a first output channel (hereinafter referred to as CH1) configured to include a line memory, and the other exposure time voltage group is supplied to a second horizontal transfer unit (not shown). The second AFE 10d is sequentially output via a second output channel (hereinafter referred to as CH2) configured to include the second line memory. Further, in the present embodiment, by reading out charges from each pixel by the non-destructive readout method, the minimum exposure time (each exposure time) of each pixel column in one exposure section (normal exposure time) by the electronic shutter function is achieved. A charge group at the time of exposure with a minimum settable time and a charge group at the time of exposure with a normal exposure time (which can be arbitrarily set by a user or the like) are read out.

Hereinafter, a method for controlling the exposure time of the image sensor 10a in the imaging processing unit 10 and a method for reading a pixel signal from the image sensor 10a will be described with reference to FIG. Here, FIG. 3 is a diagram illustrating an example of the exposure for each line of each pixel and the nondestructive readout operation of the pixel signal in the imaging processing unit 10.
Here, the control of the exposure time according to the present invention sets a clear line for emptying the stored charge for the sensor cell array, and the normal exposure written in the exposure time setting register 10b with this clear line as a reference. A first readout line for performing non-destructive readout of the pixel signal of time is set, and a second readout line for performing non-destructive readout of the pixel signal at the minimum exposure time written in advance in the exposure time setting register 10b is set. In a single exposure section (normal exposure time), nondestructive readout of the pixel signal at the normal exposure time and the pixel signal at the minimum exposure time is performed. That is, as shown in FIG. 3, when the clear line is set to the pixel line (for example, the first line) at a predetermined position, the first read line and the second read line are set to the minimum exposure time from the clear line. The first readout line is set so that the second readout line reads out the pixel signal of the first line after a time of minutes, and the pixel signal of the first line is read out from the clear line after a period of time corresponding to the normal exposure time. Is set. In the present embodiment, the pixel signal at the normal exposure time is read to the first line memory of CH1 and output to the first AFE 10c, where it is converted into digital data (hereinafter referred to as pixel data). . On the other hand, the pixel signal at the minimum exposure time is read to the second line memory of CH2 and output to the second AFE 10d, where it is converted into digital data (pixel data).

  In addition, as shown in FIG. 3, the control of the readout timing of the pixel signals of the first readout line and the second readout line is performed sequentially for the clear line, the second readout line, and the first readout line for each pixel line. After scanning (upward in FIG. 3) and clearing the charge in the clear line, non-destructive readout of the pixel signal at the minimum exposure time is performed in the second readout line for the pixel line after clearing, and the clear line Non-destructive readout of the pixel signal at the normal exposure time is performed on the first readout line with respect to the pixel line for which the normal exposure time has elapsed after clearing the charge at. In such a procedure, for all the pixel lines of the sensor cell array, for each pixel line, the pixel signal at the minimum exposure is sequentially cleared and the pixel signal at the minimum exposure is non-destructively read out. The pixel signals at the time of normal exposure are sequentially read out for the pixel lines on which non-destructive reading has been performed. Accordingly, a time difference of at least “normal exposure time−minimum exposure time” occurs in the pixel data output from CH1 and CH2.

  Returning to FIG. 2, the exposure time setting register 10b is a register for designating an exposure time (charge accumulation time) in the electronic shutter function, and is written in the exposure time setting register 10b in the present embodiment. Based on the exposure time, the minimum exposure time and the normal exposure time are set for the readout lines corresponding to the channels CH1 and CH2, respectively. The exposure with the normal exposure time is the one-time exposure section.

  The first AFE 10c and the second AFE 10d digitally output voltages (analog data) corresponding to two different types of exposure times output via CH1 of the first horizontal transfer unit and CH2 of the second horizontal transfer unit, respectively. Convert to data (pixel data). Then, the first AFE 10c and the second AFE 10d output the generated pixel data to a first buffer 12c (described later) and a second buffer 12d (described later) of the image processing unit 12, respectively.

The image processing unit 12 includes a communication unit 12a, a processing mode setting register and status register 12b, a first buffer 12c, a second buffer 12d, a difference processing unit 12e, and a frame memory access unit 12f. Composed.
The communicator 12a has a function of receiving various control signals from the host system 2, a function of transmitting a drive control signal to the image sensor 10a, a function of transmitting an exposure time setting signal to the exposure time setting register 10b, It has a function of transmitting a mode setting signal to the processing mode setting register and status register 12b. Here, the processing mode indicates a combination of operation states of the first buffer 12c, the second buffer 12d, and the difference processing unit 12e. Further, the host system 2 has a function of transmitting a status signal indicating each processing state of the first buffer 12c, the second buffer 12d, and the difference processing unit 12e.

  The processing mode setting register and status register 12b are registers for designating a processing mode to be executed by the first buffer 12c, the second buffer 12d, and the difference processing unit 12e, and the first buffer 12c and the second buffer 12d. A register for writing a status indicating each processing state of the difference processing unit 12e. The communication device 12a writes the mode information based on the mode setting signal sent from the host system 2 to the processing mode setting register and status register 12b to specify the processing mode, and sets the processing mode setting register and status. The status information written in the register for register 12b is transmitted to the host system 2 as a status signal.

  Here, in the present embodiment, the processing mode includes an OB difference pixel data generation mode for generating OB difference pixel data from pixel data in an OB (optical black) area of the image sensor 10a, CH1 at the minimum exposure time, and CH1. Minimum exposure difference pixel data generation mode for generating minimum exposure difference pixel data composed of difference values of pixel data from CH2, and difference pixel data generation mode for generating difference pixel data obtained by removing various noise components from pixel data during normal exposure There are three modes.

The first buffer 12c and the second buffer 12d temporarily store the pixel data output from the first AFE 10c and the second AFE 10d, respectively, and the stored pixel data and the frame memory 14 that stores the pixel data. It has a function of outputting an address to the frame memory access unit 12g.
The difference processing unit 12e executes generation processing of any one of the OB difference pixel data, the minimum exposure difference pixel data, and the difference pixel data according to the processing mode written in the processing mode setting register and the status register 12b. .

Specifically, in the OB difference pixel data generation mode, the pixel data of the OB area of the image sensor 10a is acquired from the imaging processing unit 10 through each of CH1 and CH2, and OB based on the pixel data of this OB area. Difference pixel data is generated.
Also, in the minimum exposure difference pixel data generation mode, pixel data at the minimum exposure time is acquired via CH1 and CH2, and for each pixel, the pixel value of the pixel data acquired via CH1 is calculated as CH2. The difference value obtained by subtracting the pixel value of the pixel data corresponding to the same pixel as CH1 acquired via is calculated, and the minimum exposure difference pixel data including the difference value is generated.

  In addition, in the differential pixel data generation mode, pixel data at the normal exposure time arbitrarily set by the user or the like in one exposure section is acquired for each line via CH1, and a preset minimum Pixel data at the time of exposure is acquired for each line via CH2, and for each pixel, a pixel corresponding to the same pixel as CH1 acquired via CH2 from the pixel value of the pixel data acquired via CH1 A difference value obtained by subtracting the pixel value of the data is calculated, and the value of the OB difference pixel data generated in the OB difference pixel data generation mode is added to the difference value, and the minimum exposure difference pixel data is calculated from the value after the addition. Difference pixel data is generated by subtracting the value of the minimum exposure difference pixel data generated in the generation mode.

  The timing control unit 12f generates a drive signal (pixel clock, horizontal synchronization signal, vertical synchronization signal), outputs it to the image sensor 10a, and outputs the synchronization signal to the first buffer 12c and the second buffer 12d. The timing control unit 12f knows the pixel position (pixel column (line) number, pixel number) of the pixel signal output from the imaging processing unit 10 in the imaging device from the horizontal synchronization signal and the vertical synchronization signal. A pixel column (line) number (hereinafter also referred to as “address information”) is generated, and the address information is output to the first buffer 12c and the second buffer 12d.

  The frame memory access unit 12g has a function of managing data transmission / reception between the first buffer 12c, the second buffer 12d, the difference processing unit 12e, and the frame memory 14. That is, when a read command is output from any of the first buffer 12c, the second buffer 12d, and the difference processing unit 12e, the frame memory access unit 12g outputs pixel data (various difference pixel data) indicated by the output read command. And the pixel data is output to the output source of the read command. When a write command is output from any of the first buffer 12c, the second buffer 12d, and the difference processing unit 12e, pixel data (including various difference pixel data) indicated by the output write command is stored in the frame memory 14. Write to.

  As shown in FIG. 2, the frame memory 14 includes pixel data at the normal exposure time acquired via CH1 (for all lines of the imaging region), and pixel data at the normal exposure time acquired via CH2 (imaging region). ), First difference pixel data (for all lines of the imaging region), OB difference pixel data, minimum exposure difference pixel data (for all lines of the imaging region), second difference pixel data (all lines of the imaging region) If a read request is received from the frame memory access unit 12g, the pixel data indicated by the request is read out. Further, when there is a write request from the frame memory access unit 12g, the frame memory 14 causes the pixel data indicated by the write request to be written.

Furthermore, the flow of OB difference pixel data generation processing will be described with reference to FIG. Here, FIG. 4 is a flowchart showing the generation process of the OB difference pixel data.
In the OB difference pixel data generation process, the data acquisition mode information of the OB difference pixel data generation mode is set (written) in the processing mode setting register and status register 12b based on the mode setting signal from the host system 2. ). Here, the communicator 12a transmits a drive signal from the host system 2 to the image sensor 10a in addition to the mode setting signal from the host system 2, and the exposure time setting signal from the host system 2 is an exposure time setting register. To 10b. This exposure time setting signal is used to set the exposure time for the first readout line of CH1 in the OB difference pixel data generation mode. Here, for example, the exposure time setting signal is set to the minimum exposure time. Note that the exposure time information for CH1 may be stored in advance in the exposure time setting register 10b in the same manner as the minimum exposure time for CH2.

  When various settings are made as described above, as shown in FIG. 4, first, the process proceeds to step S <b> 100, and in the first buffer 12 c and the second buffer 12 d, the image sensor is connected via CH <b> 1 and CH <b> 2 of the imaging processing unit 10. The pixel data of the OB area 10a is acquired, the acquired pixel data is stored in the frame memory 14 via the frame memory access unit 12g, and the process proceeds to step S102.

In step S102, the first buffer 12c and the second buffer 12d determine whether or not the process for obtaining the pixel data of the OB region has ended. If it is determined that the processing has ended (Yes), the process proceeds to step S104. If not (No), the process proceeds to step S100 and the acquisition process is continued.
When the processing proceeds to step S104, the first buffer 12c and the second buffer 12d send a status signal indicating the end of the data acquisition process to the host system 2 via the processing mode setting register and status register 12b and the communication device 12a. And the process proceeds to step S106.

In step S106, the difference processing unit 12e determines whether or not the data generation mode information of the OB difference pixel data generation mode is set in the processing mode setting register and the status register 12b (whether or not the data is written). If it is determined that it has been set (Yes), the process proceeds to step S108. If not (No), the process waits until it is set.
When the process proceeds to step S108, the difference processing unit 12e reads out the pixel data of the OB area for each channel acquired in step S100 and stored in the frame memory 14 via the frame memory access unit 12g, and reads the read out data. The OB difference pixel data is generated based on the pixel data of the OB area, the generated OB difference pixel data is stored in the frame memory 14 via the frame memory access unit 12g, and the process proceeds to step S110.

In step S110, the difference processing unit 12e determines whether or not the generation processing of the OB difference pixel data has ended. If it is determined that the processing has ended (Yes), the process proceeds to step S112, and otherwise (No ) Goes to step S108 and continues the generation process of the OB difference pixel data.
When the process proceeds to step S112, the difference processing unit 12e sends a status signal indicating the end of the OB difference pixel data generation process to the host system 2 via the processing mode setting register, the status register 12b, and the communication device 12a. Send and finish the process. Here, when the host system 2 receives the status signal indicating the end of the generation processing of the OB difference pixel data, it captures the mode setting signal indicating the data acquisition mode in the minimum exposure difference pixel data generation mode via the communication device 12a. It transmits to the processing unit 12. The imaging processing unit 12 sets (writes) the data acquisition mode information in the processing mode setting register, the status register 12b, and the communication device 12a based on the mode signal indicating the data acquisition mode received by the communication device 12a, and the minimum exposure. The mode shifts to the data acquisition mode of the difference pixel data generation mode. In the present embodiment, the OB difference pixel data generation process is a process that is performed only once as a batch process after the power is turned on.

Furthermore, the flow of the minimum exposure difference pixel data generation process will be described based on FIG. Here, FIG. 5 is a flowchart showing the minimum exposure difference pixel data generation processing.
In the generation processing of minimum exposure difference pixel data, data acquisition mode information of the minimum exposure difference pixel data generation mode is set in the processing mode setting register and status register 12b based on the mode setting signal from the host system 2 ( This is a process started by writing. Here, the communication device 12a transmits a drive signal from the host system 2 to the image sensor 10a in addition to the mode setting signal from the host system 2, and the exposure time setting signal from the host system 2 as an exposure time setting register. To 10b. This exposure time setting signal is used to set the exposure time for the first readout line of CH1 in the minimum exposure difference pixel data generation mode, and is set to the minimum exposure time here. Note that the exposure time information for CH1 may be stored in advance in the exposure time setting register 10b in the same manner as the minimum exposure time for CH2.

  When various settings are made as described above, as shown in FIG. 5, first, the process proceeds to step S <b> 200, and the image is transmitted via the CH <b> 1 and CH <b> 2 of the imaging processor 10 in the first buffer 12 c and the second buffer 12 d. Pixel data at the time of exposure with the minimum exposure time of the sensor 10a is sequentially acquired, and the acquired pixel data is stored in the frame memory 14 via the frame memory access unit 12g, and the process proceeds to step S202.

  In step S202, the pixel data acquisition process at the time of exposure is completed in the first buffer 12c and the second buffer 12d with a minimum exposure time of one frame (for all lines of pixels in the imaging region) via H1 and CH2. If it is determined that the process has been completed (Yes), the process proceeds to step S204. If not (No), the process proceeds to step S200 and the acquisition process is continued.

When the process proceeds to step S204, the first buffer 12c and the second buffer 12d send a status signal indicating the end of the data acquisition process to the host system 2 via the processing mode setting register and status register 12b and the communication device 12a. And the process proceeds to step S206.
In step S206, the difference processing unit 12e determines whether the data generation mode information of the minimum exposure difference pixel data generation mode is set (whether written) in the processing mode setting register and the status register 12b. If it is determined that it is set (Yes), the process proceeds to step S208. If not (No), the process waits until it is set.

  When the process proceeds to step S208, the difference processing unit 12e reads out the pixel data for one frame for each channel acquired in step S200 and stored in the frame memory 14 via the frame memory access unit 12g, and performs the reading. The minimum exposure difference pixel data is generated based on the pixel data, and the generated minimum exposure difference pixel data is stored in the frame memory 14 via the frame memory access unit 12g, and the process proceeds to step S210.

In step S210, the difference processing unit 12e determines whether or not the minimum exposure difference pixel data generation process has ended. If it is determined that the difference exposure unit 12e has ended (Yes), the process proceeds to step S212; In No), the process proceeds to step S208, and the generation process of the minimum exposure difference pixel data is continued.
When the process proceeds to step S212, the difference processing unit 12e sends a status signal indicating the end of the generation process of the minimum exposure difference pixel data to the host system 2 via the processing mode setting register, the status register 12b, and the communication device 12a. To finish the process. Here, when the host system 2 receives the status signal indicating the end of the generation processing of the minimum exposure difference pixel data, the host system 2 transmits a mode setting signal indicating the normal shooting mode to the imaging processing unit 12 via the communication device 12a. The imaging processing unit 12 sets (writes) the normal shooting mode information in the processing mode setting register, the status register 12b, and the communication device 12a based on the mode setting signal indicating the data acquisition mode received by the communication device 12a. Switch to shooting mode. In the present embodiment, the minimum exposure difference pixel data generation process is a process that is performed only once as a batch process after the power is turned on.

Further, the flow of the difference pixel data generation process will be described based on FIG. Here, FIG. 6 is a flowchart showing the difference pixel data generation processing.
The difference pixel data generation processing is processing that is started when normal shooting mode information is set (written) in the processing mode setting register and status register 12b based on the mode setting signal from the host system 2. . Here, the communication device 12a transmits a drive signal from the host system 2 to the image sensor 10a in addition to the mode setting signal from the host system 2, and the exposure time setting signal from the host system 2 as an exposure time setting register. To 10b. This exposure time setting signal is used to set the exposure time for the first readout line of CH1, and is set to, for example, the minimum exposure time here.
Here, in addition to the setting of mode setting information (signal) from the host system 2, the communication device 12 a transmits a drive signal from the host system 2 to the image sensor 10 a and receives an exposure time setting signal from the host system 2. It transmits to the exposure time setting register 10b. This exposure time setting signal is for setting the exposure time for the first readout line of CH1, and here, an arbitrary exposure time (normal exposure time) designated by the user or the like is set as the exposure time. .

When various settings are made as described above, as shown in FIG. 6, first, the process proceeds to step S300, and the image sensor 10a performs the relevant operation based on the exposure time for CH1 set in the exposure time setting register 10b. The normal exposure time is set in the CH1 readout line, and the process proceeds to step S302.
In step S302, in the image sensor 10a, based on the exposure time for CH2 set in the exposure time setting register 10b, the minimum exposure time is set in the readout line for CH2, and the process proceeds to step S304.

  In step S304, in the first buffer 12c and the second buffer 12d, the pixel data for one frame of the image sensor 10a at the time of exposure with the minimum exposure time is sequentially supplied for each pixel line via CH2 of the imaging processing unit 10. The pixel data for one frame of the image sensor 10a at the time of exposure with the normal exposure time is sequentially acquired for each pixel line via CH1, and the process proceeds to step S306.

In step S306, in the first buffer 12c and the second buffer 12d, the pixel data of the normal exposure time and the minimum exposure time acquired via CH1 and CH2 in step S304 are stored in the frame memory 14 via the frame memory access unit 12g. Then, the process proceeds to step S308.
In step S308, the difference processing unit 12e obtains pixel data during normal exposure (hereinafter referred to as normal exposure pixel data) acquired through CH1, and pixel data during minimum exposure (hereinafter referred to as minimum) acquired through CH2. Exposure pixel data) is read from the frame memory 14 via the frame memory access unit 12g for each pixel line, and the process proceeds to step S310.

  In step S310, for each pixel data of the line unit pixel data read out in step S308, the difference processing unit 12e calculates the minimum exposure of CH2 at the same pixel position as the pixel data from the pixel value of the normal exposure pixel data of CH1. A difference value obtained by subtracting the pixel value of the pixel data is calculated, first difference pixel data including the difference value is generated, and the process proceeds to step S312.

In step S312, the difference processing unit 12e stores the first difference pixel data generated in step S310 in the frame memory 14 via the frame memory access unit 12g, and proceeds to step S314.
In step S314, the difference processing unit 12e obtains the first difference pixel data for each pixel line, the OB difference pixel data, and the minimum exposure difference pixel data for each pixel line from the frame memory 14 via the frame memory access unit 12g. Read out and proceed to step S316.

  In step S316, in the difference processing unit 12e, for each pixel data of the first difference pixel data in units of lines read in step S314, the minimum pixel position of the same pixel position as the pixel data is determined from the pixel value of the first difference pixel data. The difference value obtained by subtracting the pixel value of the exposure difference pixel data is calculated, the second difference pixel data including the addition result obtained by adding the pixel value of the OB difference pixel data to the difference value is generated, and the process proceeds to step S318.

In step S318, the difference processing unit 12e stores the second difference pixel data generated in step S316 in the frame memory 14 via the frame memory access unit 12g, and proceeds to step S320.
In step S320, the frame memory access unit 12g outputs the second difference pixel data stored in the frame memory 14 in step S318 to the host system 2 and proceeds to step S322.

In step S322, the difference processing unit 12e completes the first difference pixel data generation process, the second difference pixel data generation process, and the second difference pixel data output process for all lines of pixels for one frame. If it is determined that the process has been completed (Yes), the process proceeds to step S300. If not (No), the process proceeds to step S304.
Here, FIG. 7 is a diagram illustrating a flow of pipeline processing of the first difference pixel data generation processing, the second difference pixel data generation processing, and the second difference pixel data output processing in the normal shooting mode. The first difference pixel data generation process (process 1 in FIG. 7) corresponds to the processes in steps S304 to S310 in the flowchart of FIG. 6, and the second difference pixel data generation process (process 2 in FIG. 7). ) Corresponds to the processing of steps S312 to S318 in the flowchart of FIG. 6, and the output processing of the second difference pixel data (processing 3 in FIG. 7) corresponds to the processing of step S322.

In the present embodiment, as shown in FIG. 7, the processing 1, processing 2 and processing 3 are pipeline processed.
Specifically, when the process 1 is started, first, the pixel data of the first line in the imaging region of the image sensor 10a at the time of normal exposure acquired through CH1, and the minimum exposure acquired through CH2 as well. First difference pixel data for the first line is generated from the pixel data of the first line at that time. When the first difference pixel data for the first line is generated, the pixel data of the second line at the time of normal exposure acquired through CH1, and the pixel of the second line at the time of minimum exposure acquired through CH2. First difference pixel data for the second line is generated from the data. That is, the process 1 is sequentially performed for each line on the pixel data corresponding to each of the first to nth lines (n is the maximum number of lines in the imaging region) in the imaging region of the image sensor 10a.

  On the other hand, when the first difference pixel data for the first line is generated in the process 1 and the generated first difference pixel data is stored in the frame memory 14, the process 2 is started and stored in the frame memory 14. Second difference pixel data for the first line is generated from the first difference pixel data for the first line, the OB difference pixel data, and the minimum exposure difference pixel data corresponding to the first line. When the second difference pixel data for the first line is generated and the first difference pixel data for the second line is stored in the frame memory 14, the first difference pixel data for the second line and the OB difference pixel data are then stored. Then, second difference pixel data for the second line is generated from the minimum exposure difference pixel data corresponding to the second line. That is, following the process 1, the process 2 is sequentially performed for each line on the first difference pixel data of the first line to the nth line.

  Further, in the second process, when the second difference pixel data for the first line is generated and the generated first difference pixel data is stored in the frame memory 14, the process 3 is started and stored in the frame memory 14. The second difference pixel data for the first line is output to the host system 2. When the second difference pixel data for the first line is output and the second difference pixel data for the second line is stored in the frame memory 14, the second difference pixel data for the second line is then output to the host system 2. Is done. That is, following the process 2, the process 3 is sequentially performed for each line on the second difference pixel data of the first line to the nth line.

As described above, the processes 1 to 3 are executed by the pipeline processing method.
Next, the actual operation of the imaging apparatus 1 of the present embodiment will be described.
When the communication device 12a receives the mode setting signal of the data acquisition mode in the generation processing mode of the OB difference pixel data from the host system 2 when the power is turned on, the imaging device 1 performs processing based on the received mode setting signal. The data acquisition mode information is set (written) in the mode setting register and status register 12b.

  When the information on the data acquisition mode is set in the processing mode setting register and the status register 12b, the imaging processing unit 10 sets the minimum exposure time for the first and second readout lines corresponding to CH1 and CH2, respectively. Set. On the other hand, the first buffer 12c and the second buffer 12d acquire pixel data at the minimum exposure time of the OB area of the image sensor 10a via CH1 and CH2, and store the acquired pixel data in the frame memory 14. (Step S100) When the first buffer 12c and the second buffer 12d acquire all the pixel data of the OB area and store them in the frame memory 14 ("Yes" branch in Step S102), a status signal indicating the end of the acquisition process Is transmitted to the host system 2 via the processing mode setting register and status register 12b and the communication device 12a (step 104).

  Then, the communication device 12a receives a data generation mode setting signal in the generation processing mode of the OB difference pixel data from the host system 2, and information on the data generation mode is stored in the processing mode setting register and the status register 12b. When set (written) (the branch of “Yes” in step S106), the difference processing unit 12e executes generation processing of OB difference pixel data (step S108).

  In the present embodiment, first, the average value of the pixel values of the pixel data of the OB area acquired via CH1 is calculated, and then the average value of the pixel values of the pixel data of the OB area acquired via CH2 is calculated. calculate. Then, a difference value obtained by subtracting the average value of CH2 from the average value of CH1 is calculated, and pixel data including the difference value is stored in the frame memory 14 as OB difference pixel data. When the OB difference pixel data is generated and stored in the frame memory 14 in this way (the branch of “Yes” in step S110), the difference processing unit 12e uses a status signal indicating the end of the generation process for setting the processing mode. The data is transmitted to the host system 2 via the register and status register 12b and the communication device 12a (step 112).

On the other hand, when the host system 2 receives the status signal indicating the end of the generation processing of the OB difference pixel data from the difference processing unit 12e via the communication device 12a, the host system 2 generates the minimum exposure difference pixel data via the communication device 12a. Data acquisition mode information in the mode is set in the processing mode setting register and status register 12b.
When the data acquisition mode information in the generation mode of the minimum exposure difference pixel data is set in the processing mode setting register and the status register 12b, the imaging processing unit 10 sets the first and second readout lines corresponding to CH1 and CH2. On the other hand, a minimum exposure time is set for each. On the other hand, the first buffer 12c and the second buffer 12d acquire pixel data at the minimum exposure time of each pixel of the image sensor 10a via CH1 and CH2, and store the acquired pixel data in the frame memory 14. (Step S200), the first buffer 12c and the second buffer 12d acquire all the pixel data at the time of the minimum exposure time for one frame and store it in the frame memory 14 (the branch of “Yes” in Step S202). A status signal indicating the end of processing is transmitted to the host system 2 via the processing mode setting register and status register 12b and the communication device 12a (step 204).

  Then, the communication device 12a receives a data generation mode mode setting signal from the host system 2 in the minimum exposure difference pixel data generation processing mode, and the processing mode setting register and status register 12b store the data generation mode information. Is set (written) ("Yes" branch of step S206), the difference processing unit 12e executes a process of generating minimum exposure difference pixel data (step S208).

  In the present embodiment, with respect to the pixel data of each line, for each pixel, pixel data of the same pixel position as the pixel data acquired via CH2 from the pixel value of the pixel data acquired via CH1. A difference value obtained by subtracting the pixel values of the pixel values is calculated, and pixel data including the difference values is stored in the frame memory 14 as minimum exposure difference pixel data. In this way, by calculating the difference value of the pixel data separately acquired via CH1 and CH2 with the minimum exposure time (substantially light-shielded state), the noise component due to the variation in the characteristics of the line memories of CH1 and CH2 can be obtained. Can be obtained.

  As described above, when the minimum exposure difference pixel data is generated for the pixel data for one frame and stored in the frame memory 14 (the branch of “Yes” in Step S210), the difference processing unit 12e generates A status signal indicating the end of processing is transmitted to the host system 2 via the processing mode setting register and status register 12b and the communication device 12a (step 212).

On the other hand, when the host system 2 receives the status signal indicating the end of the generation processing of the minimum exposure difference pixel data from the difference processing unit 12e via the communication device 12a, the host system 2 processes the normal shooting mode information via the communication device 12a. The mode setting register and status register 12b are set.
When the normal shooting mode information is set in the processing mode setting register and the status register 12b, the imaging processing unit 10 exposes the first readout line corresponding to CH1 by the exposure time setting signal from the host system 2. The normal exposure time set in the time setting register 10b is set (step S300), and the minimum exposure time set in advance in the exposure time setting register 10b is set for the second readout line corresponding to CH2. (Step S302).

  When the exposure time is set for each of the first and second readout lines corresponding to CH1 and CH2, the first buffer 12c stores the pixel data at the normal exposure time of each pixel of the image sensor 10a via CH1. The second buffer 12d sequentially acquires the pixel data at the time of the minimum exposure time of each pixel of the image sensor 10a for each line through CH2 (Step S304), and sequentially acquires each line. Pixel data is stored in the frame memory 14 (step S306). As described above, for the line of each pixel, pixel signals at the minimum exposure time and normal exposure time are read non-destructively through CH1 and CH2, respectively, in the exposure section of the normal exposure time by the electronic shutter function. .

  When the difference processing unit 12e confirms that the pixel data of the first line at the normal exposure time of the image sensor 10a is acquired and stored in the frame memory 14 via the CH1, the first difference pixel data is generated. The processing (the processing 1 above) is started, and the pixel data at the normal exposure time of the first line acquired via the CH1 from the frame memory 14 and the pixels at the minimum exposure time of the first line acquired via the CH2 (Step S308), and for each pixel, a difference value obtained by subtracting the pixel value of the pixel data at the minimum exposure time at the same pixel position as the pixel data from the pixel value of the pixel data at the normal exposure time. Calculate and generate pixel data composed of the difference value (step S310), and store this in the frame memory 14 as first difference pixel data ( Step S312). Thus, by calculating the difference value between the pixel data acquired at the exposure of the minimum exposure time (substantially light-shielded state) through CH2 and the pixel data acquired at the exposure of the normal exposure time through CH1, the image is obtained. It is possible to obtain a pixel signal from which a noise component due to variation in characteristics of elements constituting each pixel of the sensor 10a is removed.

The generation process and the storage process of the first difference pixel data as described above are sequentially performed for each line from the first line to the n-th line, which is a pixel line for one frame of the captured image.
Further, when the difference processing unit 12e confirms that the first difference pixel data for the first line has been generated and stored in the frame memory 14, the second difference pixel data generation process (the process 2) is started. First difference pixel data corresponding to the first line, OB difference pixel data, and minimum exposure difference pixel data corresponding to the first line are read from the frame memory 14 (step S314), and the first difference pixel data of the first line is read. For each pixel, a difference value obtained by subtracting the pixel value of the minimum exposure difference pixel data at the same pixel position as the pixel data from the pixel value of the first difference pixel data is calculated, and the pixel of the OB difference pixel data is added to the difference value. The values are added to generate second difference pixel data (step S316). Then, the generated second difference pixel data is stored in the frame memory 14 (step S318). Thus, by subtracting the pixel value of the minimum exposure difference pixel data from the first difference pixel data from which the noise component due to the variation in the characteristics of the elements constituting each pixel of the image sensor 10a is removed, the values of CH1 and CH2 are subtracted. It is possible to further remove noise components due to variations in line memory characteristics. Further, by adding the value of the OB difference pixel data to the pixel value of the pixel data from which the noise component due to the variation in the characteristics of the CH1 and CH2 line memories is removed, in the CH1 and CH2 line memories, respectively. When a level difference occurs in the pixel data of the read OB area, it is possible to prevent the pixel value of each pixel data from which the noise component has been removed from becoming a negative value.

The generation process and the storage process of the second difference pixel data as described above are sequentially performed for each line from the first line to the n-th line, which is a pixel line for one frame of the captured image.
Further, when it is confirmed in the frame memory access unit 12g that the second differential pixel data for the first line has been generated and stored in the frame memory 14, the pixel clock, horizontal synchronization signal, and vertical synchronization signal from the host system 2 are confirmed. The second difference pixel data for the first line is output to the host system 2 in synchronization with (step S320).

  The output process of the second difference pixel data as described above is sequentially performed for each line from the first line to the n-th line, and when the process is completed up to the n-th line (“Yes” branch of step S322), the process is again performed. The process shifts to the CH1 and CH2 exposure time setting process (may be through if there is no change), and the processes 1 to 3 in the normal photographing mode are repeated by pipeline processing.

  Note that noise generated in the image sensors 10a and the line memories of CH1 and CH2 changes due to environmental changes (particularly temperature changes). When the measurement value is measured and there is a large change in the measured value, the OB difference pixel data and the minimum exposure difference pixel data generated by the batch processing before the transition to the normal photographing mode are regenerated, It is possible to respond to changes in

  As described above, in the imaging device 1 according to the present embodiment, noise components due to variations in characteristics of elements constituting each pixel of the image sensor 10a are removed from the pixel data read by the nondestructive readout method, and CH1 and Since the noise component due to the variation in the characteristics of the CH2 line memory is removed, it is possible to improve the image quality of the captured image generated using the pixel signal read by the non-destructive readout method.

  Further, the value of the OB difference pixel data is set for the pixel data from which the noise component due to the characteristic variation of the elements constituting each pixel of the image sensor 10a and the noise component due to the characteristic variation of the CH1 and CH2 line memories are removed. As a result, in the CH1 and CH2 line memories, when the level difference occurs in the pixel data of the read OB area, the pixel value of each pixel data after removing the noise component is negative. Since it can be prevented from becoming a value, it is possible to prevent image quality deterioration due to a level difference between channels of pixel data in the OB area.

  In the above-described embodiment, the imaging processing unit 10 corresponds to the imaging unit of mode 1, and pixel data acquisition processing at the minimum exposure time from the imaging processing unit 10 in the image processing unit 12 is that of mode 1 to mode 5. The minimum exposure difference pixel data generation process in the image processing unit 12 corresponding to any one of the minimum exposure pixel data acquisition means corresponds to the minimum exposure difference pixel data generation means in the form 3, and the OB difference in the image processing unit 12 The pixel data generation process corresponds to the OB difference pixel data generation unit of the sixth aspect, and the generation process of the first difference pixel data and the second difference pixel data in the image processing unit 12 is any one of the first to sixth aspects. Corresponding to the difference pixel data generation means, the frame memory 14 corresponds to the minimum exposure difference pixel data storage means of any one of forms 3 to 5.

  In the above-described embodiment, the first output channel for reading the pixel signal at the time of normal exposure and the second output channel for reading the pixel signal at the time of the minimum exposure have a total of two output channels. Not limited to this, depending on the type of exposure time to be set, for example, (N-1) types (N-1) of the first to (N-1) output channels respectively corresponding to the exposure time (N is a natural number of 2 or more). A configuration including a plurality of channels and an Nth channel corresponding to the minimum exposure time may be employed. Accordingly, it is possible to cope with a case where HDR data is generated by multi-stage exposure in which exposure is performed with a plurality of types of exposure times in one frame period.

  In the above embodiment, the minimum exposure difference pixel data is generated for all the pixels in the imaging region of the image sensor 10a and stored in the frame memory 14. However, the present invention is not limited to this. Since the noise generated in each output channel has a characteristic of generating vertical stripes in the image, the minimum exposure difference pixel data for a predetermined one line of the sensor cell of the image sensor 10a is generated, and this is used as the first difference pixel data. It is good also as composition which generates the 2nd difference pixel data repeatedly using for a line, and generates the minimum exposure difference pixel data for predetermined plural lines (however, less than the maximum number of lines) of the sensor cell of image sensor 10a, It is good also as a structure which allocates these to each line of 1st difference pixel data, and produces | generates 2nd difference pixel data.When generating the minimum exposure difference pixel data for a plurality of lines, for example, the pixel line of the imaging region of the sensor cell is equally divided into three regions of upper, middle, and lower, and one line from each region A line is selected, and minimum exposure difference pixel data is generated only for the selected pixel line. Then, the second difference pixel data is generated by using the minimum exposure difference pixel data of each line selected from the upper, middle, and lower three regions for each first difference pixel data included in the same region. Thus, it is necessary to store the minimum exposure difference pixel data by generating only one line or a plurality of lines without generating the minimum exposure difference pixel data for all the pixels in the imaging region of the image sensor 10a. Therefore, the capacity of the memory can be reduced, and the cost of the imaging device 1 can be reduced.

  In the above-described embodiment, the example in which the OB difference pixel data is the difference value of the average values of the pixel data of the OB area acquired via CH1 and CH2 is described. , Data consisting of a difference value obtained by subtracting the difference value between the maximum value and the minimum value of the pixel data in the OB region corresponding to CH2 from the difference value between the maximum value and the minimum value of the pixel data in the OB region corresponding to CH1. Other values such as OB difference pixel data may be used.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus 1 according to the present invention. It is a block diagram which shows the internal structure of the imaging device 1 which concerns on this invention. 3 is a diagram illustrating an example of exposure for each line of each pixel and non-destructive readout operation of a pixel signal in the imaging processing unit 10. FIG. It is a flowchart which shows the production | generation process of OB difference pixel data. It is a flowchart which shows the minimum exposure difference pixel data generation process. It is a flowchart which shows a difference pixel data generation process. It is a figure which shows the flow of the pipeline process of the production | generation process of 1st difference pixel data, the production | generation process of 2nd difference pixel data, and the output process of 2nd difference pixel data in normal imaging | photography mode.

Explanation of symbols

Reference numeral 1 denotes an imaging apparatus, 2 denotes a host system, 10 denotes an imaging processing unit, 12 denotes an image processing unit, 14 denotes a frame memory, 10a denotes an image sensor, 10b denotes an exposure time setting register, 10c denotes a first AFE, and 10d denotes a first 2 AFE, 12a is a communication device, 12b is a processing mode setting and status register, 12c is a first buffer, 12d is a second buffer, 12e is a differential processing unit, 12f is a timing control unit, and 12g is a frame memory access unit.

Claims (6)

A photoelectric conversion unit in which a plurality of photoelectric conversion elements for converting exposed light into electric charges and storing them are arranged in a matrix, and non-destructive reading of charges from pixels constituting the photoelectric conversion elements in the photoelectric conversion unit An imaging device comprising imaging means for performing,
Minimum exposure pixel data acquisition means for acquiring minimum exposure pixel data, which is pixel data corresponding to charges read at the time of exposure with a settable minimum exposure time;
The difference value between the pixel value of each pixel data corresponding to the electric charge read at the time of exposure with a predetermined exposure time and the pixel value of each acquired minimum exposure pixel data corresponding to each pixel data is calculated, and the calculated An image pickup apparatus comprising: difference pixel data generation means for generating difference pixel data composed of difference values. (N-1) The pixel data corresponding to the charge for each line of the pixel read at the time of exposure with the exposure time for each type corresponding to the exposure time of each type (N is a natural number of 2 or more). (N-1) first to (N-1) output channels for storing each of them independently and outputting the stored pixel data independently for each type;
In the exposure for the (N-1) types of exposure time, the charge for each line of the pixel at the time of exposure with the minimum exposure time is read, and pixel data corresponding to the read charge is stored and stored. An Nth output channel for outputting pixel data;
The minimum exposure pixel data acquisition means acquires the minimum exposure pixel data output via the Nth output channel;
The difference pixel data generation unit includes pixel values of pixel data corresponding to (N-1) types of exposure times output from the first to (N-1) th output channels for each line of the pixels, The difference value with the pixel value of the minimum exposure pixel data acquired by the minimum pixel data acquisition unit corresponding to each pixel data is calculated, and difference pixel data composed of the calculated difference value is generated. The imaging apparatus according to claim 1, wherein: Pixel data corresponding to the charge during the exposure with the minimum exposure time, which is output through the first to (N-1) th output channels, is acquired and output through the Nth output channel. Pixel data corresponding to the charge during exposure with the minimum exposure time is acquired, and the minimum exposure acquired via each output channel for each output channel of the first to (N-1) th output channels. The difference value between the pixel value of the pixel data at the time and the pixel value of the pixel data at the minimum exposure obtained via the Nth output channel is calculated, and the minimum exposure difference pixel data including the difference value is generated. Exposure difference pixel data generating means;
Minimum exposure difference pixel data storage means for storing the minimum exposure difference pixel data,
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. A difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data is calculated, and the difference value of the pixel data and the pixel data are calculated. A difference value with the value of the minimum exposure difference pixel data stored in the minimum exposure difference pixel data storage unit corresponding to the difference value is calculated, and difference pixel data composed of the difference value is generated. The imaging device according to claim 2. The minimum exposure difference pixel data for a predetermined one line of the pixel lines is stored in the minimum exposure difference pixel data storage means,
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. While calculating a difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data, the difference value of the pixel data and the minimum exposure difference A difference value with respect to a value of minimum exposure difference pixel data for one line stored in the pixel data storage means is calculated, and difference pixel data including the difference value is generated. 3. The imaging device according to 3. The minimum exposure difference pixel data for a plurality of lines smaller than the total number of lines of the pixel is stored in the minimum exposure difference pixel data storage means,
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. While calculating a difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum exposure pixel data acquisition unit corresponding to the pixel data, the difference value of the pixel data and the minimum exposure difference A difference value with any value of the minimum exposure difference pixel data for a plurality of lines stored in the pixel data storage means is calculated, and difference pixel data composed of the difference value is generated. The imaging device according to claim 3. Pixel data corresponding to the charges of the photoelectric conversion elements in the optical black region of the photoelectric conversion unit, which are output through the first to (N-1) th output channels, respectively, are acquired, and the Nth output Obtaining pixel data corresponding to the charges of the photoelectric conversion element in the optically black region output through the channel, and for each output channel of the first to (N-1) th output channels, A difference value between the pixel value of the pixel data of the optical black area acquired through each output channel and the pixel value of the pixel data of the optical black area acquired through the Nth output channel. OB difference pixel data generation means for generating OB difference pixel data consisting of values based on
The difference pixel data generation means stores pixel data corresponding to (N-1) types of exposure times output via the first to (N-1) th output channels for each line of the pixels. The difference value between the pixel value and the pixel value of the minimum exposure pixel data acquired by the minimum pixel data acquisition unit corresponding to the pixel data is calculated, and the difference value of the pixel data is calculated as the difference value of the OB difference pixel data. 6. The image pickup apparatus according to claim 2, wherein values are added and difference pixel data including the addition result is generated.

Images