CN115023945A - Hybrid pixel circuit for capturing frame-based and event-based images - Google Patents

Abstract

The invention provides a pixel circuit. The pixel circuit includes: a current change detector for detecting a change in photocurrent from the input port; one or more pixel cells, each pixel cell comprising one or more photodiodes and a corresponding transfer gate transistor and a reset transistor, wherein a source of each transfer gate transistor is connected to a cathode of a corresponding photodiode and a drain of each transfer gate transistor is connected to a source of the reset transistor; one or more selector circuits that connect a drain of the reset transistor to the input port of the current change detector or a fixed voltage in response to a control signal. The present invention achieves good registration of frame-based images and event-based images.

Description

Hybrid pixel circuit for capturing frame-based and event-based images

technical field

The present invention relates to a pixel circuit for acquiring frame-based images and event-based images.

Background technique

Traditional image data is obtained with frame-based image sensors. Frame-based image data is light intensity data for each pixel on a frame-by-frame basis. Event-based image sensors, also known as Dynamic Vision Sensors (DVS), are expected to be primarily used in mobile devices due to their low latency (high speed), high dynamic range, and low power change detection capabilities. Components of a camera system. Event-based image data is an array of data that includes at least the location of events in the image area and their event timestamps.

A problem with event-based image sensors, however, is that traditional images cannot be obtained from such sensors alone. Therefore, their applications are limited to machine vision fields such as automotive applications. As a solution to this problem, a dual-camera system with an event-based image sensor and a frame-based image sensor can be used, but such a dual-camera system increases the size of the camera module and requires the implementation of an algorithm to register the two There are two types of images, namely event-based images and frame-based images.

For example, the prior art has at least the following two problems:

In terms of resolution, it is difficult for the event-based image sensor to improve the image resolution because the pixel circuit of the event-based image sensor is more complicated than that of the conventional frame-based image sensor. This problem is exacerbated in sensors such as Asynchronous Time-based Image Sensor (ATIS) and Active Pixel Vision Image Sensor (DAVIS) due to the additional circuitry required to measure light intensity.

In terms of image quality, since event-based pixels and frame-based pixels have different sizes from each other, the composite arrangement of event-based pixels and frame-based pixels not only makes it difficult for them to mutually adjust the periodicity of pixel arrangement, but also Differently, fixed pattern noise is also generated periodically.

SUMMARY OF THE INVENTION

In order to realize two types of image acquisition without the above-mentioned problems, the present invention discloses a pixel circuit and a method of driving the circuit. In the pixel circuit, a portion of the frame-based image sensor (eg, photodiodes) is connected to the event detection circuit without significant changes in image characteristics (resolution, color reconstruction, noise, etc.). At the same time, embodiments enable conventional color images and change detection images to be obtained from one image sensor circuit. This means that the frame-based image and the event-based image have good registration, reducing the image processing load.

The present invention provides a pixel circuit to achieve good registration of frame-based images and event-based images.

According to a first aspect, there is provided a pixel circuit comprising: a current change detector for detecting a change in photocurrent from an input port; one or more pixel units, each pixel unit including one or more photodiodes and corresponding transfer gate transistors and reset transistors, wherein the source of each transfer gate transistor is connected to the cathode of the corresponding photodiode, and the drain of each transfer gate transistor is connected to the source of the reset transistor; a or more selector circuits, each selector circuit connecting the drain of the reset transistor of one or more of the pixel cells to the input port or a fixed voltage of the current change detector in response to a control signal .

In a first possible implementation of the first aspect, clamp photodiodes are used as the one or more photodiodes of the one or more pixel units.

In a second possible implementation of the first aspect, the current change detector and the one or more selector circuits are arranged on separate substrates.

In a third possible implementation of the first aspect, there is provided a method of operating each pixel unit, comprising: connecting the drain of the reset transistor to the input of the current change detector port; as an event-based sensor mode, turn on the reset transistor and the transfer gate transistor to detect changes in light intensity; connect the drain of the reset transistor to the fixed voltage; as a frame-based sensor mode, the transfer gate transistor is turned off to accumulate photoelectrons, and each transistor in the pixel unit is switched to output a signal corresponding to the amount of accumulated photoelectrons.

In combination with the third possible implementation manner, in a fourth possible implementation manner of the first aspect, the method further includes: for each group of one or more of the Pixel units set the mode individually.

In a fifth possible implementation of the first aspect, there is provided a method of operating each pixel unit, comprising: connecting the drain of the reset transistor to the input of the current change detector port; as an event-based sensor mode, turn on the reset transistor and a portion of the transfer gate transistor to detect changes in light intensity at the photodiode connected to the transfer gate transistor; as a frame-based sensor mode, turning off the remaining transfer gate transistors to accumulate photoelectrons generated in the photodiodes connected to the remaining transfer gate transistors; connecting the drain of the reset transistor to the fixed voltage; switching all the each transistor in the pixel unit to output a signal corresponding to the accumulated photoelectron amount.

With reference to the fifth possible implementation manner, in a sixth possible implementation manner of the first aspect, the method further includes: for one or more of the optoelectronic devices of each group in the pixel unit Diode individually set mode.

In a seventh possible implementation manner of the first aspect, the image sensor includes: the pixel circuit; and a processor configured to reconstruct image data by reading out the photoelectrons accumulated by the photodiode, the image data being obtained by the current change detector.

In an eighth possible implementation manner of the first aspect, the camera system includes: the pixel circuit; and a processor for reconstructing image data by reading out the photoelectrons accumulated by the photodiode, the image data being obtained by the current change detector.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required for describing the embodiments or the prior art are briefly introduced below. It is obvious that the drawings in the following description only show some embodiments of the present invention, and those of ordinary skill in the art can still obtain other drawings from these drawings without creative efforts.

FIG. 1 shows an example of a pixel circuit 1 provided by an embodiment of the present invention;

Figure 2 shows a typical event-based pixel circuit;

Figure 3 shows a typical frame-based pixel circuit;

FIG. 4 shows the operation of the pixel circuit 1 when the reset drain is connected to the current change detector 200;

FIG. 5 shows the operation of the pixel circuit 1 when the reset drain is connected to a fixed voltage;

FIG. 6 shows an example of a pixel circuit 1 including one pixel unit 300;

FIG. 7 shows an example of a pixel circuit 1 including a plurality of pixel units 300;

FIG. 8 shows an example of a pixel circuit 1 including a plurality of selector circuits 100 and a plurality of pixel units 300;

FIG. 9 shows an example of a pixel cell 300 including one photodiode;

FIG. 10 shows an example of a pixel unit 300 including a plurality of photodiodes;

11 shows an example of a pixel cell 300 including transistors for Dual Conversion Gain (DCG) functionality;

FIG. 12 shows an example of the operation of the pixel circuit 1 during the exposure period;

FIG. 13 shows an example of the operation of the pixel circuit 1 during a readout period;

Figure 14 shows an example of switching control between frame-based sensor mode and event-based sensor mode;

FIG. 15 shows an example of a pixel cell with an Over Flow Gate (OFG);

FIG. 16 shows the binning method performed by the current change detector 100 for a plurality of pixel units 300;

17 shows the binning method performed by the current change detector 100 for a plurality of photodiodes;

FIG. 18 shows the binning method performed by the current change detector 100 for color combinations;

Figure 19 shows an example of a pixel arrangement on a substrate;

Figure 20 shows another example of a pixel arrangement on a substrate;

Figure 21 shows another example of a pixel arrangement on a substrate;

Figure 22 shows another example of a pixel arrangement on a substrate;

FIG. 23 shows an example of a stacked structure of a pixel circuit provided by another embodiment of the present invention;

Figure 24 shows an example of three stacked sensors;

Figure 25 shows an example of a dual stack sensor;

Figure 26 shows another example of a dual stack sensor;

Figure 27 shows an overview of generating overlay images for SLAM;

Figure 28 illustrates inter-image compensation of event-based images.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only some, but not all, of the embodiments of the present invention. All other embodiments obtained by persons of ordinary skill in the art according to the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

FIG. 1 shows an example of a pixel circuit 1 provided by an embodiment of the present invention. The pixel circuit 1 includes a selector circuit 100 , a current change detector 200 and one or more pixel units 300 . The pixel circuit 1 functions not only as a frame-based pixel circuit, but also as an event-based pixel circuit. The selector circuit 100 connects the drain of the reset transistor 30 in the pixel unit 300 (hereinafter referred to as the “reset drain”) to a fixed voltage such as VDD or any of the input ports of the current change detector 200 . One.

As shown in FIG. 4 , when the reset drain is connected to the current change detector 200 and the transfer gate transistors 21 to 24 and the reset transistor 30 are turned on, the pixel cell 300 is in the event-based sensor mode and the pixel circuit 1 operates as the event-based pixel circuit of FIG. 2 while the current change detector 200 monitors the photocurrent. Only one photodiode is shown in FIG. 2 , and the transfer gate transistors 11 to 14 and the reset transistor 30 are omitted.

As shown in FIG. 5 , when the reset drain is connected to the fixed voltage such as VDD, the pixel unit 300 operates in a frame-based sensor mode, and the pixel circuit 1 acts as a frame-based pixel in FIG. 3 . circuit works. In FIG. 5, the transfer gate transistors 21 to 24 are in an off state, and the photodiodes accumulate photoelectrons.

A clamped photodiode may be used as the photodiode of the pixel unit 300 . The photoelectrons are accumulated in a state where the region around the cathode of the clamp photodiode is filled with holes. Dark current can be reduced using a clamped photodiode.

The selector circuit 100 switches between two types of pixel circuits in response to a control signal (not shown in FIG. 1 ) from outside the pixel circuit 1 .

Figure 2 shows a typical event-based pixel circuit. Such circuits are called Dynamic Vision Sensors (DVS) or "event-based sensors." This circuit converts photodiode (PD) current logarithmically to a voltage, detects voltage changes, and outputs an "on event" when the voltage change exceeds a predetermined threshold voltage, or when the voltage change drops below A "shutdown event" is output below a predetermined threshold voltage. In the following, detecting an event when the current change exceeds a predetermined threshold voltage, or detecting an event when the current change falls below a predetermined threshold voltage is also referred to as "event detection". When the selector circuit 100 is connected to the current change detector 200, the photocurrent flows continuously from the photodiode, and the current change detector 200 detects the turn-on event and the turn-off event. The address and polarity, ie, where the change is detected and whether the change is above or below the upper threshold or below the lower threshold, can be obtained, and a conventional color image can also be obtained from an image sensor.

Figure 3 shows a typical frame-based pixel circuit. The circuit is a general complementary metal oxide semiconductor (CMOS) image sensor. The frame-based pixel circuit includes photodiodes 11 to 14 , transfer gate transistors 21 to 24 , reset transistor 30 , source follower amplifier transistor 40 and row gate transistor 50 .

Generally, the amount of light received by the photodiodes 11 to 14 etc. is measured as follows: First, the reset transistor 30 and the transfer gate transistors 21 to 24 are turned on, and the cathodes of the photodiodes 11 to 14 are reset to depleted state (no electrons).

Then, the transfer gate transistors 21 to 24 are turned off, and photoelectrons are accumulated at the cathodes of the photodiodes 11 to 14 . This period is called the "exposure period". During this period, the reset transistor 30 may remain on and the FD level will be pulled up for charge transfer as described below.

After accumulating the charges, the reset transistor 30 is turned off, and the transfer gate transistor 21 is switched to transfer the charges to a floating diffusion (FD) 60 . The term "switching" means switching from off to on, and after a predetermined period of time, from on to off. The row gate transistor 50 is turned on, and the source follower amplifier transistor 40 outputs a voltage corresponding to the charge amount to an Analog to Digital Converter (ADC) (not shown in FIG. 3 ). After turning off the reset transistor 30 and before switching the transfer gate transistor 21, the reference level of the floating diffusion (FD) 60 may be measured for output level calibration. The row gate transistor 50 may be turned on after the reset transistor 30 is turned off. This process is repeated to measure the amount of light received by the photodiodes 12 to 14, ie, the reset transistor 30 is switched and the transfer gate transistor 22 is switched to measure the amount of light received by the photodiode 12; the reset is switched transistor 30 and switch the transfer gate transistor 23 to measure the amount of light received by the photodiode 13 ; switch the reset transistor 30 and switch the transfer gate transistor 24 to measure the amount of light received by the photodiode 14 . This cycle is called the "readout cycle".

Here, in FIG. 1, when the reset drain is connected to the fixed voltage, the pixel circuit 1 operates as a general-purpose image sensor (frame-based sensor), and when it is connected to the current change detector At 200, the pixel circuit 1 operates as a DVS (event-based sensor). For example, if the current change detector 200 detects the turn-on event or the turn-off event while the object is moving, it can be triggered by switching from the event-based pixel circuit to the frame-based pixel circuit Start high-speed movie shooting.

As shown in FIG. 6 , the pixel circuit 1 may include one pixel unit 300 , or as shown in FIG. 7 , the pixel circuit 1 may include a plurality of pixel units 300 . In the case where a plurality of the pixel units 300 are provided in the pixel circuit 1, the selector circuit 100 may be provided for each pixel unit 300, or the current change detection may be shared by a plurality of the pixel units 300 The selector 200 , that is, the pixel circuit 1 may include one or more selector circuits 100 . In FIG. 7 , the pixel circuit 1 includes a selector circuit 100 , and the drains of a plurality of the reset transistors 30 are connected to a node and are connected to a current change detector 200 through a selector circuit 100 . In FIG. 8 , the pixel circuit 1 includes a plurality of selector circuits 100 , and the drains of the plurality of reset transistors 30 are connected to a node and are connected to a node through one of the plurality of selector circuits 100 Current change detector 200 . As described below, in FIG. 18, four pixel units 300 (each comprising four photodiodes) are connected to one selector circuit 100; in FIG. 23, sixteen pixel units 300 (each including four photodiodes) are connected to one selector circuit 100; Each of the pixel cells 300 includes four photodiodes, shown as four small squares in the figure) connected to a selector circuit 100 by inter-die connections (shown as pillars in the figure). For each group of one or more of the pixel cells 300 connected to the same selector circuit 200, the event-based sensor mode and the frame-based sensor mode can be set individually.

In FIG. 1 , the four photodiodes 11 to 14 share the pixel unit 300 . However, the number of the photodiodes is not limited to four. The pixel unit shown in FIG. 9 includes one photodiode, and the pixel unit shown in FIG. 10 includes a plurality of photodiodes.

As shown in FIG. 11 , the pixel unit may have a transistor 31 for a Dual Conversion Gain (DCG) function in order to obtain a wide dynamic range by combining two types of gains. When the output range of the source follower transistor or the input range of the ADC is insufficient, the transistor 31 connected to the source of the reset transistor is turned on.

Unlike the operation shown in FIG. 4 , as shown in FIG. 12 , a part of the transfer gate transistors may be turned on, and the rest of the transfer gate transistors may be turned off. The photodiode 11 operates in an event-based sensor mode, while the photodiodes 12 to 14 operate in a frame-based sensor mode. The photodiode (photodiodes 12 to 14 in Figures 12 and 13) connected to the transfer gate transistor in the off state accumulates photoelectrons, while the other photodiode is used for event detection (Figures 12 and 13). photodiode 11). In this case, unlike the operation shown in FIG. 5 , as shown in FIG. 13 , a part of the transfer gate transistors described above may be turned off, and the remaining transfer gate transistors may be turned on. As shown in Figures 12 and 13, the pixel function of acquiring frame-based images or detecting events is selectable for each photodiode, ie, individually controllable. For each group of one or more of the photodiodes in the pixel unit 300, an event-based sensor mode and a frame-based sensor mode can be set independently. A portion of the photodiodes are used for event detection, while the remaining photodiodes in the same optoelectronic system are used to acquire frame-based images. In FIG. 12 , the photodiodes for event detection and all for acquiring frame-based images can be separated in one of the pixel units 300 by controlling the gates of the transfer gate transistors 21 to 24 the photodiode. The pixel circuit 1 can simultaneously acquire two types of image data by the following methods.

During an exposure period, the selector circuit 100 connects the reset drain to the current change detector 200 and turns on the reset transistor 30 and a portion of the transfer gate transistor (the transfer gate transistor 21 ) , to connect the photodiode 11 to the current change detector 200 . On the other hand, the other transfer gate transistors 22 to 24 are turned off to accumulate photoelectrons. In the example shown in FIG. 12 , the photodiode 11 may be connected to the current change detector 200 during the period from resetting the photodiode to starting the readout cycle.

13, during a readout period, the selector circuit 100 connects the reset drain to the fixed voltage (VDD) and turns off the transfer gate transistor ( the transfer gate transistor 21). Then, the reset transistor 30 and the remaining transfer gate transistors 22 to 24 are driven to read out the accumulated photoelectrons as a frame-based image. As described above in conjunction with FIG. 3 , the photoelectrons accumulated by the photodiodes can be read out one by one or simultaneously, that is, the sum of the photoelectrons accumulated by a plurality of the photodiodes can be read out to Improve accuracy.

FIG. 14 shows an example of switching control between frame-based sensor mode and event-based sensor mode. The above figure shows an example of the exposure period for each row address. The following figure shows an example of signal levels for the operation of the selector circuit 100 for the row address i of the image sensor. When the signal is high, the selector circuit 100 connects the reset drain to the current change detector 200 (available for event detection). When the signal is at a low level, the selector circuit 100 connects the reset drain to the fixed voltage.

As shown in FIG. 15 , the pixel unit 300 may include an overflow gate (Over Flow Gate, OFG) or an overflow drain (Over Flow Drain, OFD) 61 , 62 . . . connected to the photodiode. In FIG. 12, the transfer gate transistors 22 to 24 are turned off, and the photocurrent from the photodiodes 12 to 14 does not flow substantially. However, when the photodiode overflows, the photoelectrons flow through the transfer gate transistor connected to the photodiode. In this case, the OFG releases the spilled photoelectrons. A constant voltage is applied to the gate of the OFG, the drain of the OFG is connected to the anode of the photodiode, and the source of the OFG is pulled up to a fixed voltage. This reduces the overflow current flowing from the photodiode in the pixel unit 300 to the current change detector 200 . Referring to Figures 12 and 13, this is an efficient topology for mitigating the effects of overflow current in the above operation.

In Figure 12, one photodiode is used for event detection and three photodiodes are used to acquire frame-based images. However, two or more photodiodes can be used for event detection in order to increase the sensitivity of the event detection. Figures 16 and 17 show examples of the binning function. In FIG. 16, the photocurrents from the reset drains of two or more pixel cells 300 may be summed and flowed into the current change detector 200 in order to improve sensitivity. Arrows indicate the flow of the photocurrent. In FIG. 17 , the photocurrents from two or more photodiodes may be summed in a floating diffusion (FD) 60 and flow into the current change detector 200 .

To improve selectivity for light of a specific wavelength or color, color filters can be arranged as follows:

(i) A transparent or complementary color filter can be placed on the photodiode for event detection to increase its sensitivity. FIG. 19 shows an example of a pixel arrangement on a substrate. The four small squares framed by thick lines in the upper left corner correspond to one pixel unit 300 . "F/G" represents the photodiode used to acquire frame-based images using the green color filter (shown with sparsely shaded rectangles), and "F/R" represents the photodiode used to acquire frame-based images using the red color filter (shaded with sparsely shaded rectangles) line rectangles), "F/B" represents photodiodes used to acquire frame-based images using blue color filters (shown with densely shaded rectangles), "E" represents photodiodes used for event detection using clear color filters . In Figure 20, each pixel cell 300 has a photodiode for event detection using a transparent color filter.

(ii) If a photodiode with a red color filter is used for event detection, only red image changes are detected. The color used for event detection is not limited to one. Signals from some different or same color pixels in the photodiodes sharing the selector circuit 100 and the current change detector 200 can be combined by binning in order to increase sensitivity or select a color for sensing. For example, if a photodiode with a red color filter, a photodiode with a blue color filter, and a photodiode with a green color filter are connected to the current change detector 200, the amount of light of the color generated by mixing the three colors can be measured . In this case, the photocurrent may become white (R+G+B=W). Color filter arrays may include Bayer arrays, quadruple Bayer arrays, RGB, RYYB, RGBW, and other arrangements. The combination of colors is not limited to this example.

The circuit shown in FIG. 18 includes one selector circuit 100 , one current change detector 200 , and four pixel cells 300 , and reset drains of the four pixel cells 300 are connected to the selector circuit 100 . For example, as shown in Figure 18, a green color filter is placed on the four photodiodes (outlined with dashed lines) in the upper left pixel cell, and a blue color filter is placed on the four photodiodes in the lower left pixel cell ( Place the red color filter on the four photodiodes in the upper right pixel cell (outlined with a solid line), and place the green color filter on the four photodiodes in the lower right pixel cell (outlined by a dotted line) framed by a dash). In Figure 18, the uppermost photodiode with the green color filter in the upper left pixel cell, the uppermost photodiode with the blue color filter in the lower left pixel cell, and the uppermost photodiode with the red color filter in the upper right pixel cell are combined and the signal of the uppermost photodiode with the green color filter in the lower right pixel cell.

Figure 21 shows an example of another pixel arrangement on the substrate. The small squares correspond to the corresponding photodiodes. The four small squares in the upper left corner, outlined in bold lines, correspond to a pixel unit 300 in which the four photodiodes have green color filters (shown as sparsely shaded rectangles). The four photodiodes in the next pixel cell 300 on the right have blue filters (shown with densely shaded rectangles), the four photodiodes in the next adjacent pixel cell 300 below have red filters (shown with hatched rectangles), The four photodiodes in the lower left pixel cell 300 have a green color filter (shown with sparsely shaded rectangles), and the color pattern of these four pixel cells 300 is repeated in FIG. 21 . 16 pixel cells 300 are shown in Figure 21, and the lower left photodiode, designated "E" in each pixel cell 300, is used for event detection. Other photodiodes denoted "F" are used to acquire frame-based images.

The lower right photodiode in each pixel unit 300 shown in FIG. 21 is used for event detection. However, to align the optical center and coordinate system, the upper left and lower right photodiodes can be used for event detection, and the upper right and lower left photodiodes can be used to acquire frame-based images. Alternatively, the upper right and lower left photodiodes can be used for event detection, while the upper left and lower right photodiodes can be used to acquire frame-based images.

In order to achieve a wide dynamic range, large-sized pixels and small-sized pixels may be used in one pixel unit 300 . The largest or larger pixel can be used as the pixel for event detection to increase its sensitivity, or as the pixel for acquiring frame-based images to increase its sensitivity. Figure 22 shows an example of another pixel arrangement on the substrate. The large and small squares in the upper left corner, outlined in bold lines, correspond to one pixel cell 301 in which two photodiodes have green filters (shown as sparsely shaded rectangles). The two photodiodes in the next pixel cell on the right have blue filters (shown with densely shaded rectangles), the two photodiodes in the adjacent pixel cells below have red filters (shown with hatched rectangles), bottom left corner The two photodiodes in the pixel cell have green color filters (shown with sparsely shaded rectangles). Four pixel cells are shown in Figure 22, the large photodiode, designated "F", is used to acquire frame-based images, and the small photodiode, designated "E," is used for event detection.

The pixel circuit provided by the embodiment is a hybrid pixel circuit for acquiring a frame-based image and detecting an event, and this circuit can be implemented by the following structure without reducing the quality of the output image. FIG. 23 shows an example of the stacked structure of the pixel circuit provided by the embodiment. In FIG. 23 , the pixel unit 300 is arranged on the upper silicon substrate, the selector circuit 100 and the current change detector 200 are arranged on the lower silicon substrate, and the pixel unit 300 and the selector circuit 100 are arranged on the lower silicon substrate. The reset drains can be connected by a stacking process such as copper-copper bonding or other silicon wafer connections, and the connections are schematically shown as four pillars in FIG. 23 .

The small squares correspond to the corresponding photodiodes. For example, each pixel unit 300 includes four photodiodes, and the positions where the four photodiodes are placed form a square, the 4 x 4 photodiodes in the lower left corner of FIG. out) are placed on the four photodiodes in the upper left pixel cell, the red color filter (shown as the hatched rectangle) is placed on the four photodiodes in the lower left pixel cell, the blue color filter (shown as the densely shaded rectangle) ) are placed over the four photodiodes in the upper right pixel cell, a green color filter (shown as a sparsely shaded rectangle) is placed over the four photodiodes in the lower right pixel cell, and this color pattern is repeated. The arrangement of upper left 8x 8 photodiodes, lower left 8x 8 photodiodes, upper right 8x 8 photodiodes, and lower right 8x 8 photodiodes used to acquire frame-based images and detect events can be compared to the arrangement shown in Figure 21 The same or different.

If each pixel cell 300 includes four photodiodes, the reset drains of 16 pixel cells 300 are connected to one selector circuit 100 through one column in FIG. 23 . The relationship between the number of photodiodes and the number of current change detectors 200 may be determined according to the minimum sizes of the photodiodes and the current change detectors 200, thereby eliminating the mismatch of their minimum sizes.

With this topology, the semiconductor process of the pixel unit 300 can basically be designed to achieve its image quality and resolution. In this case, it is only necessary to separate the reset drain from the power supply (VDD) and route it to the top metal layer of the power wiring in order to use the current change detector 200 to connect to another silicon substrate. Since the fixed voltage is applied to the reset drain, the influence of noise is relatively small. Accordingly, a hybrid image sensor can be fabricated without modifying any silicon process of the implementation. Only wiring needs to be modified in the metal layer around the reset drain of the pixel cell, as shown in FIG. 23 . Therefore, the silicon process optimized for frame-based image quality (pure NMOS structure, optimized conditions for ion implantation, oxidation process and thermal process, etc.) can be used with little need to change the circuit layout, thus maintaining the quality characteristics.

Figures 24 to 26 show specific implementations of the stacked structure. Figure 24 shows an example of a three-stacked sensor that includes a Back-Side illuminated Image sensor (BSI) photodiode layer (including pixel cells), an event detector layer, and an analog front-end (Analog) including a pillar ADC. Front End, AFE) component logic layer. The AFE also includes source follower transistors and other analog circuits. Figure 25 shows an example of a dual stacked sensor including a BSI image sensor layer (acquiring frame-based images, including AFE and some logic) and an event detector layer including other logic. Figure 26 shows another example of a dual stacked sensor including a BSI image sensor layer and an event detector layer. The BSI image sensor layer captures frame-based images and includes AFEs and some parts of logic circuits (ie, pixel source followers, etc.); the event detector layer includes other parts of AFEs (ie, ADCs) and logic circuits part.

In the above description, the inter-die connection is applied on the reset drain node, but this should not be limited. For example, the inter-die connections can be applied to various nodes such as floating diffusion (FD) 60, additional capacitor nodes for double conversion gain. On the other hand, the event detection circuit may also include other image processing units and memory units for storing event data or for other purposes.

The described embodiments provide a hybrid image sensor for acquiring frame-based images and event-based images that are well-registered with each other without degrading the quality of the output image. With this technique, high-quality images of both types can be obtained with the same optical system and coordinate system. This enables high-quality compensation of frame-based images with event-based images. The following applications are expected to gain this advantage well:

(1) Feature point extraction is performed for Simultaneous Localization And Mapping (SLAM) with low latency, high dynamic range and low power consumption, so that mobile devices can realize indoor navigation systems with high traceability. This technique enables SLAM for Augmented Reality (AR) navigation with a smaller processing load where superimposed guidance images are displayed on frame-based images. Figure 27 shows an overview of overlay images generated for SLAM.

(2) Detect the inter-frame motion of the object to compensate for frame motion blur and interpolate the inter-frame images. Figure 28 illustrates inter-image compensation using event-based images. This technique reduces motion blur. In Figure 28, during the period between frame-based image 1 and frame-based image 2, if the object moves and the image becomes blurred, the image can be interpolated by using the acquired event-based image. This technique also enables reconstruction of high-speed movies that exceed frame rate limits by reconstructing one or more images between two consecutive frame-based images.

The present invention can be used with other types of hybrid sensors, such as global shutter image sensors and rolling shutter image sensors.

The above disclosures are merely exemplary embodiments of the present invention, and are of course not intended to limit the protection scope of the present invention. It can be understood by those skilled in the art that all or part of the processes for implementing the above embodiments and equivalent modifications made according to the claims of the present invention shall fall within the scope of the present invention.

Claims (9)

1. A pixel circuit, wherein the pixel circuit comprises: A current change detector to detect changes in photocurrent from the input port; One or more pixel cells, each pixel cell including one or more photodiodes and corresponding transfer gate transistors and reset transistors, wherein the source of each transfer gate transistor is connected to the cathode of the corresponding photodiode, and each transfer gate the drain of the transistor is connected to the source of the reset transistor; one or more selector circuits, each selector circuit responsive to a control signal, connecting the drains of the reset transistors of one or more of the pixel cells to the input port or fixed Voltage. 2. The pixel circuit of claim 1, wherein a clamp photodiode is used as the one or more photodiodes of the one or more pixel cells. 3. The pixel circuit of claim 1, wherein the current change detector and the one or more selector circuits are arranged on separate substrates. 4. A method of operating each pixel unit according to claim 1, wherein the method comprises: connecting the drain of the reset transistor to the input port of the current change detector; as an event-based sensor mode, turning on the reset transistor and the transfer gate transistor to detect changes in light intensity; connecting the drain of the reset transistor to the fixed voltage; As a frame-based sensor mode, the transfer gate transistor is turned off to accumulate photoelectrons, and each transistor in the pixel unit is switched to output a signal corresponding to the accumulated photoelectron amount. 5. The operation method of claim 4, further comprising: setting a mode individually for each group of one or more of the pixel units connected to the same selector circuit. 6. A method of operating each pixel unit according to claim 1, wherein the method comprises: connecting the drain of the reset transistor to the input port of the current change detector; As an event-based sensor mode, the reset transistor and a portion of the transfer gate transistors are turned on to detect changes in light intensity at the photodiodes connected to the portion of the transfer gate transistors; as a frame-based sensor mode, off turning off the remaining transfer gate transistors to accumulate photoelectrons generated in the photodiodes connected to the remaining transfer gate transistors; connecting the drain of the reset transistor to the fixed voltage; Each transistor in the pixel unit is switched to output a signal corresponding to the accumulated amount of photoelectrons. 7 . The operating method according to claim 6 , wherein the operating method further comprises: individually setting a mode for one or more of the photodiodes of each group in the pixel unit. 8 . 8. An image sensor, characterized in that the image sensor comprises: The pixel circuit according to claim 1; a processor for reconstructing image data by reading out the photoelectrons accumulated by the photodiode, the image data being acquired by the current change detector. 9. A camera system, characterized in that the camera system comprises: The pixel circuit according to claim 1; a processor for reconstructing image data by reading out the photoelectrons accumulated by the photodiode, the image data being acquired by the current change detector.

Images