[go: up one dir, main page]

US9270911B2 - Image capturing apparatus - Google Patents

Image capturing apparatus Download PDF

Info

Publication number
US9270911B2
US9270911B2 US13/123,821 US201013123821A US9270911B2 US 9270911 B2 US9270911 B2 US 9270911B2 US 201013123821 A US201013123821 A US 201013123821A US 9270911 B2 US9270911 B2 US 9270911B2
Authority
US
United States
Prior art keywords
rows
pixels
image
focus detection
group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US13/123,821
Other languages
English (en)
Other versions
US20120038810A1 (en
Inventor
Hidenori Taniguchi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Assigned to CANON KABUSHIKI KAISHA reassignment CANON KABUSHIKI KAISHA ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: TANIGUCHI, HIDENORI
Publication of US20120038810A1 publication Critical patent/US20120038810A1/en
Application granted granted Critical
Publication of US9270911B2 publication Critical patent/US9270911B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • H04N5/374—
    • H01L27/14621—
    • H01L27/14627—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/672—Focus control based on electronic image sensor signals based on the phase difference signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/53—Control of the integration time
    • H04N25/531—Control of the integration time by controlling rolling shutters in CMOS SSIS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/702—SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
    • H04N25/704—Pixels specially adapted for focusing, e.g. phase difference pixel sets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N5/23212—
    • H04N5/3532—
    • H04N5/3696—
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10F—INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION
    • H10F39/00—Integrated devices, or assemblies of multiple devices, comprising at least one element covered by group H10F30/00, e.g. radiation detectors comprising photodiode arrays
    • H10F39/80—Constructional details of image sensors
    • H10F39/805—Coatings
    • H10F39/8053—Colour filters
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10F—INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION
    • H10F39/00—Integrated devices, or assemblies of multiple devices, comprising at least one element covered by group H10F30/00, e.g. radiation detectors comprising photodiode arrays
    • H10F39/80—Constructional details of image sensors
    • H10F39/806—Optical elements or arrangements associated with the image sensors
    • H10F39/8063—Microlenses
    • H01L27/14643—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10F—INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION
    • H10F39/00—Integrated devices, or assemblies of multiple devices, comprising at least one element covered by group H10F30/00, e.g. radiation detectors comprising photodiode arrays
    • H10F39/10—Integrated devices
    • H10F39/12—Image sensors
    • H10F39/18—Complementary metal-oxide-semiconductor [CMOS] image sensors; Photodiode array image sensors

Definitions

  • the present invention relates to an image capturing apparatus that includes an image sensor having multiple pixels arranged in two-dimensional form.
  • the contrast detection technique also called the “blur technique” and the phase difference detection technique (also called the “skew technique”) are known as general techniques for auto focus detection/adjustment methods in image capturing apparatuses using light beams that have passed through an imaging lens.
  • the contrast detection technique is a technique widely used in video movie devices that capture moving images (camcorders), digital still cameras, and so on, and in such a case, the image sensor thereof is used as a focus detection sensor.
  • This technique focuses on the signal outputted from the image sensor, and particularly information of the high-frequency components (contrast information) thereof, and uses the position of the imaging lens where the evaluation value of that high-frequency component information is the highest as the in-focus position.
  • the evaluation value is found while minutely moving the imaging lens, and it is ultimately necessary to move the lens to where the maximum evaluation value can be detected; this technique is therefore unsuited to quick focus adjustments.
  • phase difference detection technique is often employed in single-lens reflex cameras that use silver film, and is the technique that has contributed the most to the practical application of auto focus (AF) detection in single-lens reflex cameras.
  • AF auto focus
  • a light beam passing through the exit pupil of the imaging lens is divided into two parts, and the resulting light beams are received by a pair of focus detection sensors; the amount of skew in signals outputted based on the amount of light received, or in other words, the amount of relative positional skew in the division direction of the light beam, is detected.
  • the amount of skew in the focus direction of the imaging lens is detected directly.
  • the amount and direction of the focus skew can be obtained by performing a single accumulation operation using the focus detection sensor, thus making fast focus adjustment operations possible.
  • a means for dividing the optical path such as a quick return mirror, a half mirror, or the like, is generally provided in the optical path for imaging, and a focus detection optical system and AF sensor are generally provided at the end thereof. This is disadvantageous in that it increases the size and cost of the apparatus.
  • Patent Document 1 a pupil division function is provided by offsetting the sensitive region of the light-receiving portion relative to the optical axis of an on-chip microlens in some light-receiving elements (pixels) in an image sensor.
  • a configuration that performs phase difference focus detection is realized by using those pixels as focus detection pixels and disposing the focus detection pixels at predetermined intervals in groups of pixels used for imaging. Because the areas in which the focus detection pixels are arranged correspond to areas in which imaging pixels are absent, image information is produced through interpolation using the information from peripheral imaging pixels.
  • Patent Document 2 discloses an image capturing apparatus capable of switching between a thinning readout mode and an adding readout mode for output with the goal of improving the image quality of moving images and improving the sensitivity at low luminosities.
  • Patent Document 2 proposes improving the image quality of moving images by performing readout in the adding mode in order to reduce moirés when the object has a high spatial frequency and the occurrence of moirés can be foreseen, or using the thinning readout mode in the case where the luminosity is high and the occurrence of smearing can be foreseen.
  • Patent Document 3 a pupil division function is provided by offsetting the sensitive region of the light-receiving portion relative to the optical axis of an on-chip microlens in some light-receiving elements (pixels) in an image sensor, in the same manner as in Patent Document 1.
  • a configuration that performs phase difference focus detection is realized by using those pixels as focus detection pixels and arranging the focus detection pixels at predetermined intervals in groups of pixels used for imaging.
  • Patent Document 3 also proposes taking accumulation control signals from the image sensing pixel groups and the focus detection pixel groups independently and employing different accumulation times for the two pixel groups, thereby improving the frame rate of the captured image and improving the performance of the focus detection pixel group with respect to low-luminosity objects.
  • the invention disclosed in Patent Document 2 relates to switching between a thinning readout mode and an adding readout mode depending on the scene when performing readouts for moving images.
  • Focus detection pixels are not arranged in the image sensor, and thus using some of the pixels in the image sensor to perform phase difference focus detection is not considered from the outset. Even if, for example, focus detection pixels were present, those focus detection pixels could not be used for image information due to the reasons described earlier, and thus during the adding readout mode, the focus detection pixels would not be able to be added to the image capturing pixels. Furthermore, if an attempt was made to perform focus detection using the focus detection pixels, it would be necessary to read out the focus detection pixels singly even during the adding readout mode.
  • Patent Document 3 employs a configuration in which accumulation control signals from image sensing pixel groups and focus detection pixel groups are taken independently and the optimal accumulation times for the respective pixel groups can be set, thereby balancing image display refresh capabilities with rangefinding capabilities for low-luminosity objects.
  • this is problematic in that the number of signal wires arranged between pixels increases, leading to a drop in the numerical aperture of the pixels and a drop in the sensitivity thereof.
  • Patent Document 3 discloses, as a variation on the invention described therein, commonalizing the accumulation control signals between the image sensing pixels and the focus detection pixels. This is advantageous in that the wiring between the image sensing pixels and the focus detection pixels is reduced, thereby improving the numerical aperture.
  • Patent Document 3 discloses adding the output of the focus detection pixel groups multiple times in order to improve the S/N ratio of the focus detection pixels.
  • this is problematic because even if that output is added following the readout, noise from a pixel amplifier, a readout gain amplifier, or the like is added multiple times as well, and thus the S/N ratio is not improved as in the case of accumulation time control.
  • the present invention improves the S/N ratio of focus detection pixels when performing focus detection using the phase difference technique during the display of a moving picture.
  • an image capturing apparatus includes: an image sensor having multiple pixels arranged two-dimensionally, the image sensor including image sensing pixels that generate a signal for image generation by photoelectrically converting an object image formed by an imaging lens, and focus detection pixels arranged discretely in multiple image sensing pixels, the focus detection pixels dividing the pupil region of the imaging lens into pupil regions and generating a signal for phase difference detection by photoelectrically converting object images from the pupil regions obtained by the division; a switching means that switches between an all-pixel readout mode in which signals from all of the multiple pixels are read out and a thinning readout mode in which the signals of the multiple pixels are thinned and read out; and a control means that, in the case where the mode has been switched by the switching means to the thinning readout mode, controls the accumulation of charges in imaging rows used for image generation and focus detection rows including the focus detection pixels independent from each other.
  • FIG. 1 is a diagram illustrating the configuration of a camera according to a first embodiment of the present invention.
  • FIGS. 2A and 2B are a plan view and a cross-sectional view, respectively, of an image sensing pixel in an image sensor according to the first embodiment.
  • FIGS. 3A and 3B are a plan view and a cross-sectional view, respectively, of a focus detection pixel in the image sensor according to the first embodiment.
  • FIG. 4 is a diagram illustrating a pixel array in the image sensor according to the first embodiment.
  • FIG. 5 is a diagram illustrating a pixel circuit in the image sensor according to the first embodiment.
  • FIG. 6 is a block diagram illustrating the overall configuration of the image sensor according to the first embodiment.
  • FIG. 7 is a diagram illustrating accumulation and readout operations for the entirety of a screen according to the first embodiment.
  • FIG. 8 is a diagram illustrating accumulation and readout operations during thinning according to the first embodiment.
  • FIG. 9 is a diagram illustrating accumulation and readout operations during focus detection according to the first embodiment.
  • FIG. 10 is a diagram illustrating accumulation and readout operations during focus detection according to a second embodiment.
  • FIG. 11 is a diagram illustrating accumulation and readout operations during focus detection according to a third embodiment.
  • FIG. 1 is a diagram illustrating the configuration of a camera (image capturing apparatus) according to a first embodiment of the present invention, and illustrates a digital camera in which a camera body having an image sensor and an imaging lens are integrated in a single unit.
  • 101 represents a first lens group arranged at the end of an imaging optical system (image formation optical system), the first lens group being held so as to be movable forward and backward in the optical axis direction.
  • 102 represents an iris/shutter, and controlling the aperture diameter thereof enables the adjustment of the light amount during imaging; during still image capturing, the iris/shutter 102 also functions as an exposure time adjustment shutter.
  • 103 represents a second lens group. The iris/shutter 102 and the second lens group 103 move backward and forward in the optical axis direction in unity, realizing a variable power effect (zoom function) by operating in tandem with the forward/backward operation of the first lens group 101 .
  • 105 represent a third lens group that performs focus adjustment by moving forward/backward in the optical axis direction.
  • 106 represents an optical low-pass filter, which is an optical element for reducing false colors, moirés, and so on in captured images.
  • 107 represents an image sensor configured of a CMOS sensor and a peripheral circuit thereof (a CMOS image sensor).
  • 111 represents a zoom actuator, which, by rotating a barrel cam (not shown), drives the first lens group 101 and the second lens group 103 forward/backward in the optical axis direction, thereby executing variable power operations.
  • 112 represents an iris/shutter actuator, which controls the diameter of the aperture of the iris/shutter 102 so as to adjust the amount of imaging light, and controls the exposure time when capturing still images.
  • 114 represents a focus actuator, which drives the third lens group 105 forward/backward in the optical axis direction, thereby adjusting the focus.
  • 115 represents an electronic flash for illuminating objects during imaging, and although a flash illumination device employing a xenon tube is preferable, an illumination device provided with an LED that continuously emits light may be used as well.
  • 116 represents an AF assist light, which projects, onto the object field via a projection lens, a mask image having a predetermined aperture pattern, thereby improving the focus detection capabilities with respect to dark objects or low-contrast objects.
  • 121 represents a CPU, and is a CPU within the camera that performs various controls for the camera body.
  • the CPU 121 includes a processing unit, a ROM, a RAM, an A/D converter, a D/A converter, a communication interface circuit, and the like; the CPU 121 drives various circuits within the camera based on predetermined programs stored within the ROM, and executes serial operations for performing AF, imaging, image processing, recording, and so on.
  • 122 represents an electronic flash control circuit that controls the lighting of the electronic flash 115 in synchronization with imaging operations.
  • 123 represents an assist light driving circuit that controls the lighting of the AF assist light 116 in synchronization with focus detection operations.
  • 124 represents an image sensor driving circuit that controls the imaging operations of the image sensor 107 , as well as performing A/D conversion on obtained image signals and transmitting those image signals to the CPU 121 .
  • 125 represents an image processing circuit that performs processing such as ⁇ conversion, color interpolation, JPEG compression, and so on images obtained by the image sensor 107 .
  • 126 represents a focus driving circuit that controls the driving of the focus actuator 114 based on focus detection results, driving the third lens group 105 forward/backward in the optical axis direction so as to adjust the focus.
  • 129 represents a zoom driving circuit that drives the zoom actuator 111 in response to zoom operations made by a user.
  • 131 represents a display device, such as an LCD, that displays information regarding the imaging mode of the camera, pre-imaging preview images and post-imaging confirmation images, focus status display images during focus detection, and so on.
  • 132 represents an operational switch group configured of a power switch, a shutter release (imaging trigger) switch, a zoom operation switch, an imaging mode selection switch, and so on.
  • 133 represents a removable flash memory, in which captured images are stored.
  • FIGS. 2A to 3B are diagrams illustrating the structure of image sensing pixels used for image generation and focus detection pixels used for phase difference detection employed in the image sensor of the present embodiment.
  • the present embodiment employs a Bayer array, in which of four pixels in a 2 ⁇ 2 arrangement, two pixels having a G (green) spectral sensitivity are arranged opposite to each other on the diagonal, and one each of pixels having R (red) and B (blue) spectral sensitivities are arranged in the other two positions.
  • Focus detection pixels having a structure that shall be described later are distributed between the Bayer arrays with a predetermined regularity.
  • FIGS. 2A and 2B illustrate the arrangement and structure of image sensing pixels.
  • FIG. 2A is a plan view illustrating a 2 ⁇ 2 image sensing pixel.
  • G pixels are arranged opposite to each other on the diagonal, and R and B pixels are arranged in the other two pixel positions. This two-row by two-column structure is arranged repeatedly.
  • FIG. 2B illustrates a cross-section viewed along the A-A line shown in FIG. 2A .
  • ML indicates on-chip microlenses disposed on the uppermost surface of each pixel
  • CFR indicates an R (red) color filter
  • CFG represents a G (green) color filter.
  • PD schematically illustrates photoelectric conversion portions of the image sensor
  • CL indicates a wiring layer for forming signal wires that transmit various types of signals within the CMOS sensor.
  • TL schematically illustrates an imaging optical system.
  • the on-chip microlenses ML and the photoelectric conversion portions PD of the image sensing pixel are configured to effectively capture, to the greatest extent possible, light beams that have passed through the imaging optical system ML.
  • an exit pupil EP of the imaging optical system TL and the photoelectric conversion portions PD are designed to be in a conjugative relationship due to the microlenses ML, and so that the effective surface area of the photoelectric conversion portions is a large surface area.
  • FIG. 2B illustrates a light beam introduced into the R pixel, the same structure applies to both the G pixels and the B pixels. Therefore, the exit pupil EP has a large diameter in correspondence with the RGB pixels for imaging, thereby efficiently capturing light beams from an object and thus improving the S/N ratio of the image signal.
  • FIGS. 3A and 3B illustrate the arrangement and structure of focus detection pixels for dividing the pupil region on the imaging lens in the horizontal direction (sideways direction) into pupil regions.
  • FIG. 3A is a plan view illustrating a two-row by two-column pixel that includes a focus detection pixel.
  • the G pixels obtain the primary component of luminance information. Because the image recognition characteristics of humans are sensitive to luminance information, image quality degradation is apparent when there are losses in the G pixels. Meanwhile, although R and B pixels are pixels that obtain color information, humans are relatively insensitive to color information, and thus image quality degradation is not very apparent even if a certain degree of defects occur in pixels that obtain color information.
  • the G pixels are maintained as image sensing pixels, whereas the R pixels and B pixels are replaced with focus detection pixels. This is indicated by SA and SB in FIG. 3A .
  • FIG. 3B illustrates a cross-section viewed along the A-A line shown in FIG. 3A .
  • the microlenses ML and the photoelectric conversion portions PD have the same structures as those of the image sensing pixels illustrated in FIG. 2B .
  • the signals from the focus detection pixels are not used in the production of images, and thus a transparent film CFW (white) or CFG is disposed in place of the color separation color filters.
  • the opening portions of the wiring layer CL are skewed in one direction relative to the centerline of the microlenses ML.
  • a pixel SA and an opening portion OPHA thereof are skewed to the right, and thus receive a light beam that has passed through an exit pupil EPHA on the left side of the imaging lens TL.
  • a pixel SB and an opening portion OPHB thereof are skewed to the left, and thus receive a light beam that has passed through an exit pupil EPHB on the right side of the imaging lens TL. Therefore, the pixels SA are arranged with regularity in the horizontal direction, and an object image obtained by this group of pixels is taken as an image A.
  • the pixels SB are also arranged with regularity in the horizontal direction and an object image obtained by this group of pixels is taken as an image B, detecting the relative positions of the image A and the image B makes it possible to detect the amount by which the focus is off in the object image (the amount of defocus).
  • FIG. 4 is a diagram illustrating an example of the pixel arrangement in the image sensor according to the present embodiment.
  • image sensors having a high number of pixels, from several millions of pixels to more than ten million pixels, have come into practical use, and thus for the sake of simplicity, the descriptions in the present embodiment shall discuss an arrangement of 24 pixels by 12 pixels.
  • image sensors are typically provided with light-shielding optical black pixels (abbreviated hereinafter as “OB pixels”) serving as a reference for pixel output as appropriate, these OB pixels have also been omitted from FIG. 4 for the sake of simplicity.
  • OB pixels light-shielding optical black pixels
  • the pixel arrangement in the present embodiment is based on a 2 ⁇ 2 Bayer array.
  • the letters G, R, and B written in FIG. 4 indicate green, red, and blue color filters, respectively.
  • the SA and SB pixels, which are the focus detection pixels are interspersed with regularity at a predetermined ratio in some portions of the Bayer array image sensor.
  • the focus detection pixels SA which are basis pixels for phase difference AF
  • the focus detection pixels SB which are reference pixels for phase difference AF
  • Finding the image skew amount for the pixels SA and SB in the rows V 4 and V 5 makes it possible to obtain the amount of defocus of the imaging lens.
  • the configuration is such that pixels SA and SB are arranged with the same regularity in rows V 10 and V 11 as well.
  • the pixels SA and SB are discretely arranged so that normal pixels for interpolation are arranged in the periphery of the focus detection pixels and so as to suppress image degradation through this interpolation. Accordingly, these pixels are arranged discretely in the pupil division direction, and are also arranged discretely in the direction perpendicular to the pupil division direction, which corresponds, in the present embodiment, to the row direction.
  • the basis and reference pixel pairs in V 4 and V 5 and the basis and reference pixel pairs in V 10 and V 11 are arranged so as to be five rows apart. Note that the arrangement in the present embodiment is simply an exemplary arrangement, and the invention is not limited to this arrangement.
  • FIG. 5 illustrates the circuit configuration of a single pixel in the CMOS sensor.
  • 501 represents a photodiode (abbreviated as “PD” hereinafter)
  • 502 and 503 represent a floating diffusion amplifier for converting a signal charge accumulated in the aforementioned PD 501 into a voltage.
  • 502 represents a floating diffusion capacitor (abbreviated as “FD” hereinafter)
  • 503 represents an MOS transistor amplifier.
  • 504 represents a transfer gate (abbreviated as “TX” hereinafter) for transferring the signal charge of the PD 501 to the floating diffusion capacitor
  • 505 represents a reset gate (abbreviated as “RES” hereinafter) for resetting the PD 501 and the FD 502
  • 506 represents a pixel selection MOS transistor (abbreviated as “SEL” hereinafter).
  • 507 represents a common signal line for controlling TX
  • 508 represents a common signal line for controlling RES
  • 509 represents a vertical output line for outputting the voltage output of the FD amplifier.
  • 510 represents a signal line for controlling SEL
  • 511 represents a load current source of the FD amplifier connected to the vertical output line.
  • FIG. 6 is a block diagram illustrating the overall configuration of the CMOS sensor.
  • 601 represents a pixel portion, in which the various pixels illustrated in FIG. 4 are arranged. The constituent elements in each pixel are configured as illustrated in FIG. 5.
  • 602 represents a vertical scanning unit, which is capable of accumulation readout on a row-by-row basis by outputting sequential scanning signals such as ⁇ V 0 , ⁇ V 1 , ⁇ V 2 , and so on up to ⁇ V 11 .
  • 603 represents a signal selection unit, into which the reset signal RES, transfer signal TX, and selection signal SEL, which are outputted at predetermined timings, are inputted by a timing generator circuit (abbreviated as “TG” hereinafter) (not shown), and which outputs those signals as appropriate to rows specified by vertical scanning signals outputted from the vertical scanning unit 602 .
  • 604 represents a signal holding unit, which, at the time of signal readout, temporarily stores the pixel output of one row selected by the SEL signal, using a sample hold signal (not shown).
  • 605 represents a horizontal scanning unit, which causes the pixel signals temporarily stored in the signal holding unit 604 to be sequentially inputted into an output amplifier 606 and outputted from the CMOS sensor, by outputting sequential scanning signals such as ⁇ H 0 , ⁇ H 1 , ⁇ H 2 , and so on up to ⁇ H 23 .
  • FIG. 7 is a diagram for illustrating the accumulation and readout performed by the CMOS sensor, and illustrates operations performed in an all-pixel readout mode.
  • FIG. 7 also schematically illustrates control based on a known rolling shutter. Because this is known technology, this shall be described only briefly.
  • the horizontal axis expresses the passage of time, and here, the pixel signals for all of the 24 ⁇ 12 pixels are sequentially captured through the rolling shutter operations.
  • the vertical direction represents the vertical scanning order, and here, the rows from V 0 to V 12 are sequentially scanned on a row-by-row basis.
  • the diagonal broken lines in FIG. 7 indicate rolling reset operations executed prior to readout, whereas the diagonal solid lines indicate rolling readout.
  • Rolling readout includes charge transfer from PD to FD, temporary storage in the signal holding unit, and external output operations performed through horizontal scanning.
  • the accumulation time is defined by the time interval of rolling resets and rolling readouts, and can be changed by changing the timing of rolling resets. In the case where a longer accumulation than the accumulation time illustrated in FIG. 7 is desired, the rolling readout interval may be set to be longer.
  • the normal pixels and focus detection pixels are controlled under the same accumulation time, and the focus detection pixels, whose openings are partially blocked from light, have a lower signal level than the normal pixels.
  • the necessary S/N ratio cannot be obtained, and thus it is necessary to add the output of focus detection pixels and so on in order to obtain the necessary S/N ratio.
  • rolling accumulation for capturing all the pixels in this manner is typically not carried out.
  • FIG. 8 is a diagram schematically illustrating accumulation and readout during thinning operations.
  • the rolling accumulation and readout operations during thinning operations are basically the same as those shown in FIG. 7 .
  • the horizontal axis expresses the passage of time
  • the vertical axis expresses the rows that are scanned in the vertical direction.
  • the number of rows used in interlaced scanning is set so that the moving image appears smooth; for example, readout within 1/30 of a second is set so that a 30-frame moving image can be displayed.
  • the invention is not limited to 30 frames. A certain degree of smoothness can be obtained as long as there are at least approximately 20 frames.
  • the present embodiment describes an example in which the vertical scanning is interlaced scanning that scans every third row and 30 frames are read out. Therefore, in FIG. 8 , readout is executed for four rows that have been thinned in the vertical direction, and if the time T denoted in FIG. 8 is greater than or equal to 0, a 30-frame moving image can be displayed. In addition, in the moving image display, rows in which focus detection pixels are not arranged, or V 0 , V 3 , V 6 , and V 9 , are read out, and thus image degradation does not occur.
  • FIG. 9 is a diagram schematically illustrating accumulation and readout during thinning operations in the case where focus detection operations are executed.
  • the rolling accumulation and readout operations during thinning operations are basically the same as those shown in FIG. 8 .
  • the horizontal axis expresses the passage of time
  • the vertical axis expresses the rows that are scanned in the vertical direction.
  • the rows V 0 , V 3 , V 6 , and V 9 which are used for live view and capturing/recording moving images as illustrated in FIG. 8 , are first scanned by the vertical scanning unit. After this, rows V 4 and V 5 , which are used for focus detection, are vertically scanned in series, thereby performing accumulation and readout control.
  • the diagonal solid lines indicate rolling readout operations, and because the vertical output line is common among the various rows, it is necessary to scan in the vertical direction and execute sequential readouts on a row-by-row basis.
  • the imaging rows and the focus detection rows can be caused to have different reset timings and thus can be caused to have different accumulation times. For this reason, accumulation control can be executed for both simultaneously so as to achieve appropriate exposures.
  • the S/N ratio of the focus detection pixels can be improved without increasing the number of control signal lines and causing problems such as a decrease in the opening surface area of pixels.
  • the present embodiment is further advantageous in terms of the S/N ratio because noise from pixel amplifiers, readout amplifiers, or the like is not added multiple times, as with frame addition.
  • two rows are assumed to be employed as the rows to be read out during the interval of a margin time T for 30 frames, and thus the rows V 4 and V 5 are read out.
  • the number of rows with focus detection pixels to be additionally read out may be determined as appropriate based on the margin time T, or based on a permissible moving image frame rate, and may be determined even in the case where the margin time T is not available.
  • FIG. 10 is a diagram illustrating operations according to a second embodiment, and is a variation on FIG. 9 .
  • the readout cycle is caused to be different depending on the focus detection row, and the accumulation time for focus detection pixels has been extended even further than in FIG. 9 .
  • FIG. 10 illustrates an example in which a single focus detection row is read out for every two frames of a moving image.
  • the readout cycle of focus detection pixels may be determined as appropriate while referring to the output results of the focus detection pixels.
  • FIG. 11 is a diagram illustrating operations according to a third embodiment, and is a further variation on FIG. 9 .
  • the present embodiment describes an example in which the margin time T is insufficient during the live view, the capturing/recording of moving images, and so on, and switches, on a frame-by-frame basis, the focus detection rows to be read out.
  • the readout from the focus detection rows V 4 and V 5 is switched to readout from the rows V 10 and V 11 from the third frame on; this makes it possible to move the focus detection rows without increasing the readout time.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Studio Devices (AREA)
  • Automatic Focus Adjustment (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Focusing (AREA)
  • Solid State Image Pick-Up Elements (AREA)
US13/123,821 2009-03-17 2010-02-01 Image capturing apparatus Active 2030-03-30 US9270911B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2009-065221 2009-03-17
JP2009065221A JP5319347B2 (ja) 2009-03-17 2009-03-17 撮像装置及びその制御方法
PCT/JP2010/051727 WO2010106846A1 (en) 2009-03-17 2010-02-01 Image capturing apparatus

Publications (2)

Publication Number Publication Date
US20120038810A1 US20120038810A1 (en) 2012-02-16
US9270911B2 true US9270911B2 (en) 2016-02-23

Family

ID=42739512

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/123,821 Active 2030-03-30 US9270911B2 (en) 2009-03-17 2010-02-01 Image capturing apparatus

Country Status (5)

Country Link
US (1) US9270911B2 (ja)
EP (1) EP2409487B1 (ja)
JP (1) JP5319347B2 (ja)
CN (2) CN102292974B (ja)
WO (1) WO2010106846A1 (ja)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9942493B2 (en) 2016-01-21 2018-04-10 Canon Kabushiki Kaisha Image pickup apparatus and reading method for outputting signals based on light flux passing through an entire area of an exit pupil and light flux passing through part of the exit pupil
US10554913B2 (en) 2017-04-26 2020-02-04 Canon Kabushiki Kaisha Solid-state imaging device, imaging system and movable object
US11570389B2 (en) 2018-08-22 2023-01-31 Canon Kabushiki Kaisha Imaging apparatus for downsizing an image sensor and a signal processor

Families Citing this family (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5803095B2 (ja) * 2010-12-10 2015-11-04 ソニー株式会社 撮像素子および撮像装置
JP5179605B2 (ja) * 2011-02-09 2013-04-10 富士フイルム株式会社 放射線画像検出装置及び放射線の照射開始検出方法
WO2012132122A1 (ja) * 2011-03-31 2012-10-04 富士フイルム株式会社 撮像装置及びその合焦制御方法
JP5744599B2 (ja) * 2011-03-31 2015-07-08 キヤノン株式会社 撮像素子、および撮像装置
US20130075590A1 (en) * 2011-09-28 2013-03-28 Truesense Imaging, Inc. Image sensors having multiple row-specific integration times
JP2013083772A (ja) * 2011-10-07 2013-05-09 Nikon Corp カメラ
CN104012083B (zh) * 2011-12-27 2015-08-12 富士胶片株式会社 摄像装置及摄像装置的控制方法
JP5963448B2 (ja) 2012-01-13 2016-08-03 キヤノン株式会社 撮像装置
JP2013157883A (ja) * 2012-01-31 2013-08-15 Sony Corp 固体撮像素子およびカメラシステム
CN104168814B (zh) * 2012-03-28 2016-08-17 富士胶片株式会社 摄像装置以及具备该摄像装置的内窥镜装置
JP6066593B2 (ja) * 2012-06-13 2017-01-25 キヤノン株式会社 撮像システムおよび撮像システムの駆動方法
JP6172967B2 (ja) * 2013-02-20 2017-08-02 キヤノン株式会社 撮像装置、及びその制御方法
EP2782331A1 (en) * 2013-03-22 2014-09-24 Harvest Imaging bvba Image sensor with focus-detection pixel, and method for reading focus-information
JP6220148B2 (ja) * 2013-04-26 2017-10-25 キヤノン株式会社 撮像装置およびその制御方法
JP6263035B2 (ja) 2013-05-17 2018-01-17 キヤノン株式会社 撮像装置
TWI471522B (zh) * 2013-07-25 2015-02-01 Nat Univ Tsing Hua 使用陰影雲紋法線上即時檢測表面形貌與平面外變形之方法與系統
JP6257245B2 (ja) * 2013-09-27 2018-01-10 キヤノン株式会社 撮像装置及びその制御方法
JP6539015B2 (ja) 2013-12-05 2019-07-03 キヤノン株式会社 撮像装置及びその制御方法
KR102121531B1 (ko) * 2013-12-10 2020-06-10 삼성전자주식회사 초점검출 가능한 이미지 센서 제어장치 및 방법.
JP2015195235A (ja) * 2014-03-31 2015-11-05 ソニー株式会社 固体撮像素子、電子機器、および撮像方法
JP6338436B2 (ja) 2014-04-25 2018-06-06 キヤノン株式会社 撮像装置及びその制御方法
JP6609113B2 (ja) * 2014-06-18 2019-11-20 キヤノン株式会社 撮像装置及びその制御方法
KR102346622B1 (ko) * 2014-08-04 2022-01-04 엘지이노텍 주식회사 이미지 센서 및 이를 포함하는 촬상 장치
CA2902675C (en) * 2014-08-29 2021-07-27 Farnoud Kazemzadeh Imaging system and method for concurrent multiview multispectral polarimetric light-field high dynamic range imaging
JP6584059B2 (ja) 2014-09-26 2019-10-02 キヤノン株式会社 撮像装置及びその制御方法、プログラム、記憶媒体
JP6362511B2 (ja) * 2014-11-04 2018-07-25 キヤノン株式会社 撮像装置及びその制御方法
JP6600458B2 (ja) 2014-12-25 2019-10-30 オリンパス株式会社 撮像素子、焦点検出装置及び焦点検出方法
JP6476065B2 (ja) * 2015-05-14 2019-02-27 オリンパス株式会社 撮像装置及び撮像装置の制御方法
JP6412826B2 (ja) 2015-06-04 2018-10-24 オリンパス株式会社 撮像装置
US20160381274A1 (en) * 2015-06-25 2016-12-29 Novatek Microelectronics Corp. Image Sensing Module
GB2558124B (en) * 2015-09-16 2022-02-02 Canon Kk Image sensor and image capturing apparatus
GB2596029B (en) 2015-09-16 2022-07-27 Canon Kk Image sensor and image capturing apparatus
JP6632357B2 (ja) 2015-12-11 2020-01-22 キヤノン株式会社 撮像装置の駆動方法、撮像装置、撮像システム
JP6762710B2 (ja) 2015-12-14 2020-09-30 キヤノン株式会社 撮像装置及びその制御方法
WO2017192663A1 (en) * 2016-05-06 2017-11-09 Arizona Board Of Regents On Behalf Of The University Of Arizona Plenoptic imaging apparatus, method, and applications
JP6729080B2 (ja) * 2016-06-30 2020-07-22 リコーイメージング株式会社 信号処理装置
JP6808420B2 (ja) * 2016-09-27 2021-01-06 キヤノン株式会社 撮像素子および撮像装置
WO2018088121A1 (ja) * 2016-11-14 2018-05-17 富士フイルム株式会社 撮像装置、撮像方法、及び、撮像プログラム
CN108401118B (zh) * 2018-03-05 2020-10-23 北京中科虹霸科技有限公司 控制红外灯开灯时间的虹膜采集装置及方法
JP7050569B2 (ja) 2018-05-10 2022-04-08 キヤノン株式会社 撮像装置及びその制御方法
JP6807348B2 (ja) * 2018-05-16 2021-01-06 シャープ株式会社 放射線検出器及び放射線透過画像取得システム
US11818462B2 (en) * 2019-08-30 2023-11-14 Qualcomm Incorporated Phase detection autofocus sensor apparatus and method for depth sensing
EP4035942A4 (en) * 2019-09-24 2022-11-30 Sony Semiconductor Solutions Corporation IMAGING DEVICE
US11314150B2 (en) * 2020-01-08 2022-04-26 Qualcomm Incorporated Phase detection autofocus (PDAF) optical system
JP7047858B2 (ja) * 2020-03-26 2022-04-05 ソニーグループ株式会社 撮像装置及び撮像方法
WO2022070941A1 (ja) * 2020-09-29 2022-04-07 富士フイルム株式会社 撮像装置、撮像装置の駆動方法、及びプログラム
US11861874B2 (en) 2021-09-30 2024-01-02 Fujifilm Corporation Imaging apparatus, imaging control method, and imaging control program

Citations (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0981245A2 (en) 1998-08-20 2000-02-23 Canon Kabushiki Kaisha Solid-state image sensing apparatus, control method therefor, basic layout of photoelectric conversion cell and storage medium
JP2003189183A (ja) 2001-12-13 2003-07-04 Chinon Ind Inc 撮像装置
US20040090550A1 (en) * 2002-10-26 2004-05-13 Samsung Techwin Co., Ltd. Image sensing means for digital camera and digital camera adopting the same
US20040246369A1 (en) * 2003-06-04 2004-12-09 Fuji Photo Film Co., Ltd. Solid state image sensing device and photographing apparatus
US20050068455A1 (en) * 2003-09-30 2005-03-31 Matsushita Electric Industrial Co., Ltd. Solid state image pickup device
US20070126909A1 (en) * 2005-11-28 2007-06-07 Sony Corporation Solid-state image-pickup device, method of driving solid-state image-pickup device and image-pickup apparatus
US20070237511A1 (en) * 2006-04-05 2007-10-11 Nikon Corporation Image sensor, imaging device and imaging method
CN101088038A (zh) 2004-12-24 2007-12-12 佳能株式会社 焦点检测设备和焦点检测方法
US20080074534A1 (en) * 2006-09-27 2008-03-27 Nikon Corporation Image sensor and image-capturing device
CN101256261A (zh) 2007-03-01 2008-09-03 奥林巴斯映像株式会社 焦点检测装置
EP1986045A2 (en) 2007-04-18 2008-10-29 Nikon Corporation Focus detection device, focus detection method and imaging apparatus
CN101339348A (zh) 2007-07-05 2009-01-07 株式会社理光 摄像设备及摄像方法
JP2009049858A (ja) 2007-08-22 2009-03-05 Canon Inc 撮像装置
US20090086084A1 (en) * 2007-10-01 2009-04-02 Nikon Corporation Solid-state image device
JP2009089105A (ja) 2007-10-01 2009-04-23 Nikon Corp 固体撮像装置及びこれを用いた電子カメラ
US20090110383A1 (en) * 2007-10-29 2009-04-30 Sony Corporation Imaging device
US20090140122A1 (en) * 2007-10-01 2009-06-04 Nikon Corporation Solid-state imaging device, electronic camera
US20100085442A1 (en) * 2008-10-03 2010-04-08 Sony Corporation Imaging apparatus, imaging method, and program
US8218962B2 (en) * 2007-12-10 2012-07-10 Sony Corporation Image-capturing apparatus
US20130293736A1 (en) * 2012-04-16 2013-11-07 Sony Corporation Image sensor, control method of image sensor, and imaging apparatus
US20150092093A1 (en) * 2012-06-07 2015-04-02 Fujifilm Corporation Imaging device and imaging method
US20150124139A1 (en) * 2013-11-06 2015-05-07 Sony Corporation Solid-state imaging device, method of driving the same, and electronic apparatus
US20150296125A1 (en) * 2012-09-28 2015-10-15 Nikon Corporation Focus detection device and image-capturing apparatus

Patent Citations (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0981245A2 (en) 1998-08-20 2000-02-23 Canon Kabushiki Kaisha Solid-state image sensing apparatus, control method therefor, basic layout of photoelectric conversion cell and storage medium
JP2000156823A (ja) 1998-08-20 2000-06-06 Canon Inc 固体撮像装置及びその制御方法及び撮像装置及び光電変換セルの基本配列及び記憶媒体
US6829008B1 (en) * 1998-08-20 2004-12-07 Canon Kabushiki Kaisha Solid-state image sensing apparatus, control method therefor, image sensing apparatus, basic layout of photoelectric conversion cell, and storage medium
JP2003189183A (ja) 2001-12-13 2003-07-04 Chinon Ind Inc 撮像装置
US20040090550A1 (en) * 2002-10-26 2004-05-13 Samsung Techwin Co., Ltd. Image sensing means for digital camera and digital camera adopting the same
US20040246369A1 (en) * 2003-06-04 2004-12-09 Fuji Photo Film Co., Ltd. Solid state image sensing device and photographing apparatus
US20050068455A1 (en) * 2003-09-30 2005-03-31 Matsushita Electric Industrial Co., Ltd. Solid state image pickup device
CN101088038A (zh) 2004-12-24 2007-12-12 佳能株式会社 焦点检测设备和焦点检测方法
US20070126909A1 (en) * 2005-11-28 2007-06-07 Sony Corporation Solid-state image-pickup device, method of driving solid-state image-pickup device and image-pickup apparatus
US20070237511A1 (en) * 2006-04-05 2007-10-11 Nikon Corporation Image sensor, imaging device and imaging method
US20080074534A1 (en) * 2006-09-27 2008-03-27 Nikon Corporation Image sensor and image-capturing device
JP2008085535A (ja) 2006-09-27 2008-04-10 Nikon Corp 撮像素子および撮像装置
CN101256261A (zh) 2007-03-01 2008-09-03 奥林巴斯映像株式会社 焦点检测装置
EP1986045A2 (en) 2007-04-18 2008-10-29 Nikon Corporation Focus detection device, focus detection method and imaging apparatus
CN101339348A (zh) 2007-07-05 2009-01-07 株式会社理光 摄像设备及摄像方法
JP2009049858A (ja) 2007-08-22 2009-03-05 Canon Inc 撮像装置
US20090140122A1 (en) * 2007-10-01 2009-06-04 Nikon Corporation Solid-state imaging device, electronic camera
JP2009089105A (ja) 2007-10-01 2009-04-23 Nikon Corp 固体撮像装置及びこれを用いた電子カメラ
US20090086084A1 (en) * 2007-10-01 2009-04-02 Nikon Corporation Solid-state image device
US7989745B2 (en) * 2007-10-01 2011-08-02 Nikon Corporation Solid-state imaging device with focus detection and electronic camera with focus adjustment
US20090110383A1 (en) * 2007-10-29 2009-04-30 Sony Corporation Imaging device
US8218962B2 (en) * 2007-12-10 2012-07-10 Sony Corporation Image-capturing apparatus
US20100085442A1 (en) * 2008-10-03 2010-04-08 Sony Corporation Imaging apparatus, imaging method, and program
US20130293736A1 (en) * 2012-04-16 2013-11-07 Sony Corporation Image sensor, control method of image sensor, and imaging apparatus
US20150092093A1 (en) * 2012-06-07 2015-04-02 Fujifilm Corporation Imaging device and imaging method
US20150296125A1 (en) * 2012-09-28 2015-10-15 Nikon Corporation Focus detection device and image-capturing apparatus
US20150124139A1 (en) * 2013-11-06 2015-05-07 Sony Corporation Solid-state imaging device, method of driving the same, and electronic apparatus

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Apr. 3, 2013 Chine Office Action, with English Translation, that issued in Chinese Patent Application No. 201080005180.2.
Feb. 14, 2013 European Search Report of the counterpart European Patent Application No. 10753341.6.

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9942493B2 (en) 2016-01-21 2018-04-10 Canon Kabushiki Kaisha Image pickup apparatus and reading method for outputting signals based on light flux passing through an entire area of an exit pupil and light flux passing through part of the exit pupil
US10554913B2 (en) 2017-04-26 2020-02-04 Canon Kabushiki Kaisha Solid-state imaging device, imaging system and movable object
US11570389B2 (en) 2018-08-22 2023-01-31 Canon Kabushiki Kaisha Imaging apparatus for downsizing an image sensor and a signal processor

Also Published As

Publication number Publication date
JP2010219958A (ja) 2010-09-30
EP2409487A1 (en) 2012-01-25
JP5319347B2 (ja) 2013-10-16
CN103501405B (zh) 2017-03-01
EP2409487A4 (en) 2013-03-20
CN103501405A (zh) 2014-01-08
US20120038810A1 (en) 2012-02-16
CN102292974B (zh) 2014-02-26
EP2409487B1 (en) 2016-01-27
WO2010106846A1 (en) 2010-09-23
CN102292974A (zh) 2011-12-21

Similar Documents

Publication Publication Date Title
US9270911B2 (en) Image capturing apparatus
US8964061B2 (en) Image capturing apparatus with selection of thinning and readout mode in accordance with moving image recording mode
JP5276371B2 (ja) 撮像装置
JP5241355B2 (ja) 撮像装置とその制御方法
US8525917B2 (en) Image sensing apparatus with plural focus detection pixel groups
US8817165B2 (en) Image capturing apparatus
US8681261B2 (en) Image-capturing apparatus having image sensor utilizing focus detection pixel pairs
JP5746496B2 (ja) 撮像装置
JP5276374B2 (ja) 焦点検出装置
JP5739653B2 (ja) 撮像装置
US9357121B2 (en) Image capturing apparatus and control method thereof
JP2009128892A (ja) 撮像素子および撮像装置
JP6272116B2 (ja) 撮像装置及びその制御方法
JP5864989B2 (ja) 撮像素子及び撮像装置

Legal Events

Date Code Title Description
AS Assignment

Owner name: CANON KABUSHIKI KAISHA, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TANIGUCHI, HIDENORI;REEL/FRAME:026340/0205

Effective date: 20110325

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8