JP7533657B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

Conventionally, imaging units have been known in which a back-illuminated imaging chip and a signal processing chip are connected via microbumps for each cell unit that groups together multiple pixels (see, for example, Patent Document 1).

JP 2006-49361 A

However, in electronic devices equipped with conventional stacked imaging elements, there have been few proposals for capturing images in multiple block units, and electronic devices equipped with stacked imaging elements have not been user-friendly.

The present invention was made in consideration of the above problems, and aims to provide an easy-to-use imaging device.

The electronic device (1) of the present invention has an imaging element (100), an imaging section (20) capable of imaging a first area (61) and a second area (62) surrounding the first area, and a control section (70) that sets a frame rate for imaging the second area higher than the frame rate for imaging the first area.

The electronic device (1) of the present invention has an imaging element (100), an imaging section (20) capable of imaging a first area (61) and a second area (62) surrounding the first area, and a control section (70) that sets the imaging start timing of the second area to a timing earlier than the imaging start timing of the first area.

In the electronic device of the present invention, the pixels that capture the second region may include pixels that capture infrared light. Furthermore, the pixels that capture the first region may include pixels that capture infrared light. Furthermore, the control unit may make the imaging conditions when capturing the second region different from the imaging conditions when capturing the first region. Furthermore, the control unit may perform imaging of the first region depending on the imaging result of the second region. In this case, the control unit may perform imaging of the first region when capturing an image of a moving object present in the second region.

The electronic device (1) of the present invention includes an imaging unit (20) having an imaging element (100) including a first pixel (R, G, B) that captures visible light and a second pixel (IR) that captures infrared light, and a control unit (70) that sets the imaging start timing of the second pixel to a timing earlier than the imaging start timing of the first pixel.

In this case, the electronic device (1) may include an irradiation unit (90) capable of irradiating the field with infrared light. In this case, the electronic device may include a change unit (96) that changes the range in which the irradiation unit irradiates the infrared light. The control unit may limit the imaging by the first pixel while the imaging by the second pixel is being performed. The control unit may perform imaging by the first pixel according to the imaging result by the second pixel. The imaging unit may image a first region (61) and a second region (62) around the first region, and the control unit may differentiate imaging conditions when imaging the first region from imaging conditions when imaging the second region. In this case, the control unit may differentiate a gain when imaging the first region from a gain when imaging the second region.

The electronic device may also include a correction unit (30) that corrects the imaging result of one of the first and second pixels based on the imaging result of the other of the first and second pixels. The electronic device may also include a display unit that displays an image captured by the first pixel and an image captured by the second pixel.

In order to clearly explain the present invention, the above description corresponds to the reference numerals in the drawings that represent one embodiment, but the present invention is not limited to this, and the configuration of the embodiment described below may be appropriately improved, and at least a part may be replaced with other components. Furthermore, components that are not particularly limited in terms of their placement may be placed in any position that can achieve their function, not limited to the placement disclosed in the embodiment.

The imaging device of the present invention has the effect of improving usability.

1 is a diagram showing a configuration of a surveillance camera system according to a first embodiment. FIG. 1 is a diagram showing a configuration of a surveillance camera. FIG. 2 is a cross-sectional view of an imaging element. 1 is a diagram illustrating a pixel array and unit groups of an imaging chip. FIG. 2 is a diagram showing an equivalent circuit diagram of each pixel. FIG. 2 is a circuit diagram showing a connection relationship of pixels in a unit group. FIG. 2 is a block diagram showing a functional configuration of an imaging element. 5 is a flowchart showing processing of the surveillance camera according to the first embodiment. 9 is a flowchart showing the process of step S10 in FIG. 8 . 10A and 10B are diagrams for explaining the processing of FIG. FIG. 9 is a diagram for explaining step S16 in FIG. 8; FIG. 11 is a diagram showing a configuration of a surveillance camera according to a second embodiment. FIG. 2 is a diagram for explaining a function of an irradiation device. 10 is a flowchart showing a process of a surveillance camera according to a second embodiment. 15(a) and 15(b) are diagrams showing modified examples regarding the arrangement of unit groups.

First Embodiment
A surveillance camera system 200 according to a first embodiment will be described in detail below with reference to Figures 1 to 11. Figure 1 shows a schematic configuration of the surveillance camera system 200 according to the first embodiment.

As shown in FIG. 1, the surveillance camera system 200 includes multiple surveillance cameras 1 and a management device 2. The surveillance cameras 1 and the management device 2 are connected to a network 80 such as a local area network (LAN).

The surveillance camera 1 is a camera used to monitor inside buildings, on the street, etc., and has a configuration as shown in Fig. 2. As shown in Fig. 2, the surveillance camera 1 includes a lens unit 10, an imaging unit 20, an image processing unit 30, a work memory 40, an operation unit 50, a recording unit 60, a system control unit 70, a communication unit 63, and a light meter 64.

The lens unit 10 is an imaging optical system composed of multiple lens groups. The lens unit 10 guides a light beam from a subject to the imaging unit 20. The lens unit 10 may incorporate a focus lens or a zoom lens.

The imaging unit 20 has an imaging element 100 and a driving unit 21. The imaging element 100 passes the acquired pixel signals to the image processing unit 30. The driving unit 21 is a control circuit that controls the driving of the imaging element 100 according to instructions from the system control unit 70. The specific configuration of the imaging element 100 and the specific drive control of the imaging element 100 by the driving unit 21 will be described later.

The image processing unit 30 uses the work memory 40 as a workspace to perform various image processing on the RAW data consisting of pixel signals of each pixel, and generates image data. The image processing unit 30 has a first image processing unit 30A and a second image processing unit 30B. In this embodiment, as described later, the system control unit 70 divides the pixel area (imaging area) of the image sensor 100 into first and second blocks (areas) as an example. The system control unit 70 also drives and controls the image sensor 100 so that imaging is performed under different imaging conditions in the first and second blocks. In this case, for example, the first image processing unit 30A performs image processing of signals from pixels included in the first block, and the second image processing unit 30B performs image processing of signals from pixels included in the second block. Note that the pixel area (imaging area) of the image sensor 100 is not limited to being divided into two blocks, and may be divided into three or more blocks. In this case, image processing for each block is appropriately assigned to the first and second image processing units 30A and 30B.

The image processing unit 30 executes various types of image processing. For example, the image processing unit 30 generates an RGB image signal by performing color signal processing (color correction) on the signal obtained by the image sensor 100. The image processing unit 30 also performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the image signal. The image processing unit 30 also performs compression processing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary. The image processing unit 30 outputs the generated image data to the recording unit 60.

The image processing unit 30 extracts frames at a predetermined timing from the multiple frames obtained in time series from the imaging unit 20. Furthermore, the image processing unit 30 calculates one or more frames to be interpolated between each frame based on the multiple frames obtained in time series from the imaging unit 20. The image processing unit 30 then adds the calculated one or more frames between each frame. This allows for smoother movement when playing back a video.

The work memory 40 temporarily stores image data and the like when image processing is performed by the image processing unit 30.

The operation unit 50 is a switch or the like that is operated by the user. The operation unit 50 outputs a signal according to the operation by the user to the system control unit 70. The operation unit 50 may include a touch panel.

The recording unit 60 has a card slot into which a storage medium such as a memory card can be inserted. The recording unit 60 stores the image data and various data generated in the image processing unit 30 in the recording medium inserted in the card slot. The recording unit 60 has an internal memory. The recording unit 60 can also record the image data and various data generated in the image processing unit 30 in the internal memory.

The system control unit 70 centrally controls the overall processing and operation of the surveillance camera 1. The system control unit 70 has a CPU (Central Processing Unit) 70A. In this embodiment, the system control unit 70 divides the imaging surface of the imaging element 100 into multiple blocks, and acquires images with different charge accumulation times (or charge accumulation times), frame rates, and gains between the blocks. For this reason, the system control unit 70 instructs the driving unit 21 on the position, shape, and range of the blocks, and the accumulation conditions for each block. In addition, the system control unit 70 acquires images with different thinning rates, number of added rows or columns for adding pixel signals, and number of bits for digitization between the blocks. For this reason, the system control unit 70 instructs the driving unit 21 on the imaging conditions for each block (thinning rate, number of added rows or columns for adding pixel signals, and number of bits for digitization). In addition, the image processing unit 30 executes image processing with different imaging conditions (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate) between the blocks. For this reason, the system control unit 70 instructs the image processing unit 30 on the imaging conditions for each block (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate).

The system control unit 70 also causes the recording unit 60 to record the image data generated in the image processing unit 30. The system control unit 70 also issues instructions to the communication unit 63 to communicate with the management device 2 via the network 80 (see FIG. 1). The system control unit 70 transmits the image data generated in the image processing unit 30 to the management device 2.

The illuminance meter 64 detects the illuminance around the surveillance camera 1. The detection results of the illuminance meter 64 are input to the system control unit 70.

Next, the imaging element 100 will be described in detail with reference to Figures 3 to 7. The imaging element 100 of this first embodiment is a stacked imaging element.

Figure 3 shows a cross-sectional view of the imaging element 100. As shown in Figure 3, the imaging element 100 includes an imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 such as Cu.

As shown in FIG. 3, incident light mainly enters in the direction indicated by the white arrow. In this first embodiment, the surface of the imaging chip 113 on which the incident light enters is referred to as the back surface.

The imaging chip 113 is, for example, a back-illuminated MOS image sensor, and has a PD layer 106, a wiring layer 108, etc.

The PD layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 has a plurality of photodiodes (hereinafter referred to as PDs) 104 that are two-dimensionally arranged and accumulate electric charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

A filter section 102 is provided on the incident side of the PD layer 106 where the incident light is incident, via a passivation film 103. The filter section 102 has a color filter and an infrared light transmission filter. The color filter is a filter that passes a specific wavelength range of visible light, and there are multiple types that pass different wavelength ranges. The infrared light transmission filter is a filter that passes the infrared light component of the incident light. The color filter and the infrared light transmission filter have a specific arrangement corresponding to each PD 104. The arrangement of the color filter and the infrared light filter will be described later. Note that a set of the filter section 102, the PD 104, and the transistor 105 forms one pixel. A microlens 101 is provided on the incident side of the filter section 102 where the incident light is incident, corresponding to each pixel. The microlens 101 focuses the incident light toward the corresponding PD 104.

The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered and may include passive and active elements.

The signal processing chip 111 has TSVs (Through-Silicon Vias) 110 that connect the circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 may be provided in the peripheral area of the imaging chip 113 or in the memory chip 112.

Figure 4 is a diagram explaining the pixel arrangement of the imaging chip and the unit group 131. Figure 4 is a diagram showing the imaging chip 113 as viewed from the back side. The region in which pixels are arranged in the imaging chip 113 is called pixel region 113A. More than 20 million pixels are arranged in a matrix in pixel region 113A. In the example shown in Figure 4, 16 pixels, 4 pixels x 4 pixels adjacent to each other, form one unit group 131. Note that the grid lines in Figure 4 show the concept of forming unit group 131 by grouping adjacent pixels. Note that the number of pixels forming unit group 131 is not limited to this, and may be around 1000, for example 32 pixels x 64 pixels, or may be more or less than that.

As shown in the enlarged partial view of pixel region 113A (lower right view of FIG. 4), unit group 131 includes four types of pixels: green pixel G, blue pixel B, red pixel R, and infrared pixel IR. Green pixel G is a pixel having a green filter as filter section 102 (color filter) and receives light in the green wavelength band of incident light. Blue pixel B is a pixel having a blue filter as filter section 102 (color filter) and receives light in the blue wavelength band. Red pixel R is a pixel having a red filter as filter section 102 (color filter) and receives light in the red wavelength band. Infrared pixel IR is a pixel having an infrared light transmitting filter as filter section 102 and receives infrared light.

Figure 5 is a diagram showing an equivalent circuit diagram of each pixel. In the following, the pixel will be described with the reference symbol 150. Each pixel 150 has the PD 104, the transfer transistor 152, the reset transistor 154, the amplification transistor 156, and the selection transistor 158. At least a part of these transistors corresponds to the transistor 105 in Figure 3. In addition, each pixel is provided with a reset wiring 300 to which an ON signal for the reset transistor 154 is supplied, a transfer wiring 302 to which an ON signal for the transfer transistor 152 is supplied, a power supply wiring 304 to which power is supplied from the power supply Vdd, a selection wiring 306 to which an ON signal for the selection transistor 158 is supplied, and an output wiring 308 to which a pixel signal is output. In the following, each transistor will be described as an n-channel FET, but the type of transistor is not limited to this.

The source, gate, and drain of the transfer transistor 152 are connected to one end of the PD 104, the transfer wiring 302, and the gate of the amplification transistor 156, respectively. The drain of the reset transistor 154 is connected to the power supply wiring 304, and the source is connected to the gate of the amplification transistor 156. The drain of the amplification transistor 156 is connected to the power supply wiring 304, and the source is connected to the drain of the selection transistor 158. The gate of the selection transistor 158 is connected to the selection wiring 306, and the source is connected to the output wiring 308. The load current source 309 supplies a current to the output wiring 308. In other words, the output wiring 308 for the selection transistor 158 is formed by a source follower. The load current source 309 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

Figure 6 is a circuit diagram showing the connection relationship of the pixels 150 in the unit group 131. Note that in Figure 6, the transfer wiring and output wiring are shown, but other configurations of each pixel are omitted.

In this first embodiment, pixels 150 having color filters or infrared light transmitting filters of the same color within a unit group 131 form a pixel group. That is, in this first embodiment, six pixels G form a G pixel group, four pixels R form an R pixel group, four pixels B form a B pixel group, and two pixels IR form an IR pixel group.

Here, the gates of the transfer transistors are commonly connected between multiple pixels included in each pixel group. That is, the gates of the transfer transistors of the pixels included in the G pixel group are connected to a common G transfer wiring 330, the gates of the transfer transistors of the pixels included in the R pixel group are connected to a common R transfer wiring 332, the gates of the transfer transistors of the pixels included in the B pixel group are connected to a common B transfer wiring 334, and the gates of the transfer transistors of the pixels included in the IR pixel group are connected to a common IR transfer wiring 336. This allows the drive unit 21 to control the gates of the transfer transistors simultaneously within a pixel group and independently between pixel groups.

The output sides of the selection transistors of the pixels in each pixel group are connected in common. That is, the output sides of the selection transistors of the pixels in the G pixel group are connected to a common G output wiring 340, the output sides of the selection transistors of the pixels in the R pixel group are connected to a common R output wiring 342, the output sides of the selection transistors of the pixels in the B pixel group are connected to a common B output wiring 344, and the output sides of the selection transistors of the pixels in the IR pixel group are connected to a common IR output wiring 346.

Although not shown in FIG. 6, the reset wiring and power supply wiring are common to the unit group 131. In addition, 16 selection wirings are arranged one-to-one for each pixel, and are connected to the gates of the corresponding selection transistors. Furthermore, a load current source is connected to each output wiring.

As a result, the driving unit 21 can collectively control the charge accumulation time of each pixel belonging to each pixel group. Furthermore, the driving unit 21 can cause a specific pixel group to accumulate charge for an accumulation time that differs from that of other pixel groups.

Figure 7 is a block diagram showing the functional configuration of the image sensor. The analog multiplexer 411 sequentially selects six pixels G of the G pixel group of the unit group 131 and outputs each pixel signal to the G output wiring 320, etc.

The pixel signal output via the multiplexer 411 is amplified by the amplifier 416, and the amplified pixel signal is passed via the G output wiring 320 to a signal processing circuit 412 that performs correlated double sampling (CDS) and analog/digital conversion, where it is subjected to correlated double sampling signal processing and A/D conversion (conversion from analog signals to digital signals). The A/D converted pixel signal is passed to the demultiplexer 413 via the G output wiring 321 and stored in the pixel memory 414 corresponding to each pixel.

Similarly, the multiplexers 421, 431, and 441 sequentially select the pixels of the R, B, and IR pixel groups of the unit group 131, respectively, and output the pixel signals of each pixel to the output wirings 322, 324, and 326. The pixel signals output to the output wirings 322, 324, and 326 are amplified by the amplifiers 426, 436, and 446, and the signal processing circuits 422, 432, and 442 perform CDS and A/D conversion on the amplified pixel signals. The A/D converted pixel signals are passed to the demultiplexers 423, 433, and 443 via the output wirings 323, 325, and 327, and are stored in the pixel memories 414 corresponding to each pixel.

The multiplexers 411, 421, 431, and 441 are each formed on the imaging chip 113 by the selection transistor 158 and the selection wiring 306 in FIG. 5. The amplifiers 416, 426, 436, and 446 and the signal processing circuits 412, 422, 432, and 442 are formed on the signal processing chip 111. The demultiplexers 413, 423, 433, and 443 and the pixel memory 414 are formed on the memory chip 112.

The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them on to the downstream image processing unit 30. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that FIG. 6 shows the connections for one unit group 131, but in reality, these exist for each unit group 131 and operate in parallel. However, it is not necessary for there to be a arithmetic circuit 415 for each unit group 131. For example, one arithmetic circuit 415 may process sequentially while referring to the values of the pixel memories 414 corresponding to each unit group 131 in order.

Next, the blocks set in the pixel region 113A (see FIG. 2) of the image sensor 100 will be described. In this embodiment, the pixel region 113A of the image sensor 100 is divided into a plurality of blocks. The plurality of blocks are defined to include at least one unit group 131 per block. The pixels included in each block are controlled by different control parameters. In other words, pixel signals having different control parameters are acquired for a pixel group included in one block and a pixel group included in another block. Examples of the control parameters include the charge accumulation time or number of accumulations, frame rate, gain, thinning rate, number of rows or columns for adding pixel signals, and number of bits for digitization. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels.

Here, the charge accumulation time refers to the time from when PD 104 starts accumulating charge to when it finishes. Additionally, the number of times charge is accumulated refers to the number of times PD 104 accumulates charge per unit time. Additionally, the frame rate refers to a value that indicates the number of frames that are processed (displayed or recorded) in a video per unit time. The unit of frame rate is fps (Frames Per Second). The higher the frame rate, the smoother the movement of the subject (i.e., the object being imaged) in the video.

Gain refers to the gain rate (amplification rate) of the amplifiers 416, 426, 436, and 446. By changing this gain, the ISO sensitivity can be changed. This ISO sensitivity is a standard for photographic film established by ISO, and indicates how weak light a photographic film can record. However, generally, the ISO sensitivity is also used to express the sensitivity of the image sensor 100. In this case, the ISO sensitivity is a value that indicates the ability of the image sensor 100 to capture light. Increasing the gain also improves the ISO sensitivity. For example, doubling the gain also doubles the electrical signal (pixel signal), and the appropriate brightness is achieved even with half the amount of incident light. However, increasing the gain also amplifies the noise contained in the electrical signal, resulting in more noise.

The thinning rate refers to the ratio of the number of pixels whose pixel signals are not read out to the total number of pixels in a specified region. For example, if the thinning rate for a specified region is 0, this means that pixel signals are read out from all pixels in the specified region. If the thinning rate for a specified region is 0.5, this means that pixel signals are read out from half of the pixels in the specified region.

The number of added rows refers to the number of pixels in the vertical direction (number of rows) to be added when pixel signals of vertically adjacent pixels are added. The number of added columns refers to the number of pixels in the horizontal direction (number of columns) to be added when pixel signals of horizontally adjacent pixels are added. Such addition processing is performed, for example, in the arithmetic circuit 415. By performing processing in which the arithmetic circuit 415 adds pixel signals of a predetermined number of pixels adjacent in the vertical or horizontal direction, it is possible to achieve the same effect as processing in which pixel signals are read out by thinning at a predetermined thinning rate. Note that in the above-mentioned addition processing, the average value may be calculated by dividing the sum by the number of rows or columns added by the arithmetic circuit 415.

The number of digitization bits refers to the number of bits when the signal processing circuits 412, 422, 432, and 442 convert an analog signal into a digital signal in A/D conversion. The more bits in the digital signal, the more detailed the brightness and color changes are expressed.

In this embodiment, the driving unit 21 in FIG. 2 controls the frame rate for each block by controlling the timing (or timing period) of applying a reset pulse, a transfer pulse, and a selection pulse to the reset transistor 154, the transfer transistor 152, and the selection transistor 158, respectively. The driving unit 21 also sets the blocks in the pixel region (imaging region) 113A of the image sensor 100. The system control unit 70 instructs the driving unit 21 on the position, shape, range, etc. of the blocks.

Returning to FIG. 1, the management device 2 includes a terminal such as a PC (Personal Computer), and collects and displays images captured by the surveillance camera 1 via the network 80. In addition, when the management device 2 receives instructions from a user (administrator) regarding the operation and settings of the surveillance camera 1, it transmits the instructions to the surveillance camera 1 via the network 80.

Next, the processing of the surveillance camera 1 of this first embodiment will be described in detail with reference to Figures 8 to 11.

Figure 8 shows a flowchart of the processing of the surveillance camera 1 according to the first embodiment. Note that the processing in Figure 8 starts when the surveillance camera 1 is powered on and communication with the management device 2 is established.

In the process of FIG. 8, in step S10, the system control unit 70 executes a subroutine for a block frame rate setting process. In the process of step S10, a process is executed according to the flowchart of FIG. 9. Note that the block frame rate setting process is a process in which the pixel area 113A of the image sensor 100 is divided into a plurality of blocks, and a frame rate is set for each block.

In the process of FIG. 9, in step S50, the system control unit 70 determines whether or not a block setting has been input by the administrator operating the management device 2. Here, the administrator may set the block by manual input. In this case, the administrator performs input from the management device 2 or the operation unit 50 of the surveillance camera 1. In this way, if the administrator has performed a block setting, the determination in step S50 is positive, and the process proceeds to step S52.

When the process proceeds to step S52, the system control unit 70 performs block setting in response to input from the administrator. For example, as shown in FIG. 10(a), when an aisle is captured within the imaging range of the surveillance camera 1, the administrator may specify (input) a hatched block as an area where a person is likely to enter the imaging area. In this case, the system control unit 70 sets the unspecified area as the first block 61 and the specified area as the second block 62. The process then proceeds to step S56. Note that in FIG. 10(a) and other figures, the size of the unit group 131 (the size of one square) is exaggerated for ease of illustration.

On the other hand, if the determination in step S50 is negative, that is, if there is no block setting input from the administrator, the process proceeds to step S54. When the process proceeds to step S54, the system control unit 70 sets the center of the pixel area (imaging range) as the first block 61, as shown in FIG. 10(b), and sets the periphery of the first block 61 (near the outer edge of the imaging range) as the second block 62. Then, the process proceeds to step S56.

When the process proceeds to step S56 via step S52 or S54, the system control unit 70 judges whether the IR imaging conditions are met. Here, cases where the IR imaging conditions are met include when the detection result of the illuminometer 64 indicates that the surroundings of the surveillance camera 1 are darker than a predetermined level, when the light is backlit, or when a request is input from the administrator to identify the face of a person wearing sunglasses. Note that whether the light is backlit can be judged using known technology (see, for example, JP 2003-91031 A, etc.). If the judgment in step S56 is positive, the process proceeds to step S58, and if the judgment in step S56 is negative, the process proceeds to step S60.

When the judgment in step S56 is positive and the process proceeds to step S58, the system control unit 70 decides to perform IR imaging for both the first block 61 and the second block 62. That is, it decides to perform imaging using pixel IR in each unit group contained in the first and second blocks. Note that when the surroundings are dark, imaging may be performed using pixel IR only, but when there is backlighting or when it is desired to identify the face of a person wearing sunglasses, imaging may be performed using pixels R, G, and B in addition to imaging using pixel IR.

On the other hand, if the determination in step S56 is negative and the process proceeds to step S60, the system control unit 70 decides to perform visible light imaging in the first block 61 and the second block 62. The system control unit 70 may also decide to perform IR imaging in addition to visible light imaging in the first block 61 and the second block 62.

After the processing of step S58 or S60 is completed, the process proceeds to step S62. When the process proceeds to step S62, the system control unit 70 sets the imaging by the pixels in the first block 61 to a low frame rate, and sets the imaging by the pixels in the second block 62 to a high frame rate.

When the process in FIG. 9 is completed, the process proceeds to step S12 in FIG. 8.

Returning to FIG. 8, in step S12, the system control unit 70 starts imaging using the image sensor 100 in which the first block 61 and the second block 62 have been set and the frame rate of each block has been set. The system control unit 70 controls the frame rate for each block by issuing an instruction to the drive unit 21 to control the timing (or timing period) of applying the reset pulse, transfer pulse, and selection pulse to the reset transistor 154, transfer transistor 152, and selection transistor 158, respectively.

In step S12, the system control unit 70 instructs the image processing unit 30 to execute a moving object detection process using the image captured by the second block 62. In this case, the image processing unit 30 detects a moving object by performing a comparison with the image captured immediately before. Here, the range set in the second block 62 is a range in which a moving object such as a person is likely to enter the imaging range. Therefore, by setting the frame rate of this second block 62 higher than the frame rate of the first block 61, the image processing unit 30 can perform moving object detection with high accuracy. In addition, since there is no problem in setting the frame rate of the first block 61, which captures an area in which it is likely that no moving object exists, low, it is possible to reduce power consumption, etc.

In step S12, the system control unit 70 also transmits imaging data to the management device 2 via the communication unit 63. The management device 2 displays the received imaging data on a display or the like, allowing the administrator to view the images captured by the surveillance camera 1. In this case, the block (second block 62) where a moving object such as a person is likely to enter the imaging range has a high frame rate, making it easier for the administrator to find the moving object. In addition, for example, if the area around the surveillance camera 1 is dark, imaging is performed using the pixel IR of the second block 62 (S58), allowing the administrator to reliably find the moving object even in dark conditions.

Next, in step S14, the system control unit 70 waits until it is notified by the image processing unit 30 that a moving object has been detected in the second block 62. That is, until a moving object is detected, the system control unit 70 continues capturing images at the blocks and frame rate set in step S10. Then, when it is notified by the image processing unit 30 that a moving object has been detected in the second block 62, the system control unit 70 proceeds to step S16 and resets the blocks and frame rate. Specifically, as shown in FIG. 11, the system control unit 70 sets the range that includes the moving object (a person in FIG. 11) to the second block 62, and sets the other range to the first block 61. Note that the system control unit 70 maintains a high frame rate for the second block 62 and a low frame rate for the first block 61.

Next, in step S18, the system control unit 70 makes the first block 61 follow the moving object. That is, the system control unit 70 makes the pixels (unit group 131) to be included in the first block 61 and the pixels (unit group 131) to be included in the second block 62 differ from time to time in accordance with the movement of the moving object in the image obtained from the image processing unit 30.

Next, in step S20, the system control unit 70 issues an instruction to the recording unit 60 to start recording image data. Although the recording of image data may start immediately after step S12, in this embodiment, it is started after the detection of a moving object. This makes it possible to record only images in which the moving object is captured. Here, the second block 62 tracks the moving object, and since the moving object is captured at a high frame rate in the second block 62, it is possible to capture and record the appearance of the moving object in high image quality. This makes it easier for the administrator using the management device 2 to monitor the moving object. In addition, in the image processing unit 30, when recording image data of pixels R, G, and B, the pixel signal that would be obtained if pixel G was present at the position of pixel IR is interpolated using the image signal of the nearby pixel G.

Next, in step S22, the system control unit 70 waits until the moving object is no longer present in the image (within the field of view). If the moving object has gone out of the imaging range, the process proceeds to step S24, where the system control unit 70 judges whether or not to end the process. The system control unit 70 judges whether or not to end the process based on whether or not the administrator has input an end instruction from the management device 2 or the operation unit 50. If the judgment here is negative, that is, if the process is not to end, the process proceeds to step S26, where the system control unit 70 returns the first and second blocks to the state set in step S10 (the state in FIG. 10(a) or FIG. 10(b)), and returns to step S14. Thereafter, the processes and judgments in steps S14 to S26 are repeated until the judgment in step S24 is positive.

In the processes of Figs. 8 and 9, it may be possible to constantly monitor whether the IR imaging conditions are met and to switch the pixels used for imaging based on the monitoring results. For example, when the surroundings (subject scene) are dark, imaging may be performed using the IR pixel, and when the surroundings (subject scene) become bright, imaging may be started using the R, G, and B pixels.

As described above in detail, according to the first embodiment, the imaging unit 20 has an imaging element 100 and can image the first block 61 and the second block 62 that can image the periphery of the first block 61, and the system control unit 70 sets the frame rate of the second block 62 higher than the frame rate of the first block 61. As a result, in the first embodiment, the frame rate of the pixels of the second block 62 that images the area (near the outer edge of the imaging range) where a moving object is likely to enter the imaging range can be set higher, thereby improving the detection accuracy of the moving object (probability of a person discovering a moving object). It is also possible to deal with cases where the speed of a moving object entering the imaging range is fast. In addition, in the first embodiment, since moving object detection is performed based on the image obtained by the second block 62, unlike a human sensor, moving object detection can be performed with high accuracy by image recognition, and image data before or immediately after the moving object is detected can also be obtained.

In addition, in this first embodiment, the unit groups contained in the first and second blocks include pixels IR that capture infrared light, so it is possible to perform appropriate imaging in dark situations, backlit situations, when it is desired to capture an image of the face (eyes) of a person wearing sunglasses, etc.

In the first embodiment, the case where imaging of the first block 61 and imaging of the second block 62 are started simultaneously in step S12 has been described, but the present invention is not limited to this. For example, imaging of the second block 62 may be started in step S12, and imaging of the first block 61 may be started if a moving object is detected in the second block 62 (if the judgment in step S14 is positive). That is, the imaging start timing of the second block 62 may be set to a timing earlier than the imaging start timing of the first block 61. Even in this way, a moving object can be detected from the captured image of the second block 62, so that it is possible to reduce power consumption by delaying the imaging start timing of the first block 61.

In the first embodiment, a case has been described in which all unit groups 131 include a pixel IR that captures infrared light, but this is not limiting and it is sufficient that at least one unit group includes a pixel IR. For example, the pixel IR may be included in a unit group included in either the first or second block set in step S54 of FIG. 9.

In the first embodiment, the system control unit 70 has been described as making the frame rate in the imaging of the first block 61 and the frame rate in the imaging of the second block 62 different, but this is not limited to the above. The system control unit 70 may make other imaging conditions different between the first block 61 and the second block 62. For example, the system control unit 70 may make the control parameters for controlling the image sensor 100 (e.g., charge accumulation time or accumulation count, frame rate, gain), the control parameters for controlling the reading of signals from the image sensor 100 (e.g., thinning rate), and the control parameters for processing signals from the image sensor 100 (e.g., the number of rows or columns for adding pixel signals, the number of bits for digitization, and the control parameters for the image processing unit 30 to perform image processing, which will be described later), different for each block.

In the above first embodiment, in step S16, a predetermined range capable of capturing an image of a moving object is set as the second block 62 as shown in FIG. 11, and the second block 62 is caused to track the moving object, but this is not limited to the above. For example, in step S16, the entire imaging range may be set to the second block 62. In other words, imaging may be performed at a high frame rate in the entire imaging range.

In the above first embodiment, a case has been described in which a block such as that shown in FIG. 10(a) is designated in response to input from an administrator, but this is not limited to the above. For example, if the system control unit 70 can recognize an aisle or road from an image, or if, as a result of capturing images for a predetermined period of time, the system control unit 70 can recognize an area into which a moving object (such as a person) is likely to enter, the system control unit 70 may automatically set a block such as that shown in FIG. 10(a) based on the recognition result.

Second Embodiment
Next, a second embodiment will be described with reference to FIGS.

Figure 12 shows the configuration of a surveillance camera 1' according to the second embodiment. In addition to the configuration of the surveillance camera 1 of the first embodiment, the surveillance camera 1' is equipped with an irradiation device 90 that irradiates the subject field with infrared light.

12, the irradiation device 90 includes a first irradiation unit 92, a second irradiation unit 94, and a switching control unit 96. For example, as shown in FIG. 13, the first irradiation unit 92 irradiates infrared light to an area (hatched area) on the outer periphery of the field of the surveillance camera 1'. On the other hand, the second irradiation unit 94 irradiates infrared light to the entire field of the surveillance camera 1'. The switching control unit 96 switches between the irradiation of infrared light by the first irradiation unit 92 and the irradiation of infrared light by the second irradiation unit 94 under the instruction of the system control unit 70. Note that the irradiation amount (light amount) of the first irradiation unit 92 and the second irradiation unit 94 may be the same, but in this second embodiment, the irradiation amount of the second irradiation unit 94 is set to be greater than that of the first irradiation unit 92.

Next, the processing according to this second embodiment will be explained with reference to FIG.

Figure 14 shows the processing of the system control unit 70 in this second embodiment. Note that among the processes in Figure 14, the processes and decisions that differ from the processes in the first embodiment (processing in Figure 8) are indicated by thick lines. The following explanation will focus on the parts that differ from Figure 8.

In the process of FIG. 14, in step S8, the system control unit 70 determines whether the surroundings of the surveillance camera 1' are darker than a predetermined level based on the detection result of the illuminometer 64. If the determination in step S8 is positive, the process proceeds to step S10', but if the determination is negative, the process proceeds to the process of the first embodiment (the process of FIG. 8).

When moving to step S10', the system control unit 70 executes a process of setting the block, frame rate, and pixels to be used. In this process, the system control unit 70 sets, as an example, a first block 61 located in the center of the imaging range as shown in FIG. 10(b), and a second block 62 located around the first block 61. The system control unit 70 also performs imaging of the first block 61 at a low frame rate, and imaging of the second block 62 at a high frame rate. Furthermore, the system control unit 70 performs imaging using the pixels IR in the first block 61 and the second block 62.

Next, in step S11, the system control unit 70 starts irradiating infrared light by the first irradiating unit 92. That is, the system control unit 70 starts irradiating the area on the outer edge of the field shown in Fig. 13 with infrared light. Next, the system control unit 70 starts imaging in step S12, similar to the first embodiment. Note that the system control unit 70 optimizes the sensitivity of each pixel based on the received light intensity, thereby suppressing blown-out highlights and crushed shadows.
After step S12, in step S14, the system waits until a moving object is detected in the second block 62, and when a moving object is detected, in step S16', the blocks, frame rate, and pixels to be used are reset. In this case, as shown in FIG. 11, the unit group that captures images of the periphery of the moving object is the second block 62, and the rest are the first block 61. The system control unit 70 maintains the image capture of the first block 61 at a low frame rate, and the image capture of the second block 62 at a high frame rate. Furthermore, the system control unit 70 performs image capture using the pixel IR and the pixels R, G, and B in the first block 61 and the second block 62.

Next, in step S17, the system control unit 70 starts irradiating infrared light by the second irradiating unit 94. That is, the system control unit 70 starts irradiating the entire subject scene shown in FIG. 13 with infrared light.

Here, the imaging results using pixel IR and pixels R, G, and B are sent to the image processing unit 30. In the image processing unit 30, an image suitable for viewing is generated using the image captured by pixel IR and the image captured by pixels R, G, and B. For example, if the administrator requests viewing of an image in which colors can be distinguished, the image processing unit 30 increases the gain to extract colors from the image captured by pixels R, G, and B, and corrects the image captured by pixel IR using the extracted colors. This makes it possible to provide the administrator with an image with less noise and in which colors can be distinguished. Note that, conversely to the above, the image processing unit 30 may extract the contours of objects or moving objects captured from the image captured by pixel IR, and correct the image captured by pixels R, G, and B using the extracted contours.

Then, the system control unit 70 executes the processing and judgment of steps S18 to S24. If the judgment of step S24 is negative, that is, if the moving object is no longer present within the imaging range but the process is not finished, the process proceeds to step S26', where the system control unit 70 returns the range of the first and second blocks to the original range as shown in FIG. 10(b), and starts irradiating infrared light by the first irradiation unit 92. The system control unit 70 then returns to step S14, and repeats the processing and judgment of steps S14 to S26', and ends all the processing of FIG. 14 when the judgment of step S24 is positive.

As described above, according to the second embodiment, the imaging unit 20 has an imaging element 100 including pixels R, G, and B that capture visible light and pixel IR that captures infrared light, and the system control unit 70 sets the timing for starting imaging by pixel IR to be earlier than the timing for starting imaging by pixels R, G, and B (S10', S16'). This allows appropriate imaging to be performed according to the imaging situation. For example, as in the second embodiment, imaging is performed using pixel IR until a moving object is detected, and after the moving object is detected, imaging is performed using pixel IR and pixels R, G, and B, thereby reducing power consumption until the moving object is detected, and after the moving object is detected, an image in which the color can be distinguished can be provided to the administrator.

In addition, in this second embodiment, the surveillance camera 1 is equipped with an irradiation device 90 capable of irradiating the subject with infrared light, so that a moving object irradiated with infrared light can be captured by pixel IR. This makes it possible to capture a clear image by pixel IR without the moving object (e.g., a person) noticing.

In addition, in this second embodiment, the switching control unit 96 of the irradiation device 90 switches between the irradiation of infrared light by the first irradiation unit 92 and the irradiation of infrared light by the second irradiation unit 94, so that the irradiation range of the infrared light can be changed according to the image capture by the image capture unit 20. In addition, in this second embodiment, the second irradiation unit 94 has a larger irradiation amount than the first irradiation unit 92, so that the irradiation amount of infrared light can be appropriate depending on whether moving object detection is mainly performed or image capture is mainly performed.

In addition, in this second embodiment, the system control unit 70 performs imaging using pixels R, G, and B according to the imaging result using pixel IR. As a result, for example, when it is determined that the timing is appropriate based on the imaging result using pixel IR (for example, when a moving object is detected), imaging using pixels R, G, and B is performed, thereby making it possible to reduce power consumption compared to always performing imaging using pixels R, G, and B.

In addition, in this second embodiment, as in the first embodiment, the system control unit 70 sets the frame rate of the second block 62 higher than the frame rate of the first block 61, so that the frame rate of the pixels in the range (second block 62) required for moving object detection and imaging of moving objects can be set higher. This makes it possible to reduce power consumption compared to when the entire frame rate is set to a high rate.

In addition, in this second embodiment, the image processing unit 30 corrects the imaging result of one of the first and second pixels based on the imaging result of the other of the first and second pixels, so that an image suitable for viewing can be obtained using the images captured by both pixels.

In the second embodiment, the system control unit 70 has been described as making the frame rate different for the imaging of the first block and the imaging of the second block different, but this is not limited to the above. As in the first embodiment, the system control unit 70 may make other imaging conditions, such as the charge accumulation time or number of accumulations, the frame rate, the gain, the thinning rate, the number of rows or columns for adding pixel signals, the number of digitization bits, etc., different for each block.

In the second embodiment, in step S17' in FIG. 14, the case where infrared light is irradiated to the entire subject scene has been described, but the present invention is not limited to this. For example, if the second irradiation unit 94 can irradiate a moving object with pinpoint infrared light and the irradiation position of the second irradiation unit 94 can be changed by a drive unit (not shown), the switching control unit 96 may control the drive unit to track the moving object with the infrared light irradiated by the second irradiation unit 94. In addition, the first and second irradiation units 92 and 94 may be realized by a single irradiation unit that can change the range in which infrared light is irradiated.

Note that in the process of FIG. 14, if the surroundings of the surveillance camera 1 reach a predetermined brightness, the system control unit 70 may forcibly execute the process of FIG. 8 even if the processes of steps S10 to S26' are being performed.

Note that in step S10 of FIG. 14, a case where the first block 61 is set to a low frame rate is described, but this is not limiting. Image capture in the first block 61 may be turned off (OFF) until transition to step S16 is made.

In addition, in the second embodiment, in step S10', the first and second blocks may be set as shown in FIG. 10(a) in the same manner as in the first embodiment.

In the above embodiments, the surveillance camera 1 is connected to the management device 2 via the network 80, but the present invention is not limited to this, and the surveillance camera 1 does not have to be connected to the management device 2. In this case, the surveillance camera 1 may include a display unit for displaying the image captured by the imaging unit 20. This makes it possible to check the captured image on the surveillance camera 1.

In the above embodiments, the pixels in the unit group are arranged as shown in Figures 4 and 6, but the present invention is not limited to this. For example, the arrangements shown in Figures 15(a) and 15(b) may be used. Figure 15(a) shows an arrangement in which one of the two pixels G in the Bayer arrangement is pixel IR. Also, Figure 15(b) shows an arrangement in which the positions of pixel IR and pixel B in Figure 15(a) are swapped. Even if such an arrangement is adopted, the same effect as in the above embodiments can be obtained.

In addition, in each of the above embodiments, when pixel IR in a unit group is used in combination with pixels R, G, and B, the frame rate of pixel IR may be different from the frame rate of pixels R, G, and B. In this case, the frame rate of each pixel may be determined according to the imaging environment (brightness, backlighting, etc.).

In the above embodiments, the moving object is a person, but the present invention is not limited to this. The system control unit 70 may detect, for example, animals other than people, cars, and objects that may be captured during a disaster (water, fire, etc.).

The above-described embodiment is a preferred example of the present invention. However, the present invention is not limited to this embodiment, and various modifications are possible without departing from the spirit of the present invention.

REFERENCE SIGNS LIST 1 Surveillance camera 20 Imaging unit 30 Image processing unit 61 First block 62 Second block 70 System control unit 90 Irradiation device 96 Switching control unit 100 Imaging element

Claims (24)

a pixel region in which first pixels outputting first signals for detecting an intrusion of a moving object into an imaging range and second pixels outputting second signals for generating an image are arranged along a row direction;
a drive unit that controls a frame rate at which the first signal is read out from the first pixel and a frame rate at which the second signal is read out from the second pixel ;
The driving unit controls the frame rate at which the first signal is read out from the first pixel to be higher than the frame rate at which the second signal is read out from the second pixel before intrusion of a moving object into the imaging range is detected using the first signal read out from the first pixel. 2. The imaging device according to claim 1,
The first pixel is disposed closer to the periphery than the second pixel in the pixel region . 3. The imaging device according to claim 1,
The imaging device, wherein the driving unit controls so that a frame rate for reading out the second signal from the second pixel is different before an intrusion of a moving object into the imaging range is detected using the first signal read out from the first pixel and after an intrusion of a moving object into the imaging range is detected using the first signal read out from the first pixel. 4. The imaging device according to claim 3,
The imaging device, wherein the driving unit controls so that a frame rate at which the second signal is read out from the second pixel after intrusion of a moving object into the imaging range is detected using the first signal read out from the first pixel is higher than a frame rate at which the second signal is read out from the second pixel before intrusion of a moving object into the imaging range is detected using the first signal read out from the first pixel. 5. The imaging device according to claim 1,
a first output wiring to which the first signal read out from the first pixel is output;
and a second output wiring through which the second signal read out from the second pixel is output. 6. The imaging device according to claim 5,
a first load current source that supplies a current to the first output wiring;
a second load current source that supplies a current to the second output wiring. 7. The imaging device according to claim 6,
The first pixel has a first photoelectric conversion unit that converts light into an electric charge,
The second pixel has a second photoelectric conversion unit that converts light into an electric charge,
the first photoelectric conversion unit and the second photoelectric conversion unit are disposed on a first semiconductor chip;
The first load current source and the second load current source are disposed on a second semiconductor chip connected to the first semiconductor chip. 8. The imaging device according to claim 7,
The imaging device, in which the first semiconductor chip and the second semiconductor chip are stacked. 9. The imaging device according to claim 1,
a first amplifier that amplifies the first signal read out from the first pixel by a first amplification factor;
a second amplifier that amplifies the second signal read out from the second pixel by a second amplification factor. 10. The imaging device according to claim 9,
The imaging device, wherein the second amplification factor is different from the first amplification factor. 11. The imaging device according to claim 9,
The first pixel has a first photoelectric conversion unit that converts light into an electric charge,
The second pixel has a second photoelectric conversion unit that converts light into an electric charge,
the first photoelectric conversion unit and the second photoelectric conversion unit are disposed on a first semiconductor chip;
The imaging device in which the first amplifier and the second amplifier are disposed on a second semiconductor chip connected to the first semiconductor chip. 12. The imaging device according to claim 11,
The imaging device, in which the first semiconductor chip and the second semiconductor chip are stacked. 13. The imaging device according to claim 1,
a first conversion unit for converting the first signal read out from the first pixel into a digital signal having a first bit number;
a second conversion unit for converting the second signal read out from the second pixel into a digital signal with a second number of bits. 14. The imaging device according to claim 13,
The imaging device, wherein the second bit number is a different bit number from the first bit number. 15. The imaging device according to claim 13,
The first pixel has a first photoelectric conversion unit that converts light into an electric charge,
The second pixel has a second photoelectric conversion unit that converts light into an electric charge,
the first photoelectric conversion unit and the second photoelectric conversion unit are disposed on a first semiconductor chip;
The imaging device, in which the first conversion unit and the second conversion unit are disposed on a second semiconductor chip connected to the first semiconductor chip. 16. The imaging device according to claim 15,
The imaging device, in which the first semiconductor chip and the second semiconductor chip are stacked. 17. The imaging device according to claim 1,
a first arithmetic circuit that performs a first addition process on the first signal read out from the first pixel;
a second arithmetic circuit that performs a second addition process on the second signal read out from the second pixel. 18. The imaging device according to claim 17,
An imaging apparatus in which the number of additions in the second addition process is different from the number of additions in the first addition process. 19. The imaging device according to claim 17 or 18 ,
The first pixel has a first photoelectric conversion unit that converts light into an electric charge,
The second pixel has a second photoelectric conversion unit that converts light into an electric charge,
the first photoelectric conversion unit and the second photoelectric conversion unit are disposed on a first semiconductor chip;
The first arithmetic circuit and the second arithmetic circuit are disposed on a second semiconductor chip connected to the first semiconductor chip. 20. The imaging device according to claim 19,
The imaging device, in which the first semiconductor chip and the second semiconductor chip are stacked. 21. The imaging device according to claim 1,
At least one of the first pixel and the second pixel includes a filter having a spectral characteristic that transmits infrared light. 22. The imaging device according to claim 1,
The pixel region of the imaging device includes a plurality of the first pixels and a plurality of the second pixels. 23. The imaging device according to claim 22,
The first pixels are arranged along the row direction,
The second pixels are arranged along the row direction. 24. The imaging device according to claim 22 or 23,
The first pixels are arranged along a column direction,
The second pixels are arranged along the column direction.

Images