JP2008206143A - Imaging device having image processing function - Google Patents

Abstract

A high-resolution imaging device having a high-speed image processing function is provided.
An imaging apparatus includes a plurality of light receiving circuits that are pixels arranged in a matrix and that convert the amount of received light into an electric quantity, and values of electric quantities obtained by the respective light receiving circuits. Are stored in association with each light receiving circuit 101 and a processing unit 103 provided for each column of the matrix, and the processing unit 103 is arranged in at least a part of the column directions. A work memory 203 for holding a plurality of continuous values, and an image processing circuit 204 for performing predetermined image processing using the values held in the work memory 203 are provided.
[Selection] Figure 3

Description

  The present invention relates to an imaging apparatus having an image processing function, and more particularly to an imaging apparatus suitable for a Vision Chip having a pixel parallel configuration having a line-of-sight detection function.

The line-of-sight information is known to be a useful user interface as an information input method to a computer, but the current general line-of-sight detection device is an expensive and large-scale system comprising a video camera and a computer for image processing. It is. For this reason, its application is limited to special fields such as academic research. In addition, it is known that there is a very fast eye movement called a saccade (rapid eye movement) that has an angular velocity of 600 [deg / s]. This saccade is an unconsciously jumping of the line of sight, and it has been proposed that application to a user interface is useful (see Non-Patent Documents 1 and 2), but due to its speed, a video rate of about 60 Hz is proposed. Tracking is impossible with eye gaze detection at. There are systems that support saccade detection / tracking using high-speed cameras, but the system is physically large and expensive, and the delay (latency) to gaze information generation is large. Unrealistic. The inventors have studied and implemented the architecture of a high-performance image sensor for eye-gaze detection / tracking corresponding to saccades, based on the so-called Vision Chip configuration in which a light receiving element and a processing circuit are placed for each pixel, and a basic evaluation. (See Non-Patent Documents 3 and 4).
J. Trisch et al .: “What you see is what you need,” Journal of Vision, Vol. 3, No. 1, pp. 86-94 (Feb. 2003). SSIntille: “Change Blind Information Display for Ubiquitous Computing Environments,” Proc. Of UbiComp2002, pp.91-106 (Sep. 2002). J. Akita et al .: “Vision Chip Architecture for Detecting Line of Sight Including Saccade”, IEICE Transactions on Electronics, Vol.E89-C, No.11, pp.1605-1611 (Nov. 2006). Hiroaki Takagi and Junichi Akita: “Prototype and evaluation of Vision Chip with eye-gaze detection function for rapid eye movement”, ITE Technical Report, Vol.30, No.32, pp.17-20 (June 2006).

  However, in the configuration in which the light receiving element and the processing circuit are arranged for each pixel, there is a problem in that the pixel area is large and the number of pixels is as small as 16 × 16 because of the pixel parallel configuration. There is a problem that it is difficult to detect.

  In view of the above problems, the present invention provides a vision chip that realizes a resolution of about VGA and has a line-of-sight detection function corresponding to rapid eye movement, that is, a high-resolution imaging device having a high-speed image processing function. Objective.

  In order to solve the above problems, an imaging apparatus according to the present invention is an imaging apparatus having an image processing function, and is arranged in a matrix, each of which is a plurality of pixels that convert the amount of received light into an electrical quantity. A light receiving circuit, a memory that holds the value of the electric quantity obtained in each light receiving circuit in association with each light receiving circuit, and a processing unit provided for each column of the matrix, the processing unit, Predetermined image processing is performed using a working memory for holding a plurality of values that are continuous in at least a part of column directions held in the memory, and a value held in the working memory. And an image processing circuit for performing the processing.

  Also, in the present invention, the working memory has a memory area that holds three values that are continuous in the column direction around the pixel to be processed in each column of the matrix, and the imaging device further includes: The pixels to be processed in each column are shifted in the row direction for each clock so that the pixels in the same row in each column are processed in parallel, and the pixels to be processed become three pixels continuous in the column direction. A memory controller that reads a value held in each corresponding memory and writes the value to the working memory corresponding to each memory, and the image processing circuit in each column performs parallel image processing on pixels in the same row; Performs processing and holds in the working memory corresponding to each processing target pixel in the column adjacent to both sides of the column and the value of three pixels continuous in the column direction held in each working memory The memory controller writes the result of the image processing by the image processing circuit into the memory corresponding to the pixel located at the center of the three pixels continuous in the column direction. It is good.

  As described above, according to the present invention, a high-speed processing capable of handling a resolution as high as VGA and a saccade by adopting a so-called column parallel configuration in which an image processing circuit that performs processing for line-of-sight detection is arranged for each column. There is an effect that it is possible to construct an architecture of a vision parallel configuration Vision Chip that can achieve both of these.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an imaging apparatus 100 of the present invention having a so-called column parallel configuration in which image processing circuits are arranged for each column. As illustrated in FIG. 1, the imaging apparatus 100 includes a plurality of light receiving circuits (PIX) 101, a memory (MEM) 102, and a processing unit 103. The light receiving circuit 101 is arranged in a matrix in the imaging region, and converts the luminance of the received light into a voltage value (or current value) that is an electric quantity. The memory 102 is provided in the same number as the row for each column of the light receiving circuits 101, and holds a value indicating the luminance of the light received by each light receiving circuit 101. The processing unit 103 is provided for each column of the light receiving circuit 101, and performs image processing of a pixel including a central pixel and four adjacent pixels with reference to values obtained from the processing units 103 in adjacent columns.

  FIG. 2A and FIG. 2B are diagrams illustrating an example of an image captured by an infrared camera by irradiating the eyeball with infrared light. As shown in FIG. 2A, the infrared image of the eyeball includes a pupil 202, which is a black round region in the iris, and a white bright spot called a Purkinje image 201 associated with the corneal reflection of infrared light. To do. From the positional relationship between the two, the line of sight can be obtained as shown in FIG. That is, in order to obtain the line of sight, it is only necessary to obtain the center point of a black and white circular region from the infrared image of the eyeball.

  The inventors have shown that the center of the pupil 202 and the Purkinje image 201 necessary for the line-of-sight detection can be obtained by the following five processing steps based on the discussion, examination and simulation so far.

(1) Extraction of Purkinje image by detection of luminance above a certain threshold (2) Shrinkage processing of Purkinje image area and detection of its erasure point (3) Pupil by logical sum of detection of luminance below a certain threshold and Purkinje image (4) Enlargement process of several stages of pupil area (5) Contraction process of pupil area and detection of its erasure point

  In other words, image processing for line-of-sight detection can be realized by a comparator for comparison with the pixel brightness threshold, an automaton for contraction / enlargement processing, an erase point detection and its coordinate generation circuit. become.

  The contraction process refers to a process of sequentially erasing the outermost pixel row of a circular area such as a Purkinje image or pupil and determining the last remaining pixel as the center of the circular area. The erasing point refers to the position of the last remaining pixel by the contraction process.

  FIG. 3 is a diagram illustrating a configuration for one column of the imaging apparatus 100 which is a column parallel processing architecture for line-of-sight detection. The circuit for one column comprises a light receiving circuit (PIX) 301 including a photodiode and an amplifier circuit, and a memory (MEM) 102 shown in FIG. 1, and stores a flag indicating the luminance corresponding to each pixel as 0 or 1. Memory (MEM) 302, a processing unit (PE: Processing Element) 204 that performs binarization, contraction / expansion processing, and the pixel (i, j) to be processed in this column in the contraction / expansion process, And working memories WM (i, j) 203, WM (i, j-1) 203, WM (i that hold the flags of the upper (i, j-1) and lower (i, j + 1) pixels. , j + 1) 203.

  In the imaging apparatus 100, the light receiving circuit (PIX) 301 is an example of “a plurality of light receiving circuits that are arranged in a matrix and each of which is a pixel that converts an amount of received light into an electrical quantity”. The unit 103 is an example of “a processing unit provided for each column of the matrix”. In addition, the working memory WM (i, j) 203, WM (i, j-1) 203, WM (i, j + 1) 203 is “in the direction of at least a part of the columns held in the memory. It is an example of a “working memory for holding a plurality of continuous values”, and the processing unit (PE) 204 uses the values held in the working memory to perform predetermined image processing. It is an example of a “processing circuit”.

  Note that PIX 301 and MEM 302 in each column are as many as the number of rows in the pixel plane, and PE 204 is a work memory WM (i−1, j) 205, WM (i +) that holds flags of pixels to be processed in both adjacent columns. 1, j) 205 is also connected. The processing unit (PE) 204 also generates a projection line necessary for detecting an erase point and generating coordinates of the point. Details of each part will be described below.

  FIG. 4 is a diagram illustrating an example of the light receiving circuit (PIX) 301. The light receiving circuit (PIX) 301 has a general 3Tr APS (Active Pixel Sensor) structure, and includes a light receiving element PD, a capacitor (not shown) for accumulating charges generated in the light receiving element PD, and the capacitor. A reset transistor Tr1 for resetting to a constant voltage, an amplification transistor Tr2 for amplifying the charge accumulated in the capacitor, and a row selection transistor Tr3 for outputting the amplified signal to the column signal line D are provided. The light receiving circuit 301 outputs a voltage corresponding to the luminance to the column signal line D when selected by the control signal RSEL.

  The memory (MEM) 302 stores a 1-bit flag. However, since it is necessary to perform reading and writing in the same cycle when connected to the processing unit (PE) 204 described later, a two-port SRAM (static RAM) And

  Three working memories (WM) 203 are arranged per column. The working memory WM (i, j) 203, WM (i, j-1) 203, WM (i, j + 1) 203 is "in the column direction with the pixel to be processed in each column of the matrix as the center. It has an example of a memory area that holds three consecutive values ”, and the breakdown is that before the transition process of the target pixel for which the processing unit (PE) 204 performs a later-described transition process for each row in order from the top. WM (i, j) 203 holding the flag of the above, WM (i, j-1) 203 holding the flag of the pixel above it, and WM (i, j + 1) holding the flag of the pixel below it ) 203. When the processing target of the PE 204 moves to the next line, the values of WM (i, j−1) to WM (i, j + 1) are sequentially shifted by one, and the current WM (i, j−1) ) Is the previous WM (i, j) value, and the current WM (i, j) value is the previous WM (i, j + 1) value. The current value of WM (i, j + 1) is newly read from the memory (MEM). Since the value of the flag of the pixel to be processed by the processing unit (PE) 204 is only the pixel to be processed and the upper and lower three pixels, the work memory 203 is configured in this way as a 3-bit shift register. If you take it.

  FIG. 5 is a diagram showing a more detailed configuration of the processing unit (PE) 204. As shown in FIG. 5, the processing unit (PE) 204 includes the following four elements.

1. Flag generation circuit (FG) 403
2. Memory access control circuit (MC) 401
3. Automaton circuit (A) 404 for expanding and contracting
4). Projection line generation circuit (PG) 402 for X axis and Y axis

  1. The flag generation circuit (FG) 403 compares the output voltage of the light receiving circuit (PIX) 301 selected in order with the threshold value Vref by the voltage comparator in the processing steps (1) and (3) described above. A flag indicating the Purkinje image 201 is generated. The generated flags are sequentially stored in the corresponding memory (MEM) 302.

  2. The memory access control circuit (MC) 401 in the claims shifts the pixels to be processed in each column in the row direction every clock so that the pixels in the same row in each column are processed in parallel. A value held in each of the memories corresponding to three pixels continuous in the column direction with the target pixel as a center is read out, written in the working memory corresponding to each of the memories, and the three consecutive in the column direction It is an example of a “memory controller that writes the result of image processing by the image processing circuit to the memory corresponding to the pixel located at the center of the pixel”. The memory access control circuit (MC) 401 sequentially sets the flag of the memory (MEM) 302 corresponding to the pixel one pixel above in the row to be processed in the processing steps (2), (4), and (5) described above. 2. read out and store in work memory WM (i, j-1) 203; The processing result T by the automaton circuit (A) 404 is controlled to be written in the memory (MEM) 302 corresponding to the pixel PIX (i, j) to be processed.

  3. The automaton circuit (A) 404 of “the image processing is performed in parallel on the pixels in the same row, and the values of three pixels consecutive in the column direction held in each of the working memories and adjacent to both sides of the column” 2 is an example of “the image processing circuit in each column that performs image processing using values held in a working memory corresponding to each processing target pixel in the column to be processed”. The automaton circuit (A) 404 includes a pixel PIX (i, j) to be processed and four neighboring pixels PIX (i, j-1), PIX (i, j + 1), PIX (i-1, j). The result T of the expansion process or the contraction process is obtained using the value of the working memory WM storing the flag of PIX (i + 1, j). For example, in the contraction process for the pupil, the automaton circuit (A) 404 performs a process of setting the center to “1” when all of the center pixel and the vicinity of 4 are “1”. In other words, if at least one of the center and four neighborhoods includes “0”, the center is set to “0”. The transition result T is the value of the central flag obtained by such processing. FIG. 6 is a diagram illustrating an example of a transition of contraction processing by the automaton circuit (A) illustrated in FIG. FIG. 6A is a diagram illustrating an example of transition in the contraction process, for example. In FIG. 6A, the value binarized by the flag generation circuit (FG) is written in the memory (MEM) at an appropriate address. For example, a flag “1” is written at the position of the pupil, and a flag “0” is written at the other positions. In the figure, when a pixel indicated by diagonal lines is the position of the pupil, a flag “1” is written in the memory (MEM) corresponding to the pixel. When the image of FIG. 6A has N rows and M columns, the circumference of a circle having a width of one pixel is inverted to the flag “0” in N clocks. FIG. 6B is a diagram illustrating an example in which the transition is further repeated after the transition of the contraction process illustrated in FIG. Also in FIG. 6B, as in the case of FIG. 6A, the outer periphery of the circle of 1 pixel width is inverted to the flag “0” in N clocks. In this way, in the contraction process, a circular image such as a pupil is sequentially scraped from the outer periphery with a width of one pixel (inverted to “0”) and gradually reduced. FIG. 6C is a diagram illustrating an example of a case where a circular image is sequentially reduced by a width of one pixel from the outer periphery, and the last one pixel remaining at the center of the circle remains. The pixel surrounded by a circle in the figure is the erase point (center).

  4). The projection line generation circuit (PG) 402 generates a projection image of the flag of the pixel to be processed on the X axis and the Y axis. This is to calculate the X-axis coordinate and the Y-axis coordinate of the center of the pupil (pixel “1”), that is, to detect the position “1” (X-axis coordinate and Y-axis coordinate). FIG. 7A is a diagram illustrating an example of a circuit configuration of a projection line generation circuit (PG) for the X axis. The projected image Px on the X-axis is obtained by calculating the logical sum of the outputs T of the automaton circuit (A) 404 during the processing period of all the rows for one column by the processing unit (PE) 204 as shown in FIG. It can be obtained by taking in order. FIG. 7B is a diagram illustrating an example of a circuit configuration of a projecting line generation circuit (PG) for the Y axis. Further, as shown in FIG. 7B, the projected image Py on the Y axis is the logical sum of the outputs T0, T1, T2,... TN-1 of the automaton circuit (A) 404 of the processing units (PE) 204 of all the columns. Is obtained by an OR gate connected in sequence.

  In order to perform the processing described so far in order, the selection of the light receiving circuit (PIX) 301 and the generation of the flag, the selection of the memory (MEM), the enlargement / contraction processing, the projection image of the flag to the X axis / Y axis The overall control circuit controls the detection of the erasing point and the process of generating the coordinates by the priority encoder.

  As described above, the rough estimation of the performance of the Vision Chip architecture for the line-parallel configuration for line-of-sight detection described so far will be described from the viewpoint of operation speed and resolution.

  Let N be the number of rows or columns of pixels that make up the Vision Chip. In the shrinking process that occupies most of the number of processing steps, one pixel on both sides of the circular region is shrunk by one transition of the automaton of the processing unit (PE) 204, so that a maximum of N / 2 transitions are required. Since this contraction process is necessary for both the pupil and the Purkinje image, a maximum of N transitions is required. Further, the processing of the processing unit PE for one column is sequentially performed on N rows. From this, N × N automaton A transitions are required for the contraction processing for generating the coordinates of the pupil and Purkinje image necessary for detecting the line of sight for one frame. Therefore, when the frame rate of the line-of-sight detection / tracking is F, the operation clock frequency f of the processing unit PE is obtained by the following equation.

  A frame rate of about F = 500 [Hz] is required for saccade detection / tracking (see Non-Patent Document 4). Therefore, in order to realize VGA resolution (640 × 480 pixels), N = 480 It is necessary to operate at a clock frequency of about f = F × N × N = 115 [MHz]. This operation speed is considered to be a speed that can be sufficiently realized by a CMOS process of about 0.35 μm or smaller in recent years.

  The largest of the circuits for one column is the processing unit (PE). For example, the height of the standard cell of the logic gate of the austriamicrosystems 0.35μm-opt process that can be used in CMP is about 20μm, and if this standard cell is arranged side by side to create a processing unit (PE), one line including the wiring area The width of the minute is considered to be about 25 μm, which is slightly larger than 20 μm. The widths of the memory MEM and the light receiving circuit PIX are designed in accordance with the width of the processing unit PE. Therefore, if the resolution is QVGA (320 × 240 pixels), the overall width is 25 [μm] × 240 = 6 [mm], and this resolution can be sufficiently realized. In addition, it is considered that VGA resolution (640 x 480 pixels) can be realized with the same chip area by using another generation 0.18 μm process. Thus, it is considered that a very high resolution chip can be realized as a type of Vision Chip that extracts an abstract amount of an image.

  As described above, according to the present invention, a vision chip having a high-speed line-of-sight detection function that adopts a column parallel processing configuration for high resolution and can also cope with detection and tracking of saccades (rapid eye movement). Could get. As a result of rough estimation, it is possible to realize a visual line detection vision chip of the VGA resolution class with a sufficiently realizable operation clock frequency.

  In the above embodiment, the processing for detecting the center of a circle such as a pupil or a Purkinje image by the automaton circuit (A) 404 has been described. However, this is merely an example of image processing, and the present invention is not limited thereto. Not. According to the imaging apparatus of the present invention, for example, smoothing of an image, detection of an edge, and the like can be performed using the values of the center pixel and the surrounding four pixels. For example, in the case of smoothing an image, it can be realized by setting an average value of four peripheral pixels as the value of the central pixel. In the case of edge detection, the difference between the values in the working memory corresponding to the row direction between adjacent rows is calculated, and it is determined that an edge exists in the row direction when the difference value changes abruptly. be able to. Also in the column direction, it is possible to calculate the difference between the values in the working memory corresponding to the column direction between adjacent columns, and to determine that an edge exists in the column direction when the difference value changes abruptly. . Thereby, there exists an effect that it can use for the detection of a lane, etc. as a vehicle-mounted camera.

(Embodiment 2)
In the first embodiment, the imaging device for the user interface using the saccade has been described. In the second embodiment, the imaging device is mounted on the in-vehicle camera and detects an obstacle crossing the traveling direction of the vehicle. The apparatus will be described. Conventionally, images captured by an in-vehicle camera are processed by software. However, since the frame rate and processing speed are limited in the conventional method, there is a problem that an obstacle can be detected only when the vehicle is traveling at a low speed.

  FIG. 8 is a diagram illustrating a circuit configuration for one column of the imaging apparatus 800 which is a column parallel processing architecture having an optical flow calculation function.

  The imaging apparatus 800 includes addition memories (DM) 801 to 809 for each direction, a light receiving circuit 810, a selector (SEL) 811, a processing circuit 812, a selector (SEL) 813, a frame memory (MEM) 814, and a frame memory (MEM) 815. , A block memory (BM) 816, a block memory (BM) 820, a selector (SEL) 830, and a selector (SEL) 831. The block memory 820 includes block memories 821 to 829 each storing a pixel value for one pixel. However, the block memory 821 to 823 is a block memory for one column provided in the left adjacent column, and is a memory resource shared by the column of interest and the left adjacent column. The block memories 827 to 829 are block memories for one column provided in the right adjacent column, and are memory resources shared by the column of interest and the right adjacent column.

  The light receiving circuit 810 includes a number of PDs corresponding to each pixel in the row, and each PD converts the received light into an electric quantity representing a pixel value of one pixel.

  The selector 813 selects a writing line of the frame memories 814 and 815 so that the pixel value of each PD received by the light receiving circuit 810 is written to the frame memory MEM in association with each light receiving circuit PD for each frame. In the frame memory 814 and the frame memory 815, new image data is alternately written for each frame. That is, the selector 813 writes (t−1) -th frame when the (t−1) -th frame is written in the frame memory 814 and the t-th frame is written in the frame memory 815 and the processing between the frames is completed. Control is performed to write the (t + 1) -th frame, which is the next frame, in the frame memory 814 in which the) -th frame has been written. When the (t + 1) -th frame is written in the frame memory 814 when the t-th frame is written in the frame memory 815, the selector 830 causes the pixel value of the t-th frame from the frame memory 815 to be changed to the block memory 816. The readout line of the frame memory 815 is selected so that it can be read out. Further, when the (t + 1) -th frame is written in the frame memory 814, the selector 831 reads the pixel value of the (t + 1) -th frame from the frame memory 814 into the block memories 824 to 826. The readout line is selected.

  The frame memories 814 and 815 are an example of “the memory having a storage capacity for holding the value of the electric quantity in the current frame and the immediately preceding frame including the plurality of light receiving circuits”.

  The frame memory 814 includes a memory MEM equal to the number of rows for storing pixel values in each pixel in the row, and stores pixel values for one column of the (t−1) th frame which is the immediately preceding frame. .

  The frame memory 815 includes the same number of memories MEM as the frame memory 814, and stores pixel values for one column of the t-th frame that is the current frame.

  The block memory 816 stores, for example, the pixel value of one pixel at the center of a 3 × 3 block read from one MEM of the frame memory 814.

  The block memory 820 includes, for example, 3 × 3 block memories BM821 to 829 that store pixel values of a 3 × 3 block of the t-th frame. Among these, the block memories 824 to 826 store pixel values for three rows in the t-th frame of the column received by the light receiving circuit 810. The block memory 825 stores the pixel value at the same position as the pixel in which the pixel value is stored in the block memory 816 in the t-th frame. The block memories 821 to 823 store pixel values in the block memories 824 to 826 among t-th frames stored in the left adjacent column frame memory MEM (not shown). The pixel value of the pixel adjacent to the left of each pixel is stored. In the block memories 827 to 829, pixel values are stored in the block memories 824 to 826 among the t-th frames in the right adjacent column stored in the frame memory MEM in the right adjacent column (not shown). The pixel value of the pixel right next to each pixel being stored is stored.

  That is, the block memory 821 is the pixel of the pixel located at the upper left with respect to the pixel in which the block memory 825 stores the pixel value (= the central pixel of the 3 × 3 block) in the t-th frame. Store the value. The block memory 822 stores the pixel value of the pixel located to the left of the center pixel of the 3 × 3 block in the t th frame. The block memory 823 stores the pixel value of the pixel located in the lower left with respect to the center pixel of the 3 × 3 block in the t-th frame.

  The block memory 824 stores the pixel value of the pixel located directly above the center pixel of the 3 × 3 block. The block memory 825 stores the pixel value of the center pixel of the 3 × 3 block in the t-th frame. The block memory BM826 stores the pixel value of the pixel located directly below the center pixel of the 3 × 3 block in the t-th frame.

  That is, the block memory 827 stores the pixel value of the pixel located in the upper right with respect to the center pixel of the 3 × 3 block in the t-th frame. The block memory BM828 stores the pixel value of the pixel located to the right of the center pixel of the 3 × 3 block in the t-th frame. The block memory 829 stores the pixel value of the pixel located at the lower right with respect to the center pixel of the 3 × 3 block in the t-th frame.

  The processing circuit 812 is “the image processing circuit that calculates the optical flow by the block matching method from the electric quantity values in the current and immediately preceding frames while using the direction-specific addition memory as a working memory” and “optical As the direction indicating the motion of the image, block matching is performed by using blocks composed of N rows and N columns (where N is a natural number of 3 or more) in the 8 or 9 directions indicating the flow direction. This is an example of the image processing circuit that specifies a direction or one of the nine directions and outputs a value indicating the specified direction. The processing circuit 812 is provided for each column of the light receiving circuit 810, and refers to the values obtained from the frame memory and block memory of the adjacent column, and the optical flow calculation of the pixel composed of the central pixel and the adjacent 8 pixels. I do.

  The selector 811 selects a row of the direction-specific addition memories 801 to 809 for adding the absolute difference values inputted in parallel from the processing circuit 812 by 9 lines.

  The direction-specific addition memories 801 to 809 are examples of “direction-wise addition memories for holding temporary values for calculating an optical flow indicating the motion of an image in the current and previous frames”. In order to perform a search in nine directions including the stationary position by parallel processing, nine lines of addition memory are provided for three lines per line. The addition memory in each row stores the sum of absolute difference values for the directions of the upper and lower blocks. In addition, the direction-specific addition memory 801 calculates a difference between the pixel value of the immediately preceding frame in which the pixel value is stored in the block memory 816 and the pixel value of the pixel at the same position as the pixel of the immediately preceding frame in the current frame. A memory having a storage capacity for storing absolute values and sequentially adding them. The direction-specific addition memory 802 in each row is a memory having a storage capacity for adding the absolute value of the difference between the pixel value of the pixel in the immediately previous frame and the pixel value of the current frame pixel in the right direction.

  As described above, by combining the image sensor and the processing circuit, high-resolution and high-speed image processing can be realized, so even when both the vehicle and the obstacle are moving at high speed, Obstacles can be detected.

  FIG. 9 is a diagram for calculating a frame rate of the imaging apparatus 800 necessary for detecting an obstacle so that the vehicle and the obstacle can be stopped before a collision when both the vehicle and the obstacle are moving at a predetermined speed. It is.

  For example, a situation is assumed in which a vehicle traveling at 60 km / h detects a bicycle crossing 40 m ahead at 25 km / h. In this case, the stop distance required for the vehicle to stop after detecting the bicycle is the idling distance 17 m + braking distance 27 m = 44 m. In this case, the time until the vehicle stops is 2.64 seconds, and a 25 km / h bicycle travels about 20 meters in 2.64 seconds until the vehicle stops. Assuming that the shooting range of the in-vehicle camera is 20 m wide at 44 m ahead of the vehicle, the frame rate required for the imaging apparatus 800 is 200 fps (frame per second).

  FIG. 10 is an explanatory diagram for calculating a block size and a frame rate necessary for the optical flow calculation of the imaging apparatus 800.

  A captured image of the imaging apparatus 800 is considered by VGA (640 × 480 pixels). As shown in the figure, the size of one pixel is 20 m ÷ 640 = 31.25 (mm / pixel). The size of a 3 × 3 pixel block is 94 (mm) × 94 (mm).

  In the imaging apparatus 800, when the frame rate is 200 fps, scanning is performed once every 5 ms. In addition, the bicycle travels 34.7 mm in 5 ms. Accordingly, when the frame rate is 200 fps, the bicycle moves about one pixel screen in one scan. Therefore, when the frame rate is 200 fps, it is only necessary to search for a block shifted by one pixel in the surrounding 9 directions including stillness and in the surrounding 8 directions if stillness is not included.

  Conventionally, gradient methods and block matching methods are known as motion detection methods. However, the gradient method does not need to search for corresponding points, so the amount of calculation is small. However, there are disadvantages in that the optical flow error is large and the noise is weak when the luminance value changes rapidly. .

  On the other hand, the block matching method can be reduced to some extent if the search range is limited, but has a problem that the calculation time is enormous and it is vulnerable to enlargement / reduction / rotation. However, the block matching method has a merit that the calculation is simple, the optical flow error is small even in a place where the luminance value changes rapidly, and it is resistant to noise.

Therefore, in the present invention, the optical flow calculation is performed using the block matching method.
FIG. 11 is a diagram for explaining the optical flow. The optical flow is a vector that represents the movement of the density distribution between successive frames in a moving image. As shown in FIG. 11 (a), it indicates to which position in the current frame a block that was in an arbitrary position in the previous frame has moved. Specifically, the vector indicates how much the block in the previous frame has moved in which direction from the original position on the current frame. For example, in FIG. 11B, the moving direction and the moving amount to the center of the block of the current frame are represented by vectors, starting from the center of the block of the previous frame.

  FIG. 12 is a diagram for explaining an optical flow calculation method. Here, an optical flow calculation method using the block matching method will be described. FIG. 12A shows the (t−1) th frame image. FIG. 12B shows the t-th frame image. In the block matching method, an arbitrary block in the (t−1) th image that is the previous frame of the tth frame is used as a reference block. Then, as shown in FIG. 12B, in the t-th frame, which is the current frame, a method of searching for a block indicating the same pattern image is performed. If a block showing the same image as the block of the previous frame is found in the current frame, the movement amount can be expressed as a vector, assuming that the block of the previous frame has moved to the position found in the current frame.

  FIG. 13 is a diagram for explaining the block matching method of the present embodiment. In the second embodiment, assuming that the target moves only one pixel in one frame, as shown in FIG. 13 (a), a range that is shifted by one pixel from the reference block in each of the eight surrounding directions is searched. And In FIG. 13A, a block of 3 × 3 pixels surrounded by a broken line is used as a reference block, and the search range of the block in the current frame is shifted by 1 pixel in each of the surrounding 8 directions from the reference block in the previous frame. The range of the block 1201 is searched.

を算出する。この輝度の差分の総和を、図１３（ａ）の参照ブロック１２０２を中心として、例えば、まず、一画素左上に位置する斜線のハッチングを施したブロック１２０３と参照ブロック１２０２とで、上記輝度差分の絶対値の総和を算出する。次に、ブロック１２０３を１画素だけ右にずらした位置、すなわち、参照ブロック１２０２の直上にある３行３列のブロックと差分の絶対値の総和を算出する。さらに、ｔ番目のフレームのブロック１２０３をさらに１画素右にずらしたブロックと参照ブロック１２０２との差分絶対値の総和を算出する。次に、参照ブロック１２０２の左に位置するｔ番目のフレームの３行３列のブロック１２０３と参照ブロック１２０２との差分絶対値の総和を算出する。さらに、参照ブロック１２０２とｔ番目のフレームで同じ位置にある３行３列のブロック１２０３との差分絶対値の総和を算出する。同様にして、つぎには、ｔ番目のフレームで参照ブロック１２０２の１画素だけ右に位置する３行３列のブロック１２０３と参照ブロック１２０２との差分絶対値の総和を算出し、さらに、ｔ番目のフレームで参照ブロック１２０２の左下に位置するブロックとの差分絶対値の総和、ｔ番目のフレームで参照ブロック１２０２の真下に位置するブロックとの差分絶対値の総和、ｔ番目のフレームで参照ブロック１２０２の右下に位置するブロック１２０３との差分絶対値の総和を算出する。 FIG. 13B is a diagram showing a reference block 1202 in the (t−1) th frame and a block 1203 having the same size within the search range of the tth frame. In order to determine the position where the reference block has moved within the search range of the t-th frame, the absolute value of the difference in luminance between pixels at the same position in a block of the same size as the reference block is determined. Take the sum. That is, in the reference block 1202 in FIG. 13B and the block 1203 in the t-th frame, the pixels at the same position, that is, the pixel A1 and the pixel B1, the pixel A2 and the pixel B2, the pixel A3 and the pixel B3, ..., the sum of absolute values of differences in luminance between the pixels A9 and B9

Is calculated. For example, the sum of the differences in luminance is centered on the reference block 1202 in FIG. 13A. For example, the block 1203 and the reference block 1202 that are hatched in the upper left corner of one pixel and the reference block 1202 Calculate the sum of absolute values. Next, the position where the block 1203 is shifted to the right by one pixel, that is, the 3 × 3 block immediately above the reference block 1202 and the sum of absolute values of the differences are calculated. Further, the sum of absolute differences between the block in which the block 1203 of the t-th frame is further shifted to the right by one pixel and the reference block 1202 is calculated. Next, the sum of absolute differences between the block 1203 of 3 rows and 3 columns of the t-th frame located on the left of the reference block 1202 and the reference block 1202 is calculated. Further, the sum of absolute differences between the reference block 1202 and the block 1203 of 3 rows and 3 columns at the same position in the t-th frame is calculated. Similarly, next, the sum of absolute differences between the reference block 1202 and the 3 × 3 block 1203 positioned to the right by one pixel of the reference block 1202 in the t th frame is calculated. Of the difference absolute value with the block located in the lower left of the reference block 1202 in the frame of, the sum of absolute difference values with the block located directly under the reference block 1202 in the t th frame, and the reference block 1202 in the t th frame. The sum of absolute differences from the block 1203 located at the lower right of is calculated.

  In the block matching method, it is determined that the reference frame 1202 has moved to the block having the minimum sum among the nine sums of absolute differences calculated by the above nine calculations. Then, the direction from the reference block 1202 to the block having the minimum sum of absolute values of luminance differences is defined as a motion vector.

  The above calculation is performed by column parallel processing in the imaging apparatus 800 of the present invention. That is, as shown in FIG. 8, one processing circuit 812 is provided for each column of pixels, so that block matching processing is performed for each column. Then, processing is performed in order from the top in line units.

  In this case, the processing speed is slightly slower than the pixel parallel processing compared to the “pixel parallel processing” that includes a processing circuit for each pixel and is performed on a pixel basis. However, the pixel parallel processing increases the size of the pixel and the processing circuit. On the other hand, in the column parallel processing, since the imaging unit and the processing circuit can be separated, there is an advantage that the processing circuit does not become so large even if the number of pixels is increased.

  FIG. 14 is a diagram illustrating an example of a block matching procedure for column parallel processing. FIG. 14A shows one pixel constituting the reference block of the (t−1) -th frame on the left side of the figure, and is located on the right side of the figure at a position shifted by one pixel from the pixel position in the surrounding eight directions. A block on the t-th frame including pixels is shown. FIG. 14B shows one pixel one row below the pixel of the (t−1) -th frame shown on the left of FIG. 14A, and one pixel at a time in the eight directions from the pixel position on the right of FIG. A block on the t-th frame configured to include a pixel at a shifted position is shown. FIG. 14C shows one pixel further down one row from the pixel in the (t−1) -th frame shown on the left in FIG. 14B, and one pixel on the right in the figure from the pixel position to the surrounding eight directions. A block on the t-th frame configured to include pixels at different positions is shown. FIG. 14 (d) shows the processing circuit in the center column in the order of FIGS. 14 (a), (b) and (c), with the pixels on the (t-1) th frame and the left side of each figure. In other words, the sum of absolute differences of pixel values with respect to each pixel in the block is calculated.

  As shown in FIG. 14D, the column on which the block matching calculation process is performed is the processing circuit 812 in the center column. Although not illustrated in the figure, it is assumed that the same operation is simultaneously executed in the processing circuits provided in all adjacent columns in parallel with this column. The direction-specific addition memories 801 to 809 shown in FIG. 8 are for three rows, and the direction-specific addition memories 801 to 809 in the first row are different from each other with respect to the pixels in FIG. Store absolute value. The 9-direction addition memories 801 to 809 in the second row each store a difference absolute value for each direction with respect to the pixel in FIG. The 9-direction addition memories 801 to 809 in the third row each store the difference absolute value for each direction with respect to the pixel in FIG.

  The direction-specific difference absolute values of the direction-specific addition memories 801 to 809 in the right adjacent column and the left adjacent column are added to the three-row direction addition memories 801 to 809 stored therein, and the direction-specific addition memories 801 to 809 are added. By calculating the direction in which the sum of the absolute values of the differences becomes the minimum, block matching of the 3 × 3 block can be performed.

  As shown in FIG. 14A, the processing circuit 812 first calculates the sum of the absolute differences of the pixel values of the pixels shown on the left of the figure and the nine pixels constituting the block shown on the right of the figure. To do. The pixel values of the pixels in the left adjacent column and the right adjacent column in FIGS. 14A, 14 </ b> B, and 14 </ b> C are all shared with the adjacent column.

Assuming that the number of operations for one column executed by each processing circuit 812 is the number of pixels X × Y and the block size N × N,
Number of absolute difference calculations N ² × Y times Number of additions to the direction-specific addition memories 801 to 809 9N ² × Y times Number of times to calculate the minimum value Y times. On the other hand, the operable clock frequency f [Hz] is F = fp × [{N ² × Y} + {9N ² × Y} + Y] with F [fps] as the frame rate.
= F x Y x 10N ² [Hz]
It becomes. Here, for example, when F = 200 [fps], Y = 480, and N = 3,
f = 200 × 480 × 10 × 3 ²
≒ 8 [MHz]
Thus, it can be seen that it can be realized with a general CMOS circuit.

  In the above description, the sum of absolute values of pixel value differences between pixels that are stationary between frames is also calculated. However, the difference between the pixel values of pixels at the same position in the previous frame and the current frame is also calculated. The calculation of the absolute value may be omitted. In this case, only eight direction-specific addition memories are required and only eight search directions are calculated, so that both the calculation amount and the memory resources can be saved.

When the matching of the pixels at the stationary position is omitted, the number of operations for one column executed by each processing circuit 812 is the number of pixels X × Y and the block size N × N.
Number of absolute difference calculations (N ² −1) × Y times Number of additions to the direction-specific addition memories 801 to 809 8N ² × Y times Number of times to calculate the minimum value Y times. Here, when calculating the minimum value, by using seven binary comparators, it is possible to obtain the minimum value of the sum of the absolute difference values in the eight directions at once in one clock. The operable clock frequency f [Hz] is F = fp × [{(N ² −1) × Y} + {8N ² × Y} + Y] with F [fps] as the frame rate.
= F x Y x 9N ² [Hz]
It becomes. Here, for example, when F = 200 [fps], Y = 480, and N = 3, the clock frequency f [Hz] is
f = 200 × 480 × 9 × 3 ²
≒ 7 [MHz]
In fact, a general CMOS circuit can be sufficiently realized.

  In addition, here, a block consisting of pixels of 3 rows and 3 columns is used as a search unit. However, in order to further improve the search accuracy, for example, block matching is performed using a block consisting of pixels of 5 rows and 5 columns as a search unit. It is good. In this case, the block memory increases by one row on the left and right, but the processing accuracy is increased accordingly. In addition, block matching may be performed using a block including more pixels, for example, a block including N rows and N columns (where N is a natural number) as a search unit.

のように、出力する。 FIG. 15 is a diagram illustrating an example of a simulation using an actual image. In the figure, a grayscale VGA still image is used as a background in order to check whether or not motion detection is actually possible with the imaging apparatus 800. The direction of movement for each block is output as follows. For example, if you move right below,

To output.

  In the figure, the above output was obtained at the upper and lower boundaries of the rectangular parallelepiped indicated by hatching, and the downward movement was confirmed. In this case, the rectangular parallelepiped indicated by hatching is black in the actual image. In such a case, since the background motion is not detected, it can be detected even if the gradation is low.

直方体の右方向への移動が確認できた。この例では、右斜め上下の移動方向を示す動きが出力された場合もある。この理由は、斜めに移動した時のブロックの方が、参照ブロックと似ているということもありうるからである。また、動きを検出する対象物の輝度に階調がある場合には、階調数１６（４ｂｉｔ）以下では検出できないことが確認された。 FIG. 16 is a diagram illustrating another example of a simulation using an actual image. In the figure, a rectangular parallelepiped indicated by hatching is gray. Even in this case, the movement is output as follows at the left and right boundaries of the rectangular parallelepiped,

The movement of the cuboid to the right was confirmed. In this example, there is a case in which a motion indicating a moving direction that is diagonally right and up is output. This is because the block when moved obliquely may be more similar to the reference block. Further, it was confirmed that when there is a gradation in the luminance of the object for detecting the motion, it cannot be detected with the gradation number of 16 (4 bits) or less.

  In the second embodiment, control is performed so that image data of new frames are alternately written in the frame memory 814 and the frame memory 815, but the present invention is not limited to this. For example, the frame memory 814 may be controlled so that the frame immediately before the frame written in the frame memory 815 is always written. In this case, the selector 830 and the selector 831 are not necessary, the pixel values of the frame memory 814 are always stored in the block memory 816, and the pixel values read from the frame memory 815 are always stored in the block memories 824 to 826. You may make it memorize | store.

  Note that the blocks in the block diagrams (FIGS. 1, 3, 5, and 8) are typically realized as LSIs that are integrated circuits, and may be individually integrated into one chip, or part or all of them. It may be made into one chip so as to include it. For example, the functional blocks other than the memory may be integrated into one chip.

  The present invention can be applied to a user interface using a saccade. The present invention can also be applied to an imaging apparatus that performs high-speed and high-resolution image smoothing and edge detection on a captured image. Furthermore, the present invention can be applied to an on-vehicle moving obstacle detection device.

It is a block diagram which shows the structure of the imaging device of this Embodiment 1 which takes what is called a column parallel structure which arrange | positions an image processing circuit for every column. (A) is a figure which shows an example of the image which irradiated the infrared light to the eyeball and image | photographed with the infrared camera. (B) is a figure which shows the method of calculating | requiring a line of sight from the positional relationship of a pupil and a Purkinje image. It is a figure which shows the structure for 1 row | line of the imaging device which is a column parallel processing architecture for a visual line detection. FIG. 2 is a block diagram illustrating a circuit configuration of a light receiving circuit (PIX) illustrated in FIG. 1. It is a figure which shows the more detailed structure of the process part (PE) shown in FIG. It is a figure which shows an example of the transition of the contraction process by the automaton circuit (A) shown in FIG. (A) is a figure which shows an example of the circuit structure of the projection line production | generation circuit (PG) to an X-axis. (B) is a figure which shows an example of a circuit structure of the projection line production | generation circuit (PG) to a Y-axis. It is a figure which shows the circuit structure for 1 row of an imaging device provided with the optical flow calculation function of this Embodiment 2. FIG. It is a figure for calculating the frame rate of an imaging device required in order to detect an obstacle so that it can stop before a collision when both vehicles and an obstacle are moving at a predetermined speed. It is explanatory drawing for calculating a block size and a frame rate required for optical flow calculation of an imaging device. (A) is a figure which shows to which position of the present frame the block which was in arbitrary positions in the front frame moved. (B) is a diagram showing the moving direction and moving amount from the center of the block of the previous frame to the center of the block of the current frame as a vector. (A) is a figure which shows the (t-1) th frame image. (B) is a diagram showing a t-th frame image. (A) is a figure which shows searching the range which shifted 1 pixel in the surrounding 8 directions from the reference block, respectively. (B) is a diagram showing a reference block 1202 in the (t−1) th frame and a block 1203 of the same size within the search range of the tth frame. (A) The left shows one pixel constituting the reference block of the (t-1) th frame, and the right side of the figure includes pixels located at positions shifted by one pixel in the surrounding eight directions from the pixel position. It is a figure which shows the block on the t-th frame. FIG. 14B shows one pixel one row below the pixel of the (t−1) th frame shown on the left in FIG. 14A, and is shifted by one pixel from the pixel position to the surrounding eight directions on the right of FIG. It is a figure which shows the block on the t-th frame comprised including the pixel in a position. (C) shows one pixel further down one row from the pixel of the (t-1) th frame shown on the left in FIG. 14B, and is shifted by one pixel from the pixel position to the surrounding eight directions on the right in the figure. It is a figure which shows the block on the t-th flame | frame comprised including the pixel in the specified position. (D) shows the processing circuit in the center column in the order of FIGS. 14 (a), (b) and (c), the pixels on the (t-1) th frame, and the blocks shown on the left of each figure. It is a figure which shows calculating the sum total of the difference absolute value of a pixel value with each pixel of the inside. It is a figure which shows an example of the simulation using an actual image. It is a figure which shows the other example of the simulation using an actual image.

Explanation of symbols

101 Light receiving circuit (PIX)
102 Memory (MEM)
103 processing unit 201 Purkinje image 202 pupil 203 working memory WM (i, j), WM (i, j-1), WM (i, j + 1)
204 Processing unit (PE)
205 Working memory WM (i-1, j), WM (i + 1, j)
301 Light receiving circuit (PIX)
302 Memory (MEM)
401 Memory access control circuit (MC)
402 Projection line generation circuit (PG)
403 Flag generation circuit (FG)
404 Automaton circuit (A)
800 Imaging device 801-809 Addition memory for each direction (DM)
810 Light receiving circuit 811, 813, 830, 831 Selector (SEL)
812 Processing circuit 814, 815 Frame memory (MEM)
816, 820 block memory (BM)

Claims (7)

An imaging device having an image processing function,
A plurality of light receiving circuits arranged in a matrix and each pixel being a pixel that converts the amount of received light into an electrical quantity;
A memory that holds the value of the electric quantity obtained by each of the light receiving circuits in association with each of the light receiving circuits;
A processing unit provided for each column of the matrix,
The processor is
A working memory for holding a plurality of values that are continuous in at least some of the column directions held in the memory;
An image processing apparatus comprising: an image processing circuit that performs predetermined image processing using a value held in the working memory. The working memory has a memory area that holds three values that are continuous in the column direction around the pixel to be processed in each column of the matrix,
The imaging device further includes:
The pixels to be processed in each column are shifted in the row direction at each clock so that pixels in the same row are processed in parallel in each column, and three pixels that are continuous in the column direction centering on the pixel to be processed are supported. A memory controller that reads a value held in each memory and writes the value to the working memory corresponding to each memory;
The image processing circuit in each column performs image processing on the pixels in the same row in parallel, and is adjacent to both sides of the column and the value of three pixels continuous in the column direction held in each working memory. Image processing is performed using values held in the working memory corresponding to each processing target pixel in the column,
The imaging apparatus according to claim 1, wherein the memory controller writes an image processing result by the image processing circuit in the memory corresponding to a pixel located at a center of the three pixels continuous in a column direction. The image processing circuit includes:
A flag generation unit that generates a flag indicating a comparison result by comparing the value of the amount of electricity obtained by the light receiving circuit with a predetermined value;
An automaton circuit for obtaining a transition of an image enlargement or reduction process using the flag;
The imaging apparatus according to claim 2, further comprising: a projection line generation circuit that calculates an X-axis coordinate and a Y-axis coordinate in the light receiving circuit of the image using a flag obtained by the automaton circuit. The flag generation unit generates the flag by comparing the value of the electric quantity obtained by the light receiving circuit with a predetermined value for specifying a pupil or a Purkinje image,
The imaging apparatus according to claim 3, wherein the automaton circuit obtains a transition of contraction processing until only a flag corresponding to one pixel indicating a center position of the pupil or a Purkinje image becomes 1 or 0. . The memory has a storage capacity for holding the value of the amount of electricity in the current frame and the immediately preceding frame composed of the plurality of light receiving circuits,
The imaging apparatus further includes a direction-specific addition memory for holding a temporary value for calculating an optical flow indicating the movement of the image in the current and previous frames,
The image processing circuit calculates an optical flow by a block matching method from values of electricity in the current and immediately preceding frames while using the direction-specific addition memory as a working memory. The imaging device described. The image processing circuit performs block matching with a block composed of pixels of N rows and N columns (where N is a natural number of 3 or more) in 8 directions or 9 directions indicating the direction of optical flow, thereby moving the image The imaging apparatus according to claim 5, wherein one of the eight directions or the nine directions is specified as a direction indicating, and a value indicating the specified direction is output. An integrated circuit for mounting an imaging device having an image processing function,
A plurality of light receiving circuits arranged in a matrix and each pixel being a pixel that converts the amount of received light into an electrical quantity;
A memory that holds the value of the electric quantity obtained by each of the light receiving circuits in association with each of the light receiving circuits;
A processing unit provided for each column of the matrix,
The processor is
A working memory for holding a plurality of values that are continuous in at least some of the column directions held in the memory;
An integrated circuit comprising: an image processing circuit that performs predetermined image processing using a value held in the working memory.

Images