JP6809565B2 - Feature extraction element, feature extraction system, and judgment device - Google Patents

Description

The present invention relates to a feature extraction element, a feature extraction system, and a determination device.

There is a method of speeding up processing by dividing an image into blocks and extracting features for each block (see, for example, Patent Document 1).
Patent Document 1 JP-A-2008-148298

Since the data captured as an image is processed and the features are extracted, it takes time to generate the image, transfer the generated image, and the like, and it cannot be said that the speed can be sufficiently increased.

In the first aspect of the present invention, a light receiving substrate in which a plurality of light receiving elements for photoelectrically converting the received light are two-dimensionally arranged, and one or a plurality of other substrates laminated on the light receiving substrate are provided. The substrate has a plurality of multiplication circuits provided for each light receiving element or for each block composed of a plurality of light receiving elements, and the signals output from the plurality of light receiving elements are convolved using the plurality of multiplication circuits. Features include a convolution processing unit, a pooling processing unit that samples the signal output from the convolution processing unit based on predetermined conditions, and a connection wiring for passing the sampled signal to a plurality of multiplication circuits. An extraction element is provided.

In the second aspect of the present invention, the feature extraction element, the convolution calculation of the convolution processing unit, and the control unit that controls the sampling of the pooling processing unit to be repeated are provided, and the control unit repeatedly performs the convolution calculation. At that time, a feature extraction system for controlling the convolution processing unit so as to use a predetermined filter coefficient is provided.

In the third aspect of the present invention, a determination device including the above-mentioned feature extraction element and a determination unit for determining an imaging target based on a feature amount extracted based on an output from a pooling processing unit is provided.

The above outline of the invention does not list all the features of the present invention. Sub-combinations of these feature groups can also be inventions.

It is a schematic sectional view of the feature extraction apparatus 100. It is a flowchart of the process executed by the feature extraction apparatus 100. It is a schematic sectional view of the feature extraction apparatus 100. It is a schematic sectional view of the feature extraction apparatus 100. It is a timing chart of the feature extraction apparatus 100. It is a block diagram of the image pickup apparatus 500. It is a schematic sectional view of another feature extraction apparatus 101. It is a schematic sectional view of the feature extraction apparatus 101. It is a partial timing chart of the feature extraction device 101.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the inventions claimed in the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention.

FIG. 1 is a schematic cross-sectional view of the feature extraction device 100. The feature extraction device 100 includes a microlens 110, a pixel board 120, an AD conversion circuit board 130, a multiplication circuit board 140, an addition circuit board 150, a convolution calculation result addition board 160, an activation function calculation circuit board 170, and a pooling. It is a single element including a circuit board 180 and a coupling circuit board 190.

The feature extraction device 100 extracts a feature amount used when determining a subject from the received light flux of the subject. The determination of the subject includes, for example, determining what the subject is, determining the category of the subject, determining the degree of the subject (for example, the evening scene), and the like, but is not limited to these. The features extracted here correspond to the features extracted by a multi-layer neural network called deep learning or deep learning. In other words, the feature extraction device 100 can also be used for a learning process for feature extraction by deep learning.

Each of the substrates in the feature extraction device 100 has a wiring layer 12 formed by photolithography on the base substrate 11. Each wiring layer 12 includes a circuit formed by wiring, elements, and the like. That is, the pixel board 120, the AD conversion circuit board 130, the multiplication circuit board 140, the addition circuit board 150, the convolution calculation result addition board 160, the activation function calculation circuit board 170, the pooling circuit board 180, and the coupling circuit board 190 are respectively. It has a pixel 121, an AD conversion circuit 131, a multiplication circuit 141, an adder circuit 151, an adder circuit 161 with a latch B, an activation function calculation circuit 171, a pooling circuit 181 and a coupling circuit 191.

Further, the above-mentioned substrate is electrically connected to other substrates laminated adjacent to each other via through electrodes 13 penetrating each of the base substrates 11. Further, some of the substrates are directly electrically connected to the non-adjacent substrates by the through electrodes 14 formed through the plurality of substrates. This point will be described with reference to FIGS. 3 and 4.

In the feature extraction device 100, the pixel substrate 120 has a plurality of pixels 121 arranged two-dimensionally and periodically. Each of the pixels 121 has a light receiving element such as a photodiode that photoelectrically converts the incident light. In the illustrated example, the pixel substrate 120 is a back-illuminated type that receives incident light from the substrate side by removing or thinning the substrate that is the substrate for forming the pixel 121.

Further, the pixel substrate 120 may have individual light receiving elements having a transistor for instructing reset, transfer, and selection, and an element for amplifying an output signal. The microlens 110 laminated on the pixel substrate 120 collects the incident light on each pixel 121 to improve the incident efficiency.

The AD conversion circuit board 130 has a plurality of AD conversion circuits 131, a latch 132, and a changeover switch 133 that individually correspond to the pixels 121 of the pixel board 120. As a result, the AD conversion circuit board 130 outputs either a discretized value of each pixel value acquired from the pixel 121 of the pixel board 120 or a value held by the latch 132 to another board. The AD conversion circuit 131, the latch 132, and the changeover switch 133 operate at the timing when the timing trigger 210 received from the outside is received.

The multiplication circuit board 140 has a multiplication circuit 141 individually corresponding to each pixel 121 of the pixel board 120. That is, the multiplication circuit board 140 has a plurality of multiplication circuits 141 having the same number as the plurality of pixels 121 provided on the pixel board 120. The multiplier circuit 141 is a digital multiplier and may be formed by, for example, a shift register. The multiplication circuit 141 acquires and holds the filter coefficient when executing the multiplication process from the outside.

In other words, the multiplication circuit 141 can execute different multiplication processes depending on the value of the filter coefficient read from the outside. FIG. 1 shows a case where the filter coefficient a is acquired as an example. The multiplication circuit 141 on the multiplication circuit board 140 also operates at the timing when the timing trigger 210 received from the outside is received.

The multiplication circuit board 140 may have one multiplication circuit 141 for each block composed of a plurality of pixels 121 of the pixel board 120. For example, if four pixels adjacent to each other in the two-dimensional direction of the pixel 121 are regarded as one block, one multiplication circuit 141 connected to any of the four pixels 121 in the block may be provided. In this case, the multiplication circuit 141 sequentially performs multiplication processing on the outputs from the four pixels 121 in the block.

The adder circuit 151 of the adder circuit board 150 adds and outputs the values acquired from the plurality of multiplication circuits 141 on the multiplication circuit board 140. The output of the addition circuit 151 can be output to the convolution calculation result addition board 160 in the lower layer in the figure. The adder circuit 151 on the adder circuit board 150 operates at the timing when the timing trigger 210 received from the outside is received.

The convolution calculation result addition substrate 160 includes an addition circuit 161 with latch B, a latch A162, and a multiplexer 163. The adder circuit 161 with latch B, latch A162, and multiplexer 163 are interconnected, the adder circuit 161 with latch B adds to the activation function arithmetic circuit board 170 through the through electrode 13, and the multiplexer 163 adds through the through electrode 13. Each is connected to the circuit board 150. The convolution calculation result addition board 160 adds a plurality of signals output from the addition circuit board 150, and then outputs the signals to the activation function calculation circuit board 170.

The activation function calculation circuit board 170 has a number of activation function calculation circuits 171 corresponding to the addition circuit 151 of the addition circuit board 150. The activation function calculation circuit board 170 receives the output of the addition circuit board 150, executes the activation function calculation, and then outputs the output to the pooling circuit board 180. The activation function calculation circuit 171 on the activation function calculation circuit board 170 operates at the timing when the timing trigger 210 received from the outside is received.

The pooling circuit 181 of the pooling circuit board 180 and the coupling circuit 191 of the coupling circuit board 190 sequentially process the inputs from the previous stage. The output value of the coupling circuit 191 on the coupling circuit board 190 can be output as a feature amount to the outside of the feature extraction device 100. The pooling circuit 181 and the coupling circuit 191 also operate at the timing when the timing trigger 210 received from the outside is received.

In the feature extraction device 100, the pixel 121, the AD conversion circuit 131, the latch 132, the changeover switch 133, the multiplication circuit 141, the addition circuit 151, the addition circuit 161 with the latch B, the activation function calculation circuit 171 and the pooling circuit 181 , Each of the coupling circuits 191 and the like is controlled by a supplied timing trigger 210 from a control unit (not shown). The control unit may be included in the feature extraction device 100, or may also be used as a control unit of another device equipped with the feature extraction device 100, for example, an imaging device. A feature extraction system is configured by a feature extraction device 100 which is a single element and a control unit.

As described above, the feature extraction device 100 includes a multiplication circuit board 140, an addition circuit board 150, a convolution calculation result addition board 160, an activation function calculation circuit board 170, a pooling circuit board 180, and a coupling circuit board involved in the image feature extraction process. It has a structure in which a circuit board 190 is laminated on a pixel board 120 including a light receiving element. As a result, since the pixel value can be directly processed and the feature can be extracted, the process of converting the image into data and storing it and the process of transferring the stored image data become unnecessary, and the processing time can be shortened.

Further, since hardware resources such as a storage device and a transfer device for image data can be omitted, it also contributes to miniaturization of the device provided with the feature extraction device 100. Further, since the processing substrates corresponding to the pixels of the pixel substrate 120 are laminated, it is possible to prevent the processing speed from decreasing even if the number of pixels of the pixel substrate 120 increases.

Since the feature extraction device 100 receives the incident light by the plurality of pixels 121 arranged two-dimensionally, the two-dimensional luminance distribution information based on the image data is obtained from the pixel substrate 120. You can also get it. Therefore, the feature extraction device 100 can also be used as an image sensor.

FIG. 2 is a flow chart of a feature extraction process executed by the feature extraction device 100. As shown in the figure, in the feature extraction process, the convolution process S102, the activation function calculation process S103, the pooling process S104, and the join process S105 are executed for the pixel values generated by the pixel value generation process S101 to extract the pixel values. The feature amount is output to the outside (step S106).

Here, in the feature extraction process corresponding to deep learning, the pixel value generation process S101 and the feature amount output S106 are executed once for each feature extraction. However, the reading, multiplication processing, and addition processing of the filter function in the convolution processing S102 are repeatedly executed many times while changing the filter coefficient to be read. Further, the processing results of the activation function calculation process S103 and the pooling process S104 after the convolution process are subjected to the convolution process S102 again, and the processes from the convolution process S102 to the pooling process S104 are further repeated. When only the activation function calculation process S103 is repeated, when the convolution process S102 and the activation function calculation process S103 are repeated, or after the pixel value generation process S101, either or both of the convolution process S102 and the activation function calculation process S103 are repeated. May be omitted and the pooling process S104 may be performed.

FIG. 3 is a diagram illustrating the operation of the feature extraction device 100 shown in FIG. In the feature extraction device 100 shown in the figure, the function of the through electrode 13 connecting adjacent substrates is highlighted by a thick hatched line.

As shown by a thick line in the figure, in the feature extraction device 100, each of the pixels 1, the pixels 2 and the pixels 3 in the pixel substrate 120 is passed through the corresponding AD conversion circuit 131 in the AD conversion circuit board 130 and the changeover switch 133. Therefore, the multiplication circuit board 140 is connected to the corresponding multiplication circuit 141.

On the other hand, the multiplication circuit board 140 acquires the filter coefficients a, b, and c of the multiplication circuit 141 corresponding to each of the pixels 1 to 3. The pixel value output by pixel 1 on the pixel board 120 is subjected to multiplication processing using the filter coefficient a by the multiplication circuit 141 corresponding to pixel 1 on the multiplication circuit board 140, and then input to the addition circuit board 150 through the through electrode 13. To.

Similarly, the pixel value output by the pixel 2 on the pixel substrate 120 is multiplied by the multiplication circuit 141 corresponding to the pixel 2 on the multiplication circuit board 140 using the filter coefficient b, and then the addition circuit board 150 is passed through the through electrode 13. Is entered in. Further, the pixel value output by the pixel 3 on the pixel substrate 120 is subjected to multiplication processing by the multiplication circuit 141 corresponding to the pixel 3 on the multiplication circuit board 140 using the filter coefficient c, and then on the addition circuit board 150 through the through electrode 13. Entered.

The addition circuit 151 in the addition circuit board 150 adds a plurality of input multiplication processing results and outputs them to the convolution calculation result addition board 160. In this way, processing similar to filtering such as smoothing is executed. However, since the filter coefficient in the multiplication circuit 141 is predetermined by prior learning, a series of processes is executed as a convolution process.

The latch A162 on the convolution calculation result addition substrate 160 holds the signal output from the addition circuit 151 via the multiplexer 163. After that, multiplication processing is performed by the multiplication circuit 141 in which a filter coefficient different from the previous time is set, and when the multiplication processing result is added by the addition circuit 151, the addition circuit 161 with latch B adds the addition processing result of the addition circuit 151. It is held via a multiplexer 163.

The addition circuit 161 with latch B reads out the previous addition processing result held in the latch A162 and adds it to the new addition processing result, and the addition result is held again by the latch A162. By repeating this process a predetermined number of times, it is possible to add up a plurality of processing results calculated by convolution with different filter coefficients for the same pixel group (pixels 1 to 3). That is, this corresponds to the repetition of the convolution process S102 shown in FIG. Such a convolution operation is performed on all the pixels on the pixel substrate 120, that is, the entire input image.

In the feature extraction device 100, the processing result by the convolution calculation result addition board 160 is input to the activation function calculation circuit 171 of the activation function calculation circuit board 170 through the through electrode 13. The activation function calculation circuit 171 converts the information propagated from the convolution process to the pooling process. As such a function, as shown in Equation 1 below, a ReLU (Rectifier Line Unit) function in which all input values less than 0 are set to 0 and positive values are returned as they are can be exemplified.
f (x) = max (0, x) ... (Equation 1)

The output of the activation function calculation circuit 171 is propagated to the pooling circuit 181 of the pooling circuit board 180 through the through electrode 13. In the pooling circuit 181, subsampling is executed based on predetermined conditions. As a condition of subsampling executed here, a Max Polling method in which the maximum value in a certain window size is a representative value can be exemplified.

The predetermined condition may be a process of collecting a plurality of outputs from the activation function calculation circuit 171, and may be an Average Pooling method for outputting an average value. In this way, the convolution process (step S102), the activation function calculation process (step S103), and the pooling process (step S104) are executed, and the subsampling value generated from the pixel value is generated.

FIG. 4 is a diagram illustrating another operation of the feature extraction device 100 shown in FIG. In the illustrated feature extraction device 100, the through silicon vias involved in the operation described below are highlighted by thick hatched lines.

As shown by the thick line in the figure, the feature extraction device 100 can transmit and receive signals even between substrates that are not adjacent to each other. Therefore, for example, the output of the pooling circuit board 180 can be stored in the latch 132 of the AD conversion circuit board 130 through the through electrode 14 and subjected to the multiplication process again in the multiplication circuit 141. As a result, the sub-sampled value is convolved again. The repetition of such a convolution process is executed a predetermined number of times, for example, 2000 times or more while changing the filter coefficients a, b, and c.

Further, a signal processed up to a predetermined number of times between the AD conversion circuit board 130, the multiplication circuit board 140, the addition circuit board 150, the convolution calculation result addition board 160, the activation function calculation circuit board 170, and the pooling circuit board 180. Is made one-dimensional by the coupling circuit 191 on the coupling circuit board 190. As a result, a value indicating a certain feature amount of the image can be obtained for each component.

FIG. 5 is a timing chart showing the operation of the feature extraction device 100. The pulse in the figure means a signal supplied to each substrate as a timing trigger 210.

As shown in the figure, each circuit in the feature extraction device 100 is supplied with the preparation timing trigger 210 in order from the upper layer in FIG. 1, so that the processing shown in FIG. 2 is executed in the order shown in the figure. In 100, since the multiplication circuit board 140 has the multiplication circuit 141 corresponding to each of the pixels 121, the convolution process can be executed for each pixel value. As a result, features with high determination accuracy can be efficiently extracted.

The section P indicates a section in which the convolution process S102 is repeated. The area surrounded by the dotted line A indicates a section in which a pixel group (for example, 3 × 3 pixels) composed of a plurality of pixels is convolved by a filter of the same size. Further, the area surrounded by the dotted line B indicates a section in which the same pixel group is convolved by another filter.

Further, the section Q indicates a section in which the convolution process S102 to the pooling process S104 are repeated. Actually, in the calculation result addition process of the convolution process S102, a timing trigger of the addition circuit with latch A or latch B is supplied every time a signal from the addition circuit is input, but for the sake of simplification of the explanation, Only the timing trigger of the final addition process is shown.

The coupling process in the coupling circuit 191 has a smaller processing load than the processing on other substrates. Therefore, the feature extraction device 100 does not need to provide the coupling circuit board 190 for the coupling circuit 191. In this case, the feature extraction device 100 repeats up to a predetermined number of times and finally outputs a signal that has been pooled by the pooling circuit 181 to the outside. Then, the bonding process is executed on the external substrate.

FIG. 6 is a block diagram of an image pickup apparatus 500 provided with a feature extraction apparatus 100. The image pickup device 500 includes a feature extraction device 100, a system control unit 501, a drive unit 502, a photometric unit 503, a work memory 504, a recording unit 505, a display unit 506, and a main power supply 507.

Further, the image pickup apparatus 500 includes a main optical system 520 that guides the subject luminous flux to the feature extraction apparatus 100. The main optical system 520 may be an interchangeable type that can be attached to and detached from the image pickup apparatus 500.

The main optical system 520 is composed of a plurality of optical lens groups, and forms a subject light flux from the field of view in the vicinity of the focal plane thereof. In the figure, a virtual single lens arranged near the pupil represents the main optical system 520.

The drive unit 502 is a control circuit that executes charge accumulation control such as timing control and region control of the feature extraction device 100 according to an instruction from the system control unit 501. The drive unit 502 is responsible for, for example, a series of controls for the feature extraction device 100 to perform photoelectric conversion of incident light, accumulate electric charges generated, and output pixel values. Further, the drive unit 502 supplies the timing trigger 210 to the feature extraction device 100.

The feature amount of the subject output from the feature extraction device 100 is passed to the determination unit 513 of the system control unit 501. As a result, the system control unit 501 executes a determination process for determining the subject. The feature extraction device 100 may transmit information including the brightness distribution of the subject from the pixel substrate 120 to the image processing unit 511 of the system control unit 501 to generate image data of the subject. The image processing unit 511 executes processing using the work memory 504 as a workspace.

The photometric unit 503 detects the luminance distribution of the subject prior to a series of sequences in which the feature extraction device 100 photoelectrically converts the incident light to generate pixel values. The photometric unit 503 includes, for example, an AE sensor having about 1 million pixels. The calculation unit 512 of the system control unit 501 receives the output of the photometric unit 503 and calculates the brightness for each area of the scene.

Further, the calculation unit 512 determines the shutter speed, the aperture value, and the ISO sensitivity according to the calculated luminance distribution. The photometric unit 503 may also be used by the feature extraction device 100. The calculation unit 512 also executes various calculations necessary for operating the image pickup apparatus 500.

As described above, the image pickup apparatus 500 has a determination function of determining the subject based on the feature amount acquired from the feature extraction device 100. Here, the feature extraction device 100 transmits the feature amount extracted by itself to the system control unit 501. Therefore, the system control unit 501 can acquire the feature amount and determine the subject without bearing the load of the feature amount extraction process. Further, since the system control unit 501 receives the feature amount as the extraction result, the increase in the communication amount between the feature extraction device 100 and the system control unit 501 is prevented.

FIG. 7 is a schematic cross-sectional view of another feature extraction device 101. The feature extraction device 101 has the same structure as the feature extraction device 100 shown in FIG. 1, except for the portion described below. Therefore, common elements are given the same reference number to omit duplicate explanations.

The feature extraction device 101 has a structure different from that of the feature extraction device 100 in that a plurality of multiplication circuit boards 1400, 1401, 1402 ... 140n are provided between the AD conversion circuit board 130 and the addition circuit board 150. The plurality of multiplication circuit boards 1400, 1401, 1402 ... 140n have a plurality of multiplication circuits 141 individually corresponding to the plurality of pixels 121 of the pixel board 120, similarly to the multiplication circuit board 1400 of the feature extraction device 100. , It operates at the timing when the timing trigger 210 received from the outside is received.

The multiplication circuit 141 acquires and holds the filter coefficient when executing the multiplication process from the outside. Further, the multiplication circuit 141 can execute the multiplication process by holding the filter coefficients having different values for each substrate. Therefore, the multiplication circuit boards 1400 to 140n can execute the multiplication process under different conditions.

FIG. 8 is a schematic cross-sectional view illustrating the operation of the feature extraction device 101. As shown by the thick line hatched in the figure, the pixel value output by the pixel 1 in the pixel substrate 120 is multiplied by the multiplication circuit 141 formed in the multiplication circuit board 1400 in the upper part of the drawing, and then the through electrode is used. It is input to the adder circuit board 150 through 13. Further, the pixel value output by the pixel 2 on the pixel board 120 is subjected to multiplication processing by the multiplication circuit 141 formed on the multiplication circuit board 1401 in the upper part of the drawing, and then input to the addition circuit board 150 through the through electrode 13. Further, the pixel value output by the pixel 3 on the pixel substrate 120 is subjected to multiplication processing by the multiplication circuit 141 formed on the multiplication circuit board 1402 in the upper part of the drawing, and then input to the addition circuit board 150 through the through electrode 13.

The filter coefficient a1 in each of the plurality of multiplication circuits 141 provided on the multiplication circuit board 1400 has the same value among the multiplication circuits 141 provided on the same multiplication circuit board 1400. Similarly, in each of the other multiplication circuit boards 1401, 1402 ... 140n, the plurality of multiplication circuits 141 in each board have common filter coefficients b1, c1 ... n1. Therefore, when the convolution process is executed, the filter coefficients a1, b1, c1 ... n1 can be selected by selecting the substrate on which the multiplication process is executed. That is, in the example shown in FIG. 8, the pixel value output by pixel 1 is multiplied by the filter coefficient a1, the pixel value output by pixel 2 is multiplied by the filter coefficient b1, and the pixel value output by pixel 3 is filtered. Multiplication processing is performed with the coefficient c1.

FIG. 9 is a partial timing chart of the feature extraction device 101. In the feature extraction device 101, multiplication processing is performed by a multiplication circuit board 1400 to 140n that is different for each pixel 121. Therefore, the multiplication process that is executed many times in the convolution process that corresponds to one filtering for a certain pixel group is shown by the dotted lines A and B in the figure after first acquiring all the filter coefficients. In addition, it can be processed in parallel and continuously at the same time.

Therefore, since it is not necessary to acquire and set the filter coefficient each time the multiplication process is performed, the time for acquiring and setting the filter coefficient is shortened, and the throughput of the entire feature extraction device 101 can be improved. Such a feature extraction device 101 can be used as an image sensor with a feature extraction function by incorporating the feature extraction device 101 into the image pickup device 500 instead of the feature extraction device 100, for example.

The feature extraction device described in this embodiment has a multiplication circuit 141 and an addition circuit in each of the multiplication circuit board 140, the addition circuit board 150, the activation function calculation circuit board 170, the pooling circuit board 180, and the coupling circuit board 190, respectively. 151, the activation function calculation circuit 171 and the pooling circuit 181 and the coupling circuit 191 are arranged. However, the feature extraction device does not necessarily have to be provided with only one circuit for one substrate. That is, a plurality of circuits may be arranged on one board, or one circuit may be arranged over a plurality of boards.

Further, the stacking order of the plurality of substrates in the feature extraction device is not limited to the above example as long as it can be arranged using through electrodes. Further, the convolution calculation result addition substrate 160 is not limited to the above example as long as it has a configuration in which a plurality of values can be added and held.

Further, the feature extraction device described in the present embodiment realizes feature extraction using a convolutional neural network that performs convolution processing among the multi-layer neural networks with a single sensor having a laminated structure. It is also possible to realize a recursive convolutional neural network by using the same value for the filter coefficient of the convolutional process performed a plurality of times. Note that the feature extraction is not limited to these methods as long as it can be realized by the feature extraction device in the present embodiment.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of operations, procedures, steps, steps, etc. in the devices, systems, programs, and methods shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

11 Underground board, 12 Wiring layer, 13, 14 Penetration electrode, 100, 101 Feature extraction device, 110 Micro lens, 120 pixel board, 121 pixels, 130 AD conversion circuit board, 131 AD conversion circuit, 132 latch, 133 changeover switch, 140, 1400, 1401, 1402, 140n Multiplying Circuit Board, 141 Multiplying Circuit, 150 Addition Circuit Board, 151 Addition Circuit, 160 Folding Calculation Result Addition Board, 161 Addition Circuit with Latch B, 162 Latch A, 163 multiplexer, 170 Activation Function calculation circuit board, 171 activation function calculation circuit, 180 pooling circuit board, 181 pooling circuit, 190 coupling circuit board, 191 coupling circuit, 210 timing trigger, 500 imaging device, 501 system control unit, 502 drive unit, 503 metering unit , 504 Work memory, 505 Recording unit, 506 Display unit, 507 Main power supply, 511 Image processing unit, 512 Calculation unit, 513 Judgment unit, 520 Main optical system

Claims (16)

Multiple light receiving elements that photoelectrically convert the light received from the subject,
A feature extraction unit that extracts the feature amount of the subject based on the signals output from the plurality of light receiving elements, and a feature extraction unit.
A light receiving substrate in which the plurality of light receiving elements are two-dimensionally arranged, and
With at least one other substrate laminated on the light receiving substrate and having the feature extraction unit,
With
The feature extraction unit has a convolution processing unit that convolves and calculates signals output from the plurality of light receiving elements.
The convolution processing unit is a feature extraction element that has a plurality of multiplication circuits provided for each of the light receiving elements or for each block composed of the plurality of light receiving elements, and performs a convolution calculation using the plurality of multiplication circuits . The feature extraction element according to claim 1, wherein the feature extraction unit extracts the feature amount by filtering signals output from the plurality of light receiving elements. The feature extraction element according to claim 1 or 2 , wherein the feature extraction unit has a pooling processing unit that samples a signal output from the convolution processing unit based on predetermined conditions. The feature extraction element according to claim 3 , further comprising a connection wiring for passing the sampled signal to the convolution processing unit. A light receiving substrate in which the plurality of light receiving elements are two-dimensionally arranged, and
With at least one other substrate laminated on the light receiving substrate and having the convolution processing section and the pooling processing section.
The feature extraction element according to claim 4 . A plurality of the other substrates are provided, and the convolution processing unit and the pooling processing unit are provided on different substrates among the plurality of other substrates.
The feature extraction element according to claim 5 , wherein the connection wiring includes through electrodes that connect different substrates to each other. The other board has a plurality of multiplication circuit boards.
The feature extraction element according to claim 5 or 6 , wherein the plurality of multiplication circuits are provided on different multiplication circuit boards. The feature extraction element according to claim 7 , wherein the plurality of multiplication circuit boards are set with filter coefficients for convolution calculations different from each other. The convolution processing unit has an adder circuit that adds outputs from the plurality of multiplication circuits.
The feature extraction element according to any one of claims 5 to 8 , wherein the adder circuit is provided on an adder circuit board which is the other board. A latch circuit that holds the signal from the light receiving element and
The feature extraction element according to any one of claims 5 to 9 , wherein the connection wiring connects between the pooling processing unit and the latch circuit. The other board has any one of claims 5 to 10 having an activation function calculation processing unit that inputs the convolution-calculated signal to the activation function and outputs the output signal to the pooling processing unit. The feature extraction element described in the section. A control unit for controlling the convolution calculation of the convolution processing unit and the sampling of the pooling processing unit to be repeated is provided.
The feature extraction element according to any one of claims 3, 5 to 11 , wherein the control unit controls the convolution processing unit so as to use a predetermined filter coefficient when the convolution operation is repeatedly performed. .. The feature extraction element according to any one of claims 1 to 12 , which outputs the feature amount to the outside. A plurality of AD conversion circuits provided corresponding to each light receiving element or each block composed of the plurality of light receiving elements are provided.
The feature extraction element according to any one of claims 1 to 13 , wherein the plurality of AD conversion circuits are provided on another substrate laminated on a substrate provided with the plurality of light receiving elements. The feature extraction element according to any one of claims 3, 4, 5 to 11 .
It is provided with a convolution operation of the convolution processing unit and a control unit that controls so as to repeat sampling of the pooling processing unit.
The control unit is a feature extraction system that controls the convolution processing unit so as to use a predetermined filter coefficient when the convolution operation is repeatedly performed. The feature extraction element according to any one of claims 1 to 14 ,
A determination unit that determines the subject based on the extracted feature amount,
Judgment device including.

Images