CN118410844B - An optical fully connected neural network device and its working method - Google Patents

Abstract

The invention discloses an optical full-connection neural network device and a working method thereof, the device comprises an optical input module, an optical full-connection image shifting module, an optical filtering module and an optical integration module. The optical image of the input module is copied and translated to the optical filtering module through the optical full-connection image shifting module, the weights of neurons are carried out through the optical filtering module comprising a specific pattern array, and the weights are projected to the surface of the integration module for optical integration. The optical signal processing is carried out on the deep learning neural network through the optical full-connection neural network device, so that the processing speed can be greatly improved, and the power consumption can be reduced.

Description

An optical fully connected neural network device and its working method

Technical Field

The present invention belongs to the field of artificial intelligence, and specifically relates to an optical fully connected neural network device for performing deep convolutional neural network machine learning and a working method thereof.

Background Art

Deep learning is one of the latest trends in machine learning and artificial intelligence research. It is also one of the most popular scientific research trends today. Deep learning methods have brought revolutionary advances to computer vision and machine learning. New deep learning techniques are constantly being born, surpassing the most advanced machine learning and even existing deep learning techniques. In recent years, many major breakthroughs have been made in this field around the world. Deep learning is a major method derived from machine learning and artificial neural networks, and artificial neural networks are the most commonly used form of deep learning. Convolutional neural networks are a deep learning model or a multi-layer perceptron similar to artificial neural networks, which are often used to analyze visual images. A convolutional neural network mainly consists of the following 5 layers: data input layer, convolution calculation layer, excitation layer, pooling layer and fully connected layer. Among them, the fully connected layer is the last step of the convolutional neural network. All the features obtained after the convolution calculation establish a neural network connection with all the outputs in the fully connected layer.

The fully connected neural network operation based on the central computing unit (CPU) can only perform one instruction at a time, so the neural network parameters are loaded, multiplied, stored, and then added and summed. The entire calculation process requires a large number of repeated instructions, which consumes computing energy and time. Therefore, in order to increase the speed, a parallel computing method based on a graphics processing unit (GPU) can be used in hardware to increase the speed by using its large storage capacity and multi-core.

Although GPU computing can greatly improve the speed of convolution calculation, a large amount of data to be calculated still needs to be sorted and transmitted to the GPU through the CPU, and the calculation results also need to be transmitted back to the CPU for the next round of calculation preparation. Therefore, the existing calculation is still based on the CPU+GPU method, which requires a large throughput calculation channel, limited calculation speed, and huge energy consumption.

Summary of the invention

Technical problem: In view of the above-mentioned defects or deficiencies in the prior art, the present invention aims to provide an optical fully connected neural network device and a working method thereof for deep convolutional neural network machine learning, which utilizes optical principles to perform fully connected neural network calculations, thereby increasing the calculation speed while reducing the power consumption required for calculations.

Technical solution: To achieve the above functions, an optical fully connected neural network device of the present invention is as follows:

The device specifically includes a display module, a fully connected module, a weight module, and an integration module arranged in sequence; the display module is a monochrome display module with a display capacity of r×c, r is the row of the monochrome display module, and c is the column of the monochrome display module; the fully connected module includes an optical flat plate or a combination of multiple flat plates with an m×n compound eye lens array; the weight module is an optical mask plate, including a plurality of m×n pattern arrays, each pattern including r×c sub-pixels; the integration module is composed of an array of light averaging plates, and adjacent units of the light averaging plates corresponding to the pattern are provided with optical isolation devices.

Multiple optical fully connected neural network devices are combined in parallel, and the display modules of each device respectively display different channels of the signal to be processed, and the optical neural network connections are processed in parallel.

The display module is a liquid crystal display, a micro light emitting diode display, an organic light emitting diode display, a digital micromirror display, a silicon-based liquid crystal display or a projection display.

The materials of the fully connected module, weight module and integral module are glass, quartz inorganic materials, or polymethyl methacrylate, polycarbonate, polyethylene terephthalate, polypropylene, polystyrene, polyvinyl chloride, acrylonitrile/butadiene/styrene copolymer, thermoplastic polyurethane elastomer rubber, polyimide, polystyrene, polysulfone, polydiacid diamide organic materials, or a composite material of the above two organic materials, and the composite form is a single layer or a multi-layer laminate.

The display module is the output of the preceding optical system.

The device is based on a two-layer fully connected neural network, including a first layer consisting of an input layer and a hidden layer and a second layer consisting of a hidden layer and an output layer, wherein the input layer includes an input quantity X (x1, x2, ... xj), the hidden layer includes h neurons, the output layer includes an output quantity O (o1, o2, ... ok) composed of k neurons, each neuron in the hidden layer is connected to the neurons in the input quantity X and the output quantity O, a two-layer fully connected neural network is established, and the intrinsic connection between the input and the output is established through the weight and bias of each neuron; in order to facilitate optical processing, the input quantity, the hidden layer and the output quantity are converted into a two-dimensional matrix form, and the sizes thereof are r×c≥j, m×n≥h, and p×q≥k respectively; wherein r is the row of the input layer, c is the column of the input layer, j is the number of input quantities, m is the row of the hidden layer, n is the column of the hidden layer, h is the number of neurons in the hidden layer, p is the row of the output layer, q is the column of the output layer, and k is the number of output quantities.

A plurality of optical fully-connected neural network devices are serially combined, and the zth integration module of each optical fully-connected neural network device is processed by the zth nonlinear processing module and simultaneously serves as the zth display module of the subsequent device to serially process the optical fully-connected neural network multiple times.

The working method of an optical fully connected neural network device of the present invention is specifically as follows:

Step 1: The display module displays the r×c image of the signal to be processed;

Step 2: According to the focal length of the compound eye lens, by adjusting the distance between the display module, the fully connected module and the weight module, the r×c image displayed by the display module is copied to the positions of the m×n r×c weight patterns corresponding to the weight module through the fully connected module, and the size and position of the copied shifted image coincide with each weight pattern;

Step 3: After the copied image block passes through the weight pattern, it is projected onto the surface of the integration module for optical integration and the optical fully connected neural network integration result is displayed.

An optical fully connected neural network device and a working method thereof of the present invention are specifically as follows, where z is greater than or equal to 1:

Step 1: The zth display module displays the r×c image of the signal to be processed;

Step 2: According to the focal length of the zth compound eye lens, by adjusting the distance between the zth display module, the zth fully connected module and the zth weight module, the r×c image displayed by the zth display module is copied to the positions of the m×n r×c zth weight patterns corresponding to the zth weight module through the zth fully connected module, and the size and position of the copied shifted image coincide with each zth weight pattern;

Step 3: After the copied image block passes through the zth weight pattern, it is projected onto the surface of the zth integration module for optical integration and the zth integration result is displayed;

Step 4: According to the focal length of the zth compound eye lens, by adjusting the distance between the zth nonlinear processing module, the z+1th fully connected module and the z+1th weight module, after the zth integral result passes through the zth nonlinear processing module, the displayed m×n image is copied through the z+1th fully connected module to the position of the p×q m×n z+1th weight pattern corresponding to the z+1th weight module, and the size and position of the copied shifted image coincide with each z+1th weight pattern;

Step 5: After the copied image block passes through the z+1th weight pattern, it is projected onto the surface of the z+1th integration module for optical integration and the z+1th integration result is displayed;

Step 6: The z+1th integral result passes through the z+1th nonlinear processing module and displays the z+1th layer fully connected neural network result.

Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

1. The compound eye lens array replaces the window sliding in the traditional convolution calculation, and uses the optical mask and homogenizer made of the convolution kernel array to realize optical convolution, which greatly improves the calculation speed.

2. The convolution calculation process does not involve high-speed data transmission, which is more energy-efficient.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various figures unless otherwise specified.

Figure 1 is a schematic diagram of a two-layer fully connected neural network.

FIG. 2 is a schematic diagram of a device according to the present invention.

FIG3 is a schematic diagram illustrating a method for serial operation of devices according to the present invention;

The figure includes: a display module 1, a fully connected module 2, a weight module 3, an integration module 4; a compound eye lens 20, a weight pattern 30, a sub-pixel 300, a light averaging plate 40, and a light isolation device 41.

DETAILED DESCRIPTION

The following is a description of the specific embodiments in conjunction with the accompanying drawings. Specific exemplary embodiments in which the present invention may be implemented are described by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the concepts disclosed herein, and it should be understood that the various disclosed embodiments may be modified and other embodiments may be utilized without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be considered restrictive.

The present invention aims to provide an optical fully connected neural network device and a working method thereof, which uses optical principles to perform convolution calculations, thereby increasing the calculation speed and reducing the power consumption required for calculations. To achieve the above functions, the present invention adopts the following examples:

Embodiment 1: A two-layer fully connected neural network as shown in FIG1 includes an input layer, a hidden layer and an output layer, wherein the input layer includes an input quantity X (x ₁ , x ₂ , ... x _j ), the hidden layer includes h neurons, and the output layer includes an output quantity O (o ₁ , o ₂ , ... _ok ). Each neuron in the hidden layer is connected to the input quantity X and the output quantity O, and a two-layer fully connected neural network is established, and the intrinsic connection between the input and the output is established through the weight and bias of each neuron. In order to facilitate optical processing, the input quantity, the hidden layer and the output quantity can usually be converted into a two-dimensional matrix form, and the dimensions thereof are r×c=j, m×n=h, and p×q=k, respectively.

In combination with the 2-layer fully connected neural network shown in FIG1 , as shown in FIG2 , the present invention discloses a single-layer optical fully connected neural network device, including a display module 1, a fully connected module 2, a weight module 3, and an integration module 4, wherein the display module 1 is a monochrome display module with a display capacity of r×c, and may be a liquid crystal display, a micro-light emitting diode display, an organic light emitting diode display, a digital micromirror display, a silicon-based liquid crystal display, or a projection display; the fully connected module 2 is an optical flat plate or a combination of multiple flat plates including an array of m×n compound eye lenses 20, each lens in the compound eye may be single-sided or double-sided, and the compound eye lens 20 is a cylindrical lens, a spherical lens, a single convex lens, a concave lens, or a composite lens composed of a combination of multiple lenses; the weight module 3 is an optical mask plate, including an array of several m×n patterns 30, each pattern 30 including r×c sub-pixels 300, whose transmittance represents the weight coefficient of the neuron; the integration module 4 is an array of multiple light averaging plates, and in order to prevent light crosstalk, the light averaging plates have optical isolation devices 41 for adjacent units corresponding to the pattern 30, and the transmittance of each unit represents the bias coefficient of the neuron. The materials of the fully connected module 2, the weight module 3 and the integral module 4 are glass inorganic materials, or organic materials such as polymethyl methacrylate, polycarbonate, polyethylene terephthalate, polypropylene, polystyrene, polyvinyl chloride, acrylonitrile/butadiene/styrene copolymer, thermoplastic polyurethane elastomer rubber, polyimide, polystyrene, polysulfone, polydiacid diamine, etc., or composite materials of the above two organic materials, and the composite form is a single layer or a multi-layer laminate.

The working method of the optical fully connected neural network device is as follows:

Step 1: the display module 1 displays the image of the signal to be processed;

Step 2: According to the focal length of the compound eye lens 20, by adjusting the distances between the display module 1, the fully connected module 2 and the weight module 3, the image displayed by the display module 1 is formed into a real image at the position of the pattern 30 corresponding to the weight module 3, and the size and position of the image shifting coincide with the pattern 30;

Step 3: The image block after image shifting is filtered by the pattern 30 and then projected onto the surface of the integration module 4 for optical integration and the result of the optical fully connected neural network is displayed on the homogenizer 40 .

Embodiment 2:

According to an optical fully-connected neural network device described in Example 1, multiple optical fully-connected neural network devices are combined in parallel, and the display module 1 of each device displays different channels of the signal to be processed respectively, and optical convolution is processed in parallel.

The working method is as follows:

Step 1: the display module 1 displays the r×c image of the signal to be processed;

Step 2: According to the focal length of the compound eye lens 20, by adjusting the distances between the display module 1, the fully connected module 2 and the weight module 3, the r×c image displayed by the display module 1 is copied to the positions of the m×n r×c weight patterns 30 corresponding to the weight module 3 through the fully connected module 2, and the size and position of the copied shifted image coincide with each weight pattern 30;

Step 3: After the copied image block passes through the weight pattern 30, it is projected onto the surface of the integration module 4 for optical integration and displays the optical fully connected neural network integration result.

Embodiment 3:

According to an optical fully-connected neural network device described in Example 1, multiple optical fully-connected neural network devices are serially combined, and the zth integration module 4 _z of each optical convolution device passes through the zth nonlinear processing module 5 _z and serves as the z+1th display input module 1 _z+1 of the subsequent device to serially process the optical fully-connected neural network multiple times.

The working method is as follows, taking z greater than or equal to 1:

Step 1: The zth display module 1 _z displays the r×c image of the signal to be processed;

Step 2: According to the focal length of the zth compound eye lens _20z , by adjusting the distances between the zth display module _1z , the zth fully connected module _2z and the zth weight module _3z , the r×c image displayed by the zth display module _1z is copied to the positions of the m×n r×c zth weight patterns 30z corresponding to the zth weight module _3z through the _zth fully connected module _2z , and the size and position of the copied shifted image coincide with each zth weight pattern _30z ;

Step 3: After the copied image block passes through the zth weight pattern 30 _z , it is projected onto the surface of the zth integration module 4 _z for optical integration and the zth integration result is displayed 40 _z ;

Step 4: According to the focal length of the zth compound eye lens _20z , by adjusting the distance between the zth nonlinear processing module _5z , the z+1th fully connected module 2z ₊₁ and the z+1th weight module 3z ₊₁ , after the zth integral result _40z passes through the zth nonlinear processing module _5z , the displayed m×n image is copied to the position of the p×q m×n z+1th weight pattern _30z+1 corresponding to the z+1th weight module 3z ₊₁ through the z+1th fully connected module 2z ₊₁ , and the size and position of the copied shifted image coincide with each z+1th weight pattern 30z ₊₁ ; The z+1th compound eye lens 20z ₊₁ is located in front of the z+1th fully connected module 2z ₊₁ ;

Step 5: After the copied image block passes through the z+1th weight pattern 30 _z+1 , it is projected onto the surface of the z+1th integration module 4 _z+1 for optical integration and the z+1th integration result 40 _z+1 is displayed;

Step 6: The z+1th integral result 40 _z+1 passes through the z+1th nonlinear processing module 5 _z+1 and then displays the z+1th layer fully connected neural network result.

Embodiment 4: According to an optical convolution device and a working method thereof described in Embodiment 1, the integration module 4 can also reduce the averaging effect by controlling the thickness of the light averaging plate or implement optical maximum pooling by using a nonlinear optical selection module.

Embodiment 5: According to the optical fully-connected neural network device described in Embodiment 1, multiple optical fully-connected neural network devices are combined in series or in parallel.

Embodiment 6: According to the optical fully-connected neural network device described in Embodiment 1, the display module 1 is the output of the preceding optical system.

Embodiment 7: According to an optical convolution device and a working method thereof described in embodiments 1 to 6, the weight module 3 and the integration module 4 are integrated into one component, and the graphic of the filter module is directly attached to a plane of the integration module.

Claims (7)

1. An optical fully connected neural network device, characterized in that the device specifically comprises a display module 1, a fully connected module 2, a weight module 3, and an integration module 4 arranged in sequence; the display module 1 is a monochrome display module with a display capacity of r×c, r is the row of the monochrome display module, and c is the column of the monochrome display module; the fully connected module 2 comprises an optical flat plate or a combination of multiple flat plates with an array of m×n compound eye lenses 20; the weight module 3 is an optical mask plate, comprising an array of several m×n patterns 30, each pattern 30 comprising r×c sub-pixels 300; the integration module 4 is composed of an array of light averaging plates 40, and the adjacent units of the light averaging plates 40 corresponding to the pattern 30 have optical isolation devices 41; m is the row of the hidden layer, and n is the column of the hidden layer; A plurality of the optical fully connected neural network devices are combined in parallel, and the display module 1 of each device displays different channels of the signal to be processed respectively, and the optical neural network connection is processed in parallel; The display module 1 is a liquid crystal display, a micro light emitting diode display, an organic light emitting diode display, a digital micromirror display, a liquid crystal on silicon display or a projection display. 2. An optical fully-connected neural network device according to claim 1, characterized in that the materials of the fully-connected module 2, the weight module 3 and the integral module 4 are glass inorganic materials, or polymethyl methacrylate, polycarbonate, polyethylene terephthalate, polypropylene, polystyrene, polyvinyl chloride, acrylonitrile/butadiene/styrene copolymer, thermoplastic polyurethane elastomer rubber, polyimide, polystyrene, polysulfone, polydiacid diamide organic materials, or a composite material of the above two organic materials, the composite form is a single layer or a multi-layer laminate. 3. An optical fully connected neural network device according to claim 1, characterized in that the display module 1 is the output of the preceding optical system. 4. An optical fully connected neural network device according to claim 1, characterized in that the device is based on a 2-layer fully connected neural network, including a first layer consisting of an input layer and a hidden layer, and a second layer consisting of a hidden layer and an output layer, wherein the input layer includes an input quantity X (x1, x2, ... xj), the hidden layer includes h neurons, and the output layer includes an output quantity O (o1, o2, ... ok) composed of k neurons, each neuron in the hidden layer is connected to the neurons in the input quantity X and the output quantity O, and a 2-layer fully connected neural network is established, and the intrinsic connection between the input and the output is established through the weight and bias of each neuron; the input quantity, the hidden layer and the output quantity are converted into a two-dimensional matrix form, and the sizes thereof are r×c≥j, m×n≥h, and p×q≥k respectively; wherein r is the row of the input layer, c is the column of the input layer, j is the number of input quantities, m is the row of the hidden layer, n is the column of the hidden layer, h is the number of neurons in the hidden layer, p is the row of the output layer, and q is the column of the output layer. 5. An optical fully-connected neural network device according to claim 1, characterized in that a plurality of optical fully-connected neural network devices are serially combined, and the zth integration module 4 _z of each optical fully-connected neural network device is processed by the zth nonlinear processing module 5 _z and simultaneously serves as the z+1th display module 1 _z+1 of the subsequent device to serially process the optical fully-connected neural network multiple times. 6. A method for operating an optical fully connected neural network device as claimed in claim 1 or 2, characterized in that the method is specifically as follows: Step 1: The display module 1 displays the r×c image of the signal to be processed, wherein r is the row of the input layer and c is the column of the input layer; Step 2: According to the focal length of the compound eye lens 20, by adjusting the distance between the display module 1, the fully connected module 2 and the weight module 3, the r×c image displayed by the display module 1 is copied to the positions of the m×n r×c weight patterns 30 corresponding to the weight module 3 through the fully connected module 2, and the size and position of the copied image coincide with each weight pattern 30; m is the row of the hidden layer, and n is the column of the hidden layer; Step 3: After the copied image block passes through the weight pattern 30 , it is projected onto the surface of the integration module 4 for optical integration and the optical fully connected neural network integration result is displayed on the homogenizer 40 . 7. A method for operating an optical fully connected neural network device as claimed in claim 3, 4 or 5, characterized in that the method is specifically as follows, taking z to be greater than or equal to 1: Step 1: The zth display module 1 _z displays the r×c image of the signal to be processed; Step 2: According to the focal length of the zth compound eye lens _20z , by adjusting the distances between the zth display module _1z , the zth fully connected module _2z and the zth weight module _3z , the r×c image displayed by the zth display module _1z is copied to the positions of the m×n r×c zth weight patterns 30z corresponding to the zth weight module _3z through the _zth fully connected module _2z , and the size and position of the copied shifted image coincide with each zth weight pattern _30z ; Step 3: After the copied image block passes through the zth weight pattern 30 _z , it is projected onto the surface of the zth integration module 4 _z for optical integration and the zth integration result is displayed 40 _z ; Step 4: According to the focal length of the z-th compound eye lens 20 _z , by adjusting the distances between the z-th nonlinear processing module 5 _z , the z+1-th fully connected module 2 _z+1 and the z+1-th weight module 3 _{z+1 ,} after the z-th integral result 40 _z passes through the z-th nonlinear processing module 5 _z , the displayed m×n image is copied to the positions of the p×q m×n real images of the z+1-th weight module 3 _z+1 corresponding to the p×q m×n z+1-th weight pattern 30 _{z+1 through the z+1} -th fully connected module 2 z+1, and the size and position of the copied shifted image coincide with each z+1-th weight pattern 30 _z+1 ; Step 5: After the copied image block passes through the z+1th weight pattern 30 _z+1 , it is projected onto the surface of the z+1th integration module 4 _z+1 for optical integration and the z+1th integration result 40 _z+1 is displayed; Step 6: The z+1th integral result 40 _z+1 passes through the z+1th nonlinear processing module 5 _z+1 and then displays the z+1th layer fully connected neural network result.