CN115220211A - Microscopic imaging system and method based on deep learning and light field imaging - Google Patents

Abstract

The invention discloses a microscopic imaging system and a microscopic imaging method based on deep learning and light field imaging, and belongs to the technical field of light field microscopic imaging. The microscopic imaging system comprises: the system comprises a microscopic imaging system, a deep learning network module and an image output module. The invention obtains a two-dimensional image array through a light field microscope system composed of a micro lens array and a camera sensor, obtains multi-view image information from the two-dimensional image array after passing through a deep learning network, further carries out super-resolution reconstruction on the image and finally obtains a high-resolution three-dimensional reconstruction image with uniform intensity distribution. Compared with the existing super-resolution reconstruction method based on the convolutional neural network, the image generated by the method has higher spatial resolution, smaller reconstruction artifact and higher reconstruction throughput, so that the imaging definition is effectively improved; in addition, under the lens array system, the invention also has the advantages of low cost, low system complexity and super-resolution imaging without scanning.

Description

Microscopic imaging system and method based on deep learning and light field imaging

technical field

The invention relates to a microscopic imaging system and method based on deep learning and light field imaging, and belongs to the technical field of light field microscopic imaging.

Background technique

The light field imaging system mainly collects and captures the four-dimensional light field of the spatial distribution through the optical device, and then calculates the corresponding image according to different application requirements. Computational light field imaging technology is based on four-dimensional light field, aiming to establish the relationship of light in multiple dimensions such as spatial domain, viewing angle, spectrum and time domain, realize coupled perception, decoupled reconstruction and intelligent processing, and is used for multi-dimensional dynamic scenes in a wide range. Multiscale imaging. Light field imaging technology is gradually being used in life sciences, industrial detection, national security, unmanned systems, and virtual reality/augmented reality, and has important academic research value and industrial application prospects.

Light-Field Microscopy (LFM) is achieved by inserting a microlens array capable of capturing light field information on the relay image plane of a traditional optical microscope. Multi-view images and multi-layer focal plane images can be reconstructed through the inversion of 4D light field data, and three-dimensional microscopic imaging can be achieved by introducing deconvolution algorithms and tomographic reconstruction. Since these post-processing can be achieved with a single exposure, it has unique advantages for observing moving microorganisms and light-sensitive samples. Optical microscopes have long been an important tool in biomedical research due to their non-contact and non-invasive advantages. However, since 1873, it has been believed that the resolution limit of optical microscopy is about 200 nm, and it cannot be used to clearly observe biological structures with dimensions within 200 nm.

The use of light field microscopy to obtain high-resolution images is of great significance in biological research and medical treatment. Digital microscopy imaging technology is different from traditional optical microscopy imaging. The biological parameters and morphology information of cells can be obtained according to the reconstructed hologram. , is an effective non-contact non-destructive 3D imaging technology. With the development of image sensors and the improvement of hardware computing capabilities, digital holographic microscopy imaging technology has made significant progress and breakthroughs in the detection of living biological cells, especially in the field of red blood cell detection. Today, digital microscopy imaging techniques are widely used for cell migration analysis and abnormal cell behavior studies, and are also widely used in instruments for delineating medical images.

However, on the one hand, due to the limitations of equipment and imaging technology, the images obtained by digital microscopy imaging technology cannot have very high accuracy; Limited by the ability of light field cameras to differentiate between two adjacent objects, light field cameras typically trade spatial resolution (the size or size of the smallest unit that can be distinguished in detail) for angular resolution (the minimum separation between two adjacent objects that an imaging system or system element can differentially distinguish). ability), resulting in unclear images. Therefore, the limited spatial resolution is the difficulty in the development of light field cameras.

In order to solve the above problems, Yoon et al. first proposed the super-resolution reconstruction of light field data based on convolutional neural network (Yoon Y, Jeon H G, Yoo D, et al. Light Field Image SuperResolution using Convolutional Neural Network [J]. IEEE Signal Processing Letters, 2017.), its network can be divided into spatial resolution reconstruction convolutional neural network and angular resolution reconstruction convolutional neural network, but this model does not make full use of the effective information between multi-view images, resulting in the inability to obtain higher resolution Even if the reconstructed image is high, the reconstructed image with no artifacts and uniform intensity distribution cannot be quickly obtained.

SUMMARY OF THE INVENTION

In order to solve the problems of artifacts, non-uniform resolution, and slow reconstruction speed in current microscopic imaging, the present invention provides a microscopic imaging system and method based on deep learning and light field imaging, and the technical solutions are as follows:

The first object of the present invention is to provide a microscopic imaging system based on deep learning and light field imaging, the microscopic imaging system includes: a microscopic system, a deep learning network module, and an image output module connected in sequence;

The microscope system is used to collect multiple two-dimensional data of the image, including: a microscope head 1, a first dichroic mirror 2, a lens 3, a second dichroic mirror 4, a bandpass filter 5, and a first camera sensor 6 , a microlens array 7, a relay lens 8, a second dichroic mirror 9, a second camera sensor 10, and a third camera sensor 11;

The microscope head 1 collects image data, filters the interference light through the first dichroic mirror 2, refracts the light through the lens 3 to focus the light, and then passes through the second dichroic mirror 4 to make the collected signal. A part is collected to the first camera sensor 6 for wide-field imaging to perform wide-field imaging; the other part passes through the microlens array 7, the relay lens 8, and the second dichroic mirror 9, respectively, and then passes through the second camera. Light field imaging is performed on the sensor 10 and the third camera sensor 11;

The deep learning network module is used to reconstruct the two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11 into a high-resolution three-dimensional image;

The image output module is used for outputting the reconstructed high-resolution three-dimensional image.

Optionally, the deep learning network module uses the trained VCD deep network VCD-Net to reconstruct high-resolution three-dimensional images.

Optionally, the first camera sensor 6 , the second camera sensor 10 and the third camera sensor 11 use sCMOS cameras.

Optionally, the training process of the VCD deep network VCD-Net includes:

Step 1: Initialize VCD-Net, including: network parameters and loss function;

Step 2: Acquire high-resolution 3D images from real static samples and concomitant microscopy of synthetic data;

Step 3: constructing a wave optics model, inputting the high-resolution three-dimensional image obtained in step 2 into the wave optics model, and outputting a corresponding two-dimensional image;

Step 4: Construct a training set and a test set based on the two-dimensional image obtained in step 3 and the high-resolution three-dimensional image obtained in step 2, take the two-dimensional image as input, and use the high-resolution three-dimensional image as output. The above VCD-Net is trained until convergence, and the optimal VCD-Net network model is obtained.

Optionally, when performing light field imaging on the sample, a 1:1 relay lens 8 is used to focus the second camera sensor 10 and the third camera sensor 11 on the back focal plane of the microlens array 7 .

Optionally, the wave optics model is:

F=Hg

Among them, the vector F represents the collected original light field image, the vector g represents the reconstructed 3D discrete point cloud of the object, and H is the point spread function matrix representation of the imaging process.

The second object of the present invention is to provide a microscopic imaging method based on deep learning and light field imaging, including:

Step 1: Use a microscope to collect image data, and input the image data into multiple camera sensors in the microscope system and sensors composed of a microlens array and a CCD for wide-field imaging and light-field imaging, respectively, to obtain multiple two-dimensional images. ;

Step 2: Input the two-dimensional image obtained in the first step into a trained deep neural network, and obtain a reconstructed high-resolution three-dimensional image through the trained deep neural network.

Optionally, the microscope system includes: a microscope head 1, a first dichroic mirror 2, a lens 3, a beam splitter 4, a bandpass filter 5, a first camera sensor 6, a microlens array 7, a relay lens 8. The second dichroic mirror 9, the second camera sensor 10, the third camera sensor 11;

The microscope head 1 collects image data, passes through the first dichroic mirror 2 to filter and removes interfering light, then passes through the lens 3 to refract the light to focus, and then passes through the beam splitter 4 to gather a part of the collected signal. The other part passes through the microlens array 7, relay lens 8, and second dichroic mirror 9, respectively, to the second camera sensor 10. and light field imaging on the third camera sensor 11 .

Optionally, the deep neural network adopts a VCD-Net network to reconstruct the two-dimensional image data imaged by the first camera sensor 6 , the second camera sensor 10 and the third camera sensor 11 into a high-resolution three-dimensional image.

Optionally, the training process of the VCD-Net network includes:

Step 1: Build and initialize VCD-Net;

Step 2: Acquiring high-resolution 3D images from confocal microscopy of real static samples and their synthetic data;

Step 3: constructing a wave optics model, inputting the high-resolution three-dimensional image obtained in step 2 into the wave optics model, and outputting a corresponding two-dimensional image;

Step 4: Construct a training set and a test set based on the two-dimensional image obtained in step 3 and the high-resolution three-dimensional image obtained in step 2, take the two-dimensional image as input, and use the high-resolution three-dimensional image as output. The above VCD-Net is trained until convergence, and the optimal VCD-Net network model is obtained.

The beneficial effects of the present invention are:

The invention constructs a light field microscopic imaging system based on light field imaging and deep learning network, acquires multiple two-dimensional images through a microscopic system composed of a microlens array and a camera sensor, and then the deep learning network acquires multiple two-dimensional images from these two-dimensional images. Perspective image information performs super-resolution reconstruction on the image to obtain a high-resolution three-dimensional image. Compared with the existing super-resolution reconstruction method based on convolutional neural network, the image generated by the present invention has higher spatial resolution (1.0 + 0.15um), smaller reconstruction artifacts and larger reconstruction throughput (200HZ), thereby effectively improving imaging clarity; and under the lens array system, the deep neural network-based super-resolution of the present invention The high-rate light field image generation model also has the advantages of low cost, low system complexity, and super-resolution imaging without scanning, which effectively improves the reconstruction speed.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic plan view of a microscopic imaging system according to an embodiment of the present invention.

Among them, 1-microscope; 2-first dichroic mirror; 3-reflector; 4-beam splitter; 5-bandpass filter; 6-first camera sensor; 7-microlens array; 8-relay lens; 9-second dichroic mirror; 10-second camera sensor; 11-third camera sensor.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

First, the basic theoretical knowledge involved in the present invention is introduced as follows:

VCD-Net network:

The VCD network is: In a general convolutional neural network (CNN), a certain Nth convolutional layer receives feature maps from the previous (N-1) layers and uses different convolution kernels to generate new feature maps. The network eventually produces a multi-channel output. where each channel is a non-linear combination of the original input, a concept that has similarities to digital reconvergence algorithms in light field photography, where each composite plane of the reconstructed volume can be thought of as a different view extracted from the light field superposition. By cascading layers, our model is expected to gradually transform raw angular information from light-field raw images to depth features, eventually forming a traditional 3D image stack and reconstructing the scene. In the implementation process, the customized VCD-Net is based on the modified U-Net architecture (refer to https://cloud.tencent.com/developer/article/1520224). It contains a downsampling path and a symmetric upsampling path, along two paths, each layer has three parameters: n, f, and s, which represent the number of output channels, the filter size of the convolution kernel, and the stride of the moving kernel, respectively.

Example 1:

This embodiment provides a microscopic imaging system based on deep learning and light field imaging, the microscopic imaging system includes: a microscopic system, a deep learning network module, and an image output module connected in sequence;

The microscope system is used to collect multiple two-dimensional data of an image, see FIG. 1 , including: a microscope head 1, a first dichroic mirror 2, a mirror 3, a beam splitter 4, a bandpass filter 5, a first camera sensor 6, microlens array 7, relay lens 8, second dichroic mirror 9, second camera sensor 10, third camera sensor 11;

The microscope head 1 collects image data, passes through the first dichroic mirror 2 to filter and removes interfering light, and then passes through the reflecting mirror 3 to make the light enter the horizontal direction from the vertical direction, and then passes through the beam splitter 4 to make the light beam. A part of the collected signal is collected to the first camera sensor 6 for wide-field imaging to perform wide-field imaging; the other part passes through the microlens array 7, the relay lens 8, and the second dichroic mirror 9, respectively, and then passes through the microlens array 7, the relay lens 8 and the second dichroic mirror 9, respectively. Perform light field imaging on the second camera sensor 10 and the third camera sensor 11;

The deep learning network module is used to reconstruct the two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11 into a high-resolution three-dimensional image;

The image output module is used for outputting the reconstructed high-resolution three-dimensional image.

Embodiment 2:

This embodiment is a microscopic imaging system based on deep learning and light field imaging, the microscopic imaging system includes: a microscopic system, a deep learning network module, and an image output module connected in sequence;

The microscope system is used to collect multiple two-dimensional data of an image, see FIG. 1 , including: a microscope head 1, a first dichroic mirror 2, a mirror 3, a beam splitter 4, a bandpass filter 5, a first The camera sensor 6 , the microlens array 7 , the relay lens 8 , the second dichroic mirror 9 , the second camera sensor 10 , and the third camera sensor 11 ; the three camera sensors all use sCMOS cameras.

The microscope head 1 collects image data, passes through the first dichroic mirror 2 to filter and removes interfering light, and then passes through the reflecting mirror 3 to make the collected light enter the horizontal direction from the vertical direction, and then passes through the beam splitter 4, A part of the collected signal is collected to the first camera sensor 6 for wide-field imaging, and wide-field imaging is performed; Light field imaging is performed on the second camera sensor 10 and the third camera sensor 11; when performing light field imaging, the second camera sensor 10 and the third camera sensor 11 are focused on the 1:1 relay lens 8. The back focal plane of the microlens array 7 .

The deep learning network module adopts the trained VCD deep network VCD-Net, and reconstructs a high-resolution three-dimensional image according to the two-dimensional image data imaged by the first camera sensor 6, the second camera sensor 10 and the third camera sensor 11;

The training process of VCD-Net includes:

Step 1: Build and initialize the VCD-Net. In this example, Tensorflow 1.15.0, Tensorlayer 1.8.5 and Python 3 are used to build the VCD-Net deep learning model in the Windows 10 environment;

Step 2: Acquire high-resolution 3D images from static samples or concomitant microscopy of synthetic data;

Step 3: constructing a wave optics model, inputting the high-resolution three-dimensional image obtained in step 2 into the wave optics model, and outputting a corresponding two-dimensional image;

The wave optics model is: F=Hg, where the vector F represents the light field, the vector g represents the discrete volume being reconstructed, H is the measurement matrix modeling of the forward imaging process, and H is mainly determined by the point spread function of the light field microscope.

Using wave optics to model the spatially varying point spread function of an optical microscope, the light field point spread function maps the transition from a 3D object to a 2D plane, and is also spatially variable, for each point in the region of interest A unique point spread function is considered. To generate the point spread function, multiple points in the volume are imaged wavefronts through a microlens array using scalar diffraction theory calculations.

Step 4: Construct a training set and a test set based on the two-dimensional image obtained in step 3 and the high-resolution three-dimensional image obtained in step 2, take the two-dimensional image as input, and use the high-resolution three-dimensional image as output. The VCD-Net described above is trained incrementally by iteratively minimizing the difference between its intermediate output and the reference high-resolution image, by setting appropriate loss parameters, such as the mean squared error of pixel intensities, to obtain optimized kernels for each layer parameters, and effectively converge to an optimal VCD-Net network model.

The final high-resolution reconstructed image can be obtained by inputting multiple two-dimensional images collected by the microscope system (the images of C. elegans collected in this embodiment) into the trained VCD-Net network model.

The image output module is used to output the reconstructed high-resolution three-dimensional image.

Embodiment three:

This embodiment provides a microscopic imaging method based on deep learning and light field imaging, which is implemented by the deep learning and microlens array-based microscopic imaging system described in the second embodiment, including the following steps:

Step 1: collecting image data with a microscope head, and inputting the image data into multiple camera sensors and microlens arrays in the microscope system for wide-field imaging and light-field imaging, respectively, to obtain multiple two-dimensional images;

Step 2: Input the two-dimensional image obtained in the first step into the trained VCD-Net network model, and obtain a reconstructed high-resolution three-dimensional image through the trained VCD-Net network model.

Some steps in the embodiments of the present invention may be implemented by software, and corresponding software programs may be stored in a readable storage medium, such as an optical disc or a hard disk.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. a microscopic imaging system based on deep learning and light field imaging, is characterized in that, described microscopic imaging system comprises: microscopic system, deep learning network module, image output module which are connected successively; The microscope system is used for collecting a plurality of two-dimensional data of an image, including: a microscope head (1), a first dichroic mirror (2), a lens (3), a beam splitter (4), a band-pass filter ( 5), a first camera sensor (6), a microlens array (7), a relay lens (8), a second dichroic mirror (9), a second camera sensor (10), and a third camera sensor (11); The microscope head (1) collects image data, passes through the first dichroic mirror (2) to filter and removes interfering light, and then passes through the lens (3) for refraction to focus the light, and then passes through the beam splitter (4) , so that a part of the collected signal is collected to the first camera sensor (6) for wide-field imaging to perform wide-field imaging; the other part passes through the microlens array (7), the relay lens (8), the second a dichroic mirror (9), respectively performing light field imaging on the second camera sensor (10) and the third camera sensor (11); the deep learning network module, configured to reconstruct the two-dimensional image data imaged by the first camera sensor (6), the second camera sensor (10) and the third camera sensor (11) into a high-resolution three-dimensional image; The image output module is used for outputting the reconstructed high-resolution three-dimensional image. 2 . The microscopic imaging system according to claim 1 , wherein the deep learning network module adopts the trained VCD deep network VCD-Net to reconstruct high-resolution three-dimensional images. 3 . 3. The microscopic imaging system according to claim 1, wherein the first camera sensor (6), the second camera sensor (10) and the third camera sensor (11) are sCMOS cameras. 4. The microscopic imaging system according to claim 1, wherein the training process of the VCD deep network VCD-Net comprises: Step 1: Initialize VCD-Net, including: network parameters and loss function; Step 2: Acquire high-resolution 3D images from real static samples and concomitant microscopy of synthetic data; Step 3: constructing a wave optics model, inputting the high-resolution three-dimensional image obtained in step 2 into the wave optics model, and outputting a corresponding two-dimensional image; Step 4: Construct a training set and a test set based on the two-dimensional image obtained in step 3 and the high-resolution three-dimensional image obtained in step 2, take the two-dimensional image as input, and use the high-resolution three-dimensional image as output. The above VCD-Net is trained until convergence, and the optimal VCD-Net network model is obtained. 5. The microscope imaging system according to claim 1, characterized in that, when performing light field imaging on the sample, the second camera sensor (10), the third camera sensor (10), the third camera sensor (10), the third camera sensor (10), the third camera sensor (10), the third The camera sensor (11) is focused on the back focal plane of the microlens array (7). 6. The microscopic imaging system according to claim 4, wherein the wave optics model is: F=Hg Among them, the vector F represents the collected original light field image, the vector g represents the reconstructed 3D discrete point cloud of the object, and H is the point spread function matrix representation of the imaging process. 7. A microscopic imaging method based on deep learning and light field imaging, wherein the microscopic imaging method comprises: Step 1: Use a microscope to collect image data, and input the image data into multiple camera sensors in the microscope system and sensors composed of a microlens array and a CCD for wide-field imaging and light-field imaging, respectively, to obtain multiple two-dimensional images. ; Step 2: Input the two-dimensional image obtained in the first step into a trained deep neural network, and obtain a reconstructed high-resolution three-dimensional image through the trained deep neural network. 8. The microscopic imaging method according to claim 7, wherein the microscopic system comprises: a microscope head (1), a first dichroic mirror (2), a lens (3), a beam splitter (4) ), bandpass filter (5), first camera sensor (6), microlens array (7), relay lens (8), second dichroic mirror (9), second camera sensor (10), Triple camera sensor (11); The microscope head (1) collects image data, passes through the first dichroic mirror (2) to filter and removes interfering light, and then passes through the lens (3) for refraction to focus the light, and then passes through the beam splitter (4) , so that a part of the collected signal is collected to the first camera sensor (6) for wide-field imaging to perform wide-field imaging; the other part passes through the microlens array (7), the relay lens (8), the second A dichroic mirror (9), respectively performing light field imaging on the second camera sensor (10) and the third camera sensor (11). 9 . The microscopic imaging method according to claim 8 , wherein the deep neural network adopts a VCD-Net network, and the first camera sensor ( 6 ), the second camera sensor ( 10 ) and the third camera sensor ( 10 ) and the third The two-dimensional image data imaged by the camera sensor (11) is reconstructed into a high-resolution three-dimensional image. 10. The microscopic imaging method according to claim 9, wherein the training process of the VCD-Net network comprises: Step 1: Build and initialize VCD-Net; Step 2: Acquiring high-resolution 3D images from confocal microscopy of real static samples and their synthetic data; Step 3: constructing a wave optics model, inputting the high-resolution three-dimensional image obtained in step 2 into the wave optics model, and outputting a corresponding two-dimensional image; Step 4: Construct a training set and a test set based on the two-dimensional image obtained in step 3 and the high-resolution three-dimensional image obtained in step 2, take the two-dimensional image as input, and use the high-resolution three-dimensional image as output. The above VCD-Net is trained until convergence, and the optimal VCD-Net network model is obtained.

Images