CN110286482A - A Grouped Fourier Stack Microscopic Reconstruction Method - Google Patents

Abstract

The invention proposes a kind of micro- method for reconstructing of packet type Fourier lamination, can overcome the shortcomings of conventional Fourier lamination method for reconstructing in speed, realize Fourier's lamination and rebuild promotion of the speed on the order of magnitude.The micro- method for reconstructing of packet type Fourier's lamination of the invention, using by slightly being rebuild to packet type Fourier's lamination of essence, and operation is rebuild come parallel processing using graphics processor, conventional Fourier lamination method for reconstructing is overcome the shortcomings of in speed, is realized Fourier's lamination and is rebuild promotion of the speed on the order of magnitude；And it can easily be combined with existing Fourier's lamination algorithm for reconstructing, without excessive hardware resource or special operating procedure is added.

Description

A Grouped Fourier Stack Microscopic Reconstruction Method

technical field

The invention belongs to the technical field of computational imaging, and in particular relates to a grouped Fourier stack high-resolution microscopic reconstruction method, the object of which is to realize fast reconstruction of Fourier stack high-resolution microscopic images.

Background technique

High resolution and wide field of view have always been the goals pursued by optical microscopy technology. However, due to the limitations that the optical system itself cannot break through, high resolution and wide field of view cannot be combined. This problem greatly limits optical microscopy. The application of technology in many fields. Fourier laminated microscopic imaging (FPM) is a new computational imaging technology developed this year. By introducing phase recovery and synthetic aperture technology, it realizes high-resolution reconstruction of wide-field images. Fourier stack microscopic imaging technology has received a lot of research and attention in the fields of optical microscopy, clinical medicine, and biological sciences since it was proposed because it combines many advantages of wide field of view, high resolution, and phase imaging.

However, the improvement of spatial resolution by Fourier stack imaging technology comes at the expense of temporal resolution. Different from the traditional imaging technology of what you see is what you get, the result of Fourier stack microscopic imaging technology is the reconstruction of hundreds of multi-angle illumination images. The process of collecting original images and image reconstruction consumes a lot of time. The time spent on reconstruction is often measured in hours, which seriously affects the practical application of Fourier stack microscopic imaging technology. In order to solve the problem of long acquisition time, a large number of scholars have proposed methods. For example, the multiplexed illumination method disclosed in the paper "Multiplexedcoded illumination for Fourier Ptychography with an LED array microscope" published by Lei Tian et al. can double the time consumed in the image acquisition process. In order to solve the problem of long reconstruction time, some scholars have proposed some methods, but these methods are often at the expense of reconstruction quality. For example, the paper "Content adaptive illumination for Fourierptychography" published by Liheng Bian et al. analyzed the spectral information of the sample and eliminated the intensity images containing less high-frequency information, thereby reducing the number of original images to improve the speed of image reconstruction. Although this method can improve the reconstruction speed to a certain extent, excluding the original images containing effective information will cause the degradation of reconstruction quality. How to increase the speed of reconstruction under the condition of ensuring the quality of reconstruction is an urgent problem to be solved in Fourier stack research.

Contents of the invention

In view of this, the present invention proposes a grouped Fourier stack high-resolution microscopic reconstruction method, which can overcome the shortcomings of the traditional Fourier stack reconstruction method in terms of speed, and realize the Fourier stack reconstruction speed An increase in magnitude.

To achieve the above object, the technical scheme of the present invention is as follows:

Step 1, building a Fourier stack microscopic imaging system, the lighting module in the imaging system is a programmable LED array, and the center of the LED array is aligned with the optical axis of the microscopic system;

Step 2, imaging the sample to be observed with the imaging system described in step 1, shielding the ambient light during imaging, lighting up the LEDs in the LED array one by one, and recording the original image under a single LED;

Step 3, take the center of the LED array as the center, light the LEDs outward layer by layer, and the original image of the LED layers that are lit in turn

Corresponding records are the first group, the second group...the nth group of original images, where the second group includes the LEDs of the first group, the third group includes the LEDs of the second group...the nth group includes all the LEDs;

Step 4, initialize the spectrum to obtain an empty spectrum; use the results of step 2 to obtain the corresponding position of each LED in the LED array in the spectrum; use the results of step 3 as a unit, and use them sequentially from the first group Each group of original images updates the frequency spectrum, wherein, for each group of LEDs, according to the corresponding position of each LED in the frequency spectrum, the frequency spectrum of the corresponding position is updated, until the frequency spectrum corresponding to all groups is updated, and the Fourier stack is completed. layer microscopic reconstruction.

Wherein, in the step 4, the specific steps for updating the frequency spectrum by using each group of original image data are:

Step 41, using the original image corresponding to the center position LED to obtain the initial value of the complex amplitude of the image, and then converting to the frequency domain to obtain the initialized Fourier spectrum;

Step 42, for the jth group of original image data, j=1, 2, 3...n, select the corresponding area in the Fourier spectrum and obtain the complex amplitude estimate corresponding to each LED through inverse transformation, and then use the jth group The intensity of the original image data updates the complex amplitude, and then transforms back to the frequency domain to replace the corresponding area of the spectrum corresponding to each LED;

Step 43, repeat step 42 in the group until the judging condition for entering the next group of iterations is satisfied, wherein the judging condition for the iteration is:

where mean represents the array mean function, abs represents the absolute value function, I _k represents the high-resolution intensity image obtained after the kth iteration in the current group reconstruction, and I _k-1 represents the k-1th iteration in the current group reconstruction Obtained high-resolution intensity images;

Step 44, repeat steps 42 to 43 until all n groups of image data are updated;

Step 45, inversely transforming the reconstructed Fourier spectrum into the space domain, so as to obtain the reconstructed intensity and phase images.

Wherein, in the step 1, the reflective or active lighting system of a typical biological optical microscope is removed, and a programmable LED array is installed with a bracket or a base to replace the original lighting system.

Wherein, in the step 2, the LED array is controlled by the Arduino microcontroller, including the specific position and color of the lighted LED; the original image is collected by the CMOS image sensor camera, and the Arduino microcontroller and the CMOS image sensor camera are all controlled by the PC. machine control.

Wherein, the sample to be observed is a biological tissue section or a blood smear.

Wherein, the LED array is a rectangular array or a circular array, and the center of the array is one LED or a 2×2 matrix LED array.

Beneficial effect:

The grouped Fourier stack microscopic reconstruction method of the present invention utilizes the grouped Fourier stack to reconstruct from coarse to fine, and uses a graphics processor to process the reconstruction operation in parallel, which overcomes the traditional Fourier stack reconstruction Insufficient speed of the method realizes an order-of-magnitude improvement in the speed of Fourier stack reconstruction, and at the same time realizes high-resolution reconstruction; and can be easily combined with the existing Fourier stack reconstruction algorithm without Need to add too many hardware resources or special operation steps.

Description of drawings

Fig. 1 is the flowchart of grouped Fourier stack microscopic reconstruction method of the present invention;

Fig. 2 is the basic composition of the Fourier stack microscope system of the present invention;

Fig. 3 is each group of calculation flowcharts of the grouping type Fourier stack reconstruction algorithm of the present invention;

Fig. 4 is the simulation schematic diagram of traditional Fourier stack reconstruction algorithm;

Fig. 5 is a simulation schematic diagram of the grouped Fourier stack reconstruction algorithm of the present invention.

Detailed ways

In order to better illustrate the purpose and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and examples.

Fourier stacking technology models the sample imaging process under obliquely incident quasi-monochromatic light illumination, and thus develops a method for recovering the high-resolution complex amplitude of the sample by using multiple intensity images for inverse reconstruction. The grouped Fourier stack reconstruction algorithm is based on the development and improvement of the existing Fourier stack imaging principle, and the grouped Fourier stack reconstruction method of the present invention is based on the existing Fourier stack imaging The basic process of imaging and reconstruction in the technique.

Among them, the imaging process of the existing Fourier stack imaging technology is as follows:

In Fourier stack imaging, for a thin sample, its optical properties can be expressed as a spatial domain transfer function o(r). When the sample is illuminated by an obliquely incident plane wave whose wave vector is u _l , the complex amplitude reaching the sample plane is exp(i2πu _l r), and the complex amplitude of the outgoing light after modulation by the sample can be expressed as:

e(r)=o(r)exp(i2πu _l r),

Converting this complex amplitude into the frequency domain yields:

It can be found that the result is the result of shifting the center of the original frequency spectrum from zero to u _l . If the pupil function of the objective lens is expressed as P(u), the complex amplitude reaching the sensor surface can be expressed as:

This formula is also known as the forward model of Fourier stack imaging, which establishes the relationship between the illumination of the LED and the acquired image;

The reconstruction process of the existing Fourier stack imaging technology is as follows:

The main process of Fourier stack image restoration is to synthesize multiple images containing different frequency domain information in the frequency domain, so as to obtain frequency domain information far exceeding that of a single imaging, and then improve the spatial resolution. However, since the image sensor can only collect the intensity information of the complex amplitude, but not the phase information, the Fourier stacking technology also introduces a phase recovery method to estimate the lost phase information during the synthesis, so that the restored The complex amplitude is more accurate. In the traditional method, the actual recovery process is to use the collected intensity image to continuously update the corresponding sub-region in the spectrogram, expressed as:

In order to simplify the formula form, the forward model and recovery formula of Fourier stack imaging can be expressed in vector form as:

g _le =F ^-1 (PO _l )

and

Finally, transform the reconstructed frequency domain into the spatial domain to obtain the high-resolution intensity image and phase image of the sample:

I _h =abs(F ^-1 (O ^u )) ² , G _h =angle(F ^-1 (O ^u )).

Traditional restoration algorithms are generally divided into two categories: sequential and global. The sequential method only estimates and updates one spectral sub-region at a time, while the global method estimates and restores all spectral sub-regions at one time. The sequential method has the characteristics of less memory consumption and fast convergence of results, while the global method can obtain better reconstruction results despite the large amount of calculation and memory usage.

The grouped Fourier stack reconstruction algorithm proposed by the present invention combines the advantages of the traditional sequential reconstruction method and the global reconstruction method, and simultaneously updates a part of the frequency domain sub-regions in one cycle, and gradually expands the update region from low-frequency To high frequency to speed up the reconstruction process, taking into account the reconstruction speed and effect. The idea of grouping the collected original images to achieve fast convergence is based on the fact that in existing studies, researchers have found that a better initial value can greatly speed up the convergence speed, such as using low-resolution Graph initialization converges faster than initialization with zero values. If multiple initial values can be set and the frequency spectrum is updated in steps, the speed of convergence to the final result can be greatly accelerated. The present invention proposes a grouped frequency spectrum iterative update strategy. When the convergence speed of iteration in one group becomes slower and slower, the data of the next group is introduced, so that the iteration speed will be greatly improved.

In order to use the grouping strategy for reconstruction, it is necessary to group the LEDs according to their positions in the array, and determine the group to which the original image belongs according to the grouping of the corresponding LEDs. Expand the LED array layer by layer from the center of the array to the outermost position, and record it as the first group, the second group...the nth group, when the LED array is a (2n+1)×(2n+1) rectangular array, the first A group contains the center 3×3 LEDs, the second group contains the center 5×5 LEDs, and so on until the nth group contains all (2n+1)×(2n+1) LEDs. When the LED array is a 2n×2n rectangular array, the first group contains 2×2 LEDs in the center, the second group contains 4×4 LEDs in the center, and so on until the nth group contains all 2n×2n LEDs.

The reason for grouping LEDs in this way is that images captured with LEDs at the center correspond to lower frequency information in the spectrogram, while images captured with LEDs at the edges correspond to higher frequencies in the spectrogram. Information. In order to achieve a coarse-to-fine update strategy, the update range of the spectrum must be gradually extended from low frequency to high frequency during iteration, and it cannot be grouped arbitrarily. In the traditional global reconstruction method, all intensity images need to be calculated in each iteration, which consumes a large amount of calculation. Although the reconstruction quality is high, the reconstruction speed is slow. In the present invention, the grouped intensity images are substituted into the reconstruction process from less to more, and thanks to the less data volume in the previous iteration, the reconstruction speed is accelerated; in the end, all intensity images participate in the reconstruction, which also ensures the quality of reconstruction .

With reference to Fig. 1, a kind of grouping type Fourier stack high-resolution microscopic reconstruction method that the present invention proposes comprises the following steps:

Step 1, build a Fourier stack high-resolution microscopic imaging system:

The traditional optical microscope mainly includes the main parts such as illumination system, objective lens, tube lens, camera and bracket, and the hardware structure of the Fourier stack microscope is different from that of the traditional optical microscope, which can be realized by modifying the existing optical microscope. . The specific modification method is to remove the original lighting system and install a programmable LED array with a bracket or base. The LED array is mounted at a distance of approximately 10 cm below the stage, with the module axis aligned with the microscope optical axis. The typical type of LED array is a printed circuit board soldered with SMD LED components. The LED components are arranged in a matrix. The LED array is a rectangular array or a circular array. The center of the array is 1 LED or 1 2 ×2 matrix LED array. The LED component adopts a programmable full-color LED with a built-in SK6812 chip, so that the position, color and time of LED lighting can be controlled by programming. In order to obtain a larger field of view and a higher resolution enhancement effect, the Fourier stack microscope generally chooses to use a low-power objective lens such as 2× or 4×. Refer to Figure 2 for the structure of a typical Fourier stack microscope.

Step 2, collect the original image data of the Fourier stack microscope:

The image acquisition process of the Fourier stack microscope is also different from that of the traditional optical microscope. When imaging the sample to be observed, the Fourier stack microscopic imaging system needs to be shielded to reduce the influence of ambient light on the LED light source. The LED array is controlled by an Arduino microcontroller, and receives instructions from the PC through the serial port to control the specific position, color, and exposure time of the LEDs. The original image is collected by a high-sensitivity CMOS image sensor, and the data is transmitted between the CMOS image sensor acquisition card and the PC. The CMOS image sensor accepts the instructions of the PC to adjust the number of sampling bits, shutter speed and exposure time. The control program of the LED array and the image sensor is implemented by programming on the PC, which lights up the LEDs one by one and collects the corresponding images. Perform preprocessing such as noise reduction on the collected images to improve the final reconstruction effect.

Step 3, group the original images of Fourier stack imaging:

The grouping takes the center of the LED array as the center and gradually expands from the center to the outermost position. Expand the rectangular LED array of 2n×2n or (2n+1)×(2n+1) layer by layer from the center of the array to the outermost edge, and record it as group 1, group 2...n group in sequence. When the LED array is a (2n+1)×(2n+1) rectangular array, the first group contains 3×3 LEDs in the center, the second group contains 5×5 LEDs in the center, and so on until the nth group contains all (2n+1)×(2n+1) LEDs. When the LED array is a 2n×2n rectangular array, the first group contains 2×2 LEDs in the center, the second group contains 4×4 LEDs in the center, and so on until the nth group contains all 2n×2n LEDs. The image collected by the LED at the center corresponds to the information of the lower frequency in the spectrogram, while the image collected by the LED at the edge corresponds to the information of the higher frequency in the spectrogram. The intensity image corresponding to the lower frequency information has a higher signal-to-noise ratio, so the convergence is fast and the reconstruction error is small. In the traditional global reconstruction method, all intensity images need to be calculated in each iteration, which consumes a large amount of calculation. Although the reconstruction quality is high, the reconstruction speed is slow. The present invention substitutes the grouped intensity images into the reconstruction process according to the number of iterations from few to many, benefiting from the small amount of data in the previous iteration, the reconstruction speed is accelerated; finally all the intensity images participate in the reconstruction, which also ensures Quality of reconstruction.

Step 4, perform grouped Fourier stack reconstruction:

Initialize the spectrum to obtain an empty spectrum; use the results of step 2 to obtain the position of each LED in the LED array in the spectrum; use the results of step 3 as a unit, and use each group's original The image updates the frequency spectrum. For each group of LEDs, the frequency spectrum of the corresponding position is updated according to the position of each LED in the frequency spectrum, until the frequency spectrum corresponding to all groups is updated, and the Fourier stack microscopy is completed. reconstruction. This step is the core of the grouped Fourier stack high-resolution microscopic reconstruction method. The specific steps to update the spectrum by using each group of original image data are as follows:

Step 41, using the original image corresponding to the center position LED to obtain the initial value of the complex amplitude of the image, and then obtain the initialized Fourier spectrum O ₀ in the frequency domain through Fourier transform F;

Step 42, for the jth group of original image data, j=1,2,3...n, select the sub-region O _l(j-1) in the Fourier spectrum and obtain the complex amplitude corresponding to each LED through inverse transformation Estimate g _lj , then use the intensity I _lj of the jth group of original images to update the complex amplitude estimate, and then transform back to the frequency domain to replace the spectral sub-region corresponding to each LED;

Step 43, repeat step 42 in the group until the judgment condition for entering the next group iteration is met, so that the update process of the subsequent group can be significantly accelerated;

Step 44, repeat steps 42 to 43 until all n groups of image data are updated;

In step 45, inverse Fourier transform F ^-1 is performed on the reconstructed Fourier spectrum O _j into the space domain, so as to obtain a reconstructed high-resolution intensity and phase image. The entire reconstruction process can be expressed as:

Where j represents the grouping of the image and also represents the grouping of the reconstruction process, P represents the pupil function of the microscope objective, and W _j is the weight for correcting spectral overlap.

The formula for judging whether to enter the next set of iterations is:

where mean represents the array mean function, abs represents the absolute value function, I _k represents the high-resolution intensity image obtained after the kth iteration in the current group reconstruction, and I _k-1 represents the k-1th iteration in the current group reconstruction Acquired high-resolution intensity images. The physical meaning of this formula is that the update effect of the kth iteration on the image has been lower than a certain threshold. This formula can also be applied to traditional Fourier stack reconstruction as a criterion for terminating iterations.

Similar to the traditional reconstruction method, the high-resolution intensity image and phase image of the sample can be obtained by transforming the reconstructed frequency domain into the spatial domain:

I _h =abs(F ^-1 (O _j )) ² , G _h =angle(F ^-1 (O _j )).

In the grouped Fourier stack reconstruction algorithm of the present invention, most of the image processing operations can be processed in parallel by a graphics processor, and the calculation speed far exceeds that of the traditional CPU, thereby greatly improving the efficiency of the algorithm. Referring to FIG. 3 is a flow chart of the reconstruction algorithm for each group of the present invention. Referring to FIG. 4 and FIG. 5 , they are schematic diagrams of simulation using the traditional method and the grouped Fourier stack reconstruction algorithm, respectively. The method proposed in the present invention combines the advantages of the sequential reconstruction method and the global reconstruction method, and greatly speeds up the reconstruction speed on the basis of maintaining good reconstruction results. Compared with the global method, the speed-up effect is about 10 times, Compared with the sequential method, the speed-up effect is about 8 times.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (6)

1. a grouping type Fourier stack microscopic reconstruction method, is characterized in that, comprises the steps: Step 1, building a Fourier stack microscopic imaging system, the lighting module in the imaging system is a programmable LED array, and the center of the LED array is aligned with the optical axis of the microscopic system; Step 2, imaging the sample to be observed with the imaging system described in step 1, shielding the ambient light during imaging, lighting up the LEDs in the LED array one by one, and recording the original image under a single LED; Step 3, take the center of the LED array as the center, light up the LEDs layer by layer, and record the original images of the successively lit LED layers as the first group, the second group...nth group of original images, where the second group includes Group 1 LEDs, Group 3 includes LEDs from Group 2... Group n contains all LEDs; Step 4, initialize the spectrum to obtain an empty spectrum; use the results of step 2 to obtain the corresponding position of each LED in the LED array in the spectrum; use the results of step 3 as a unit, and use them sequentially from the first group Each group of original images updates the frequency spectrum, wherein, for each group of LEDs, according to the corresponding position of each LED in the frequency spectrum, the frequency spectrum of the corresponding position is updated, until the frequency spectrum corresponding to all groups is updated, and the Fourier stack is completed. layer microscopic reconstruction. 2. a kind of grouping type Fourier stack microscopic reconstruction method according to claim 1, is characterized in that, in described step 4, utilizes each group of original image data to update the specific steps of spectrum as: Step 41, using the original image corresponding to the center position LED to obtain the initial value of the complex amplitude of the image, and then converting to the frequency domain to obtain the initialized Fourier spectrum; Step 42, for the jth group of original image data, j=1, 2, 3...n, select the corresponding area in the Fourier spectrum and obtain the complex amplitude estimate corresponding to each LED through inverse transformation, and then use the jth group The intensity of the original image data updates the complex amplitude, and then transforms back to the frequency domain to replace the corresponding area of the spectrum corresponding to each LED; Step 43, repeat step 42 in the group until the judging condition for entering the next group of iterations is satisfied, wherein the judging condition for the iteration is: where mean represents the array mean function, abs represents the absolute value function, I _k represents the high-resolution intensity image obtained after the kth iteration in the current group reconstruction, and I _k-1 represents the k-1th iteration in the current group reconstruction Obtained high-resolution intensity images; Step 44, repeat steps 42 to 43 until all n groups of image data are updated; Step 45, inversely transforming the reconstructed Fourier spectrum into the space domain, so as to obtain the reconstructed intensity and phase images. 3. A grouped Fourier stack microscopic reconstruction method according to claim 1, characterized in that, in said step 1, by removing the reflective or active illumination system of a typical biological optical microscope, Install a programmable LED array with a bracket or base to replace the original lighting system. 4. A kind of grouped Fourier stack microscopic reconstruction method according to claim 1, characterized in that, in said step 2, the LED array is controlled by the Arduino microcontroller, including the specific position of lighting the LED and Color; the original image is collected by a CMOS image sensor camera, and both the Arduino microcontroller and the CMOS image sensor camera are controlled by a PC. 5. A grouped Fourier stack high-resolution microscopic reconstruction method according to claim 1, wherein the sample to be observed is a biological tissue section or a blood smear. 6. A grouped Fourier stack high-resolution microscopic reconstruction method according to claim 1, wherein the LED array is a rectangular array or a circular array, and the center of the array is one LED Or a 2×2 matrix of LED arrays.