RU2829111C1 - Structured illumination microscopy systems and methods - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to structured illumination microscopy. Capturing and processing in real time is carried out on a plurality of image fragments on a large image plane by dividing them into sub-fragments, processing the sub-fragments and generating an image with improved resolution from the sub-fragments. Images with improved resolution can be combined into improved images and used in subsequent stages of analysis.

EFFECT: invention provides improved images, which can be used separately or in combination.

37 cl, 57 dwg

Description

PRIORITY APPLICATIONS

[0001] This application claims the benefit of and incorporates references to U.S. Provisional Patent Application No. 62/924,130, entitled “Systems and Methods for Structured Illumination Microscopy” (Accession Number ILLM 1012-1), filed October 21, 2019, and U.S. Provisional Patent Application No. 62/924,138, entitled “Improved Calculation Efficiency for Structured Illumination Microscopy” (Accession Number ILLM 1022-1), filed October 21, 2019. This application also claims the benefit of and incorporates references to regular U.S. Patent Application No. 17/075,692, entitled “Systems and Methods for Structured Illumination Microscopy” (Accession Number ILLM 1012-2), filed October 21, 2020 and U.S. Patent Application No. 17/075,694, entitled “Improved Calculation Efficiency for Structured Illumination Microscopy” (Accession Number ILLM 1022-2), filed October 21, 2020, both of which claim the benefit of the above-mentioned provisional applications.

COMBINATION OF FEATURES OF THE INVENTION

[0002] The following applications and materials are incorporated herein by reference in all respects as if fully set forth herein: U.S. Provisional Patent Application No. 62/692,303, entitled “Luminescence Imaging Device,” filed June 29, 2018 (unpublished), and U.S. Regular Patent Application entitled “Luminescence Imaging Device,” filed June 29, 2019 (Accession Number P-1683-US).

AREA OF THE DESCRIBED TECHNOLOGY

[0003] The described technology relates to structured illumination microscopy (SIM). In particular, the described technology relates to capturing and processing in real time a plurality of image fragments on a large image plane by dividing them into sub-fragments, efficiently processing the sub-fragments, and generating an image with improved resolution from the sub-fragments. The images with improved resolution can be combined into improved images and used in subsequent stages of analysis.

[0004] The described technology relates to structured illumination microscopy. In particular, the described technology relates to reducing the computations for real-time processing of multiple image fragments on a large image plane and obtaining images with improved resolution from image fragments/sub-fragments. During some intermediate transformations in the SIM technological chain, almost half of the multiplication and division required in other cases can be replaced by search operations using specific applications of symmetry.

PREREQUISITES FOR THE CREATION OF THE INVENTION

[0005] The subject matter of the invention described in this section should not be considered a known technical solution solely by virtue of its mention in this section. Likewise, the problem mentioned in this section or related to the subject matter of the invention presented in this section should not be considered to have been previously recognized in known technical solutions. The subject matter of the invention in this section merely represents various approaches, which in themselves may also correspond to embodiments of the claimed technology.

[0006] More than a decade ago, the pioneers of structured illumination microscopy won the Nobel Prize in Physics. The breakthrough was achieving image resolution above the Abbe diffraction limit.

[0007] SIM 2D and 3D microscopy has been used to visualize biological samples, such as parts of individual cells. Much effort has been put into studying the internal structure of cells, resulting in many alternative technical variations.

[0008] Resolving millions of sources distributed over an image plane presents a completely different challenge than studying the internal structure of cells. For example, one new approach that has been developed is to combine multiple spectral illumination images to produce an image with improved resolution after extensive computation. Processing a large image plane in real time with modest resources requires a radically different approach than is currently used.

[0009] Accordingly, the possibility arises of introducing new methods and systems adapted to processing large image planes with lower requirements for computing resources.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

[0010] Color drawings in priority applications that correspond to the black and white drawings in this application may be requested from the United States Patent and Trademark Office (USPTO) in PAIR under the Supplemental Content tab. The same reference numbers in different drawings indicate similar components. In addition, the drawings may not be to scale; the focus of their preparation is usually on illustrating the principles of the described technology. The description below presents various embodiments of the described technology with reference to the following drawings, which illustrate the following:

[0011] FIG. 1A illustrates the formation of a moire interference fringe using a one-dimensional (1D) modulation grating.

[0012] FIG. 1B is a graphical illustration of illumination intensities produced by a two-dimensional (2D) structured lighting pattern.

[0013] FIG. 1C illustrates an example of a geometric structure for arranging nanowells.

[0014] FIG. 2 illustrates a structured illumination microscopy imaging system that can use spatially structured excitation light to form an image of a sample.

[0015] FIG. 3 is a diagram of an optical system illustrating an example of an optical configuration of a dual-arm SIM imaging system.

[0016] FIG. 4A and 4B are diagrams illustrating an optical configuration of a SIM imaging system with two optical arrays for slides.

[0017] FIG. 5A illustrates unwanted changes in the spacing parameter that may occur in SIM imaging systems.

[0018] FIG. 5B illustrates unwanted phase parameter changes that may occur in SIM imaging systems.

[0019] FIG. 5C illustrates unwanted orientation angle changes that may occur in SIM imaging systems.

[0020]

[0021] FIG. 6 illustrates simplified patterns of interference fringes of illumination that can be projected onto a sample plane using the vertical and horizontal gratings of the SIM imaging system.

[0022] FIG. 7 illustrates simplified patterns of interference fringes of illumination that can be projected onto a plane using the first and second gratings of a SIM imaging system with two optical gratings for slides.

[0023] FIG. 8A illustrates a simplified image of parallel lines curved due to distortion of a magnifying lens.

[0024] FIGS. 8B and 8C illustrate the spacing between nominally parallel lines.

[0025] FIG. 8D illustrates an example of sub-portions or sub-fields of a full field of view (FOV) image.

[0026] FIG. 9 illustrates a general image of a surface with color coding that closely matches the observed data points shown in red.

[0027] FIGS. 10A and 10B in the form of an inverted bowl illustrate the measured spacing relative to the spacing in a subfield located close to the center.

[0028] FIGS. 11A, 11B, and 11C show a comparison of the measured spacing distortion with a quadratic and cubic surface approximation.

[0029] FIG. 12A illustrates measured data points without smoothing by curve fitting.

[0030] FIG. 12B illustrates actual data compared to a quadratically fitted surface.

[0031] FIGS. 13A-13E illustrate improvement of curve approximation by truncation along the sensor boundary.

[0032] FIG. 14A graphically illustrates the reduction in mean square error (MSE) for different shrinkage ratios.

[0033] FIGS. 14B-14G illustrate an improvement in the quadratic approximation for the angular distortion model by gradually increasing the shrinkage factor applied to the full FOV image data.

[0034] FIG. 14H is a graph illustrating a slight improvement in the approximation of a quadratic surface when the shrinkage factor value is six.

[0035] FIGS. 15A and 15B illustrate a quadratic approximation of a reaction surface obtained by approximating the fringe spacing distortion (A) and the angle spacing distortion (B) from full FOV image data.

[0036] FIG. 16A illustrates the transfer of a sub-fragment phase offset from the sub-fragment coordinate system to the full FOV image coordinate system.

[0037] FIG. 16B illustrates the position of a point in a sub-fragment relative to the coordinate system of the sub-fragment and the coordinate system of the full FOV image.

[0038] FIG. 17 is an example of a phase shift lookup table for sub-tiles in a full FOV image.

[0039] FIG. 18 is an overview of the process steps in a non-redundant SIM fragment image recovery algorithm.

[0040] FIG. 19 illustrates symmetry in Fourier space for matrices with even and odd numbers of rows and columns.

[0041] FIG. 20 illustrates non-redundant and redundant halves of full matrices in the frequency domain representing three images obtained using a single illumination peak angle.

[0042] FIG. 21 illustrates a change in the configuration of non-redundant halves of three matrices in the frequency domain representing three images obtained using a single illumination peak angle.

[0043] FIG. 22 illustrates a process of dividing a band by multiplying an inverse band division matrix by the reshaped matrix of FIG. 21.

[0044] FIG. 23A illustrates an example of a (1, 1) shift operation applied to a matrix.

[0045] FIGS. 23B and 23C illustrate averaging of the first and middle columns in non-redundant reconstruction of a SIM image.

[0046] FIG. 24 is a simplified block diagram of a computer system that may be used to implement the described technology.

[0047] FIGS. 25A and 25B illustrate a reduction in the error rate and the cluster percent bandpass filter (%PF) metric with updates in the SIM image recovery algorithm.

[0048] FIGS. 26A-26C graphically illustrate the reduction in error rate over multiple cycles of sequencing runs.

[0049] FIG. 26D graphically illustrates the reduction in error rate by incorporating phase learning and local problem elimination techniques into the original SIM algorithm.

DETAILED DESCRIPTION

[0050] The following description is provided to enable any person skilled in the art to make and use the described technology. The description is provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described technology. Thus, the described technology is not limited to the embodiments shown, but should encompass a variety of possible variations consistent with the principles and features described herein.

Introduction

[0051] Structured illumination microscopy in two and three dimensions has made it possible to probe the internal structure of living cells and even the molecular structure of biological materials, as it improves the resolution of image acquisition systems. See, for example, Sydor, Andrew & Czymmek, Kirk & Puchner, Elias & Mennella, Vito. (2015). Super-Resolution Microscopy: From Single Molecules to Supramolecular Assemblies. Trends in Cell Biology. 25. 10.1016/j.teb.2015.10.004; also Lal et al. 2015, A, Shan, C, & Xi, P. (2015). An image reconstruction algorithm using structured illumination microscopy. IEEE Journal of Selected Topics in Quantum Electronics, 22(4), 50–63. For readers who want to refresh their memory on the SIM principle, Lal et al. 2015 is an excellent combination of illustrations and mathematical explanation, not to mention the derivation of the equations used. This description covers SIM technology for flow cell imaging using several technologies that can be used separately or in combination.

[0052] SIM enables the deployment of densely packed samples from flow cells with fluorescent signals from millions of image points, thereby reducing the amount of reagents required for processing and increasing the throughput of imaging. The technique is to deploy densely packed fluorescent samples near the Abbe diffraction limit to deploy adjacent SIM light sources. The samples may be in regularly spaced nanowells or may be randomly distributed clusters. Much of the following description applies to nanowells with an ordered structure, but the SIM technology is also applicable to randomly distributed clusters.

[0053] Technical issues related to lens distortion and the computational resources required arise from the deployment of densely packed sources. This description addresses these technical issues.

[0054] Structured illumination can create images that have several times more deployed illumination sources than under normal illumination. The information is not created by itself. Instead, multiple images with varying angles and phase shifts of structured illumination are used to transform closely spaced, otherwise non-deployable, high spatial frequency features into lower frequency signals that can be sensed by an optical system without violating the Abbe diffraction limit. This limit is physically inherent in image formation by the nature of light and optics and is expressed as a function of the wavelength of the light and the numerical aperture (NA) of the final objective lens. Using SIM reconstruction, information from multiple images is transformed from the spatial domain to the Fourier domain, combined and processed, and then reconstructed into an enhanced image.

[0055] In SIM, a grating is used or an interference pattern is formed between the illumination source and the sample to form an illumination pattern, such as a pattern whose intensity varies according to a sine or cosine wave. In the context of SIM, in addition to referring to a surface that creates a structured illumination pattern, the term "diffraction grating" sometimes refers to a projected structured illumination pattern. In an alternative embodiment, the structured illumination pattern may be formed as an interference pattern between portions of a split coherent beam.

[0056] Projection of structured illumination onto a sample plane, such as shown in FIG. 1, mixes the illumination pattern with fluorescent (or reflective) sources in the sample and produces a new signal, sometimes called a moire fringe or aliasing. The new signal shifts high spatial frequency information to a lower spatial frequency that can be captured without violating the Abbe diffraction limit. After capturing images of a sample illuminated with a one-dimensional (1D) intensity modulation pattern, as shown in FIG. 1A, or a two-dimensional (2D) intensity modulation pattern, as shown in FIG. 1B, a system of linear equations is solved and used to extract from a plurality of moire fringe or alias images portions of the new signal that contains information shifted from a higher to a lower spatial frequency. To solve the linear equations, three or more images are captured with a stepwise shift or offset of the structured illumination pattern. Often, images with varying phases (3-7 phases) per angle (1-5 angles) can be captured and then striped to shift the Fourier domain and recombine. Increasing the number of images can improve the quality of the reconstructed images by increasing the signal-to-noise ratio. However, it can also increase the computation time. The Fourier representation of the striped images is shifted and summed to obtain a reconstructed sum. Finally, a new high-resolution image is reconstructed from the reconstructed sum using the inverse fast Fourier transform (FFT).

[0057] The use of multiple images and shifting only improves the image information in a specific direction along which the 1D structured illumination pattern is shifted, so the structured illumination or diffraction grating is rotated and the shifting procedure is repeated. A rotation of the 1D pattern, for example by 45, 60, 90 or 120 degrees, can be applied to obtain six or nine images in three steps. Environmental factors such as damage to molecules on the surface of the flow cells due to blue and green lasers, tilting of fragments on the surface of the flow cells, etc., can cause distortions in the phase shift estimates for image sub-tiles or sub-windows of the full field of view (FOV). They can also cause differences between tiles on the substrate. More frequent estimates of the SIM image reconstruction parameters can be performed to compensate for these environmental factors. For example, the phase shifts of the sub-tiles can be re-estimated for each tile to minimize these errors with each reconstruction cycle. The angle and offset parameters do not change as frequently as the phase offset, and therefore increasing the frequency of their evaluation may introduce additional calculations that are not necessary. However, the angle and offset parameters can be calculated more frequently if required.

Application of SIM to Flow Cell Image Analysis

[0058] Imaging a flow cell with millions of fluorescent samples is more like scanning a telescope than studying the microstructures of a living cell. Scanning a flow cell with low-cost optics is more like capturing images with an instamatic camera than using the adaptive optics at Palomar Observatory. Scanning a flow cell or the sky involves many images that cover portions of the target, as opposed to imaging a living cell in the sweet spot of a lens. The number of images needed to cover any given target depends on the field of view of each image portion and the degree of overlap. The low-cost lens has distortions at the edges of the field of view, making it difficult to create enhanced images.

[0059] The calculations used in SIM reconstruction are sensitive to lens distortions, pattern alignment in a particular set of images, and thermal effects for reference alignment over imaging cycles that progress over several hours. Increasing the field of view by using more of the lens instead of the sweet spot in the center makes SIM image reconstruction susceptible to distortions caused by lens aberrations. These aberrations, such as asymmetric aberration, distort the structured illumination pattern and make parallel lines of illumination peaks look like curves, as generally shown in FIG. 8A, changing the distance between the pattern brightness peaks and the apparent corners of the pattern.

[0060] In practice, instruments must have characteristics that allow distortions and offsets to be compared between parts of a fragment. Due to the duration of flow cell imaging over several cycles, some instruments will benefit from updated characteristics at the beginning of a cycle or even during a cycle. Thermal distortion over several hours of flow cell imaging over several hundred cycles may allow temporary or regular regeneration of the angle, spacing, and phase (offset) of the structured illumination.

[0061] The angle (rotation) and spacing (scale) of the projected structured illumination pattern can be determined using the peak fringe estimate in device characterization. The estimated phase or offset of the repeating pattern in the execution direction can be expressed as a spatial shift from 0 to 360 degrees or in radians. Lens distortion complicates the parameter estimation, leading to proposals for computationally expensive reconstructions from multiple images. See, e.g., Ayuk et al. (2013) Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIM algorithm. Optics letters. 38. 4723-6. 10.1364/OL.38.004723; and Mudry et al. (2012) Structured illumination microscopy using unknown speckle patterns. Nature Photonics. 6. 312-315. The described technology simplifies the evaluation of repeated parameters and reduces the number of necessary calculations in repeated evaluation cycles.

[0062] Real-time parameter estimation for SIM reconstruction is a computationally challenging task. The computational power required for SIM reconstruction increases with the cube of the number of pixels in an image field or subfield. For example, for an image with a width of M pixels and a height of N pixels, the Fourier transform may have the following computational complexity: k*M*N(log(M*N)). Thus, the order of magnitude of the resources for SIM image reconstruction may increase in proportion to the square of O(N ² ) or the cube of O(N ³ ) as the number of pixels in the images increases. Thus, doubling the image dimensions, for example from 512 × 512 to 1024 × 1024, may result in an eightfold increase in computational costs. Particularly challenging is reconstructing an enhanced image from six or nine 20-megapixel images on a CPU at a rate of one completed reconstruction within 0.3 seconds during a scan. Real-time processing is desirable to reduce storage requirements and to track the quality of flow cell processing over the hours that scanning and sequencing occurs. The described technology reduces the number of key calculations by approximately half by exploiting the symmetry and approximate symmetry in the Fourier domain coefficient matrices.

Eliminating lens distortion

[0063] The described technology solves the problem of minor lens distortions for an optical system within budget manufacturing tolerances by dividing the captured image fragment into sub-fragments and processing the sub-fragment located near the center in a different way compared to the other sub-fragments. An image captured by an optical sensor may be called a fragment. An imaging cycle for a flow cell captures multiple image fragments with some overlap. Each image fragment is divided into sub-fragments, which are evaluated independently. The sub-fragments may be reconstructed independently of each other, even in parallel. Reconstructions from the improved sub-fragments may be stitched together to create a reconstructed fragment with improved spatial resolution.

[0064] The interpolation technique described for the ratio of the reconstruction parameters of the closest sub-fragment to the other sub-fragments approximates a non-linear function, such as a quadratic curve, using a fragmentary approximation. The described technique breaks an image fragment into sub-fragments such that the peak lines are approximately evenly spaced within the sub-fragment, thereby achieving better image quality from the reconstructed sub-fragments in the field of view of the lens.

[0065] Sub-tile creation to reduce fringe distortion introduces a new problem: reconstruction parameters must be estimated for each sub-tile. Parameter estimation is the most expensive set of values in SIM reconstruction, and sub-tile division increases the execution time of parameter estimation by at least one order of magnitude, e.g., sub-tile division of an image into 5×5 sub-tiles will produce an algorithm that is 25 times slower. Sub-tile division of a tile into 8×11 sub-tiles requires 88 sets of reconstruction parameters instead of one set.

[0066] The described technology learns a function that matches the distortion and thereby reduces the amount of computation in repeated cycles. The found function is valid for all cycles and image sets, since it matches the optical characteristics of the optical system that change when the optical system is changed, for example, during repeated alignment. During the evaluation cycles between the instrument characteristics, the expensive calculation of the parameters is focused on one sub-patch closest to the center. The parameters for each sub-patch are matched with measurements in cycles distant from the central reference sub-patch and other sub-patch.

[0067] Three parameters are matched for sub-patterns: illumination peak angle, illumination peak spacing, and phase shift. The illumination peak angle is also called the grating angle. The illumination peak spacing is also called the grating spacing. The described technique matches the angle and spacing using quadratic models of surface distortion. The phase shift, or simply phase, is the shift of the structured illumination pattern or grating when projected onto the sample plane.

[0068] Reusable models of spacing and angle distortion are computed a priori using a sub-fragment approach to characterize the full FOV. In a window or sub-fragment approach, a fragment is divided into overlapping windows or image sub-fragments and SIM parameters are estimated on each sub-fragment. The results of the sub-fragment parameter estimates are then fed to a least-squares regression, which generates quadratic distortion surfaces. Stored coefficients of the equations can then be used to extrapolate the parameters of the sub-fragments or at any location in the full field of view. In alternative embodiments, the coefficients can be stored and used, and transferred to a sub-fragment lookup table that relates the sub-fragment closest to the center to other sub-fragments. The sub-fragment closest to the center is used as a reference because the distortion at the center of the lens is less than at the edges.

[0069] For the phase relationship between sub-fragments, the lookup table is better suited than the curve fitting, as shown below.

[0070] During a run that takes several hours, periodic reclassification can be used to protect against thermal instability and the resulting bias. After redefining the extrapolation coefficients, the process resumes extrapolating the parameters of the closest sub-chunk to the center with other sub-chunks by extrapolation and without the cost of extrapolating the parameters for other sub-chunks.

Using symmetry to reduce computational costs

[0071] When imaging a single cuvette, SIM reconstruction from multiple images is often performed on specialized hardware such as a GPU, FPGA, or CGRA. Due to the computational overhead, reconstruction algorithms operate on fully redundant Fourier transforms that are biased toward the center. Using a CPU instead, the authors describe a SIM reconstruction using non-redundant coefficients of Fourier-transformed images. An embodiment is described based on the symmetry of the data in the corner-shifted Fourier transform space. Using a certain symmetry increases the complexity of the program, but can reduce the amount of data to compute in the key set by half and the number of required computations by half while maintaining nearly the same accuracy. The number of computations required to bias the Fourier representations with respect to the combined images is also reduced.

Adaptation to 2D Lighting Drawings

[0072] Standard algorithms for one-dimensional (1D) modulated illumination require modification when used with a two-dimensional (2D) modulated illumination pattern. This includes estimating the illumination peak spacing and the illumination peak angle, which requires a two-dimensional (2D) strip separation rather than a one-dimensional (1D) one. In addition, the modification includes estimating the Wicker phase, which requires using two points (instead of one) to estimate the phase in two dimensions. The one-dimensional (1D) interference pattern can be formed by a one-dimensional diffraction grating, as shown in FIG. 1A, or by an interference pattern of two beams.

[0073] FIG. 1B shows an intensity distribution that can be obtained by a two-dimensional (2D) diffraction grating or by interference of four light beams. Two light beams create an intensity pattern (horizontal bright and dark lines) along the y-axis and are therefore called a y-pair of incident rays. Two more light beams create an intensity pattern (vertical bright and dark lines) along the x-axis and are therefore called an x-pair of incident rays. The interference of the y-pair with the x-pair of light beams creates a two-dimensional (2D) illumination pattern. FIG. 1B shows the intensity distribution of such a two-dimensional (2D) illumination pattern.

[0074] FIG. 1C shows the arrangement of nanowells on the surface of a flow cell at the corners of a rectangle. When using one-dimensional (1D) structured illumination, the angle of the peak illumination is selected such that images are obtained along a line connecting diagonally opposite corners of the rectangle. For example, two sets of three images (a total of six images) can be obtained at angles of +45 degrees and -45 degrees. Since the distance along the diagonal is greater than the distance between any two sides of the rectangle, a higher resolution image can be obtained. The nanowells can be arranged in other geometric shapes, such as a hexagon. In this case, three or more images can be obtained along each of the three diagonals of the hexagon, which ultimately gives, for example, nine or fifteen images.

[0075] The described new and specialized technologies can be used individually or in combination to improve scanning performance in detecting fluorescence of several million samples distributed over a flow cell over several runs. In the following sections, the authors introduce terminology, describe an imaging tool that can be improved by the described technology, and describe new image enhancement technologies that can be used individually or in combination.

Terminology

[0076] The term "frequency" as used herein in relation to a structured illumination parameter means the reciprocal of the spacing between fringes or lines of a structured illumination pattern (e.g., a fringe pattern or grating), since frequency and period are inversely related. For example, a pattern having a larger spacing between fringes will have a lower frequency than a pattern having a smaller spacing between fringes.

[0077] The term "phase" as used herein in relation to a structured illumination parameter means the phase of the structured illumination pattern illuminating the sample. For example, the phase can be changed by moving the structured illumination pattern relative to the illuminated sample.

[0078] The term "orientation" as used herein in relation to a structured illumination parameter means the relative orientation between a structured illumination pattern (e.g., an interference fringe or grating pattern) and a sample illuminated by that pattern. For example, the orientation can be changed by rotating the structured illumination pattern relative to the illuminated sample.

[0079] The terms "predict" or "prediction" as used herein in relation to a structured illumination parameter mean either calculating values of the parameter without directly measuring the parameter or estimating the parameter from a captured image corresponding to the parameter. For example, the phase of a structured illumination pattern at time t1 may be predicted by interpolating between phase values directly measured or estimated (e.g., from captured phase images) at times t2 and t3, where t2<t1<t3. As another example, the frequency of a structured illumination pattern at time t1 may be predicted by extrapolating from frequency values directly measured or estimated (e.g., from captured phase images) at times t2 and t3, where t2<t3<t1.

[0080] The term "order" or "ordinal number" as used herein with respect to light diffracted by a diffraction grating means the number of integer wavelengths that represent the difference in path length of light from adjacent slits or structures of the diffraction grating for constructive interference. The interaction of an incident beam of light with a repeating row of grating structures or other beam-splitting structures can redirect or diffract portions of the beam of light in predictable angular directions relative to the original beam. The term "zero order" or "zero order maximum" means a central bright interference fringe emitted by a diffraction grating in which there is no diffraction. The term "first order" means two bright interference fringes diffracted to either side of a zero order interference fringe, where the path length difference is equal to ±1 wavelength. Higher order fringes are diffracted at higher angles from the original beam. The properties of the grating can be manipulated to control the degree of beam intensity directed into fringes of different orders. For example, a phase grating can be designed to maximize the transmission of ±1 order beams and minimize the transmission of zero order beams.

[0081] The term "element" as used herein in relation to a sample means a point or region in a structure that can be distinguished from other points or regions based on relative position. An individual element may include one or more molecules of a particular type. For example, an element may include a single target nucleic acid molecule having a particular sequence, or an element may include multiple nucleic acid molecules having the same sequence (and/or a complementary sequence).

[0082] As used herein, the term "xy plane" means a 2-dimensional region defined by the straight x and y axes in a Cartesian coordinate system. When used in relation to a detector and an object observed by the detector, the region may be further defined as orthogonal to the beam axis or the viewing direction between the detector and the object being detected.

[0083] As used herein, the term "z coordinate" is intended to refer to information that specifies the location of a point, line, or region along an axis orthogonal to the XY plane. In particular embodiments, the z axis is perpendicular to the region of the object observed by the detector. For example, the focal direction of an optical system may be set along the z axis.

[0084] As used herein, the term "optically coupled" means one element adapted to directly or indirectly transmit light to another element.

SIM equipment

[0085] This section is based on the disclosure previously provided by Applicant in U.S. Provisional Application No. 62/692,303, filed June 29, 2018 (unpublished). Structured illumination microscopy (SIM) describes a method by which spatially structured (i.e., patterned) light can be used to visualize a specimen and increase the lateral resolution of a microscope by two orders of magnitude or more. FIG. 1A shows the formation of a moiré interference fringe (or moiré pattern) using a 1D modulated grating. A surface containing a specimen is illuminated with a structured pattern of light intensity, typically sinusoidal, to affect the formation of the moiré interference fringe. FIG. 1A shows two sinusoidal patterns in which their frequency vectors, in frequency or reciprocal space, are (a) parallel and (b) non-parallel. FIG. 1A is a typical illustration of the shift of high frequency spatial information to a lower frequency that can be detected optically. The new signal is called a moiré fringe or alias. In some cases, during sample imaging, three images of the fringe patterns are acquired at different pattern phases (e.g., 0°, 120°, and 240°) so that each location on the sample is exposed to a range of illumination intensities, and this procedure is repeated while rotating the pattern orientation around the optical axis by 2 (e.g., 45°, 135°) or 3 (e.g., 0°, 60°, and 120°) separate angles. The resulting images (e.g., six or nine images) can be assembled into a single image with an expanded spatial frequency bandwidth, which can be re-transformed into real space to produce an image with a higher resolution than that obtained with a standard microscope.

[0086] In some embodiments of SIM systems, a linearly polarized beam of light is directed through an optical beam splitter that splits the beam into two or more separate orders that can be combined and projected onto a sample, the image of which is formed, in the form of a pattern of interference fringes with a sinusoidal change in intensity. Examples of beam splitters are diffraction gratings that can form beams with a high degree of coherence and stable propagation angles. When two such beams are combined, the interference between them can create a pattern of uniform, regularly repeating interference fringes, and the distance between them is determined by factors including the angle between the interfering beams.

[0087] FIG. 1B shows an example of a two-dimensional (2D) structured illumination. The two-dimensional (2D) structured illumination can be formed by two orthogonal one-dimensional (1D) diffraction gratings superimposed on each other. As with one-dimensional (1D) structured illumination patterns, two-dimensional (2D) illumination patterns can be formed either by using two-dimensional (2D) diffraction gratings or by interference between four beams of light that create a regularly repeating pattern of interference fringes.

[0088] During the acquisition and/or subsequent assembly or reconstruction of multiple images into a single image with an extended spatial frequency bandwidth, the following structured illumination parameters must be taken into account: the orientation or angle of the interference fringe pattern is also referred to as the illumination angle relative to the illuminated sample, with the spacing between adjacent interference fringes referred to as the illumination peak spacing (i.e., the frequency of the interference fringe pattern) and the phase shift of the structured illumination pattern. In an ideal imaging system that is not subject to factors such as mechanical instability and thermal variations, each of these parameters does not drift or otherwise change over time, so the exact frequency, phase, and orientation parameters of the SIM associated with a given image sample must be known. However, due to factors such as mechanical instability of the excitation beam path and/or thermal expansion/contraction of the sample being imaged, these parameters may drift or otherwise change over time.

[0089] Thus, it may be necessary to estimate the structured illumination parameters of a SIM imaging system to account for their change over time. Since many SIM imaging systems do not perform real-time SIM image processing (e.g., they process captured images offline), such SIM systems may spend a significant amount of time computing to process a SIM image and estimate the structured illumination parameters for that image.

[0090] FIGS. 2-4B illustrate three such examples of a SIM rendering system. It should be noted that, although these systems are described primarily in the context of SIM rendering systems that generate one-dimensional (1D) lighting patterns, the technology described herein may be implemented using SIM rendering systems that generate higher-dimensional lighting patterns (e.g., two-dimensional lattice patterns).

[0091] FIG. 2 shows a structured illumination microscopy (SIM) imaging system 100 that can implement structured illumination parameter prediction in accordance with some embodiments described herein. For example, the system 100 may be a structured illumination fluorescence microscopy system that uses spatially structured excitation light to form an image of a biological sample.

[0092] In the example of FIG. 2, the light emitter 150 is configured to emit a beam of light that is collimated by a collimator lens 151. The collimated light is patterned by a light structuring optical unit 155 and directed by a dichroic mirror 160 through an objective lens 142 onto a sample in a sample container 110 that is placed on a movable stage 170. In the case of a fluorescent sample, the sample fluoresces in response to the structured excitation light, and the resulting light is collected by the objective lens 142 and directed to an image sensor of a camera system 140 for detecting fluorescence.

[0093] The optical light structuring assembly 155 includes one or more optical diffraction gratings or other beam-splitting elements (e.g., a cube or a beam-splitting plate) for forming a light pattern (e.g., interference fringes, typically sinusoidal) that is projected onto samples in the sample container 110. The diffraction gratings may be one-dimensional or two-dimensional transmitting or reflecting gratings. The diffraction gratings may be diffraction gratings with a sinusoidal amplitude or gratings with a sinusoidal phase.

[0094] In some embodiments, the diffraction gratings may not use a rotating stage to change the orientation of the structured illumination pattern. In other embodiments, the diffraction grating(s) may be mounted on a rotating stage. In some embodiments, the diffraction gratings may be stationary during operation of the imaging system (i.e., do not require rotational or linear motion). For example, in a particular embodiment, described further below, the diffraction gratings may include two fixed one-dimensional transmission diffraction gratings oriented perpendicular to each other (e.g., a horizontal diffraction grating and a vertical diffraction grating).

[0095] As shown in the example of FIG. 2, the optical light structuring unit 155 emits diffuse light rays of the first order (e.g., m=±1 order), while blocking or minimizing rays of all other orders, including rays of the zero order. However, in alternative embodiments, light rays of additional orders may be projected onto the sample.

[0096] During each imaging cycle, the imaging system 100 uses the optical light structuring unit 155 to acquire multiple images at different phases with a lateral shift of the interference fringe pattern in the modulation direction (e.g., in the x-y plane and perpendicular to the interference fringes), and this procedure is repeated one or more times, rotating the orientation of the pattern around the optical axis (i.e., relative to the x-y plane of the sample). The captured images can then be reconstructed using computations to form an image with a higher resolution (e.g., an image having approximately twice the lateral spatial resolution compared to individual images).

[0097] In the system 100, the light emitter 150 may be an incoherent light emitter (for example, emitting light beams output by one or more excitation diodes) or a coherent light emitter, such as an emitter of light emitted by one or more lasers or laser diodes. As shown in the example of the system 100, the light emitter 150 includes an optical fiber 152 for directing an optical beam to be emitted. However, other configurations of the light emitter 150 may be used. In embodiments using structured illumination in a multi-channel imaging system (for example, in a multi-channel fluorescence microscope using multiple wavelengths of light), the optical fiber 152 may be optically coupled to a plurality of different light sources (not shown), wherein each light source emits light with a different wavelength. Although the system 100 is shown to have one light emitter 150, some embodiments may include a plurality of light emitters 150. For example, a plurality of light emitters may be included in the case of a structured illumination imaging system using a plurality of arms, which is discussed further below.

[0098] In some embodiments, the system 100 may include a tubular lens 156, which may include a lens element for pivoting along the z-axis to adjust the shape and path of the structured beam. For example, the tubular lens component may be pivotally moved to accommodate a range of sample thicknesses (e.g., different cover glass thicknesses) of the sample in the container 110.

[0099] In an example of system 100, fluid delivery module or device 190 may direct a flow of reagents (e.g., fluorescently labeled nucleotides, buffers, enzymes, cleavage reagents, etc.) into (and through) sample container 110 and drain valve 120. Sample container 110 may contain one or more supports on which samples are presented. For example, in the case of a system for analyzing a large number of different nucleic acid sequences, sample container 110 may include one or more supports to which nucleic acids to be sequenced are attached or linked. The support may include any inert support or matrix to which nucleic acids can be attached, such as, for example, glass surfaces, plastic surfaces, latex, dextran, polystyrene surfaces, polypropylene surfaces, polyacrylamide gels, gold surfaces, and silicon wafers. In some applications, the substrate is located within a channel or other region at a plurality of locations formed in a matrix or array throughout the sample container 110. The system 100 may also include a temperature unit actuator 130 and a heater/cooler 135 that can, if necessary, regulate the temperature of the fluids in the sample container 110.

[00100] In particular embodiments, the sample container 110 may be implemented as a structured flow cell that includes a transparent overlay plate, a substrate, and a liquid contained therebetween, and a biological sample may be placed on an inner surface of the transparent overlay plate or an inner surface of the substrate. The flow cell may comprise a large number (e.g., thousands, millions, or billions) of wells (also referred to as nanowells) or regions that are structured into an array of a certain shape (e.g., a hexagonal array, a rectangular array, etc.) in the substrate. Each region may form a cluster (e.g., a monoclonal cluster) of a biological sample, such as DNA, RNA, or other genomic material, that may be sequenced, such as using sequencing by synthesis. The flow cell may be further divided into a number of spaced lanes (e.g., eight lanes), with each lane comprising a hexagonal array of clusters.

[00101] The sample container 110 may be mounted on the sample stage 170 to allow movement and alignment of the sample container 110 relative to the objective lens 142. The sample stage may have one or more actuators that allow it to move in any of three dimensions. For example, in terms of a Cartesian coordinate system, actuators may be provided that allow the stage to move in the X, Y, and Z directions relative to the objective lens. This may allow one or more sample locations to be positioned on the sample container 110 so as to optically align them with the objective lens 142. Movement of the sample stage 170 relative to the objective lens 142 may be achieved by moving the sample stage itself, the objective lens, some other components of the imaging system, or any combination of the above. Other embodiments may also include movement of the entire imaging system over a stationary sample. In an alternative embodiment, the sample container 110 may be stationary during imaging.

[00102] In some embodiments, a focusing component 175 (z-axis) may be included to control the positioning of the optical components relative to the sample container 110 in a focal direction (typically referred to as the z-axis or z-direction). The focusing component 175 may include one or more actuators physically coupled to the optics stage or the sample stage, or both, to move the sample container 110 on the sample stage 170 relative to the optical components (e.g., the objective lens 142) to provide proper focusing for the imaging operation. For example, the actuator may be physically coupled to the corresponding stage, such as by direct or indirect mechanical, magnetic, hydraulic, or other connection to the stage or contact with the stage. One or more actuators may be configured to move the stage in the z direction while maintaining the position of the sample stage in the same plane (e.g. maintaining a level or horizontal position perpendicular to the optical axis). One or more actuators may also be configured to tilt the stage. This may be done, for example, to dynamically align the sample container 110 taking into account any tilt of its surfaces.

[00103] Structured light emitted from the test sample at the location of the sample, the image of which is formed, can be directed by means of a dichroic mirror 160 to one or more detectors of the camera system 140. Some embodiments can include a filter switching unit 165 with one or more emission filters, wherein the one or more emission filters can be used to pass radiation with specific wavelengths and block (or reflect) radiation with other wavelengths. For example, one or more emission filters can be used to switch between different channels of the imaging system. In a particular embodiment, the emission filters can be implemented as dichroic mirrors that direct emitted light of different wavelengths to different image sensors of the camera system 140.

[00104] The camera system 140 may include one or more image sensors for monitoring and tracking the imaging (e.g., sequencing) of the sample container 110. The camera system 140 may be implemented, for example, as a camera with a charge-coupled device (CCD) image sensor, but other image sensor technologies (e.g., an active-pixel sensor) may be used.

[00105] The output data (e.g., images) from the camera system 140 may be transmitted to the SIM imaging component 191 in real time, which may be implemented as a software application that, as further described below, may reconstruct the images captured during each imaging cycle to create an image having a higher spatial resolution. The reconstructed images may take into account changes in the structured illumination parameters that are predicted over time. In addition, the SIM imaging component 191 may be used to track the predicted SIM parameters and/or to predict the SIM parameters given the previously estimated and/or predicted SIM parameters.

[00106] A controller 195 may be provided to control the operation of the structured illumination imaging system 100, including the synchronization of various optical components of the system 100. The controller may be implemented to control aspects of the operation of the system, such as, for example, configuring the light structuring optical assembly 155 (e.g., selecting and/or linearly moving the diffraction gratings), moving the tubular lens 156, focusing, moving the stage, and imaging operations. The controller may also be configured to control elements of the equipment of the system 100 to correct for changes in the parameters of the structured illumination over time. For example, the controller may be configured to transmit control signals to motors or other devices that control the configuration of the light structuring optical assembly 155, the movement of the stage 170, or some other elements of the system 100 to correct or compensate for changes in the phase, frequency, and/or orientation of the structured illumination over time. In embodiments, these signals can be transmitted in accordance with the parameters of the structured illumination predicted using the SIM imaging component 191. In some embodiments, the controller 195 can include a memory device for storing the predicted or estimated parameters of the structured illumination corresponding to different moments of time and/or positions of the sample.

[00107] In various embodiments, the controller 195 may be configured to use hardware, algorithms (e.g., computer-executable instructions), or a combination of the above. For example, in some embodiments, the controller may include one or more central processing units (CPUs), graphics processing units (GPUs), or processors with associated memory. As another example, the controller may be hardware or other electrical circuitry for controlling operation, such as a computer processor and a non-transitory computer-readable medium with computer-readable instructions stored thereon. For example, this circuit solution may include one or more of the following: a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logic device (CPLD), a programmable logic array (PLA), a programmable array logic (PAL), and other similar processing devices or circuit solutions. As another example, the controller may be a combination of this circuit design with one or more processors.

[00108] FIG. 3 shows a diagram of an optical system illustrating an example of an optical configuration of a two-arm SIM imaging system 200 that can implement structured illumination parameter prediction in accordance with some embodiments described herein. The first arm of the system 200 comprises a light emitter 210A, a first optical collimator 220A for collimating light from the emitter 210A, a diffraction grating 230A in a first orientation relative to an optical axis, a rotating mirror 240A, and a second optical collimator 250A. The second arm of the system 200 includes a light emitter 210B, a first optical collimator 220B for collimating light from the emitter 210B, a diffraction grating 230B in a first orientation relative to the optical axis, a rotating mirror 240B, and a second optical collimator 250B. Although diffraction gratings are illustrated in this example, in other embodiments, other beam splitting elements, such as a cube or beam splitter plate, may be used to split the light received at each arm of the SIM imaging system 200.

[00109] Each light emitter 210A-210B may be a non-coherent light emitter (e.g., emitting light beams output by one or more excitation diodes) or a coherent light emitter, such as an emitter of light emitted by one or more lasers or laser diodes. In the example of system 200, each light emitter 210A-210B is an optical fiber that outputs an optical beam collimated by a corresponding collimator 220A-220B.

[00110] In some embodiments, each optical fiber may be optically coupled to a corresponding light source (not shown), such as a laser. During imaging, each optical fiber may be turned on or off using a high-speed shutter (not shown) located in the optical path between the fiber and the light source, or by pulsing the corresponding light source on the fiber channel during imaging at a given frequency. In some embodiments, each optical fiber channel may be optically coupled to the same light source. In such embodiments, a beam splitter or other suitable optical element may be used to direct light from the source into each optical fiber channel. In such examples, each optical fiber channel may be turned on or off using a high-speed shutter (not shown) located in the optical path between the fiber channel and the beam splitter.

[00111] In an example of a SIM imaging system 200, a first arm comprises a fixed vertical grating 230A for projecting a structured illumination pattern or a grating pattern in a first orientation (e.g., a vertical pattern of interference fringes) onto a sample, and a second arm comprises a fixed horizontal grating 230B for projecting a structured illumination pattern or a grating pattern in a second orientation (e.g., a horizontal pattern of interference fringes) onto a sample 271. The gratings of the SIM imaging system 200 do not need to be mechanically rotated or moved, and this can improve the speed, reliability, and repeatability characteristics of the system.

[00112] In alternative embodiments, the gratings 230A and 230B may be mounted on respective linear moving stages that may be moved to change the optical path length (and hence the phase) of the light emitted by the gratings 230A and 230B. The axis of linear motion of the stages may be perpendicular or otherwise offset from the orientation of their respective grating so that the grating pattern can be transferred along the sample 271.

[00113] The gratings 230A-230B may be translucent diffraction gratings with a plurality of diffraction elements (e.g., parallel slits or grooves) formed in a glass substrate or other suitable surface. The gratings may be implemented as phase gratings that provide a periodic change in the refractive index of the grating material. The pitch of the grooves or elements may be selected so that it scatters light at suitable angles and is adjusted for a minimum resolution size of the samples, the image of which is formed, for the operation of the SIM visualization system 200. In other embodiments, the gratings may be reflective diffraction gratings.

[00114] In the example of the SIM imaging system 200, the vertical and horizontal patterns are offset by approximately 90 degrees. In other embodiments, other orientations of the gratings can be used to create an offset of approximately 90 degrees. For example, the gratings can be oriented such that they project images that are offset by ±45 degrees from the x or y plane of the sample 271. The configuration of the example of the SIM imaging system 200 can be particularly advantageous when using a sample 271 with a uniformly formed pattern and elements on a rectangular grating, since a structured improvement in resolution can be achieved using only two perpendicular gratings (e.g., vertical and horizontal).

[00115] The gratings 230A-230B in the example system 200 are configured to scatter the input beams into a number of orders (e.g., 0 order, ±1 order, ±2 orders, etc.), from which the ±1 order beams can be projected onto the sample 271. As shown in this example, the vertical grating 230A scatters the collimated light beam into first order (±1 order) scattered beams, distributing the first orders on the plane of the page, and the horizontal grating 230B scatters the collimated light beam into first order scattered beams, distributing the orders on the plane of the page (i.e., in a plane perpendicular to the page). To improve the efficiency of the system, it is possible to block the zero-order beams and all other higher-order scattered beams (e.g., ±2 orders and higher), such as by filtering the illumination pattern projected onto the sample 271. For example, a beam blocking element (not shown), such as an order filter, can be inserted into the optical path after each diffraction grating to block the zero-order beam and higher-order beams. In some embodiments, the diffraction gratings 230A-230B can be configured to scatter beams only into first-order beams, and the zero-order beam (the unscattered beam) can be blocked by some beam blocking element.

[00116] Each arm comprises an optical phase modulator or phase shifter 240A-240B for phase shifting the scattered light of each of the gratings 230. For example, during structured imaging, the optical phase of each scattered beam may be shifted by a certain fraction (e.g., 1/2, 1/3, 1/4, etc.) of the pitch (λ) of each interference fringe of the structured pattern. In the example of FIG. 3, the phase modulators 240A and 240B are implemented as rotating windows that may use a galvanometer or other rotary drive mechanism to rotate and modulate the optical path length for each scattered beam. For example, window 240A may rotate about a vertical axis to shift the image projected by vertical grating 230A onto sample 271 to the left or right, and window 240B may rotate about a horizontal axis to shift the image projected by horizontal grating 230B onto sample 271 up or down.

[00117] In other embodiments, other phase modulators may be used that change the length of the optical path of the scattered light (e.g., linear moving tables, wedges, etc.). In addition, although the optical phase modulators 240A-240B are shown after the array 230A-230B, in other embodiments they may be placed in other places in the lighting system.

[00118] In alternative embodiments, the single-phase modulator may operate in two different directions for different fringe patterns, or the single-phase modulator may use a single motion to adjust the length of both paths. For example, a large rotating optical window may be positioned after the mirror 260 with apertures 261. In this case, the large window may be used instead of the windows 240A and 240B to modulate the phases of both sets of scattered beams emitted by the vertical and horizontal diffraction gratings. The axis of rotation of the large rotating window may not be parallel to the optical axis of one of the gratings, but may be offset by 45 degrees (or other angular offset) from the optical axis of each vertical and horizontal grating to provide a phase shift along both directions along one common axis of rotation of the large window. In some embodiments, the large rotating window may be replaced by a wedge-shaped optical device rotating about the nominal beam axis.

[00119] In the example of the system 200, the mirror 260 with the openings 261 combines the two arms into a lossless optical path (e.g., without a significant loss of optical power, except for a small absorption by the reflective coating). The mirror 260 can be arranged such that the scattered rays of different orders from each of the gratings are spatially deployed, and the unwanted rays of different orders can be blocked. The mirror 260 transmits the first order rays emitted by the first arm through the openings 261. The mirror 260 reflects the first order rays emitted by the second arm. In this way, the structured illumination pattern can be switched from a vertical orientation (e.g., grating 230A) to a horizontal orientation (e.g., grating 230B), by turning on or off each emitter or by opening and closing an optical shutter that directs light from the source through an optical fiber cable. In other embodiments, the structured illumination pattern may be switched using an optical switch to change the arm that illuminates the sample.

[00120] Also as an example, the imaging system 200 is a tubular lens 265, a semi-reflective mirror 280, an object 270 and a camera 290. For example, the tubular lens 265 can be configured to rotate along the z-axis to adjust the shape and trajectory of the structured beam. The semi-reflective mirror 280 can be a dichroic mirror for reflecting the structured light received from each arm to the objective 270 for projection onto the sample 271 and passing through the light emitted by the sample 271 (for example, fluorescent light, the wavelength of which differs from the excitation wavelength) to the camera 290.

[00121] The output data (e.g., images) from the camera 290 may be transmitted to a real-time SIM imaging component (not shown), which may be implemented as a software application that, as further described below, may reconstruct the images captured during each imaging cycle to create an image having a higher spatial resolution. The reconstructed images may take into account changes in the structured illumination parameters that are predicted over time. In addition, the real-time SIM imaging component may be used to track the predicted SIM parameters and/or to predict the SIM parameters given the previously estimated and/or predicted SIM parameters.

[00122] A controller (not shown) may be provided to control the operation of the structured illumination imaging system 200, including the synchronization of various optical components of the system 200. The controller may be implemented to control the operation of the system, such as, for example, the configuration of each optical arm (e.g., turning each optical arm on/off during phase image acquisition, actuating the phase modulators 240A-240B), moving the tubular lens 265, moving the stage (if used) of the sample 271, and imaging operations. The controller may also be configured to control hardware elements of the system 200 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to transmit control signals to devices (e.g., phase modulators 240A-240B) that control the configuration of each optical arm or some other elements of the system 100 to correct or compensate for changes in phase, frequency, and/or orientation of the structured illumination over time. As another example, when the gratings 230A-230B are mounted on linear sliding stages (e.g., instead of using the phase modulators 240A-240B), the controller can be configured to control the linear sliding stages to correct or compensate for phase changes. In embodiments, these signals can be transmitted in accordance with the structured illumination parameters predicted using the SIM imaging component. In some embodiments, the controller can include memory for storing the predicted or estimated structured illumination parameters corresponding to different times and/or sample positions.

[00123] It should be noted that for simplicity, the optical components of the SIM imaging system 200 may be omitted from the foregoing description. Furthermore, although in this example the system 200 is illustrated as a single-channel system, in other embodiments it may be implemented as a multi-channel system (e.g., using two different cameras and light sources emitting two different wavelengths).

[00124] Although the system 200 illustrates a structured illumination system with two arms that includes two gratings oriented at two different angles, it should be noted that in other embodiments, the technology described herein can be implemented in the form of systems that use more than two arms. In the case of a sample with a regular pattern and elements on a rectangular grid, an improvement in resolution can be achieved with only two perpendicular angles (e.g., a vertical grating and a horizontal grating), as described above. On the other hand, three illumination angles can be used to improve the image resolution in all directions for other samples (e.g., samples with rectangular patterns). For example, a three-arm system can include three light emitters and three fixed diffraction gratings (one per arm), where each diffraction grating is oriented around the optical axis of the system to project a corresponding pattern orientation onto the sample (e.g., a pattern at an angle of 0°, a pattern at an angle of 120°, or a pattern at an angle of 240°). In such systems, additional apertured mirrors may be used to combine additional images of additional gratings into a lossless system. In an alternative embodiment, such systems may use one or more polarizing beam splitters to combine images of each of the gratings.

[00125] FIG. 4A and 4B show a diagram illustrating an example of an optical configuration of a SIM imaging system 500 with two optical gratings for slides, which can implement a structured prediction of illumination parameters in accordance with some embodiments described in this document. In the example system 500, all changes to the structured illumination pattern or grating pattern projected onto the sample 570 (e.g., phase shifts or pattern rotations) can be performed by linearly moving the movable stage 530 along one axis of motion to select the grating 531 or 532 (i.e., select the orientation of the grating) or a phase shift of one of the gratings 531-532.

[00126] The system 500 comprises a light emitter 510 (e.g., a fiber optic channel optically coupled to a light source), a first optical collimator 520 (e.g., a collimating lens) for collimating the light output of the emitter 510, a linear moving stage 530 with a first diffraction grating 531 (e.g., horizontal) and a second diffraction grating 532 (e.g., vertical), a tubular lens 540, a semi-reflective mirror 550 (e.g., a dichroic mirror), an object 560, a sample 570, and a camera 580. For simplicity, the optical components of the SIM imaging system 500 may not be shown in FIG. 4A. Furthermore, although in this example the system 500 is illustrated as a single-channel system, in other embodiments it may be implemented as a multi-channel system (e.g., using two different cameras and light sources emitting two different wavelengths).

[00127] As shown in FIG. 4A, the grating 531 (for example, a horizontal diffraction grating) can disperse the collimated light beam into first-order scattered light rays (on the plane of the page). As shown in FIG. 4B, the diffraction grating 532 (for example, a vertical diffraction grating) can deflect the beam into first-order rays (above and below the plane of the page). In this configuration, only one optical arm with one emitter 510 (for example, an optical fiber) and one linear moving stage are required to image the sample 570, which can provide system advantages such as a reduction in the number of moving parts for increased speed, simplicity and reduced cost. In addition, in the system 500, the absence of a polarizer can provide the previously mentioned advantage: high optical efficiency. The configuration of the example SIM imaging system 200 may be particularly advantageous when using a sample 570 with a regular pattern and elements on a rectangular lattice, since a structured resolution improvement can be achieved using only two perpendicular lattices (e.g., vertical and horizontal).

[00128] To improve the efficiency of the system, it is possible to block the zero-order rays and all other higher-order scattered rays (e.g., ±2 orders and higher) generated by each grating (e.g., to filter the illumination pattern projected onto the sample 570). For example, after the movable stage 530, a beam blocking element (not shown), such as an order filter, can be inserted into the optical path. In some embodiments, the diffraction gratings 531-532 can be configured to scatter rays only into first-order rays, and the zero-order beam (the unscattered beam) can be blocked by some beam blocking element.

[00129] In an example of system 500, two gratings may be located at an angle of approximately ±45° from the axis of motion (or at another angle from the axis of motion, such as about +40°/-50°, about +30°/-60°, etc.), so that a phase shift can be implemented for each grating 531 and 532 along one axis of linear motion. In some embodiments, two gratings may be combined into one physical optical element. For example, one side of the physical optical element may have a structured illumination pattern or a grating pattern in a first orientation, and an adjacent side of the physical optical element may have a structured illumination pattern or a grating pattern in a second orientation located at a right angle to the first orientation.

[00130] The linear moving stage 530 with a single axis may include one or more actuators that allow it to move along the X-axis relative to the plane of the sample or along the Y-axis relative to the plane of the sample. During operation, the linear moving stage 530 can provide sufficient movement (for example, approximately 12-15 mm) and accuracy (for example, repeatability with an approximate error of less than 0.5 micrometers) to cause the projection of accurate illumination patterns for effective image reconstruction. In embodiments in which the moving stage 530 is used in an automated imaging system, such as a fluorescence microscope, it can provide high speed of operation, minimal vibration and a long service life. In embodiments, the linear moving stage 530 may include crossed roller supports, a linear motor, a high-precision linear encoder and/or other components. For example, the movable table 530 can be implemented as a high-precision movable table with a step or piezo drive, which can be moved using a controller.

[00131] The output data (e.g., images) from the camera 580 may be transmitted to a real-time SIM imaging component (not shown), which may be implemented as a software application that, as further described below, may reconstruct the images captured during each imaging cycle to create an image having a higher spatial resolution. The reconstructed images may take into account changes in the structured illumination parameters that are predicted over time. In addition, the real-time SIM imaging component may be used to track the predicted SIM parameters and/or to predict the SIM parameters given previously estimated and/or predicted SIM parameters.

[00132] A controller (not shown) may be provided to control the operation of the structured illumination imaging system 500, including the synchronization of various optical components of the system 500. The controller may be implemented to control the operation of the system, such as movement of the linear stage 530, movement of the tubular lens 540, movement of the stage (if used) of the sample 570, and imaging operations. The controller may also be configured to control hardware elements of the system 500 to correct for changes in structured illumination parameters over time. For example, the controller may be configured to transmit control signals to devices (e.g., the linear stage 530) to correct for or compensate for changes in phase, frequency, and/or orientation of the structured illumination over time. In embodiments, these signals may be transmitted in accordance with structured illumination parameters predicted using the SIM imaging component. In some embodiments, the controller may include memory for storing predicted or estimated structured illumination parameters corresponding to various times and/or positions of the sample.

[00133] Although the example in FIGS. 4A-4B illustrates a two-grating slide imaging system that can implement structured prediction of illumination parameters, structured prediction of illumination parameters can be implemented in SFM imaging systems that use a linear motion actuator mounted with more than two diffraction gratings.

Estimation of projection parameters

[00134] Parameter estimation for SIM image processing may be required to correct for unwanted changes in structured illumination parameters over time. As an example, FIGS. 5A-5C illustrate unwanted changes in the frequency or spacing of the illumination peak (FIG. 5A), the phase or phase shift (FIG. 5B), and the orientation or angle of the illumination peak (FIG. 5C) that may occur over time in a SIM imaging system that projects a one-dimensional structured illumination pattern onto a sample with a regular pattern. In particular, FIG. 5A shows a sample 50 with elements 51 illuminated by a one-dimensional structured illumination pattern with interference fringes 60 before and after frequency shifts. Before any frequency shifts, adjacent interference fringes 60 have a pitch or center-to-center distance P corresponding to an initial frequency f. Over time, as the temperature in the system changes, the pitch P may increase or decrease. For example, thermal expansion can lead to an increase in the pitch P to P+ΔP ₁ , correspondingly decreasing the frequency f to f-Δf ₁ . Conversely, thermal contraction can lead to a decrease in the pitch P to P-ΔP ₁ , correspondingly increasing the frequency f to f+Δf ₂ .

[00135] FIG. 5B shows a sample 50 illuminated with a one-dimensional structured illumination pattern with interference fringes 60 before and after phase changes. As shown, before the phase shift, the first phase state Φ may correspond to interference fringes that completely illuminate every other column of elements 51 of the sample 50. Over time, the position of the interference fringes 60 relative to the sample 50 may shift such that all phase images are shifted by ΔΦ. For example, mechanical vibrations in the SIM imaging system (e.g., in the excitation beam path), an error in the translation stage used by the grating or sample stage, thermal variations and/or other factors may cause an unwanted phase change. After the phase change by ΔΦ, the first phase state changes to Φ+ΔΦ, the interference fringes are shifted from the center in every second column of elements.

[00136] FIG. 5C shows a sample 50 illuminated with a one-dimensional structured illumination pattern with interference fringes 60 before and after orientation changes. As shown, before the change, the orientation of the interference fringes relative to the sample 50 is completely vertical. Over time, the orientation may change due to factors such as changes in the excitation beam path, sample movements, thermal variations, and/or other factors. After the orientation is rotated by an angle Δθ, the interference fringes are no longer completely vertical relative to the sample.

[00137] Estimation of parameters during the SIM imaging process to accurately account for changes in structured illumination parameters, as described above, helps to ensure the absence of artifacts and accurate reconstruction of an image from a sample of images. However, such a process involves expensive computations and is often performed after image acquisition. For mission-critical SIM imaging systems that involve real-time image processing and reconstruction, and thus real-time estimation of parameters such as frequency, phase, orientation, and modulation order, these computational requirements can result in a loss of data throughput (e.g., less data can be processed per unit of time). In such systems, the rate at which samples are generated may exceed the rate at which the structured illumination parameters can be directly estimated from the acquired images. Thus, there is a need for a method for generating parameter estimates with low complexity and low processing time.

[00138] To this end, embodiments of the technology described herein are directed to predicting structured lighting parameters for a specific point in time, space, and/or temperature using estimates of structured lighting parameters obtained from images captured by a structured lighting system. Specific embodiments are directed to predicting structured lighting, phase, orientation, and/or modulation order parameters.

[00139] According to some embodiments, a structured illumination parameter may be predicted for a given point in time, space, and/or temperature by interpolating estimates of the structured illumination parameter from an image capture. For example, a first frequency may be estimated from a first selected image, a second frequency may be estimated from a second selected image, and a frequency corresponding to a point in time between the first captured image and the second captured image (e.g., a frequency for an image made between the first and second images) may be predicted by interpolation using at least a certain rate of change of frequency between the first captured image and the second captured image.

[00140] According to some embodiments, a structured illumination parameter may be predicted for a given point in time, space, and/or temperature by extrapolation using estimates of the structured illumination parameter obtained from two image captures. For example, a first orientation may be estimated from a first selected image, a second orientation may be estimated from a second selected image, and an orientation corresponding to a point in time after the first and second captured images (e.g., an orientation for a third image obtained after the first and second images) may be predicted by extrapolation using at least a certain rate of change of orientation from the first captured image to the second captured image. In a second example, the first orientation may be estimated from the first selected image, the second orientation may be estimated from the second selected image, and an orientation corresponding to a point in time after the first and second captured images (e.g., an orientation for a third image obtained after the first and second images) may be predicted by holding a value from the second captured image.

[00141] In example implementations, the estimated and predicted structured lighting parameters may be used to narrow the search space for other structured lighting parameters that are predicted. For example, given an estimated value of a structured lighting parameter for a first point in time, space, and/or temperature, a value of a structured lighting parameter for a second point in time, space, and/or temperature that is near the first point in time, space, and/or temperature may be predicted based on the estimated or predicted value at the first point in time, space, and/or temperature.

[00142] In example implementations, the estimated and predicted parameters of the structured lighting may be stored in a memory device of the structured lighting system for subsequent use by the system. For example, the predicted and estimated parameters may be stored in a history file, such as a lookup table. The predicted parameters that are stored in the memory may be determined from the estimated parameters, or they may be established based on the physical characteristics of the structured lighting system. For example, a nominal spacing between gratings of the structured lighting system may be stored. The stored parameters may then be associated with performing operations such as: calibrated image restoration, providing feedback to a hardware component to correct for changes in the structured lighting parameters, and narrowing the search space when predicting additional structured lighting parameters.

Drawings of interference fringes of illumination

[00143] FIG. 6 shows simplified patterns of interference fringes of illumination that can be projected onto a plane of a sample 271 by a vertical grating 230A and a horizontal grating 230B of a SIM imaging system 200 during a single imaging cycle to use structured light to create a high resolution image. In this example, three phase images with a vertical illumination orientation can be captured using the vertical grating 230A, and three-dimensional images with a horizontal illumination orientation can be captured using the horizontal grating 230B. For each orientation, the projected interference fringes can be shifted in 1/3λ increments (e.g., by setting the phase modulator 230A or 230B to three different positions) to capture three phase images of the orientation pattern.

[00144] During the capture of each phase image, any light emitted by the sample may be captured by the camera 290. For example, fluorescent dyes located on various elements of the sample 271 may fluoresce, and the resulting light may be collected by the objective lens 270 and directed to the image sensor of the camera 290 to detect fluorescence. The resulting six images may be used to form an image of the entire sample or to determine the location of a larger sample.

[00145] After capturing all images for an imaging cycle (in this example, six images), a high-resolution image can be composed from the captured images. For example, a high-resolution image can be reconstructed from the six images shown in FIG. 6. Suitable algorithms can be used to combine these different images to synthesize a single image of the sample with a significantly higher spatial resolution than any of the individual component images.

[00146] During high-resolution image composition, unwanted shifts or changes in structured illumination parameters (such as angle, spacing, and phase) can be compensated for using an algorithm with structured illumination parameters predicted in accordance with the present invention (e.g., predicted changes in angle, spacing, or phase). For example, shifts in phases, angle, and/or spacing of vertical illumination images and/or horizontal illumination images can be compensated for.

[00147] In some embodiments, unwanted shifts or changes in the structured illumination parameters may be compensated for prior to image capture by controlling one or more hardware elements of the system 200 to compensate for these changes in the SIM imaging system. For example, prior to an imaging sequence and/or between image captures of an imaging sequence, a phase change may be compensated for each optical arm by adjusting a phase shift element (e.g., a rotating mirror, a linear actuator, etc.). In some embodiments, a combination of hardware and algorithmic compensations may be implemented.

[00148] FIG. 6 shows an embodiment in which the flow cell comprises symmetrically arranged nanowells. In some embodiments, the nanowells may be arranged asymmetrically, for example, the nanowells may have a shape or configuration to form an elongated structure. An elongated structure refers to a shape in which the dimension along the first axis is larger than the dimensions along the second axis. Asymmetrically arranged nanowells may be elliptical, rectangular, etc. In such embodiments, the sample may be deployed along the direction or axis of the long side, and SIM is used to increase the resolution along the short side. If the shorter side of the nanowells is located along the x-axis and the longer side along the y-axis, the pitch Px of the asymmetrically arranged flow cell is narrow or tight, which leads to an increase in resolution, while the pitch Py is larger along the y-axis. Accordingly, the resolution is increased along the x-axis and only three images are captured for proper deployment of the sample contained in the nanowells of the flow cell.

[00149] FIG. 7 shows simplified patterns of interference fringes of illumination that can be projected onto a plane of a sample 570 by a first diffraction grating and a second diffraction grating of a SIM imaging system 500 with two optical gratings for imaging slides during an image capture process for a structured illumination imaging cycle. For example, the SIM imaging system 500 can use a first diffraction grating 531 and a second diffraction grating 532 to create the illumination patterns shown in FIG. 7. As shown in the example of FIG. 7, the two gratings project perpendicular patterns of interference fringes onto the surface of the sample 570 and are located at an angle of approximately ±45° from the axis of motion of the linear moving stage 530.

[00150] A first grating (e.g., grating 531) can project interference fringes of a first order of illumination onto a sample 570. Any light emitted by the sample can be captured by camera 580, and a first phase image of a first pattern (e.g., a pattern at an angle of +45°) can be captured to create a first phase image. In order to capture additional phase-shifted images, the pattern projected by the grating can be phase-shifted by moving the linear sliding stage. These phase-shifting movements are illustrated as steps 1 and 2 in FIG. 7. Moving the phase shifts can provide small (e.g., approximately 3 to 5 micrometers or less) movements of the gratings to slightly shift the interference fringe pattern projected onto the grating.

[00151] After capturing all phase-shifted images for a diffraction grating, the system 500 may switch diffraction gratings by moving the linear moving stage 530 to optically couple another diffraction grating to the light source of the imaging system (e.g., moving from FIG. 4A to 4B). This movement is shown as step 3 in the example of FIG. 7. In the case of changing the diffraction grating, the linear moving stage may provide a relatively large movement (e.g., on the order of 12-15 mm).

[00152] A series of phase images for the next grating can then be captured. For example, as shown in FIG. 7, the second diffraction grating can project first-order light interference fringes onto the sample, and the position of the projected interference fringes can be changed by moving the linear moving stage 530 to capture three phase images of the grating pattern (e.g., steps 4 and 5 in FIG. 7).

[00153] After capturing all images for an imaging cycle (in this example, six images), a high resolution image can be composed from the captured images. For example, a high resolution image can be reconstructed from the six images shown in FIG. 7. Suitable algorithms can be used to combine these different images to synthesize a single image of the sample with a significantly higher spatial resolution than any of the individual component images.

[00154] During high-resolution image composition, unwanted shifts or changes in structured illumination parameters (such as angle, spacing, and phase) can be compensated for using an algorithm with structured illumination parameters estimated in accordance with the present invention (e.g., predicted changes in angle, spacing, or phase). For example, shifts in phases, angle, and/or spacing of vertical illumination images and/or horizontal illumination images can be compensated for. In one embodiment, one or more shifts in phases, angle, and/or spacing of images can be calculated periodically, such as in the first cycle of a sequencing run. The frequency of these evaluations can be increased or decreased based on environmental factors that affect the stability of the process.

[00155] In some embodiments, unwanted shifts or changes in the structured illumination parameters may be compensated for prior to image capture by controlling one or more hardware elements of the system 500 to compensate for these changes in the SIM imaging system. For example, prior to an imaging sequence and/or between image captures of an imaging sequence, a phase change may be compensated for by moving the linear sliding stage 530. In some embodiments, a combination of hardware and algorithmic compensations may be implemented.

[00156] According to embodiments described herein, structured illumination parameters may be predicted for a particular point in time using estimates of structured illumination parameters obtained from images captured before and/or after that time. For example, computational resource limitations may limit the rate at which a SIM imaging system (e.g., system 100, 200, or 500) can directly estimate structured illumination parameters, such as phase, frequency, and/or orientation, from captured images. In some cases, a SIM imaging system may directly estimate or measure a structured illumination parameter in each phase image, in which case prediction of the structured illumination parameters may not be required. However, in other cases, the SIM imaging system may directly estimate or measure a structured illumination parameter for some phase images of an imaging cycle, once per imaging cycle, or even less frequently (e.g., every 3, 5, 10, 50, or 100 imaging cycles). In such cases, to maintain the image sampling rate of the system, it may be appropriate to use a direct estimate of the structured illumination parameter that was obtained for a particular point in time and/or space to make predictions about the structured illumination parameter at other points in time and/or space.

[00157] To mathematically describe one example of this fundamental value, one way to estimate the parameters of structured lighting is by correlation with respect to a reference.

where h(x) is a reference that may be either known or obtained from image data, c(x) is obtained from image data that correlate with the reference, and ƒ is the value to be estimated (in this example, frequency). It should be noted that other alternative estimation techniques may be used in accordance with the present invention.

[00158] In the example of equation (1), one output correlation signal can be generated for each set of hypothetical values of ƒ. The estimate of the parameters ƒ can be obtained as the value of ƒ that maximizes the magnitude of the correlation. However, in many cases it is necessary to try to increase the number of hypothetical values of ƒ in order to maximize the output correlation signal. A large search space can increase the computational requirements and as a result lead to a decrease in the performance of the system (i.e., less data can be processed per unit of time).

Distortion of an optical device (in particular, a lens)

[00159] The next two sections present parameter estimation techniques for reconstructing a SIM image. Some of the disclosed techniques compensate for fringe peak lines that are distorted or curved due to lens imperfections. Pattern lines that should be parallel remain so near the center of the image, but typically converge or become non-parallel near the edges of the lens. This affects the angle or orientation of the illumination peak, the spacing of the illumination peak, and the phase shift. FIG. 8A shows dividing an image fragment into overlapping regions called sub-fragments, sub-windows, or sub-regions. The sub-fragments are small enough that parameters can be set that will give a satisfactory reconstruction for the entire sub-fragment. Parameter estimation can be performed in two steps. First, parameter estimation can be performed for the sub-fragment of the image closest to the center. The parameters for the other sub-fragments can then be estimated and compared with the closest sub-fragment to determine distortion and distortion corrections relative to the parameters for the closest sub-fragment.

[00160] To illustrate the implementation of these techniques, the present inventors present an example of pseudocode for estimating parameters in a "sub-fragment angle and spacing matching" algorithm (also referred to as Algorithm 2A) and in a "sub-fragment phase matching" algorithm (also referred to as Algorithm 2B). As an example, the inventors also present pseudocode for determining distortions and correcting distortions in the estimated parameters for non-central sub-fragments compared to sub-fragments closest to the centre in a "final distortion calibration algorithm" (also referred to as Algorithm 1).

[00161] FIGS. 8A-8C show the physical aspects of a full field of view (FOV). In one embodiment, a rectangular sensor of 5472 pixels by 3694 pixels is used. Of course, a square or other sized sensor can be used, such as 5472 x 5472 pixels or 4800 x 4800 pixels. When using a rectangular sensor, the distortion is greatest at the edges of the lens. The lens is often circular, so the rectangular sensor does not approach the edge of the lens on the long side as closely as on the short side.

[00162] FIG. 8A provides two illustrations that show the distortion of the spacing of the interference fringes across the field of view (FOV). Figure 801 on the left is a simplified illustration 801 of parallel lines that are curved due to the distortion of a magnifying lens. The lines shown should be parallel in the image plane. When viewed through the lens, they appear to converge at the right and left ends compared to being spaced apart in the center. Figure 805 on the right is another exaggerated example. In this figure, the interference fringes are oriented diagonally between the upper left and lower right corners. The spacing of the interference fringes is exaggerated to make them easier to see. The interference fringe lines converge at the upper left and lower right corners relative to the center. For a particular manufacturer's lens, the interference fringe pattern may not be uniform.

[00163] FIGS. 8B and 8C illustrate measurements of the image spacing between nominally parallel fringe peaks in the image plane for green and blue laser illumination. The color scale indicates the variation in spacing between 2.08 and 2.22 (sensor pixels). In both figures, the color scale indicates that the central spacing between parallel lines is approximately 2.14. Non-uniformity for green wavelength illumination is visible in the upper right corner of FIG. 8B. More significant non-uniformity for blue wavelength illumination is visible in FIG. 8C along the right and left edges. In these figures, the fringe pattern is a series of parallel lines from the lower left corner to the upper right corner of the figures. Thus, the spacing is measured in a normal position relative to the general direction of the arrow in FIG. 8C. These figures substantiate the correction for lens-induced distortions. Since lenses are manufactured and installed individually, calibration and correction of individual systems is desirable after assembly.

[00164] FIG. 8B in grayscale illustrates three conceptual contour ovals with the main axis along the diagonal surface from the upper left to the lower right corner. The contour lines approximately indicate areas of different spacing between the parallel lines. In the provisional priority application with color drawings, the evaluated variations are drawn in more detail on the color scheme. The interference fringe pattern is a series of parallel lines at an angle of 45° from the lower left corner to the upper right corner of the figure. The spacing between the parallel lines increases in the central area, as shown by the lighter shade of gray. As one passes from the central area to the upper right or lower left areas, the spacing between the interference fringe lines decreases. The main axes of the three contour lines are approximately perpendicular to the direction of the interference fringe lines. The distance between the dimensions decreases as one exits the area in the central contour line. FIG. 8C in grayscale also contains contour lines. The contour lines in FIG. 8C have the same meaning as described above and indicate areas with different distortion ranges between spacings.

[00165] FIG. 8D shows sub-tiles or sub-fields of a full field of view (FOV) in a fragment of an image. This figure shows a sub-tile of 512 pixels by 512 pixels. These sub-tiles may split the field of view as shown, or may overlap. The sub-tiles may be larger or smaller. For example, it has been shown that it is possible to work with sub-tiles of 400 x 400 and 1024 x 1024. The figure shows 5 x 7 sub-tiles. The larger sensor mentioned above may have 8 x 11 sub-tiles. Other sub-fragment configurations may be used, such as 3 x 3, 5 x 5, 5 x 7, 9 x 9, 9 x 16. Larger sensors may be divided into more sub-fragments. The sub-fragments may overlap by at least 2 pixels of the optical sensor. More and fewer pixels may be used for the overlap between the sub-fragments. For example, for a sub-fragment with a width of 512 pixels, an overlap of up to 256 pixels may be used, and for a sub-fragment with a width of 1024 pixels, an overlap of up to 256 pixels may be used. According to FIGS. 8B and 8C, there are several potentially suitable sub-fragments 803 near the center, all of which are in the sweet spot of the lens, including the central sub-fragment in an array of sub-fragments with an odd number of rows and columns. As used herein, the sub-tile closest to the center includes the central pixel of the sensor or is adjacent to a sub-tile that includes the central pixel. In some optical systems that are flat and have a small error, a sub-tile further from the sub-tiles adjacent to the central sub-tile may be used as a reference without affecting the overall distortion compensation.

[00166] The described technique involves matching a distortion measured over substantially the entire full field of view captured by the sensor. Three parameters that determine the resolution-enhanced SIM reconstruction based on properly structured illumination include the fringe spacing, the fringe angle, and the fringe pattern phase shift. These variables are also referred to as the spacing, angle, and phase shift of the structured illumination or grating pattern. The deviations of the spacing and angle from the value of the central patch can be approximated over the entire field of view using polynomial surfaces. Both quadratic and cubic surfaces have been investigated. Higher-order polynomial surfaces can also be used.

[00167] Both the fringe spacing and the fringe angle on an image fragment can be approximated by quadratic surfaces. Sensitivity analysis shows that quadratic surfaces are approximated very closely, as are cubic surfaces. A quadratic surface corresponds to the equation:

In subsequent imaging runs, the compression or expansion of the spacing between the central sub-fragment due to temperature changes can be characterized and the same offset correction can be applied to the rest of the image. Depending on the contraction and expansion near the central sub-fragment, the offset correction can be modified and transferred accordingly. The authors observed relative offsets in the sub-fragments with respect to the central or closest sub-fragment, which is used as a reference.

[00168] FIG. 9 illustrates a color-coded image of a surface that corresponds to measured data points shown in red. Since this is a three-dimensional surface, it is recommended to refer to the color version of the figure to understand the contours. The dark areas in FIG. 9, shown in grayscale, represent the dark colors at the top and bottom of the three-dimensional graph. For example, the dark upper right portion of the graph in FIG. 9 represents the dark red color, and the dark area at the bottom represents the blue colors in the color graph. At the time of filing of the priority application, the USPTO (United States Patent and Trademark Office) stored the color version in the additional content tab of the PAIR online system, where it will later be publicly available.

[00169] In practice, it has been noted that the difference in distortion in the field of view can be represented in the form of a bowl or an inverted bowl, while the distortion of the angle is usually represented in the form of a saddle. FIGS. 10A and 10B in the form of an inverted bowl illustrate the measured spacing relative to the spacing in a sub-fragment located close to the center. FIGS. 12A and 12B in the form of a saddle illustrate the measured angles relative to the angle of the interference fringe pattern in a sub-fragment located close to the center.

[00170] In FIGS. 10A and 10B, the x-axis corresponds to the length of the sensor. The y-axis runs along the width of the sensor. The labels on the x-axis and y-axis are sub-section indices. The vertical axis spacing is normalized to 1.000 of the z-axis spacing between regularly spaced fringes observed in the near-center sub-section. FIG. 10A shows the measured data points without curve-fitting smoothing. This shows a change of approximately 1.2 percent in the observed fringe spacing between measurements in the center and lower right portions of the sensor. In FIG. 10B, the actual data points, shown in blue, are shown compared to a quadratically fitted surface. In the next section, the authors present the measurement of spacing and angle distortion for the full FOV compared to the near-center sub-section. Black and white versions of FIGS. 10A and 10B have dark areas at the top and bottom of the graphs. The dark shade at the top represents red, and the dark shade at the bottom represents blue.

Calibration of spacing and angle distortion

[00171] This section further describes methods for calibrating the full field of view (FOV) of the sensor used to capture structured illumination and ensure that the function matches the nonlinear effects caused by lens distortion. Structured illumination includes projecting a pattern with a regular spacing and tilt onto an image plane. An image fragment is a captured portion of the projected structured illumination pattern. From the full field of view, a sub-fragment closest to the center is selected. The spacing and angle of the sub-fragment are measured, including the variation in the sub-fragment itself. The spacing and angle of the pattern of other sub-fragments in the full field of view are then determined. The spacing and angle of the pattern in the full field of view are expressed relative to the spacing and angle of the pattern of the sub-fragment closest to the center. Polynomial surfaces are repeatedly fitted to the spacing and angle of the pattern over substantially the entire full field of view.

[00172] FIGS. 11A, 11B, and 11C show a comparison of the measured spread distortion with a quadratic and cubic surface approximation. The color change is virtually imperceptible in the three figures because the spread deviation in the field of view is less than 2% in all three figures. Visual analysis of these figures and calculation of the approximation errors suggest that the quadratic polynomial surface provides a reasonably good approximation of the spread distortion. The spirit and methodology of the present disclosure encompass higher-order polynomial surfaces and alternative approximation schemes that may provide somewhat better or worse results than using a quadratic polynomial surface. Black and white versions of FIGS. 11A, 11B, and 11C include dark tones in the central areas that represent deep red, while the outer areas at the top right and bottom left have lighter tones that represent light red, fading to orange and yellow near the borders.

[00173] FIGS. 12A and 12B have the same x- and y-axis configuration as FIGS. 10A and 10B. The vertical axis in these figures indicates the orientation angle of the interference fringes observed in the field of view. The nominal angle can be taken as 45°. The range of measured angles is from 44.85 to 45.15 degrees, which represents a deviation of 0.6%. FIG. 12A shows the measured data points without smoothing by curve fitting. In FIG. 12B, the actual data points, shown in blue, are shown in comparison with the quadratically fitted surface. The black and white versions of FIGS. 12A and 12B have dark areas at the top and bottom of the graphs. The dark tones at the top represent dark red, and the dark tones at the bottom represent dark blue.

[00174] FIGS. 13A-13E and 14B-14H show how cropping along the boundary of the sensor can improve the curve approximation for the spacing and angle of the interference fringes in the range of the sensor. The x- and y-axes of the graphs in these figures represent the positions of the pixels along the length and width of the sensor. As shown in FIG. 8C, cropping is indicated by the boundary near the edges of the sensor. The figure shows that this cropping or shrinkage removes distortion near the edges of the image captured by the sensor. The following figures show how this image reduction provides a good approximation of the FOV data on the surface model. The shrinkage factor or shrinkage unit can be equal to the shrinkage of the image by one step of the sliding window. The step value can be selected based on the size of the sub-fragments or empirically based on the number of pixels at the edges of the sensor that can cause distortion. For example, if a step of 200 pixels is selected, one shrink block can remove the outer 200 pixels of low-quality image data from all sides of the image. You can use a step of less than 200 pixels, such as 25, 50, or 100 pixels, or more than 200 pixels, such as 225, 250, or 300 pixels.

[00175] In FIG. 13A, the blue dashed line shows a quadratic surface of a model for approximating the spacing distortion from the full field of view (FOV) image data without cropping. For this full FOV image data, no shrinkage factor is applied, i.e., the shrinkage factor is zero. As shown in FIG. 13A, the full FOV data does not provide a good approximation to the model. As the shrinkage factor increases from 1 to 4 (FIGS. 13B-13E), the approximation between the model and the uncropped data points can be improved. Since cropping eliminates outer data points, the z-axis scale decreases, indicating less variation among the sub-tiles. As shown in FIG. 8C, the outer data points are located along the edges of the sensor where cropping is applied. In FIG. 13C, many more outer data points are visible than in FIGS. 13D-E. As shown in FIG. 13E, the MSE value has decreased to 0.000001 and the z-scale has dropped slightly lower than in FIG. 13D. The black and white versions of FIGS. 13A-13E, 14B-14H, and 15A-15B have dark areas at the top and bottom of the graphs. The dark tones at the top represent the dark colors at the top and bottom of the graphs. Some of these graphs have light and dark circles located on or near the graphs. The light and dark colors of the circles do not indicate any difference between the circles. Some circles are shown dark for clarity.

[00176] FIGS. 14B-14H illustrate an improvement in the quadratic surface approximation for angular distortion by gradually increasing the shrinkage factor applied to the full FOV image data. As the shrinkage factor increases, the z-scale is again reduced and the array of data points moves closer to the approximated surface. Eliminating the outer distorted values at the edges of the sensor significantly improves the surface approximation. In FIGS. 14B-E, the distortion is noticeable as blue data points in the plane below the approximated surface. As the shrinkage factor is increased from 1 (FIG. 14C) to 3 (FIG. 14E), the z-scale is reduced by half: from [-45, -20] to [-44, -40]. In FIG. 14F, the cropping eventually allows the data points and the approximated surface to be brought closer together. In FIG. 14F The MSE is reduced to 0.000033. In this paper, the quadratic surface model for the angle distortion is matched to the measured data, shown as blue dots. In FIG. 14H, increasing the shrinkage factor by 6 does not improve significantly. In the figures, a shrinkage factor of 4 or more gives a positive result. A threshold based on a percentage or absolute improvement in MSE can be used to select the shrinkage factor or to determine the effect of shrinkage factors on resolution. The selection of the shrinkage factor can be automated using a ratio-based or threshold-based approach.

[00177] Characterization of the imaging system by fitting distortion surfaces can be performed before and/or during operation of the instrument. The spacing and pattern angle are periodically evaluated in the sub-tile closest to the center and using extrapolated factors to determine the spacing and pattern angle in the full field of view. The extrapolation factors can be applied without evaluating the spacing and pattern angle beyond the sub-tile closest to the center. The sub-tiles of the full field of view can overlap each other. During calibration, the spacing and pattern angle are calculated independently for the overlapping sub-tiles. In this approach, two orientation angles can be used for the structured illumination pattern such that the two angles are substantially aligned with opposite corners of the rectangular or quadrangular nanowell pattern.

[00178] Thus, the described technology includes calibration and recalibration of the instrument to determine the variation among sub-fragments of the fragment. The characteristic distortion of the spacing and angles is applied in such a way as to significantly reduce the need for parameter estimation during the operation of the instrument. The characteristic values/calibration can be improved by cropping pixels at the edge of the sensor that are so distorted that they are not approximated on the matching surface, i.e., cropping pixels that prevent approximation.

Using calibrated spacing and angle

[00179] The described technology allows for efficient reconstruction of an improved image over virtually the entire area of a full field of view. For this purpose, the full field of view is divided into at least nine sub-fragments or sub-fields for independent reconstruction based on the sub-fragment. It is possible to estimate or obtain pre-estimated spacing parameters and pattern angle parameters by viewing a structured illumination pattern with a regular spacing and angle for the sub-fragment closest to the center. Extrapolated coefficients are used to transfer the parameters of the sub-fragment closest to the center to other sub-fragments in the field of view. To perform an improved image reconstruction from a plurality of images, the estimation parameters of the sub-fragment closest to the center and the extrapolated parameters of the other sub-fragments are used. The plurality of images are viewed under a plurality of pattern angles in periodically shifting phases of the pattern of the corresponding pattern angles by applying structured illumination microscopy (SIM) analysis to the plurality of images. In one embodiment, the above-described technology allows for additionally estimating a periodic shift in the phase pattern of the corresponding pattern angles. The above technique can be extended to merge sub-fragments by identifying a regular noise pattern along the merge boundaries and removing the noise.

[00180] FIGS. 15A and 15B illustrate examples of distortion models fitted from real data that was collected from sequencing instruments. FIG. 15A illustrates a quadratic fit to a reaction surface obtained by fitting the distortion of the spacing of interference fringes in a full field of view (FOV). FIG. 15B illustrates a quadratic fit to a reaction surface obtained by fitting the distortion of the angle of interference fringes in the FOV.

Estimation of phase shift

[00181] The phase shift is the remaining, third input parameter for the SIM image reconstruction. In this section, existing techniques are adapted to estimate the phase shift. The described technology allows for sub-fragment based estimation, so a technique is additionally developed to establish common criteria for sub-fragment based phase shift estimation relative to a full FOV fragment. One embodiment of the phase estimation uses the technique proposed by Wicker et al. 2013 in their paper entitled Phase optimisation for structured illumination microscopy, section 3. The equations from Lal et al. 2015 entitled Structured illumination microscopy image reconstruction algorithm and Wicker et. al. 2013 provide a better understanding of the Wicker phase estimation.

[00182] The following equation (3), taken from Lal et al. 2015, separates the three frequency component bands: from the obtained images The mixing matrix uses estimates of the phases φ ₁ , φ _{2 ,} and φ ₃ of images captured using a pattern with sinusoidal illumination intensity I _θ,Ф (r) corresponding to the angle or orientation of the pattern equal to θ. Wicker et al. 2013 refer to the phase for the ^nth image at orientation φ _n. If the phases are not known with sufficient accuracy, the process of demixing or band splitting will not perfectly separate the spatial frequency components based on the observed images in the frequency domain. In practice, three spatial-frequency components to one degree or another will contain residual information from other components, which is referred to as “noise”.

This three-component expression follows from the Fourier transform for sine or cosine illumination. Other lighting functions may modify these equations.

[001831 Therefore, information about the phases of the illumination pattern with sinusoidal illumination intensity is important. Since it is not always possible to precisely control these phases in the experimental setup, it is desirable to determine the phases of the illumination pattern from the acquired image data. Wicker et. al. 2013 presented a phase estimation technique for SIM data acquired using coherent sinusoidal illumination at a selected frequency. Coherent illumination gives good pattern contrast from gratings with a very small illumination peak spacing "s", which improves the resolution in the reconstruction. The phase of the illumination pattern ^{of the nth} image is reconstructed using the peak frequency of the illumination pattern. The peak frequency of the illumination pattern is also called the Fourier peak.

[00184] The following equation (4) from the paper by Wicker et. al. 2013 represents a generalized form of equation (3) with the resulting images across all frequencies in the frequency domain. Each image contains three components, labeled superimposed with different phases. It should be noted that these three components are the same as in equation (4).

[00185] It should be noted that "c" in equation (4) is called the contrast of the illumination pattern. In the absence of noise, "c" coincides with the modulation coefficient "m" in the mixing matrix M in equation (3). To determine φ _n , the frequency in equation (4) is replaced by which is the peak frequency of the illumination pattern obtained from equation (5).

[00186] Equation (5) shows that the phase of the pattern φ _n is approximately equal to the phase of the resulting image on frequency This rough estimate of the pattern phase φ _n can give good results if three recommendations are observed. First, the contrast "c" of the illumination pattern should be sufficiently high. Second, the spectral power of the sample should decrease rapidly with increasing frequency. When these two recommendations are observed, the last term in equation (5) dominates and can therefore be simplified:

For any real-valued sample, the central frequency is will be real-valued. Furthermore, if the point spread function (PSF) is real and symmetric, then the optical transfer function (OTF) will be real. The OTF is a convolution of the point spread function (PSF). The point spread function is the spatial domain version of the optical transfer function of the imaging system. The name "point spread function" indicates that all physical optical systems blur (spread) a point of light to some extent, with the amount of blur determined by the quality of the optical components. The resolution of the imaging system is limited by the size of the PSF. For asymmetric PSFs, the phases of the OTF must be taken into account.

[00187] Third, OTF at pattern frequency must be large enough to overcome the noise. If the OTF is too small, the noise in the resulting image can significantly alter the phase measured at This method of phase estimation cannot be used for frequencies. pattern beyond the OTF detection support limits. For such frequencies

[00188] The OTF of an optical system can be determined experimentally. For example, Lal et al. (2015) calculated the OTF by acquiring multiple images of samples using spatially distributed 100 nm fluorescent microspheres. The intensity distribution corresponding to more than 100 microspheres was then superimposed and averaged to obtain an approximation for the PSF of the system. The Fourier transform of this PSF yields an estimate of the OTF of the system. The phase estimation technique described for sub-fragments can be used for this purpose.

Transfer of subfragment phases to common criteria

[00189] It may be useful to estimate the phase shift of the tiles relative to the entire field of view (FOV), so that the phase measurement in one subtile can be extrapolated to other subtiles in the tile. The illumination peak angle and illumination peak spacing for the entire FOV can be estimated from the illumination peak angle and illumination peak spacing of the subtile using the quadratic models presented above. The phase shift may be less regular since it depends on the geometry of the subtile pixels, which may yield a function with an uneven step instead of a smooth function. Each phase estimate has a "criterion" anchored to the upper left corner of the image used for the phase estimate. As a result, if it is desired to correct for phase differences geometrically across the entire image, the phases estimated from each subtile need to be compared to a reference phase (from the central subwindow or estimation subtile) in a common criterion. The phase estimate of each sub-patch is transferred to the same geometric location to achieve a common criterion. The phase transfer requires an accurate estimate of the fringe angle and spacing for each sub-patch. The fringe angle and spacing can be obtained from quadratic surface distortion models that have been calibrated by the distortion calibration process.

[00190] The phase estimation is specified using common criteria across all sub-tiles of the full FOV image. The coordinate space of the sub-tiles is mapped to the coordinate space of the full FOV. This section illustrates the transfer of coordinate points from one space to another. FIG. 16A shows a common setting of a sub-tile window in a full FOV image window and the corresponding variables, as well as an adjusted phase shift formula that transfers coordinates between local and common references. FIG. 16A shows a boundary 1603 representing a full field of view that contains a sub-tile shown by boundary 1605. The cropping angle a is defined by a line 1606 connecting the upper left corner of the full FOV window 1603 and the upper left corner of the sub-tile window 1605. The length of the line 1606 is measured in pixels and is denoted by " a ".

[00191] The illumination peak angle β is formed between the upper boundary of the sensor, window 1603, and a line 1607 perpendicular to the fringe peak line 1609 in the upper left corner of the sub-fragment window 1605. This fringe peak line will be parallel to the actual fringe peaks and will likely be offset from the peaks. The length of the line 1607 from the upper left corner 1603 to 1609 is the projected distance "b". The phase difference φ between the upper left corners of fragment 1603 and sub-fragment 1605 is a function of the spacing "s" and the distance "b". In formula (7), the illumination peak spacing is represented as "s" in pixels, and the phase is in degrees:

This formula is derived and described below. It can be used to transfer the phase shift from the sub-fragment coordinates to the full FOV coordinates. The transfer between coordinate systems is shown in the following two examples.

[00192] In SIM reconstruction, a one-dimensional (1D) lattice can be described using an intensity function, such as:

where the parameters have the following meaning:

• M – modulation depth;

• s - lighting phase shift;

• - unit vector;

• β - peak angle of illumination.

[00193] The highlighted values x, u in the equation are vectors: x is a vector in image space expressing the position in image space at which the intensity is determined; u is a unit vector. Sinusoidal illumination intensity values are positive over a scale range from +M (bright) to -M (dark).

[00194] Once the phase shift difference between sub-fragments is characterized, it is only necessary to estimate the phase shift in the sub-fragments closest to the center, since the phase shifts of the other sub-fragments can be extrapolated from the sub-fragment closest to the center.

[00195] The following calculations relate to the estimation of the phase shift φ. Instead of estimating the phase shift using the full image (full FOV) data, it is possible to use data from only a sub-image. The phase shift of a sub-image is related to, but not identical to, the phase shift of the full FOV. To reconstruct the phase shift of the full FOV from the phase shift of a sub-image, it is necessary to determine how the phase shifts change during spatial translation. The following mathematical model represents this spatial translation from a sub-image to the full FOV.

Derivation of coordinate transformation

[00196] Consider the following function that generates the grid lines in the orientation shown in FIG. 16A. The two-dimensional (2D) space of the full FOV is represented by the coordinate space _Ox with coordinates ( _x1 , _x2 ), as shown in FIG. 16B. The point 1611 is represented by the position ( _x1 , _x2 ) relative to _Ox . The function f( _x1 , _x2 ) can be defined below, as shown in equation (9). This function can be used to generate the lines of the illumination pattern in the orientation shown in FIG. 16A.

[00197] The one-dimensional periodicity of the above-described function ƒ is represented by the function in equation (10). As shown in FIG. 16A, this function is the repetition of the lines of the illumination pattern that extend along line 1607 in FIG. 16A, which is perpendicular to the lines of the illumination pattern. This function can determine the spacing "s" between two lines by determining the position at which a particular intensity value is repeated. Equation (10) shows the periodicity of the intensity of the illumination pattern. In equation (10), the line perpendicular to the intensity peaks has an angle of 45 degrees extending downward and to the right from the upper left corner, so the intensity at point (x ₁ ,x ₂ ) is repeated at the next point down and to the right along line 1607.

[00198] The illumination peak space "s" defined by the above function is shown in equation (11). For convenience, the coordinate system can be scaled to match the spacing of the illumination pattern, as shown in equation (11).

[00199] The peak angle of illumination is defined by equation (12). The peak angle of illumination is formed between the upper boundary (horizontal line) of the full FOV and the line perpendicular to the lines of the illumination pattern, as shown in FIG. 16A. Equation (12) shows this angle of -45 degrees for the structured illumination pattern generated by the 1D periodicity function in equation (9).

[00200] The zero set of lines of the patterns formed by the periodicity function from equation (9) is a connection of lines and is represented by equation (13).

[00201] The zero set contains the lines of the structured illumination pattern, located at a distance "s" from each other. The lines of the patterns are formed by the points at which the value of the 1D function ƒ(x ₁ , x ₂ ) in equation (9) is zero. The unit vector perpendicular to the grating is defined by equation (14).

[00202] Now we offer two examples of transferring the phase shift of a subfragment from the coordinate space O _{y of} the subfragment defined by the coordinates (y ₁ ,y ₂ ) to the coordinate space O _x of the full FOV. FIG. 16B shows a point 1611, which can be represented in the coordinates of the subfragment by the vector coming out of the upper left corner of the subfragment window 1605. The same point 1611 can be represented in the coordinates of the full FOV by the vector coming out of the top left window 1603 for a full FOV image Vector connects the upper left corner of the full FOV image window 1603 to the sub-fragment window 1605, as shown in FIG. 16B. The following sections provide examples of the position of a point 1611 that can be transferred from one coordinate system to another.

First example

[00203] Let a=(a, -a)∈R ² , represent the upper left corner of the sub-fragment 1605 in FIG. 16B. This means that the upper left corner of the sub-fragment 1605 is offset from the upper left corner of the full FOV window 1603 by a distance " a " along the horizontal x ₁ axis and by a distance "- a " along the vertical x ₂ axis. Now we translate the point (x ₁ ,x ₂ ) from the _x -coordinate space of the full FOV to the point (y ₁ ,y ₂ ) in the y-coordinate space _of the sub-fragment. Consider the change in coordinates as:

[00204] Now we can represent the 1D periodicity function of equation (10) in the coordinate space of the Oy subfragment as:

[00205] As can be seen, the phase shift introduced by changing the coordinates is given by:

Second example

[00206] Now consider another position of the subfragment relative to the upper left corner of the full FOV window. Let a=(2a, -a)∈R ² , in this case the change in coordinates is given by:

[00207] The 1D periodicity function from equation (10) can now be represented in the coordinate space Oy of the subfragment as:

[00208] In this case, the phase shift introduced by changing the coordinates is given by:

where (u, a ) is the scalar product of the vector u and the vector a . The scalar product is defined as the sum of the corresponding components of the two vectors.

General example

[00209] Theorem 1.

Let s>0, φ ₀ ∈R, au∈R ² be a single vector. Consider the function

a) The spacing "f" between the gratings is equal to "s".

b) For any a ∈R ² there exists φ _a ∈R, thus

[00210] Equation (21) shows the periodicity function with respect to the Ox coordinate space of the full FOV and has a phase shift of φ ₀ . Equation (22) shows the periodicity function for the subfragment coordinate space and has a phase shift of φ ₀ .

c) Any displacement φ _a satisfies the equation

[00211] Proof (a) The peak illumination spacing is the length of the fundamental period for ƒ. The vector su is a period of length c, since

Any smaller period would mean a period less than 2π for sine.

(b) One of the solutions:

because

c) If sin(t- a )=sin(tb) for all t∈R, then

[00212] The relationships obtained above can be used to transfer the phase shift estimated in the first coordinate space to a second coordinate space, for example between a sub-tile and the full FOV image. This is useful because the phase shift is also subject to distortion between the sub-tile closest to the center and the edge of the lens. In the next section, the authors present a phase deviation lookup table that can capture the relationship between the phase shift values for off-center sub-tiles relative to the center sub-tiles.

Phase Deviation Estimation

[00213] The described technique includes estimating the phase for each sub-tile and then transferring the phase offset to the full FOV coordinate system to present the predicted phase offset values in common criteria. The difference between the phase offset values for the off-center sub-tiles and the sub-tile closest to the center is stored in a phase offset or phase difference look-up table for each angle of the illumination pattern. In some cases, the influence of environmental factors can increase the instability of the system. In such cases, the phase offset look-up table is generated more frequently to compensate for the increased instability of the system. For example, the phase offset table can be generated in the first cycle of each sequencing run. The frequency of generating the phase offset look-up table can be increased or decreased depending on the stability of the system.

[00214] FIG. 17 shows an example of a phase deviation look-up table 1701 with color coding of phase differences from the sub-tile closest to the center at 3, 5, which has a value of zero. The look-up table 1701 contains the same number of cells as the number of sub-tiles in the full FOV image, in this example - 7 x 9 in sub-tiles. The position of a cell in the table 1701 may correspond to the position of a sub-tile in the full FOV image. The phase look-up table stores the differences in phase between each of the sub-tiles relative to the calculated phase of the center window (3, 5). The example table contains the phase deviation for one angle of the structured illumination. A similar phase deviation table would be stored for each angle of the structured illumination pattern. The phase values depend on the overall error in the calculated values of the other parameters (angle and spacing). It should be noted that phases can have values from 0 degrees to 360 degrees. The phase of the central sub-fragment is shown as 0 degrees in the example table. The color coding indicates the degree of difference in the phases of the sub-fragments corresponding to the central sub-fragment. Estimation of phase deviation requires the parameters angle, spacing, fringe phase of the central sub-fragment or sub-window, and a distortion model (which is assumed to be constant). Examples of algorithms for estimating the parameters angle, spacing, and phase are presented in the following sections.

[00215] In the above sections, the authors described methods for estimating the parameters for sub-tiles in a full FOV image. In the pseudocode that follows, the authors further illustrate estimating the angle, spacing, and phase parameters for the sub-tiles. An example of pseudocode for estimating the angle and spacing parameters is presented below and is referred to as "sub-tile angle and spacing matching" (also referred to as Algorithm 2A). To estimate the distortion coefficients of the angle and spacing parameters for the sub-tiles, the authors use a polynomial approximation of a quadratic or cubic surface. An example of pseudocode implementing the phase estimation method is presented below and is referred to as the "sub-tile phase matching" algorithm (also referred to as Algorithm 2B). The phase offset estimates can be stored in a phase offset lookup table for each angle or orientation of the illumination pattern. In one embodiment, a separate lookup table may be used for each sequencing fragment on the flow cell to account for minor differences in flow cell orientation, surface defects, and other non-ideal properties that may distort the calculation results. Pseudocode for an example final distortion calibration algorithm is presented last and is called "final distortion calibration using quadratic approximation and lookup table" (also called Algorithm 1 and Algorithm 2C).

Algorithm 1. Final distortion calibration using quadratic approximation

[00216] In the first pseudocode, we can apply cropping to remove distortion near the edges of the image. This provides a good approximation of the full FOV data to quadratic surface models for angle and spacing estimation for calibration/characterization.

Input data A set of images, for example for 2 angles and 3 phases per angle.

Output data Model distortion coefficients.

Algorithm

1. Create the following images/filters (size corresponds to the sub-area).

a.OTF

2. Dividing the image into overlapping subregions of equal size, such as 512 × 512, using the sliding window method.

3. Calculated parameters in the center of the image used, for example 512 × 512 windows (see subfragment and spacing matching algorithm and subfragment phase matching algorithm below).

a. Calculated angle and peak illumination spacing for each angle.

b. Calculated modulation for each angle.

c. Calculated phase for each image.

4. For each sub-region

a. Performing the estimation of the parameters of stage 3

b. Storing the x, y coordinates of the subfragment, the fringe angle (per angle), the spacing (per angle) and the phase 0 (per angle) of the Wicker phase estimate in arrays (2d arrays).

5. For the spacing and angle of interference fringes

a. Subtract the value from the table in step 4 from the calculated values of the central window from step 3.

b. Performing an optimal shrinkage estimate for a quadratic surface.

i. Approximation of the quadratic surface by least squares regression in the functional form (x,y)=a+b* ^x2 +c* ^y2 +d*xy+e*x+f*y for each array (call this Z, for example, the spacing of the interference fringes from the 0 angle) in Step 4 (using the same X and Y array that corresponds to the coordinates of the center x,y of the subregion).

ii. Estimation of the ^R2 coefficient of the determination from step i.

iii. Compress the array X, Y, Z (removing outermost elements) and repeating step (i) until the number of compressions N is performed (N=3-5, and this parameter can be adjusted).

iv. As the dataset is compressed towards the central part of the data, the quality of the approximation is expected to improve, and the coefficients are selected at the inflection point (minimizing the number of compressions while simultaneously increasing R ² ) where the approximation of the model is good enough (current criteria define it as less than 5% relative to the R ² value).

Other compliance criteria that may be used in this document include:

• Improvement of the mean square error of approximation (the next iteration of the mean square error is less than the current iteration of the mean square error by some pre-set threshold value).

• Improvement of the maximum approximation error of the quadratic surface (each subsequent iteration should decrease this error and increase the approximation if successive improvements are found to stabilize).

• A regularized error measure, such as MAE (Mean Absolute Error) instead of MSE (Root Mean Square Error), using an improvement threshold similar to the previous two criteria.

6. The optimal coefficients in step 5 are the output of the final distortion calibration algorithm.

Algorithm 2A. Matching the angle and spacing of a subfragment

[00217] The following pseudocode is used to implement the peak illumination angle and sub-fragment spacing estimation used in steps 3-4 above and during operation. Input A collection of images for the same angle that was used for the phases of each image, uniformly distributed from 0 to 360 degrees.

Output data

1. The location of the peak in Fourier space (or the spacing and angle of the grating).

2. Fourier peak modulation.

3. Phase shift of the first image.

Algorithm

1. Fast Fourier transform of each image.

2. Carrying out the separation of lanes.

3. Images are taken from two components:

a. 0 (wide field) and +1 (high resolution part).

b. Masking all Fourier components of each image that do not overlap (also called common region).

c. Masking of low frequency components around DC and around the calculated peak of the interference fringes.

d. Using inverted OTF.

4. Repeat to reduce the thickness of the lattice around the starting point.

a. For each point in a 10×10 grid, a certain range is uniformly selected.

i. Applying a phase shift to one of the two phases of the images.

ii. Multiplication of one image by the conjugate of another.

iii. Addition of the result.

iv. Normalization with respect to the sum of the masked wide-field Fourier image.

v. Preserving the shift ensures the maximum absolute value of the normalized complex correlation.

1. The absolute value of this number represents the modulation.

2. The angle of this complex number represents the phase shift.

Algorithm 2B. Subfragment phase matching

[00218] Pseudocode for an example of implementing the estimation of the phase shift of sub-fragments. The described technology allows using recent best estimates of the values of the spacing and angle parameters for such a comparison. The system can also obtain estimated values of the spacing and angle parameters using a distortion model and the parameters of the central subwindow or sub-fragment.

Input data

1. Collecting images for one angle.

2. Peak spread and illumination angle calculated using the previously described algorithm.

Output: Phase shift of each image in the set.

Algorithm

1. Perform a Fast Fourier Transform (FFT) on each image.

2. Create a copy of the FFT image.

3. Apply OTF to each image.

4. Shifting one image.

5. Multiply both images using the conjugate and the sum of the absolute value of each complex number.

6. The phase is the angle of the resulting complex number.

Algorithm 2C. Studying the phase deviation lookup table

[00219] To perform phase deviation, the following steps are performed for a fragment on the flow cell during imaging of the first cycle of the sequencing run.

Input data A set of images, e.g. for 2 angles and 3 phases per angle, from each fragment of the sequencing flow cell, the calibrated distortion model from Algorithm 1.

Output data Phase deviation lookup table.

Algorithm

• For each sequencing fragment (not to be confused with the SIM reconstruction subfragment).

i. Application of stage 3 of algorithm 1 to estimate the parameters of the SIM central window (spacing, angle, phases of each “angle-channel” pair) of the fragment.

ii. Split the image into sub-fragments; for each sub-fragment:

a. Applying the distortion model using Algorithm 2A to obtain the spacing and angle of the sub-fragment interference fringes.

b. Perform Wicker phase estimation on the first phase image of each channel-angle pair using the specified fringe parameters from step a. on a local sub-fragment.

c. Calculate the difference between the phase obtained in step iib and the phase 0 of the central window obtained in step i, and store the phase deviation in the corresponding 2d array for phase deviation.

iii. Store the completed phase deviation lookup table in memory with the fragment identifier as a record for later use.

• Repeat step 1 for every second subfragment encountered in the entire flow cell.

Note: To improve efficiency, we evaluate the phase drift search for each fragment in the first cycle of the sequencing run; however, the training frequency can also be adjusted to train at every N cycle if stability issues exist.

[00220] In these pseudo codes, the authors presented methods for estimating and matching the peak illumination angle, peak illumination spacing, and phase shift parameters. The estimated parameters are used as input for structured illumination microscopy (SIM) image reconstruction. The next section presents the proposed SIM image reconstruction algorithm, which can efficiently reconstruct high-resolution sub-tile images and then combine these sub-tile images to reconstruct a high-resolution image of the full FOV.

SIM image reconstruction

[00221] The disclosed SIM image reconstruction technique uses angle, spacing, and phase estimates of sub-tiles to reconstruct high-resolution images. The disclosed technique can create a high-resolution image of the full FOV in two steps. First, a high-resolution SIM image is reconstructed for a sub-tile using the sub-tile parameters. Then, the high-resolution images for the sub-tiles can be combined to construct a full FOV image. In an alternative embodiment, the high-resolution sub-tiles can be processed without combining. This section begins with an example of pseudo-code for reconstructing a SIM image of a sub-tile. This is called a sub-tile reconstruction algorithm. The disclosed technique exploits symmetries in Fourier space using non-redundant data in the frequency domain. This reduces the amount of computation required for some key operations by almost half. This section also presents an example of pseudo-code for combining SIM images for the sub-tiles used to reconstruct the high-resolution image for the full image. Such a method is called a full field of view reconstruction algorithm. It should be noted that the parameter estimation methods described above can be used with traditional redundant or non-redundant SIM image reconstruction methods described above.

Algorithm 3. Subfragment recovery

[00222] The following describes pseudocode for one embodiment of high-resolution image reconstruction using sub-slice structured illumination microscopy.

Input data

a. A set of images of a sample with some grating profile (I _{angle, phase} ).

For each angle we need at least 3 phase images, spaced evenly from 0 to 360 degrees.

b. Parameters:

i. Peak spacing and illumination angle for a group of images corresponding to the angle.

ii. Phase for each image.

Output: One high-resolution reconstructed image per sub-slice.

Algorithm

1. Image pre-processing (e.g. high-pass filter).

2. Create a large Fourier image buffer (its size will depend on the final value of the upsampling factor) and initialize it to 0.

3. For each corner

a. For a set of images with a unique phase for a grating at a given angle.

i. Perform a Fast Fourier Transform on each image.

b. Perform stripe splitting on a set of images (n>=3 input images → 3 output images or component images).

i. Two of the three output images contain high resolution information.

ii. The remaining image contains all the standard information (original resolution image).

iii. The required phase for each image and modulation for the entire set

c. Apply MTF to each component image.

d. Shift each component image from the stripe split to its correct location in the larger Fourier image buffer and sum the components.

4. Create a Wiener filter from the input MTF by shifting and summing the 2D MTF image into a large Fourier image.

a. We do this for each input image at each angle.

b. There is no need to perform this operation for each reconstruction, it can be performed once and used as an input parameter.

5. Application of Wiener filters and apodization in Fourier space to the large Fourier representation of the reconstructed image.

6. Performing inverse fast Fourier transform on a large Fourier image buffer to obtain high resolution image.

[00223] This non-redundant pseudo-code for sub-chunk reconstruction is described in more detail than the other pseudo-code because the described technology cannot be explained by reference to the prior art. The described technology uses symmetry in the Fourier space to improve the efficiency of the SIM image reconstruction and to reduce the computational load. FIG. 18 shows a high-level overview of the non-redundant sub-chunk construction technology. For each sub-chunk per corner, three images 1811, 1813, 1815 are obtained, preferably with uniformly separated phases. We apply a discrete fast Fourier transform (FFT) to transform the image data from the spatial domain (real space) to the frequency domain (complex space). We exploit the symmetry of the FFT of the data representing the three images in the frequency domain. We improve the efficiency of the high-resolution SIM image reconstruction of the sub-chunk by using non-redundant parts of the image data in the frequency domain. The details of the steps of the non-redundant sub-chunk reconstruction processing algorithm are presented below.

Input data

1st Set of Images

[00224] Sample images divided into sub-fragments are given as input to the sub-fragment reconstruction algorithm. The images are divided into sub-fragments, sub-windows or sub-regions of a predetermined size, such as 512 × 512 pixels. Sub-fragments of other sizes can be used.

[00225] The image subset extends previous technologies because multiple subsets are required to capture and construct an image plane. In one embodiment, the image subsets are obtained using a diffraction grating that produces a sinusoidal illumination intensity pattern I _θ,Φ (r). The intensity varies with location (x,y) represented by vector t, which is a two-dimensional spatial position vector. For comparison, Lal et al. 2015 presents a basic structured illumination microscopy reconstruction algorithm, which is referred to as “SIM-RA algorithm 1”. To explain the steps of our SIM image reconstruction algorithm, we used the mathematical description from Lal et al. 2015. It should be noted that the reconstruction algorithm presented by Lal et al. 2015 is applied to the full image, while our sub-slice reconstruction technique reconstructs high-resolution SIM sub-slices.

[00226] The mathematical description for I _θ,Ф (r) is given in equation (1) of Lal et al. 2015. For each orientation θ of the sinusoidal illumination pattern, three SIM images of the sample are acquired, corresponding to three different illumination phases. In one embodiment, two sets of images are acquired using two orientation angles of the sinusoidal illumination pattern, i.e. θ ₁ = 45° and θ ₂ = 135°, respectively. The orientation angle θ is also referred to as the peak illumination angle β. In such an embodiment, the nanowells can be located at the corners of a square or, more generally, a rectangle or quadrilateral. The orientation angles are chosen such that images are acquired along lines connecting opposite corners of the rectangle. This increases the apparent distance between the nanowells.

[00227] In another embodiment, three sets of images can be obtained at three orientation angles, such as θ ₁ = 0°, θ ₂ = 60°, θ ₃ = 120° or θ ₁ = 30°, θ ₂ = 90°, θ ₃ = 150°. In this embodiment, the nanowells can be located at opposite corners of the hexagon. The images are taken along the lines connecting the opposite ends of the hexagon. Typically, the sets of images can be selected to correspond to the number of diagonals connecting the opposite ends of the figure in which the nanowells are located.

[00228] For each orientation 6, at least three images are obtained, spaced from 0° to 360°, corresponding to three different illumination phases φ ₁ = 0°, φ ₂ = 120° and φ ₃ = 240°. A uniform phase spacing is preferred. Additional images, for example five, six or seven, can be obtained using a smaller phase step.

1-b(i). Parameters of angle and spacing of structured lighting pattern

[00229] The illumination peak spacing "s" which defines the grating periodicity or the illumination of the pattern with a sinusoidal intensity I _θ,Ф (r) is also input. The illumination peak spacing "s" and the illumination peak angle θ for the sub-fragments are estimated using the "sub-fragment angle and spacing matching" algorithm presented above (also referred to as Algorithm 2A). The "final distortion calibration using quadratic approximation and look-up table" algorithm (also referred to as Algorithm 1) is used to compensate for errors in estimating the angle and spacing parameters for off-center sub-fragments due to optical perception distortions. Pseudocode examples for the "sub-fragment angle and spacing matching" and "final distortion calibration using quadratic approximation and look-up table" algorithms are presented above.

1-b (ii). Phase shift parameter

[00230] The phase offset (also called grating phase) "φ" for the structured illumination intensity pattern I _θ,Ф (r) is also input. The phase values for the sub-fragments are estimated using the "sub-fragment phase matching" algorithm (also called Algorithm 2B). The "finite distortion calibration using quadratic approximation and look-up table" algorithm (also called Algorithm 1) is used to compensate for errors in estimating the phase parameter for the off-center sub-fragments due to optical perception distortions. The phase difference between the off-center sub-fragments and the central sub-fragment by an angle is stored in a look-up table. An example of pseudocode for the "sub-fragment phase matching" algorithm is given above. We proceed to processing using the described input data.

Processing step 1. Image pre-processing (high-pass filter)

[00231] The processing for sub-patch reconstruction can be started with image pre-processing. The pseudo-code that essentially corresponds to this description follows the explanation. The reconstruction of the enhanced image is performed in Fourier space. But before applying the Fourier transform to obtain the image data from the spatial domain to the frequency domain, we can apply a pre-processing step, as described below, to the image patches. The descriptions below are based on three images with different phase values along one angle. Two or more orientations will be used, as described above.

[00232] The acquired images can be pre-processed to remove noise. Noise has many sources. For example, exposure to intense excitation light can lead to photobleaching, which reduces the emission intensity of the fluorophores. An example of noise in the acquired SIM images is shown in Fig. 6 of Lal et al. 2015. The upper left image in Fig. 6, labeled (a), shows the raw SIM image with background fluorescence. This results in large errors in the frequency content of the raw SIM images in a small neighborhood around the zero frequency or DC frequency. A high-pass filter, such as a Gaussian high-pass filter or a Butterworth high-pass filter, can be used to pre-process the raw SIM images. The second image in Fig. 6, labeled (b), shows the raw SIM image after background removal. As can be seen, the Fourier spectrum of both images (a) and (b) is the same. For a detailed description of high-pass filters, see Chapter 4 of Gonzales and Woods 2008, Digital Image Processing, ^3rd edition. Chapter 4 of Gonzales and Woods provides an illustration in Fig. 4.52, which includes plots, images, and cross-sections of ideal Butterworth and Gaussian high-pass filters.

Processing Step 2: Creating a Large Fourier Image Buffer

[00233] It is possible to reserve and initialize a buffer in the memory to obtain an enhanced Fourier image. The size of the memory buffer depends on the frequency enhancement factor, the degree of enhancement. According to the SIM image reconstruction method, it is theoretically possible to create an enhanced resolution along each angle or orientation of three images, which is 2 times the resolution of the obtained images along the orientation. For two or three sets of three images, we can reserve a buffer to store a high-resolution image with an enhancement factor of 2.

Processing step 3. Cyclic execution of steps 3-a - 3-d for each angle θ

[00234] The processing steps 3-a to 3-d are performed cyclically. In one iteration, the processing steps are performed for three images of the sub-fragment obtained at the corner. The details of the cyclic processing steps are described below.

Stage 3-a (Fast Fourier Transform)

[00235] A discrete fast Fourier transform (FFT) is applied to a set of images acquired at the same angle or orientation 9. As described above, we obtain at least three images with different phases for each orientation. Each image is transformed from the spatial domain to the frequency domain. For an N × N image, the order of complexity of computing the two-dimensional Fourier transform (DFT) is O(N ⁴ ), which can be reduced to O(N ³ ) by using common expressions from row to row and from column to column. An additional speed increase can be achieved if N has a power of 2. The fast Fourier transform (FFT) algorithm can achieve order of complexity O(N ² log ₂ N) for an N × N image when N has a power of 2. An example of using common expressions in rows and columns is the use of "butterfly calculations". For more information, see Heckbert 1995, Fourier Transforms and the Fast Fourier Transform (FFT) Algorithm at http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/2091/pub/www/notes/fourier/fourier.pdf. Thus, by choosing a subfragment size of, say, 512 × 512 pixels, higher frequency resolution is obtained at the expense of increased computation time. Subfragments of 128, 256, 512, 1024, 2048, 4096, or a range from 128 to 4096 pixels can be used.

[00236] The image data in the spatial domain are represented as real numbers. When the Fourier transform is performed, the obtained data in the frequency domain are represented as complex numbers. A complex number z can be represented as z=x+iy, where x and y are its real and imaginary parts. The conjugate of said complex number z is represented as z*=x-iy.

[00237] Two examples of a Fourier domain matrix with an angular shift are shown in FIG. 19, depicting matrices with even (1910) and odd (1920) N values to illustrate symmetry in Fourier space. These frequency domain images are represented by N x N matrices. The top row and left column of both matrices 1910 and 1920 are called the DC row and DC column, corresponding to the components of the zero frequency or DC frequency. In contrast to these matrices with angular shifted values, the frequency domain graphs typically show the zero frequency or DC frequency at the center of the circle.

[00238] The matrices in FIG. 19 demonstrate symmetries in the Fourier domain or frequency domain. The DC row and DC column in matrices 1910 and 1920 have a conjugate symmetry from left to right and from top to bottom. The matrix 1910 with an even number of rows and columns has a middle column that also has a conjugate symmetry from top to bottom. In addition to the top row and left column (1910, 1920) and the middle column (1910) in the even matrix, the remaining portions of each matrix have a 180 degree conjugate symmetry. These portions of the two matrices 1910 and 1920 are shown in bold lines.

[00239] Splitting the full matrix in the frequency domain into two halves after setting the row and column(s) yields a right half which, after rotating by 180 degrees, is a conjugate of the left half. FIG. 19 illustrates the (1911, 1912) and (1921, 1922) half matrices for even and odd matrices (1910, 1920). The values for the right half can be considered from the data for the left half. Taking this into account, the left half-matrix represents the so-called non-redundant data, and the right half-matrix represents the redundant data. The number of columns in the left half (slightly more than half) is ((N/2)+1), and the columns in the right half are (N-(N/2)-1). The described technique allows tracking symmetries to reduce the computational load.

[00240] When reconstructing SIM images, we operate on the unreserved left half of the full image matrices in Fourier space. This improves efficiency since the number of required computations is significantly reduced. In the following steps of SPM reconstruction, the authors describe the use of symmetries in Fourier space in more detail.

Stage 3-b (splitting lanes)

[00241] Each captured image contains shifted and unshifted components because the source positions are randomly distributed relative to the peaks of the illumination fringes. In this step, the shifted and unshifted components are separated in the frequency domain representation of the acquired images using a fringe separation matrix M ^-1 . The fringe separation matrix is also called the inverse mixing matrix (Wicker et. al. 2013) or the reconstruction matrix (Lal et al. 2015). Gustafsson et. al. 2000 in the paper “Doubling the lateral resolution of wide-field fluorescence microscopy using structured illumination” illustrate the fringe separation process in Fig. 5 of the paper. The figure shows the individual frequency components from the observed images. The fringe separation can be understood from equation (31) below, taken from Lal et al. 2015. The equation shows that the three components separated from the three observed images on the right side of the equation. The three images are obtained using the 0 angle of the illumination pattern.

100242] The first component extracted from equation (31) is the so-called DC component This is the component with spatial frequencies without offset. When separated using precise phase and modulation parameters, this component does not include frequencies above the Abbe diffraction limit. The second and third components contains high-frequency content shifted to the region of the Abbe diffraction limit bands.

[00243] In the strip separation step, the described SIM image reconstruction technology is used, as shown in FIGS. 20-22. Assume that the full matrix of each sub-patch image in the Fourier space has 512 rows and 512 columns, as shown in FIG. 20. We divided the matrix positions in the three frequency domain matrices corresponding to the three images obtained in three phases for one angle into non-redundant and redundant halves, as described above. Each non-redundant half-matrix has 512 rows and 257 columns, including two columns and one row of the DC components, and is dominated by non-redundant coefficients that have the response coefficients of the conjugate in the redundant half-matrix. We modified each non-redundant half-matrix into a vector and combined the three vectors for the three half-matrices into a matrix having 3 rows and 131584 columns, as shown in FIG. 21. We then multiplied the inverse of the band-separation matrix M ^-1 with the combined matrix to obtain the final component of the split matrix of the same size, also with 3 rows and 131584 columns. The split matrix corresponds to the left-hand side of equation (31). We then modified the three rows of the matrix to obtain the non-redundant left halves of the matrices corresponding to the three frequency components. The three non-redundant matrices are shown in FIG. 22 (and FIG. 18) as 1831, 1833, and 1835. In our case, the separated components are in the form of non-redundant half-matrices.

[00244] As shown in FIG. 18, after the processing step 3-b, the complete matrix for the so-called DC frequency component can be represented as a square matrix that combines the non-redundant half-matrix 1831 (left half) and the redundant half-matrix 1832 (right half), or it can be a rectangular matrix. The DC frequency matrix is not subject to the same bias as the high-frequency matrices, so the DC halves can be assembled earlier or later without additional computation. For the two high-frequency components, symmetry is handled differently, reducing the computational load.

[00245] After the processing step 3-b, the full matrix of high-frequency components is configured to combine parts from two bands, such as the non-redundant half-matrix (1) 1843 from the first high-frequency image and the redundant half-matrix (2') 1844 from the second high-frequency image. Similarly, it is possible to combine the non-redundant half-matrix (2) 1845 from the second high-frequency component and the redundant half-matrix (1') 1846 from the first high-frequency component.

[00246] Although FIG. 18 shows three full matrices for the DC and high frequency components, the operations in steps 3b, 3c and 3d may be processed using symmetry, processing the frequency component bands using half of the data.

[00247] A similar method can be applied to five images with 1D structured illumination with phase stages along an angle divided into frequency components, or to a larger number of images along an angle. A function other than sinusoidally or cosinely varying illumination can be used. The described technology can also be applied to sets of images created using two-dimensional structured illumination.

Step 3 (Application of the modulation transfer function)

[00248] Two variants of applying the modulation transfer function (MTF) and/or optical transfer function (OTF) are described below: a two-stage application of an independent MTF followed by Wiener filtering for noise reduction, and a one-stage application of the MTF embedded in the Wiener filter. Both of these variants can take advantage of the described non-redundant computations.

[00249] In this two-stage implementation, we multiply (MTF), which is the amplitude or absolute value of the optical transfer function (OTF), with each of the three components obtained after stage 3-b.

It should be noted that steps 3-b and 3-e are combined as shown in FIG. 18, since the application of MTF does not affect the symmetry in the matrices.

[00250] The modulation function of a lens is a measurement of its ability to transfer contrast at a given resolution from an object to an image. In other words, MTF is a way of including resolution and contrast in a single description. Resolution is the ability of an imaging system to resolve detail in an object. It can be expressed as the number of line pairs per millimeter (lp/mm), where a line pair is a sequence of one black and one white line. This method of measuring line pairs per millimeter is also called frequency. The reciprocal of frequency gives the spacing in millimeters between two swept lines. Columnar targets with a series of equally spaced, alternating white and black columns (also known as a USAF 1951 target or Ronchi grid) are often used to test system performance. As the spacing between lines decreases (i.e., as the frequency on the target increases), it becomes more difficult for the lens to effectively transfer this reduction in contrast; as a result, the MTF decreases.

[00251] For more information on the modulation transfer function, see the paper "Introduction to Modulation Transfer Function" at https://www.edmundoptics.com/resources/application-notes/optics/introduction-to-modulation-transfer-function/. Figure 5 in the above paper shows that as the spatial frequency of the lines increases, the image contrast decreases. For an image to be sharp, black must be truly black and white must be truly white, with a minimum amount of gray. Figure 6 in this paper shows the MTF of an aberration-free image with a rectangular lens. The MTF decreases as the spatial resolution increases. It is important to note that the available systems are not completely aberration-free.

[00252] This step of two-step application of MTF and Wiener filtering can be replaced by a single step, which is presented in step 15 of SIM-RA (Algorithm 1) of Lal et al. 2015. Again, step 3-e is the first part of the algorithm, performing two steps of Wiener filtering. Alternatively, this step can be performed after step 3-d without any impact on the quality of the reconstructed SIM image.

[00253] This two-step approach (i.e., applying the MTF separately from the Wiener filtering) has been recommended in the SIM image reconstruction literature. For example, M.G.L. Gustafsson recommended this two-step approach in his paper entitled "Surpassing the lateral resolution limit by factor of two using structured illumination microscopy" published in the Journal of Microscopy, Volume 198, Part 2, May 2000, pp. 82-87. To reconstruct SIM images, this paper suggests multiplying each of the three frequency components by the microscope OTF. The components are then added together where they overlap. Finally, the sum of the components is divided by the sum of the squares of the OTF and a small constant. The paper further states that this process is equivalent to one-step Wiener filtering.

[00254] The Wiener filtering of the three components in Equations 32, 33 and 34 is shown below (adapted from Lal et al. 2015):

[00255] Three frequency components obtained on the left-hand side of equations 32, 33 and 34 are referred to in Lal et al. 2015 as the so-called unclassified estimates of noise estimates by applying Wiener filtering. The term “unclassified” is likely borrowed from the unclassified meshes of finite element analysis. Algorithm 1 (SIM-RA) of Lal et al. 2015 operates under the assumption that the raw SIM images are corrupted by white noise. The expression in square brackets in equations 32, 33, and 34 is called the Wiener filter, and m is the modulation coefficient.

[00256] Calculated noise values Gonzales and Woods in their book "Digital Image Processing)) in section 5.7 on page 351 call distorted images. Gonzales and Woods refer to the OTF quality degradation function In an ideal noise-free system, a simple "inverse filter" can be used to obtain an undistorted image. However, in most real systems, noise is present in the resulting images. Thus, an inverse filter cannot be used to reconstruct undistorted components. The Wiener filter reconstructs a given image in such a way that the mean square error between the corrupted image and the classified image is minimized. Thus, the Wiener filter is also called the "minimum mean square error" filter. For more information about the Wiener filter, see Section 5.8 on page 352 of the book "Digital Image Processing" by Gonzales and Woods.

[00257] Wiener filtering or Gustafsson's two-stage method can recover components close to the undistorted ones, which are called unclassified calculations in the left-hand sides of equations 32, 33, and 34 of Lal et al. 2015, as shown above.

[00258] Another important observation from equations 32, 33, and 34 of Lal et al. 2015 is that the numerators of equations 32, 33, and 34 are OTF conjugates If the point distribution function (PSF) H(r) of an optical system is real and symmetric, then it is implied that the OTF is real. For real values, OTF is equal to its conjugate, i.e. Furthermore, if the OTF is positive, then the MTF (absolute value of the OTF) is equal to the OTF, i.e. Based on our previous observation, this further implies that This observation shows that MTF can be used as the numerator in Equation 32, 33, 34 (when OTF is real and positive), which means that the 3rd processing step is equivalent to multiplying the numerator of Equations 32, 33, and 34 by the three distorted components respectively in three equations 32, 33 and 34.

[00259] In the algorithm for restoring the SIM sub-fragment without redundancy, we multiply the non-redundant matrices 1831, 1833, and 1835 obtained after the strip division step (step 3-b). This is a scalar multiplication by elements and does not change the sizes of the obtained non-redundant matrices. After this multiplication, we obtain three non-redundant half-matrices 1831, 1833, and 1835, which are shown in FIG. 18. After applying the OTF, the symmetries in the three full matrices remain the same as described in the strip division step.

Averaging the first and middle columns

[00260] We now propose a computational function that can improve the quality of reconstructed images using a non-redundant SIM image reconstruction algorithm. FIG. 23B shows the Fourier matrices of two high-frequency images. The non-redundant algorithm uses the left halves of the two matrices, labeled 1843 (left half-block 1) and 1845 (left half-block 2). These two matrices are shown in the top row of FIG. 23B along with the separated right half-matrices 1846 (right block 1') and 1844 (right half-block 2'). The labeling of the left and right halves of the two matrices corresponds to FIG. 18 (fourth row from the top). The non-redundant algorithm uses the left halves of the two high-frequency matrices during image reconstruction. Due to the Fourier symmetry, the symmetries of the right half-matrices can be reconstructed by searching for analogues of the left half-matrices. In the second row of matrices in FIG. 23B shows the connection of the left half-matrix of one high-frequency image with the right half-matrix of a second high-frequency image during image reconstruction. These connected matrices are also shown in ^{the 3rd} and ^4th rows of FIG. 18.

[00261] The left half-matrices of the two high-frequency images in this embodiment have two more columns than their corresponding right half-matrices. These two columns are the first and middle columns, which are shown with a shaded pattern in FIG. 23B. The first and middle columns 1843 and 1845 represent DC components that should match when using ideal instruments and measurements. Since the instruments used to capture the images (at different phases) are imperfect and noise may be present in the process, these two high-frequency matrices may have different values in the first and middle columns. The reconstruction can be improved by forcing the values of the first and middle columns of matrices 1843 and 1845 to match.

[00262] Averaging the first and middle columns of the non-redundant components of the matrices for the two high-frequency components is one of the methods for aligning the values of these columns in the matrices 1843 and 1845 to improve the quality of the reconstructed image by introducing symmetry in the values of these columns. FIG. 23C shows averaging the first and middle columns from the left half-matrices of the two high-frequency images in the Fourier space. The averaged values in the first and middle columns of the two matrices make the resulting matrices satisfy the symmetry of the Fourier space. The first and last columns after averaging are shown shaded in FIG. 23C.

[00263] Other techniques may also be used to align the values of these columns in matrices 1843 and 1845. For example, we may use the values in the first and middle columns of either the first or second high frequency image and use these values in both matrices 1843 and 1845. It is also possible to use a weighted combination of the values of the first and middle columns of two high frequency images. For example, by combining the values by assigning two-thirds of the weight to the values of one high frequency image and one-third of the weight to the values of the other high frequency image. It should be understood that different relative weights may be assigned for combining the values of the corresponding columns.

Stage 3-d (subpixel shift)

[00264] At this stage, each of the high frequency components are shifted to overcome the effects of discrete fast Fourier transform quantization. Since the pixel positions in the Fourier space matrix correspond to discrete frequency blocks, we denote this as subpixel shifting. To explain this, Karras et. al. 2019, in their paper “Successful optimization of reconstruction parameters in structured illumination microscopy - A practical guide” published in Optics Communications, demonstrate a shift of higher frequency components into the OTF bandwidth when using structured illumination (see Fig. 1 in Karras et al. 2019). During subpixel shifting, the true positions of the high-frequency components are centered at frequencies p _θ and p _{θ ,} respectively, in the frequency domain. This shift is illustrated in Equations 35 and 36 taken from Lal et. al. 2015.

[00265] In prior art image restoration algorithms, such as the SIM-RA algorithm 1 presented in Lal et al. 2015, two Fourier transform operations are performed on a single high-frequency image to shift the high-frequency content to the desired position, as shown in Equations 35 and 36. The shift property of the Fourier transform allows us to shift the signal. The shift can be done by transferring information from the frequency domain to the spatial domain and then applying a multiplication (also called applying a shift vector). The Fourier transform is then applied to this shifted signal to return it back to the frequency domain. More information about the shift property of the Fourier transform can be found at http://www.thefouriertransform.com/transform/properties.php. The shift of the high-frequency content is actually performed in real space by performing the inverse Fourier transform on the unclassified images so as not to restrict the shift vector to whole pixels in discrete Fourier transforms. In this way, we can achieve pixel displacement in real space. Then the displaced high-frequency components are transformed into the frequency domain by Fourier transform.

[00266] We improve the efficiency of SIM image reconstruction and reduce the required computational resources by using symmetry in the Fourier space. Thus, the so-called DC frequency component does not have to be shifted in the form of a subpixel, as shown in FIG. 18. The high-frequency component matrices are shifted relative to the DC frequency component matrix.

[00267] In the description of the redundancy-free algorithm, we shift only one matrix of high-frequency components after combining the non-redundant halves of the response matrices with shifted phases of the high-frequency components shown in FIG. 18. The combined halves of the high-frequency components are shown as the left half 1833 and the right half 1834, or the matrix with the label 1|2' shown in the middle column of the third row of FIG. 18. This matrix is shifted from the Fourier domain to the spatial domain and back, instead of two matrices in the prior art SIM image reconstruction algorithms. The shifted matrix 1|2' is shown in the middle column of the fourth row of FIG. 18 by two arrows pointing diagonally upward toward the upper left corners. This matrix consists of the left half of the matrix 1843 and the right half of the matrix 1844. The second high-frequency matrix, which consists of the left half of the matrix 1845 and 1846 or denoted as 2|1', is obtained using the symmetry of the Fourier space and the conjugation of the matrix 1|2'. We shift it the same distance as the matrix 1|2', but in the opposite direction, which is denoted by the two diagonal arrows towards the lower right corner.

[00268] The processing steps 3a-3d are repeated cyclically along the angles of the illumination peak. In this example, we use two angles or orientations, so we perform the above steps for a second set of three images obtained with a second orientation of the structured illumination pattern.

Processing step 4. Reconstruction sum and Wiener filter creation

[00269] The Wiener filter can be created once and then applied repeatedly to the filter images even if the parameters change. The supplementary material for Karras et al. 2019, “Successful optimization of reconstruction parameters in structured illumination microscopy A practical guide,” published in Optics Communications, illustrates the adjustment of the Wiener filter parameter (w), which is the constant part of the denominator in Equations 32, 33, and 34. Fig. S1 of this supplementary material for Karras et al. 2019 illustrates the effect of different values of the Wiener filter parameter (w) on the extracted image spectra. A low value of the Wiener filter parameter may result in noise in the extracted spectra. Too high a value may result in loss of image data. The value of the Wiener filter parameter can be determined empirically, as described in Lal et al. 2015 and the supplementary material of Karras et al. 2019. In an alternative embodiment, other types of Fourier low-pass filters can be used to reduce noise. For example, the Gaussian low-pass filter, the Butterworth low-pass filter, or the Hamming low-pass filter. Detailed information about these filters is available at http://fourier.eng.hmc.edu/e101/lectures/Fourier_Analysis/node10.html.

[00270] Only the portions of the matrices that include high-frequency components overlap with the so-called DC frequency matrix. The expanded matrix, in which the coefficients from a plurality of Fourier domain frequency matrices are combined, is wider and larger than the individual matrices before combining using a gain factor, which reflects the gain of the unfolded image. The frequency enhancement factor can be from 1.0 to 2.0 times the width and height of the individual matrices. If the frequency enhancement factor is 1.0, there is no improvement in image resolution. The high-frequency components and images at all phases along the orientation do not overlap each other. When the frequency enhancement factor is 2.0, the offset matrices of the high-frequency components are separated from each other in such a way that their edges (or boundaries) touch the center of the matrix without offset. This means that the width and height of the reconstructed high-resolution image are increased by a factor of two compared to the individual image matrices. This combined matrix is shown as a large matrix 1850 in FIG. 18. The matrix 1850 is twice as wide and tall as the individual matrices labeled 1|2' and 2|1' to indicate that the reconstructed image has been enhanced. When the frequency enhancement factor for improving the image resolution is greater than 1.0 and less than 2.0, the offset matrices 1|2' and 2|1' are located closer to and overlap each other in the center of the large matrix 1850.

[00271] FIG. 18 shows the construction of a matrix 1850 from matrices for three images in a first corner. The non-redundant half of the DC component for the first corner is shown as 1841 in FIG. 18. We then place the subpixel offset coefficients for one high-frequency image in the upper left portion of the matrix 1850, and for the other high-frequency image in the lower right portion. The redundant components of the conjugate needed to complete 1850 can be considered in complementary non-redundant components, maintaining tracking of the positions of the matrix of complementary components (coefficients) and applying a changed sign to any imaginary part of the copied coefficients.

[00272] By adding the second, perpendicular corner and three additional images (not shown), the square is filled in. We place the full matrix for one high-frequency component for the second corner in the lower left of the 1850 matrix, and the second high-frequency component in the upper right.

[00273] Combining nine images is illustrated in Lal et al. 2015. The coefficients in the positions of the overlapping matrices are combined into a matrix of 1850. This is a change from six images for two angles instead of three.

[00274] The visualization of the matrices being combined is clearest when the coefficients are depicted with an offset from the center, where the average value of the frequency range is at the center of the matrix, but it is possible to actually perform calculations using matrices with an angular offset with the smallest frequency component, such as in the upper left or lower right corner. A reconstruction of the image matrix 1850 with improved resolution is illustrated in FIG. 18 by placing the low-frequency components in the center and the high-frequency components outside the center. This arrangement of the frequency content is called an off-center spectrum. Applying a fast Fourier transform (FFT) to the image data in the spatial domain results in a spectrum offset to the corner at which the amplitudes of the low-frequency components end at the corners of the two-dimensional spectrum in Fourier space (or frequency domain). The high-frequency components are located closer to the center of the two-dimensional spectrum. For ease of understanding, it is useful to visualize the frequency content in the frequency domain as an off-center spectrum, in which the low-frequency components are close to the center and the high-frequency components are off-center. The symmetry of the quadrants in the corner-offset spectrum allows the quadrants to be swapped diagonally to move the low-frequency components closer to the center. Thus, a corner-offset spectrum can be directly transformed into an off-center spectrum and vice versa. Conjugate symmetry and redundant regions exist in both the corner-offset and off-center spectrum. The off-center spectrum in FIG. 18 is presented to facilitate the reader's understanding.

[00275] The Wiener filter can be constructed for the combined image using one stage (Lal et al. 2015) or two stages (Gustafsson 2000) presented above for the band-split images in stage 3. For example, in one embodiment, the Wiener filter can be constructed using the MTF (modulation transfer function), which is the amplitude or absolute value of the OTF (optical transfer function). In this embodiment, we perform a reconstruction process to calculate the overlapping frequency data in the Wiener filter. We apply the weighting coefficients of the Wiener filter for the band-split images to the OTF of the combined image to construct a Wiener filter for the combined image. The Wiener filter is applied to the combined image in the next stage.

Processing step 5. Applying the Wiener filter and the apodization filter

[00276] In this step, we apply the Wiener filter or the alternative filter created in the previous step to the registered image in Fourier space. In step 3, we split the Wiener filtering into two steps by applying an independent MTF. See equations 32, 33, and 34 in step 3. As a two-step filtering process, this step involves applying the remaining portions of the Wiener filter (presented in equations 32, 33, and 34) to the registered image in Fourier space. In an alternative embodiment, we may apply the Wiener filter to the Fourier images of the three frequency components at an angle before combining them to form the registered image. When applied in the early steps, we may apply the Wiener filter only to the non-redundant halves of the band-split frequency matrices.

[00277] The use of a Wiener filter enhances the signal, but it can also increase noise in images. A pre-designed apodization filter helps to eliminate sharp edges in images that arise due to high frequency. The apodization filter is a low-pass filter that removes frequency content outside a pre-specified radius in Fourier space. The apodization filter reduces the errors introduced by the Wiener filter. In addition, other low-pass filters can be applied to reduce errors due to the Wiener filter, such as a low-pass filter in real Hamming space or a Gaussian filter. It should be noted that the apodization filter is optional and is not used by some SIM reconstruction algorithms, such as Algorithm 1 (SIM-RA) of Lal et al. 2015. In an alternative embodiment, the apodization filter can be applied only to the non-redundant halves of the matrix for the three frequency components earlier in the chain.

Processing step 6. Performing inverse FFT

[00278] In this step, we apply the inverse Fourier transform to the 1850 matrix for the full frequency domain image. Applying the inverse fast Fourier transform (IFFT) transforms the matrix components from the Fourier domain into a spatial domain (real space) image. The resulting 1860 matrix is a high-resolution reconstructed SIM image. When using sub-fragments, it is part of a larger fragment.

SIM image reconstruction for full FQV image

[00279] The high-resolution SIM image sub-tiles may be combined to reconstruct a high-resolution image of the full FOV. The pseudocode for one embodiment of the described technique for reconstructing a high-resolution full field of view from a plurality of sub-tile images is presented below. The algorithm presented below is called the field of view reconstruction algorithm (Algorithm 4). It extends Algorithm 3 for sub-tile reconstruction. The input data includes the obtained images and distortion coefficients for the estimation parameters. The estimation of the distortion coefficients for the angle and spacing parameters (input data 2) and the phase shift table (input data 3) is presented above. The algorithm divides the full FOV images into sub-tiles of equal-sized regions. The SIM images are reconstructed for the sub-tiles presented above in Algorithm 3. Finally, the high-resolution SIM images of the sub-tiles are combined to reconstruct a high-resolution SIM image of the full FOV (step 4b of the algorithm).

Algorithm 4. Reconstruction of the full field of view

Input data

1. A set of images, for example for 2 angles and 3 phases per angle.

2. Distortion coefficients for angle/spacing (see the algorithm "Distortion calibration using quadratic approximation and look-up table").

3. Distortion lookup table for phase (see the algorithm “Distortion calibration using quadratic approximation and lookup table”).

4. Wiener filter and apodization parameters. Output data Reconstructed image.

Algorithm

1. Create the following images/filters (size corresponds to the sub-area).

a.OTF

b. Wiener

c. Apodization

2. Dividing the image into sub-regions of equal size, for example 512 × 512.

a. If the image cannot be divided evenly by the size of the subregion, on the right edge or at the bottom, move the subregion to the left (or up) so that it is in the image.

b. Set up the ledger for the boundary subareas so that only the non-overlapping portions for the right or bottom edge subarea are copied.

3. Calculated parameters in the center of the image used, for example 512 × 512 windows (see subfragment and spacing matching algorithm and subfragment phase matching algorithm).

a. Calculated angle and peak illumination spacing for each angle.

b. Calculated modulation for each angle.

c. Calculated phase for each image.

4. For each sub-region

a. Recovering a higher resolution sub-region using a sub-fragment recovery algorithm

i. Using the grating and angle obtained in step 3 and the given distortion coefficients to estimate the local grating and angle.

ii. Using the phase obtained in step 3 and the given distortion table to calculate the correct phase.

iii. Given the previously constructed OTF, applying the Wiener filter and apodization to the subdomain.

b. Copy the output from (a) to a large image.

i. In case of a boundary sub-area, only the missing data will be copied.

[00280] In the context of the previous descriptions, this pseudocode is self-explanatory.

Potential improvements to the SIM image reconstruction algorithm

[00281] We provide potential improvements to the SIM image reconstruction (SIR) algorithm described in U.S. Provisional Patent Application Ser. No. 62/924,130, entitled SYSTEMS AND METHODS FOR STRUCTURED ILLUMINATION MICROSCOPY (Accession Number ILLM 1012-1); and U.S. Provisional Patent Application Ser. No. 62/924,138, entitled INCREASED COMPUTATION EFFICIENCY FOR STRUCTURED ILLUMINATION MICROSCOPY (Accession Number ILLM 1022-1). Both provisional applications were filed on October 21, 2019.

[00282] The first possible improvement relates to the phase deviation lookup table. Based on the SIR algorithm, a phase deviation lookup table is used in the original mode. The assumption in the original model is that the phase deviation and distortion can be obtained once and stored in a lookup table. It is assumed that the values in the lookup table are stable and applicable to multiple sequencing runs. The original SIR algorithm assumes that the phase deviation lookup table (LUT) remains stable in the spatial domain. This means that the same lookup table can be applied to fragments of the flow cell. The second assumption in the original algorithm is that the LUT remains stable over multiple sequencing runs. Experiments have shown that these assumptions may not be true for all flow cells and sequencing runs. Fragments in the flow cell may have differences or other distortions. The values in the lookup table may be affected by various environmental factors that change over multiple sequencing runs.

[00283] Thus, the universal SIM algorithm includes the generation of a phase offset lookup table for each sequencing run per fragment when a new flow cell is loaded into the sequencer and sequencing is started. This may increase the processing time required for the first sequencing run. However, there is a significant improvement in the error rate and an increase in the percentage of clusters that pass filtering (or %PF), as shown in the following paragraphs.

[00284] The second potential improvement relates to geometric distortion, which causes parallel lines to bend or otherwise become distorted. Geometric distortion can cause distortions in the imaging or optical path and in the flow cell. In some cases, all or some fragments in the flow cell may have tilted features and not be flat. This can lead to geometric distortion. Characterizing all fragments in the flow cell is preferable because there may be fragment-to-fragment variations in the flow cell. An early version of the correction assumes that the geometric distortion can be obtained once and used for multiple sequencing runs. Experiments have shown that this assumption may not be true for all sequencing runs. Thus, the universal algorithm involves obtaining the geometric distortion for each fragment in the flow cell in the first cycle of the sequencing run.

[00285] A third potential improvement relates to combining sub-tile images to obtain a full field of view (FOV) image. In the original SIM algorithm, when the inverse fast Fourier transform (IFFT) is applied to the sub-tiles in the frequency domain, the edges of the sub-tiles appear in the spatial domain. Thus, the combined sub-tiles in the full field of view show merging lines at the edges of the sub-tiles. The merging problem causes dark artifacts to appear along the boundary of the sub-tiles. The universal SIM algorithm solves the merging problem by overlapping the sub-tiles. For example, the overlap of the sub-tiles may be from 5 to 15 pixels. In one embodiment, the sub-tiles overlap by 8 pixels along the boundary. When reconstructing the full field of view (FOV) image, the universal algorithm may ignore the outer 8 pixels for the sub-tiles and use the inner region to reconstruct the full FOV image in the spatial domain. In this embodiment, the sub-fragment size can be made larger than 512x512 pixels by adding border pixels (e.g., 8 pixels) on each side. In another embodiment, the border pixels of the sub-fragments can be averaged instead of removing the border pixels.

[00286] The fourth potential improvement is related to the optimization of the Wiener and apodization filters. The filter parameters are dependent on the upsampling. The original SIM algorithm uses a 1.2x upsampling, which can clip a partial pixel. When combining images of sub-tiles in the spatial domain, this can cause image shifting and alignment problems. The pixel clipping is associated with an increase in the upsampling factor (i.e., 1, 2), which provides a non-integer number of pixels. The universal SIM algorithm uses a 1.5x upsampling to overcome the problem of partial pixel clipping by using an upsampling factor that generates an integer number of pixels. This provides clearer sub-tile images for combining. Other implementations may use other upsampling factors, such as a 1.25x upsampling, which can also generate an integer number of pixels.

[00287] In some embodiments, the disclosed technology may use a dose ramp to compensate for nucleotide damage caused by exposure cycles in a sequencing run. With a dose ramp, the optical power of the blue and green channel lasers increases.

Performance comparison results

[00288] We compared the performance of the original SIM algorithm with the universal SIM algorithm. The two algorithms are compared in terms of the quality of the sequencing results, as indicated by the error rate and the percentage of clusters passing the filter (%PF). The error rate measures the percentage of base calls that were in error compared to a known reference genome during the sequencing process. The clusters passing the filter, or %PF, indicates the quality of the signal from each cluster of molecules in the flow cell. Low error rates and higher %PF values are desirable because they indicate good sequencing output with high quality base calls. From over hundreds of fragments on the flow cell, we selected the results for the best fragment in each sequencing run to compare the algorithms. Fragments in the flow cell can have differences in flatness, tilt, surface smoothness, and other environmental factors. The reason for selecting the best fragment for comparison from more than 100 fragments is to reduce the differences caused by other external factors through statistical aggregation.

[00289] FIG. 25A shows the error rate for the baseline and all algorithms for three sequencing runs labeled A, B, and C. For each sequencing run, the error rate for the best fragment is shown for two algorithms. The top column indicates the error rate for the best fragment in the overall algorithm, and the bottom column indicates the error rate for the best fragment in the baseline algorithm for the corresponding sequencing run. The table at the bottom of the graph shows the quantitative improvements in the error rate of the best fragment when a universal algorithm is used. For example, the error rate of the best fragment for sequencing run A is 1.85% when the baseline algorithm is used. The error rate of the best fragment for sequencing run A is 1.18% when using all algorithms, indicating higher quality sequencing with a lower error rate. Similar improvements in the form of reduced error rates are observed in sequencing runs B and C when the universal algorithm is used.

[00290] FIG. 25B shows the clusters passing the filter (or %PF) for obtaining the best fragment in three sequencing runs for two algorithms. The lower table shows the %PF values for the best fragment in the corresponding sequencing for the two algorithms. The universal algorithm results in higher %PF values for all sequencing runs. In particular, the %PF value for the first cycle A of the sequencing run is 82.7, which indicates that 82.7% of the clusters on the flow cell fragment are considered high-quality clusters for generating base calling information. The %PF value for the best fragment in the sequencing run is only 70.1. The higher the %PF value, the more clusters pass the filter, thereby increasing the sequencing throughput. Patterned flow cells may have a theoretical maximum limit on the number of clusters equal to the number of nanowells in which clusters of molecules can form. The results in FIGS. 25A and 25B show that the results of the generalized algorithm result in a lower error rate and higher throughput (due to higher %PF values) compared to the original algorithm.

[00291] FIGS. 26A, 26B and 26C show the best fragment error rate for the cycles in sequencing runs A, B and C, respectively. A separate time series of the sequencing run error rates is shown for the original and universal algorithms. The results in FIGS. 26A-26C illustrate that the universal algorithm reduces the error rate in the sequencing runs. The reduction in error rate using the universal algorithm for the best fragment can vary from cycle to cycle. As shown in FIG. 26A, for the initial 25 cycles, the error rates of the original and universal algorithms are almost the same, however, as the number of sequencing runs increases, the error rate for the universal algorithm decreases compared to the error rate for the original algorithm. FIG. 26B shows that the error rate for the universal algorithm is slightly lower in the initial cycles. In later sequencing run cycles, the general-purpose algorithm performs better than the original algorithm. FIG. 26C plots the best fragment error rates for the cycles in sequencing run C for the two algorithms. The general-purpose algorithm performs significantly better than the original algorithm for most runs. The graphs also illustrate specific runs for which the general-purpose algorithm has a lower error rate. These results may help analysts or researchers focus on quality issues, resulting in lower error rates for specific runs.

[00292] FIG. 26D shows the improvements in error rate achieved by specific updates to the general algorithm compared to the original algorithm. For example, the left graph shows the improvements in error rate for sequencing run A by sequentially incorporating phase learning and local problem fixing techniques. Phase learning refers to a phase offset estimate that can capture the relationship between the phase offset values for non-center sub-tiles relative to the center sub-tiles. A separate phase offset lookup table is generated and stored for each flow cell tile. The error rate for the original algorithm for sequencing run A is 1.85, as shown in the first column of FIG. 24A. The error rate reduction when incorporating phase learning into the baseline algorithm is 0.65 or 32%, while when incorporating local fixing into the baseline model, the error rate reduction in the baseline algorithm is 0.19 or 10%. The graph can also be interpreted as a 0.65 error rate reduction with phase learning and 0.19 with local fixing in the baseline algorithm. The residual error rate after incorporating the two techniques is 1.18.

[00293] The graph on the right in FIG. 26D demonstrates the reduction in error rate using the phase learning and local fix methods for sequencing run B. Incorporating phase learning into the baseline algorithm reduces the error rate by 0.227, or 33%. Incorporating local fix into the baseline algorithm reduces the error rate by 0.039, or 6%. The residual error rate after incorporating both techniques is 0.414. A discussion of the impact of the described technique when using SIM parameter calculation and image reconstruction methods to process images from a sequencing run is presented below.

The impact of the described technology

[00294] Simplified examples are provided to illustrate how the described technology can reduce the computational requirements for improving the structured illumination image in a flow cell over several cycles and several hours. Machine capabilities vary, so round numbers are chosen for ease of interpretation of the calculation examples. It should be remembered that the described technologies can be applied independently or in combination with each other. In some embodiments, parameters can be calculated more frequently to compensate for system instability due to environmental factors. For example, as described above, a universal SIM algorithm is used to calculate a phase deviation lookup table in the first cycle of each sequencing run. This can increase the duration of the first cycle by up to 30 percent. However, a significant reduction in the error rate and an improvement in %PF can be achieved.

[00295] Assume a sequencing run includes 150 analysis cycles with two illumination frequencies at a rate of 6 minutes per cycle. The basic image acquisition of 150 cycles takes 15 hours. Assume that two angles of +/- 45 degrees are used for structured illumination with three images per angle in each cycle. The number of images per fragment over 150 cycles is 1800. Assume that there are 70 sub-tiles in a 7 x 10 lattice. The processing requirement is 126,000 sub-tiles over 900 minutes.

[00296] In the first embodiment, it is assumed that the system is calibrated before the run, and that a quadratic approximation and a look-up table generated for the spacing, angle, and phase differences can be used throughout the run. In this embodiment, the sub-section closest to the center will be subjected to interference fringe peak evaluation 1800 times. This is the number of times that the position of the flow cell or the illumination source, or the projected phase of the structured illumination, changes.

[00297] In this first example, the system will not perform peak fringe estimation 69 out of 70 times. For the 124,200 sub-tile images, the other sub-tiles will use a pre-computed relationship with the closest sub-tile to the center to determine the reconstruction parameters. Estimating the reconstruction parameters is more expensive than the actual reconstruction. Thus, the computational requirements are reduced by more than half. Further reduction can be achieved by exploiting the described dependence on symmetry.

[00298] The use of symmetry allows the values of a pair of conjugates to be considered rather than calculated using FFT and other methods. Particular care must be taken with sub-fragment sections that have reflection symmetry rather than 180 degree rotation and sign reversal symmetry that dominates. When there is an even number of rows and columns in a corner-shifted Fourier transform space, the top row and the left and middle +1 columns are treated differently due to reflection symmetry rather than rotational symmetry. See FIG. 19. When there is an odd number of rows and columns in a corner-shifted Fourier transform space, the top row and the left column are treated differently due to reflection symmetry rather than rotational symmetry. See FIGS. 19 and 23A. Other symmetries may be applied to off-center Fourier transform space.

[00299] The center column of FIG. 18 illustrates how half of the frequency domain calculations can be avoided at various stages by focusing on non-redundant data. Suppose this reduces the calculated magnitude by 40 percent and the image accumulation coefficient by 50 percent during these stages before the entire image is reconstructed at 1850, 1860. The conjugate pairs do not necessarily have to be included in the calculations during stages 3b-3d, since value estimation can be applied to the conjugate pairs if values are needed at 1850, 1860.

[00300] Thus, one described technology can save 50 percent of the required computing resources, and the other can save 40 percent. In total, the estimated costs of SIM reconstruction can be reduced by more than two-thirds.

[00301] In a second embodiment, assume that the system is calibrated at the beginning of the run and again at the midpoint to update the quadratic fit and the look-up table. Assume that this calibration takes 8 minutes, since it includes a peak evaluation of the interference fringes of all 70 sub-tiles, and then the fit and creation of the look-up table. This evaluation continues from 15 hours to 15 hours 16 minutes. If, instead of using the described technology, an evaluation was performed for each sub-tile and exposure frequency, once for each trio of images, for a total of 300 parameters, the run time would be more than three times longer.

[00302] In the second embodiment, the effect of applying symmetry does not change.

[00303] The combined effect of the described technologies is expected to be to provide real-time processing of image fragments using a CPU without adding a GPU, FPGA, or CGRA to the system. In an alternative embodiment, simpler computing resources can be used even if a GPU, FPGA, or CGRA is selected. This can reduce the complexity of the system and make it easier to maintain the form factor of existing machines while increasing their real-time processing capabilities.

Specific implementation options

Sub-fragments of the full field of view captured by the sensor

[00304] The described sub-fragment technology may be described in terms of calibration and/or production using a calibrated scanner.

[00305] In one embodiment of the disclosed technology, a method is described for calibrating or characterizing a scanner that determines fluorescence of millions of samples distributed over a flow cell, or more generally, an imaging plane, in images collected over multiple cycles. Structured illumination is used to improve the resolution between fluorescent samples that are closer to each other than the Abbe diffraction limit for optical resolution. This method for calibrating a scanner to use substantially the entire field of view projected by a lens onto an optical sensor in the scanner includes capturing image fragments under structured illumination at multiple angles and phase shifts of the structured illumination. Variants of the method may use 2D structured illumination and a different phase shift pattern. The method includes calculating optical distortion from an image fragment, including measuring the distance between intensity peaks and the angle of intensity peaks in at least 9 image sub-fragments, including the sub-fragment closest to the center, and approximating the spacing surface and the angle surface to the measured spacing and angle, respectively. Of course, other configurations can be used to divide the image, for example, 3 × 3, 5 × 5, 5 × 7, 9 × 9, 8 × 11, 9 × 16 sub-fragments. Images captured by larger sensors are divided into a larger number of sub-fragments, not limited to the sub-fragment arrays listed in the list. The approximated spacing surface and the approximated angle surface express the correction of distortions in the sub-fragments of the image fragment. The obtained fit factors are stored, including the results of the approximation, which relate to the spacing of the intensity peaks and the angle of the intensity peaks in at least 9 sub-plots relative to the sub-plot closest to the center. The calibration further includes measuring the phase shifts of the structured illumination at a variety of angles and the phase shifts within the sub-plots and storing a look-up table, which expresses the differences between the phase shift in the sub-plot closest to the center and the phase shift in the other sub-plots of the image patch. This calibration prepares the scanner for production, for the analysis of fluorescence of millions of samples distributed over the flow cell or imaging plane, in images collected over several cycles.

[00306] In an embodiment, the method provides improved fluorescence images of millions of samples distributed over a flow cell or imaging plane from images collected over multiple cycles. This includes processing captured image elements on the flow cell, wherein image fragments are captured at multiple angles and structured illumination phase offsets over multiple cycles. This processing is for sub-fragments of each image fragment. It includes accessing stored approximation evaluation results or generating estimated approximation results that express the spacing between the sub-fragment closest to the center and other sub-fragments. Suitable results may be approximation coefficients or polynomial surface values in a look-up table. The method also includes accessing a stored look-up table or creating a look-up table that expresses the phase difference between the sub-fragment closest to the center and other sub-fragments. The results of the approximation and lookup table are combined with measurements for the closest-to-center fragment of each acquired image fragment, the peak-to-peak intensity spacing, the peak-to-peak intensity angle, and the structured illumination phase shift. The reconstruction parameters are determined on a sub-fragment basis for each acquired image fragment and then applied to the SIM reconstruction. This method produces images of improved resolution for sub-fragments in the plane of the flow cell or in the imaging plane over several cycles, with a degree of resolution exceeding the Abbe diffraction limit.

[00307] For example, for an image fragment divided into 9 sub-fragments, the method includes obtaining estimated reconstruction parameters for at least 8 additional image sub-fragments in addition to the central sub-fragment. Estimated spacing and angle reconstruction parameters for the additional sub-fragments are obtained by combining the available or estimated parameters for the sub-fragment closest to the center with stored approximation results that relate to the spacing parameters and angle parameters for at least 8 additional sub-fragments relative to the sub-fragment closest to the center. Estimated phase shift reconstruction parameters for the additional sub-fragments are obtained by combining the available or estimated parameters for the sub-fragment closest to the center with a stored look-up table for phase shift measurements for at least 8 additional sub-fragments relative to the sub-fragment closest to the center. As described above, other configurations can be used for dividing the image without affecting the parameter estimation process described above.

[00308] The enhanced resolution images can be used to sequence samples over multiple cycles or to study fluorescence over an image plane captured by several tens, hundreds, or even thousands of image fragments.

[00309] In a combined calibration and production implementation, the actions described above are combined, so that the scanner is calibrated and then used in production. The described technology includes the implementation of a method that improves the efficiency of the scanner, which determines the fluorescence of millions of samples distributed over a flow cell, in images collected over several cycles. Structured illumination is used to improve the resolution between fluorescent samples located closer to each other than the Abbe diffraction limit for optical resolution. Using this technology, calibrating the scanner to use substantially the entire field of view projected by the lens onto the optical sensor in the scanner includes capturing image fragments under structured illumination at a variety of angles and phase shifts of the structured illumination. It includes calculating the optical distortion from an image fragment, including measuring the distance between intensity peaks and the angle of intensity peaks in at least 9 image sub-fragments, including the sub-fragment closest to the center, and approximating the spacing surface and the angle surface to the measured spacing and angle, respectively. Of course, the image of the fragment can be divided into a greater number of sub-fragments, as described above. The approximated spacing surface and the angle surface express the correction of distortions in the sub-fragments of the image fragment. The obtained approximation coefficients are stored. The calibration additionally includes measuring the phase shifts of the structured illumination at a plurality of angles and the phase shifts within the sub-fields and storing a look-up table that expresses the differences between the phase shift in the sub-fragment closest to the center and the phase shift in other sub-fragments of the image fragment.

[00310] The method is implemented by processing captured image elements on a flow cell, wherein image fragments are captured at a plurality of angles and phase shifts of structured illumination over several cycles. It includes combining stored approximation results and a stored look-up table with measurements for each captured image fragment in the fragment closest to the center, the spacing between intensity peaks, the angle of intensity peaks and the phase shift of structured illumination, to determine reconstruction parameters for each captured image based on a sub-fragment. It further includes obtaining images with improved resolution for the sub-fragments at positions along the flow cell over several cycles at a higher resolution than the Abbe diffraction limit, and using images with improved resolution for a series of samples over several cycles.

[00311] Each of the embodiments of the combined calibration and manufacturing implementation presented above has a separate utility. For example, a first entity may deliver and calibrate a scanner for use by a second entity. The first entity may use a different calibration approach to determine and express offsets and tilt angle dependence and phase differences between the closest sub-tile to the center and other sub-tiles, and the described manufacturing technology may still be used. The described calibration technology may be used, and then the second entity may use the calibrated scanner in a different manner. Thus, the described technology includes calibration and/or manufacturing options.

[00312] This method and other embodiments of the described technology may include one or more of the following features and/or features described in connection with additional described methods. For the sake of brevity of the description, combinations of features described in this application are not individually numbered and are not repeated with each basic set of features. The reader will understand how the features specified in this section can be easily combined with sets of basic features specified in other embodiments.

[00313] When the imaging plane is a flow cell, the samples can be distributed in millions of nanowells across the flow cell. At least some of the adhesion pairs of nanowells can be located closer to each other than the Abbe diffraction limit for optical resolution. Structured illumination ensures such close arrangement. In an alternative embodiment, the samples can be randomly distributed across the flow cell.

[00314] Each fragment comprises at least 512 x 512 pixels of the optical sensor. Larger or smaller quantities may be used, including 256, 400, 1024, 2048 and 4096 or in the range of 256 to 4096 pixels. Efficiency may be improved when the power of the sub-fragment size is 2, practical ranges that have been found suitable are 512 and 1024 pixels.

[00315] The sub-fragments may overlap with at least 2 pixels of the optical sensor. More or less may be used. For example, for a window with a width of 512 pixels, an overlap of up to 256 pixels may be used, and for a width of 1024 pixels, an overlap of up to 512 pixels may be used.

[00316] The results of the approximation of the spread surface and the angle surface may be stored in the form of coefficients of a quadratic, cubic, or higher order approximation, as well as in a lookup table calculated from a polynomial approximation of the spread and angle surfaces.

[00317] In some embodiments, a sequencing run may be performed one or more times to re-measure the spacing between intensity peaks and the angle of intensity peaks in sub-tiles, re-fit the spacing surface and the angle surface, and store the re-fitting results for use in processing subsequent captured image tiles. This may be performed, for example, at an intermediate cycle, such as the 50th cycle or more within at least 100 cycles. In some embodiments, the re-measurement may be performed when computing resources are available, such as at the end of a sequencing run or between two reads when performing a chemical analysis step for the next read.

[00318] The nanowells can be arranged in a regular repeating pattern. In the case of a rectangular pattern, two angles of structured illumination can be used, substantially along two diagonals connecting opposite corners of the rectangle in the pattern, so that the peaks of the intensity of the structured illumination are oriented substantially normally to the two diagonals. In the case of a repeating hexagonal structure of nanowells with three diagonals connecting opposite corners of the hexagons in the pattern, three angles of structured illumination can be used with intensity peaks that are oriented substantially normally to the three diagonals. In an alternative embodiment, the samples can be randomly distributed on the imaging plane without nanowells. Or the samples can be arranged in a regular order on the imaging plane in a pattern other than the nanowell structure.

[00319] The method may further include combining the enhanced resolution images of the sub-fragments into an enhanced resolution image of the fragment and using the enhanced resolution image fragment for further analysis. Further analysis may include sequencing the samples at one position per cycle.

[00320] As an improvement in the approximation of surfaces, the method may further include determining an edge trim and applying the edge trim to remove pixels around the edges of the sensor from the calculations before approximating the spacing surface and the angle surface to the measured spacing and angle values. The edge may be uniform around the edges of the sensor or may vary along the edge, such as being wider at the ends of a rectangular sensor than along a long edge.

[00321] The method may further include measuring the spacing between intensity peaks and the angle of intensity peaks on at least 8×11 image sub-fragments. Other configurations may be used to divide the image fragment into sub-fragments, as described above.

[00322] The described sub-tile technology can be used to obtain an image of improved resolution on target images captured under structured illumination. In one embodiment of the described technology, a method is provided for dividing a captured target image captured by an optical sensor into 9 or more sub-tiles and processing each of the 9 or more sub-tiles independently. As described above, more sub-tiles can be used to further divide the captured image. The method includes transforming at least three target images in the Fourier domain in a corresponding sub-tile to obtain at least three frequency domain matrices of the corresponding sub-tile. The images are captured by the sensor in the spatial domain by applying at least three phase translations of the structured illumination along one angle. An inverse mixing matrix is calculated using the calculated reconstruction parameters. The inverse matrix is applied to the frequency domain matrices to obtain at least three phase-separated matrices in the Fourier domain, which are unbiased and biased matrices. Subpixels are shifted on at least one shifted matrix. The subpixel shifting includes transforming the shifted matrix from the Fourier domain to the spatial domain, applying the shift vector to the data in the spatial domain, and transforming the transferred data from the spatial domain data. Subpixel shifting results in obtaining two or more reconstructed shifted matrices in the Fourier domain. The method includes aligning and summing the overlap values of the unshifted matrix and the reconstructed shifted matrices to obtain an extended frequency coverage matrix. The extended frequency coverage matrix is inversely transformed from the Fourier domain to obtain a sub-fragment of improved resolution in the spatial domain. Additionally, the target image of improved resolution can be obtained by combining the sub-fragments of the extended resolution for 9 or more sub-fragments.

[00323] This method and other embodiments of the described technology may include one or more of the following features and/or features described in connection with additional described methods. For the sake of brevity of the description, combinations of features described in this application are not individually numbered and are not repeated with each basic set of features. The reader will understand how the features specified in this section can be easily combined with sets of basic features specified in other embodiments.

[00324] The steps of the method can be applied repeatedly to obtain a sequence of images of improved resolution. The sequence of images of improved resolution can be used for a sequence of samples visualized by the sensor over several cycles.

[00325] Each fragment comprises at least 512 x 512 pixels of the optical sensor. Larger or smaller quantities may be used, including 256, 1024, 2048 and 4096 or in the range of 256 to 4096 pixels. Efficiency may be improved when the power of the sub-fragment size is 2, practical ranges that have been found suitable are 512 and 1024 pixels.

[00326] The sub-fragments may overlap with at least 2 pixels of the optical sensor. More or less may be used. For example, for 512, an overlap of 256 pixels may be suitable, and for 1024, an overlap of 512 pixels may be suitable.

[00327] The method may further include combining the enhanced resolution images of the sub-fragments into an aggregated enhanced resolution image and using the aggregated enhanced resolution image to sequence the sample over multiple cycles.

[00328] The method can be further applied to types of images at positions in the flow cell that capture fluorescence from millions of samples distributed throughout the flow cell. Such application of the method provides improved images of the flow cell.

[00329] When the imaging plane is a flow cell, the samples can be distributed in millions of nanowells across the flow cell. At least some of the adhesion pairs of nanowells can be located closer to each other than the Abbe diffraction limit for optical resolution. Structured illumination provides such a close location. In an alternative embodiment, the samples can be randomly distributed across the flow cell. In such an embodiment, the method further includes obtaining enhanced image sequences for the flow cell and using the enhanced image sequences to call sample sequences.

[00330] The above-described computation methods may be implemented in a system that includes computer hardware. One or more of the methods described above may be implemented in a system with a computer. The system with a computer may include any of the methods described immediately above or within the framework of this application that are applicable to the method implemented by the system. For brevity, alternative combinations of system elements are not listed separately. Features applicable to systems, methods and articles of industrial production are not repeated for each set of basic features of a class of patentable subject matter of the invention. The reader will understand how the features indicated in these examples of implementation can be easily combined with basic features indicated in other classes of patentable subject matter of the invention.

[00331] As an article of manufacture rather than a method, a non-volatile computer-readable medium (CRM) may be loaded with program instructions executable by a processor. When executed, the program instructions implement one or more computer-implementable methods described above. In an alternative embodiment, the program instructions may be loaded onto a non-volatile CRM and, when combined with appropriate hardware, may become a component of one or more computer-implementable systems that practice the described methods.

[00332] Each of the features discussed in a particular implementation section for other embodiments is equally applicable to this embodiment. As noted above, all features are not repeated herein for brevity and should be considered as repeated by reference.

Application of SIM image reconstruction without redundancy

[00333] The described technology relates to reducing the computational resources required to obtain an improved resolution image from structured illumination of a target.

[00334] In one embodiment, the described sub-tile technology can be used to obtain an image of improved resolution on target images captured under structured illumination. This method applies one or more transformations to non-redundant data and then reconstructs the redundant data from the non-redundant data after the transformations.

[00335] The method includes transforming at least three target images captured by a sensor in the spatial domain. This sensor captures images from at least three structured illumination phase shifts along one angle. The three captured images in the spatial domain are transformed into the Fourier domain to obtain at least three frequency domain matrices. These frequency domain matrices include at least non-redundant components and redundant components of the conjugate. The redundant components of the conjugate are conjugates at positions of complementary matrices of the non-redundant components. The method further includes using the calculated reconstruction parameters to create an inverse mixing matrix and applying the inverse mixing matrix to at least the non-redundant components. This allows obtaining at least three phase-separated matrices in the Fourier domain, which are unbiased and biased matrices. These matrices include non-redundant components without bias and non-redundant spatial frequency components with bias, obtained by applying the inverse mixing matrix to the non-redundant components of the frequency domain matrix.

[00336] The method also includes performing one or more intermediate transformations of at least the non-redundant components of the phase-separated matrices. One intermediate operation is a subpixel shift. It includes a transformation from the Fourier domain to the spatial domain of at least the non-redundant components of the spatial frequency of the shifted matrices. Then, a shift vector is applied to the data in the spatial domain and the transferred data is transformed from the spatial domain data into one or more reconstructed shifted matrices back to the Fourier domain. Applying the shift vector effectively aligns the overlap values in the shifted matrices in the Fourier domain with the unshifted matrix more accurately than the positions of the entire matrices of the discrete Fourier transforms. The redundant components of the conjugate can be extracted after the intermediate transformations from the non-redundant components by copying the transformed values from the non-redundant components to the position of the complementary matrices from the redundant components of the conjugate and changing the sign of the imaginary parts of the copied values to obtain the transformed redundant components of the conjugate. This reduces the resources required during intermediate transformations. Constructing an enhanced resolution image further involves aligning and summing the overlapping values of the unbiased matrix and the biased matrices to obtain an extended frequency coverage matrix, and then inversely transforming the extended frequency coverage matrix from the Fourier domain to obtain an enhanced resolution image in the spatial domain.

[00337] In some embodiments, a series of images of improved resolution are obtained by repeated application of the preceding steps. Additionally, for example in sequencing by synthesis, the described technology can be applied to use a series of images of improved resolution to sequence samples obtained by the sensor over several cycles.

[00338] This method and other embodiments of the described technology may include one or more of the following features and/or features described in connection with additional described methods. For the sake of brevity of the description, combinations of features described in this application are not individually numbered and are not repeated with each basic set of features. The reader will understand how the features specified in this section can be easily combined with sets of basic features specified in other embodiments.

[00339] The described technology can be applied to components located in a matrix offset to a corner, when the positions of the complementary matrices of the redundant components of the conjugate are rotated by 180 degrees relative to the position of the matrix of non-redundant components.

[00340] The method may further include applying one- or two-stage Wiener filtering to reduce noise during intermediate operations. When applying two-stage Wiener filtering, an independent modulation function for separating phases may be applied to the separated phases as an intermediate transform to compensate for optical contrast transfer by an objective lens that has a reduced contrast transfer capability with increasing spatial frequency. The noise removal components of the Wiener filtering may be applied separately later. The intermediate operations may further include applying apodization filtering.

[00341] In some embodiments, the phase-separated Fourier domain matrices include an even number of rows and an even number of columns. For these even matrices, column averaging may be applied as an intermediate operation to average the DC column, the DC components with the middle column, one column outside the center of the even matrix. This column averaging may precede the subpixel offset.

[00342] In some embodiments, the phase-separated matrices in the Fourier domain are shifted toward the corner. The DC components then appear in the top row or bottom row, as well as in the left column or right column of the matrices, and the redundant components of the conjugate appear in a block rotated 180 degrees relative to the orientation of the non-redundant components.

[00343] In some embodiments, the phase-separated matrices in the Fourier domain include an odd number of rows and an odd number of columns with no column between the blocks of non-redundant components and the redundant components of the conjugate. In odd matrices, the redundant components of the conjugate also appear in a block rotated 180 degrees relative to the orientation of the non-redundant components.

[00344] In embodiments in which the target includes regularly spaced nanowells arranged in a rectangular pattern, the method can be applied to sets of at least three images from a phase shift along two corners of the target's structured illumination, wherein the two corners are located along diagonals between opposite corners of the rectangles. Furthermore, in embodiments in which the target includes regularly spaced nanowells arranged in a hexagonal pattern, the method can be applied to sets of at least three images from a phase shift along three corners of the target's structured illumination, wherein the three corners are located along diagonal channels between opposite corners of the hexagons.

[00345] The method may further include applying Wiener filtering during intermediate operations in two stages. Performing the two stages includes applying an independent modulation transfer function to the separated phases to compensate for the optical contrast transfer, which decreases with increasing spatial frequency, using an objective lens that transfers the target under structured illumination to the sensor. It further includes applying noise removal components from Wiener filtering or another equivalent filter.

[00346] The method may further include applying an apodization filter during intermediate operations.

[00347] The phase-separated matrices in the Fourier domain include an even number of rows and an even number of columns. For such even matrices, the method may further include applying column averaging during intermediate operations to average the DC column, the DC components with the middle column, between blocks of non-redundant components and redundant components of the conjugate. The column averaging may precede the subpixel shift.

[00348] When the phase-separated matrices in the Fourier domain are shifted to the upper left corner, the DC components may be displayed in the upper row and left column of the matrices, and the redundant components of the conjugate in a block rotated 180 degrees relative to its orientation as non-redundant components. In an alternative embodiment, the phase-separated matrices may be shifted to the lower right corner.

[00349] Fourier domain phase separation matrices may also include an odd number of rows and an odd number of columns. Such odd matrices do not contain a middle column between the blocks of non-redundant components and redundant components of the conjugate.

[00350] The method can be further applied to sets of at least three target images captured by a lens using a sensor in a spatial domain, from phase shifts along two angles of structured illumination of the target. When the target comprises regularly spaced nanowells, the two angles can be oriented mainly along the diagonals between opposite corners of quadrangles among the nanowells, located at an equal distance from each other. These quadrangles can be squares, and the two angles can be mainly rectangular with an error within one to three degrees. The method can also be applied to sets of images from phase shifts along three angles of structured illumination of the target. This is useful when the target includes nanowells located at an equal distance from each other in the form of a hexagon. In this pattern, the three angles can be along the diagonals between opposite corners of the hexagons.

[00351] The above-described computation methods may be implemented in a system that includes computer hardware. One or more of the methods described above may be implemented in a system with a computer. The system with a computer may include any of the methods described immediately above or within the framework of this application that are applicable to the method implemented by the system. For brevity, alternative combinations of system elements are not listed separately. Features applicable to systems, methods and articles of industrial production are not repeated for each set of basic features of a class of patentable subject matter of the invention. The reader will understand how the features indicated in these examples of implementation can be easily combined with basic features indicated in other classes of patentable subject matter of the invention.

[00352] As an article of manufacture rather than a method, a non-volatile computer-readable medium (CRM) may be loaded with program instructions executable by a processor. When executed, the program instructions implement one or more computer-implementable methods described above. In an alternative embodiment, the program instructions may be loaded onto a non-volatile CRM and, when combined with appropriate hardware, may become a component of one or more computer-implementable systems that practice the described methods.

[00353] Each of the features discussed in a particular implementation section for other embodiments is equally applicable to this embodiment. As noted above, all features are not repeated herein for brevity and should be considered as repeated by reference.

[00354] The above description is provided to enable the production and use of the described technology. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described technology. Thus, the described technology is not limited to the presented embodiments, but should cover the entire number of possible variations consistent with the principles and features described herein. The scope of the described technology is defined by the appended claims.

Computer system

[00355] FIG. 24 is a simplified block diagram of a computer system that can be used to implement the described technology. The computer system typically includes at least one processor that communicates with several peripheral devices via a bus subsystem. These peripheral devices may include a storage subsystem, including, for example, memory devices, and a file storage subsystem, user interface input devices, user interface output devices, and a network interface subsystem. The input and output devices provide for user interaction with the computer system. The network interface subsystem provides an interface with external networks, including an interface with corresponding interface devices in other computer systems.

[00356] In one embodiment, a parameter estimation tool 2401 for calculating three parameters (angle, offset, and phase shift) is coupled to a storage subsystem and user interface input devices for data exchange.

[00357] User interface input devices may include a keyboard; pointing devices such as a mouse, trackball, touch pad, or graphics tablet; a scanner; a touch screen integrated into a display; audio input devices such as voice recognition systems and microphones; and other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for entering information into a computer system.

[00358] The user interface output devices may include a display subsystem, a printer, a facsimile machine, or non-visual display devices such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide non-visual display such as audio output devices. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from a computer system to a user or to another machine or computer system.

[00359] The data storage subsystem stores software constructs and data constructs that provide the functionality of some or all of the modules and methods described herein. Typically, these software modules are executed by the processor alone or in combination with other processors.

[00360] The memory used in the data storage subsystem may include a number of memory devices, including a main random access memory (RAM) for storing instructions and data during program execution, and a read-only memory (ROM) in which fixed instructions are stored. The file storage subsystem may provide a read-only memory for program files and data and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical disk, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by the file storage subsystem in the data storage subsystem or in other machines accessible to the processor.

[00361] A bus subsystem provides a mechanism for allowing various components and subsystems of a computer system to interact with each other in accordance with their intended purpose. Although the bus subsystem is schematically shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[00362] The computer system itself may be of various types, including a personal computer, a portable computer, a workstation, a computer terminal, a network computer, a television, a central system, a group of servers, a widely dispersed set of loosely coupled network computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the example computer system described in FIG. 24 is intended only as a specific example for the purpose of illustrating the described technology. Many other computer system configurations may contain more or fewer components than the computer system shown in FIG. 24.

[00363] Deep learning processors may be GUs or FPGAs and hosted on a deep learning cloud platform such as Google Cloud Platform, Xilinx, and Cirrascale. Examples of deep learning processors include Google's Tensor Processing Unit (TPU), rack-mounted solutions such as the GX4 Rackmount Series, GX8 Rackmount Series, NVIDIA DGX-1, Microsoft's Stratix V FPGA, Graphcore's Intelligent Processor Unit (IPU), Qualcomm's Zeroth platform with Snapdragon processors, NVIDIA Volta, NVIDIA DRIVE PX, NVIDIA JETSON TX1/TX2 MODULE, Intel Nirvana, Movidius VPU, Fujitsu DPI, ARM DynamicIQ, IBM TrueNorth, and others.

Claims (59)

1. A method for producing images with improved resolution for samples from a scanner that detects fluorescence from millions of samples distributed over a flow cell in images collected over multiple runs, wherein the samples are located closer to each other than the Abbe diffraction limit, for optical resolution between at least some adjacent pairs of samples, comprising: calibration of the scanner using virtually the entire field of view projected by the lens onto the optical sensor on the scanner, including: capturing at least one image fragment under structured illumination at multiple angles and phase shifts of the structured illumination; calculating optical distortion from an image fragment, including measuring the distance between intensity peaks and the angle of intensity peaks in at least 9 image sub-fragments, including a sub-fragment closest to the center, and approximating a spacing surface and an angle surface to the measured spacing and angle, respectively, where the spacing surface and the angle surface express the correction of distortion on the sub-fragments of the image fragment, and storing the results and measuring structured illumination phase shifts at multiple angles and phase shifts within sub-fragments and storing a look-up table of coefficients that expresses the differences between the phase shift in the sub-fragment closest to the center and the phase shift in other sub-fragments of the image fragment; processing of captured image elements on a flow cell, wherein image fragments are captured at multiple angles and phase shifts of structured illumination over multiple cycles, including: using the coefficients of the stored approximation results and the stored lookup table to extrapolate from the measurements, for the near-center sub-fragment of each captured image, the interval between intensity peaks, the angle of the projection pattern of the illumination, and the phase offsets of the structured illumination, for the reconstruction parameters for the near-center sub-fragment and for at least 8 additional sub-fragments; applying certain reconstruction parameters to obtain improved resolution images for sub-fragments over multiple cycles at a power resolution higher than the Abbe diffraction limit; and using high-resolution images for sub-samples of samples over multiple cycles. 2. The method of claim 1, wherein the samples are distributed into several million nanowells through a flow cell and at least some of the adhesion pairs of nanowells are located closer to each other than the Abbe diffraction limit for optical resolution. 3. The method according to any one of claims 1, 2, wherein each subfragment contains at least 512 × 512 pixels of the optical sensor. 4. The method according to any one of paragraphs 1-3, in which the sub-fragments overlap by at least 2 pixels of the optical sensor. 5. The method according to any one of paragraphs 1-4, further comprising storing the results of the approximation of the spread surface and the angle surface as coefficients of the quadratic surface. 6. The method according to any one of paragraphs 1-4, further comprising storing the results of the approximation of the spread surface and the angle surface as coefficients of the cubic surface. 7. The method according to any one of paragraphs 1-4, further including storing the results in the form of a reference table calculated using a quadratic approximation of the spread surface and the angle surface. 8. The method according to any one of paragraphs 1-4, further including storing the results in the form of a reference table calculated using a cubic approximation of the spread surface and the angle surface. 9. The method according to any one of claims 1-8, further comprising, in an intermediate cycle among the plurality of cycles, re-measuring the spacing between intensity peaks and the angle of intensity peaks in the sub-fragments, re-approximating the spacing surface and the angle surface, and storing the results for use in processing image fragments captured later. 10. The method according to claim 9, characterized in that the intermediate cycle is the 50th cycle or more within at least 100 cycles. 11. The method of claim 2, wherein the nanowells are arranged in a repeating rectangular pattern with two diagonals connecting opposite corners of the rectangle, in the pattern, further including the use of two structured illumination angles in which the intensity peaks are oriented predominantly perpendicular to the two diagonals. 12. The method of claim 2, wherein the nanowells are arranged in a repeating hexagonal pattern with three diagonals connecting opposite corners of the hexagon, in the pattern, further including the use of three structured illumination angles in which intensity peaks are oriented predominantly perpendicular to the three diagonals. 13. The method according to any one of claims 1-12, further comprising combining the enhanced resolution images of the sub-fragments into an enhanced resolution image of the fragment and using the enhanced resolution image fragment to sequence samples over multiple cycles. 14. The method according to any one of claims 1-13, further comprising using edge cropping to remove pixels around the edges of the sensor from the approximation of the span surface and the corner surface. 15. The method according to any one of claims 1-14, further comprising measuring the spacing between intensity peaks and the angle of intensity peaks on at least 8×11 sub-fragments of an image. 16. A method for generating sub-slice calibration parameters used with a scanner to obtain improved fluorescence resolution images of samples located closer to each other than the Abbe diffraction limit for optical resolution between at least some adjacent pairs of samples, comprising: calibration of the scanner using virtually the entire field of view projected by the lens onto the optical sensor on the scanner, including: capturing at least one image fragment under structured illumination at multiple angles and phase shifts of the structured illumination; calculation of optical distortion of an image fragment, including: measuring the distance between the intensity peaks and the angle of the intensity peaks of the projected illumination pattern as shown in the intensity peaks in at least 9 sub-sections of the image section, including the sub-section closest to the center, and approximation of the spacing surface and the angle surface to the measured spacing and angle, respectively, wherein the spacing surface and the angle surface express the correction of distortions in the sub-fragments of the image fragment and measuring structured illumination phase shifts at multiple angles and phase shifts within sub-fragments and creating a look-up table that expresses the differences between the phase shift in the sub-fragment closest to the center and the phase shift in other sub-fragments of the image fragment; storing sub-tile calibration parameters, including a look-up table and approximation results, that relate to the intensity peak distance and the illumination projection pattern angle of at least 8 additional sub-tiles in the near center sub-tile, for use by a scanner in production. 17. The method of claim 16, wherein the samples comprise several million samples distributed across the flow cell, in images collected over several cycles. 18. The method according to any one of claims 16, 17, wherein each subfragment contains at least 512 × 512 pixels of the optical sensor. 19. The method according to any one of paragraphs 16-18, in which the sub-fragments overlap by at least 2 pixels of the optical sensor. 20. The method according to any one of paragraphs 16-19, further comprising storing the results of the approximation of the spread surface and the angle surface as coefficients of the quadratic surface. 21. The method according to any one of paragraphs 16-19, further comprising storing the results of the approximation of the spread surface and the angle surface as coefficients of the cubic surface. 22. The method according to any one of paragraphs 16-19, further comprising storing the results of the approximation in the form of a lookup table calculated using a quadratic approximation of the spread surface and the angle surface. 23. The method according to any one of paragraphs 16-19, further comprising storing the approximation results in the form of a lookup table calculated based on the cubic approximation of the spread surface and the angle surface. 24. The method according to any one of paragraphs 16-23, further comprising using edge cropping to remove pixels around the edges of the sensor from the approximation of the span surface and the corner surface. 25. The method according to any one of claims 16-24, further comprising measuring the spacing between intensity peaks and the angle of intensity peaks on at least 8×11 sub-fragments of an image. 26. A method of using a calibrated scanner to obtain images of improved resolution of samples located closer to each other than the Abbe diffraction limit for optical resolution between at least some adjacent pairs of samples, including: for the closest to the center sub-fragment of the image captured by the scanner's optical sensor, access to or generation of reconstruction calculation parameters, including: spacing parameters for the spacing between intensity peaks and pattern angle parameters, for the angle of intensity peaks, of the projected illumination pattern as shown in intensity peaks, spacing parameters generated by imaging measurements with uniform separation and an oblique pattern of structured illumination projected onto the sub-tile closest to the center, and phase shift parameters for the relative phase relationship between sub-fragments at a sub-fragment reference point obtained from additional visualization measurements of multiple phase shifts along one corner of the structured lighting pattern projected onto the sub-fragment closest to the center; obtaining estimated reconstruction parameters for at least 8 additional sub-fragments of the image fragment, in addition to the closest-to-center sub-fragment, by combining the available or estimated reconstruction parameters for the closest-to-center sub-fragment with: saved approximation results that relate to the spacing parameters and angle parameters of at least 8 additional sub-fragments relative to the sub-fragment closest to the center and a stored look-up table for measuring the phase shift of at least 8 additional sub-fragments relative to the sub-fragment closest to the center; and applying the calculated reconstruction parameters for the closest sub-fragment and at least 8 additional sub-fragments to obtain images of improved resolution for the sub-fragments in order to obtain an optical power above the Abbe diffraction limit. 27. The method of claim 26, further comprising repeatedly applying the sequence of actions of claim 26 to obtain a sequence of images of improved resolution for the sub-fragments and using the sequence of images of improved resolution to sequence samples visualized by the sensor over several cycles. 28. The method according to any one of claims 26, 27, wherein each subfragment contains at least 512 × 512 pixels of the optical sensor. 29. The method according to any one of paragraphs 26-28, in which the sub-fragments overlap by at least 2 pixels of the optical sensor. 30. The method according to any one of claims 26-29, further comprising combining the enhanced resolution images of the sub-fragments into an enhanced resolution image of the fragment and using the enhanced resolution image fragment to sequence samples over multiple cycles. 31. The method according to any one of claims 26-30, further comprising, in an intermediate cycle among the plurality of cycles, re-measuring the spacing between intensity peaks and the angle of intensity peaks in the sub-fragments, re-approximating the spacing surface and the angle surface, and storing the results for use in processing image fragments captured later. 32. The method according to claim 31, characterized in that the intermediate cycle is the ^50th cycle or more within at least 100 cycles. 33. The method of any one of claims 26-32, further applied to types of images at positions in the flow cell that capture fluorescence from millions of samples distributed throughout the flow cell, including creating enhanced images for the flow cell. 34. The method of claim 33, wherein the samples are distributed into several million nanowells through a flow cell and at least some of the adhesion pairs of nanowells are located closer to each other than the Abbe diffraction limit for optical resolution. 35. The method of any one of claims 26-34, further comprising obtaining enhanced image sequences for the flow cell and using the enhanced image sequences to call up sample sequences. 36. The method of claim 35, wherein the nanowells are arranged in a repeating rectangular pattern with two diagonals connecting opposite corners of the rectangle, in the pattern, further including the use of two structured illumination angles in which intensity peaks are oriented predominantly perpendicular to the two diagonals. 37. The method of claim 35, wherein the nanowells are arranged in a repeating hexagonal pattern with three diagonals connecting opposite corners of the hexagon, in the pattern, further including the use of three structured illumination angles in which intensity peaks are oriented predominantly perpendicular to the three diagonals.