JP2020042283A - Microscope apparatus - Google Patents

Abstract

To provide a microscope apparatus for improving quality of a reconstructed image using imaging result captured at a plurality of imaging positions.SOLUTION: A microscope apparatus 1 includes: an illumination optical system 4 that radiates activation light L2 to activate a part of fluorescent material included in a sample W and excitation light L1 to excite at least a part of the activated fluorescent material; an image forming optical system 5 that has an objective lens 21 and an astigmatic optical system 61 that generates astigmatism to at least a part of fluorescence from the fluorescent material; and forms an image of the fluorescence; an image-capturing unit 6 that captures an image formed by the image forming optical system 5; a drive unit 50 that moves an image-capturing position in the sample W along an optical axis-direction 21a of the objective lens 21; and a control unit 42, where the control unit 42 causes the image-capturing unit 50 to capture images in a plurality of numbers of frames respectively at a first image-capturing position and at a second image-capturing position different from the first image-capturing position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a microscope device, an imaging method, and a program.

There is a microscope apparatus using a single-molecule localization microscopy method such as STORM or PALM (for example, see Patent Document 1).
Patent Document 1 US Patent Application Publication No. 2008/0182336

  In order to reconstruct the structure of the sample in a wide range in the thickness direction of the sample with the microscope device, it is necessary to move the sample and the optical system relatively in the optical axis direction and image fluorescence at a plurality of imaging positions. There is. Therefore, there is a need for improving the quality of a reconstructed image using imaging results obtained at a plurality of imaging positions.

  In the first aspect of the present invention, an activating light for activating a part of the fluorescent substance contained in the sample, and an illumination optical system for irradiating excitation light for exciting at least a part of the activated fluorescent substance, An objective lens, an imaging optical system that forms an image of fluorescent light, and an imaging optical system that has an astigmatism optical system that generates astigmatism for at least a part of the fluorescent light from the fluorescent substance. A microscope that has an imaging unit that captures the captured image, a driving unit that moves the imaging position of the sample along the optical axis direction of the objective lens, and a control unit, wherein the control unit includes a first imaging unit, At a second imaging position different from the imaging position and the first imaging position, imaging is performed with a plurality of frames, respectively.

  In a second aspect of the present invention, an activating light for activating a part of a fluorescent substance contained in a sample, and an illumination optical system for irradiating excitation light for exciting at least a part of the activated fluorescent substance, An objective lens, an imaging optical system that forms an image of fluorescent light, and an imaging optical system that has an astigmatism optical system that generates astigmatism for at least a part of the fluorescent light from the fluorescent substance. In a microscope apparatus having an imaging unit that captures a captured image and a driving unit that moves the imaging position of the sample along the optical axis direction of the objective lens, the imaging unit includes a first imaging position and a first imaging position. An imaging method is provided in which imaging is performed with a plurality of frames at different second imaging positions.

  In a third aspect of the present invention, an activating light for activating a part of a fluorescent substance contained in a sample, and an illumination optical system for irradiating excitation light for exciting at least a part of the activated fluorescent substance, An objective lens, an imaging optical system that forms an image of fluorescent light, and an imaging optical system that has an astigmatism optical system that generates astigmatism for at least a part of the fluorescent light from the fluorescent substance. A computer that controls a microscope device having an imaging unit that captures the captured image and a driving unit that moves the imaging position of the sample along the optical axis of the objective lens. At a second imaging position different from the first imaging position, a program is provided for executing a procedure for imaging with a plurality of frames.

  The above summary of the present invention is not an exhaustive listing of all features of the present invention. Further, a sub-combination of these feature groups can also be an invention.

FIG. 2 is a diagram illustrating a microscope device 1 according to the embodiment. FIG. 2 is a diagram illustrating an imaging optical system 5 and an image sensor 40. FIG. 4 is a diagram illustrating an optical path of fluorescence from a position Z2 and an image of the fluorescence in a sample W. FIG. 4 is a diagram showing an optical path of fluorescence from a fluorescent substance present at a position Z1 in a sample W and an image of the fluorescence. FIG. 7 is a diagram illustrating an optical path of fluorescent light from a fluorescent substance present at a position Z3 in a sample W and an image of the fluorescent light. FIG. 3 is a diagram illustrating an example of image processing by an image processing unit 7. It is an example of the data configuration 45 of the three-dimensional distribution information of the center of gravity position Q. FIG. 3 is a schematic diagram of an operation sequence of the microscope device 1. 4 is a flowchart illustrating an imaging process of the microscope device 1. It is an example of a screen for inputting an imaging condition in an imaging process. It is a schematic diagram which shows the three-dimensional distribution information of the gravity center position of fluorescence. 13 is an example of a screen 150 for setting a display image of the reconstruction of the sample W. The display image CL1 color-coded for each layer is shown. It is a schematic diagram which shows the thickness direction overlap of the sample between layers. It is a schematic diagram explaining the display image CL2 which displays the position of the center of gravity included in the overlap Lw. FIG. 7 is a schematic diagram of another operation sequence of the microscope device 1. It is another example of the screen which inputs the imaging conditions in an imaging process. It is a schematic diagram which shows the three-dimensional distribution information of the gravity center position of fluorescence. The display image CL3 color-coded for each layer is shown. The display image CL4 color-coded for each pass is shown.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all combinations of the features described in the embodiments are necessarily essential to the solution of the invention.

  The microscope apparatus according to the embodiment is, for example, a microscope apparatus using a single-molecule localization microscopy method such as STORM or PALM. The microscope device according to the present embodiment can generate a three-dimensional super-resolution image. The microscope apparatus according to the embodiment can be used for both fluorescence observation of a sample labeled with one kind of fluorescent substance (label) and fluorescence observation of a sample labeled with two or more kinds of fluorescent substances.

  The sample may include living cells (live cells), may include cells fixed using a tissue fixing solution such as a formaldehyde solution, or may be a tissue. The fluorescent substance may be a fluorescent dye such as a cyanine dye or a fluorescent protein. The fluorescent dye includes a reporter dye that emits fluorescence when receiving excitation light in an activated state, which is an activated state. The fluorescent dye may include an activator dye that receives the activating light and turns the reporter dye into an activated state. If the fluorescent dye does not contain an activator dye, the reporter dye will be activated upon receiving the activating light. The fluorescent dye is, for example, a dye pair obtained by binding two kinds of cyanine dyes (eg, a Cy3-Cy5 dye pair (Cy3 and Cy5 are registered trademarks), and a Cy2-Cy5 dye pair (Cy2 and Cy5 are registered trademarks). , Cy3-Alexa Fluor 647 dye pair (Cy3, Alexa Fluor is a registered trademark)) and one type of dye (eg, Alexa Fluor 647 (Alexa Fluor is a registered trademark)). The fluorescent protein is, for example, PA-GFP, Dronpa, or the like.

  FIG. 1 is a diagram illustrating a microscope device 1 according to the present embodiment. The microscope device 1 includes a stage 2, a light source device 3, an illumination optical system 4, an imaging optical system 5, an imaging unit 6, an image processing unit 7, and a control device 8. The control device 8 includes a control unit 42 that comprehensively controls each unit of the microscope device 1. The control device 8 may be a computer that executes a procedure described below by reading a software program.

  The stage 2 holds the cover glass 51. The cover glass 51 holds the sample W to be observed. More specifically, as shown in FIG. 1, a cover glass 51 is placed on the stage 2, and a sample W is placed on the cover glass 51. The stage 2 may or may not move within the XY plane.

  The light source device 3 includes an activation light source 10a, an excitation light source 10b, a shutter 11a, and a shutter 11b. The activation light source 10a emits activation light L2 for activating a part of the fluorescent substance contained in the sample W. Here, it is assumed that the fluorescent substance contains a reporter dye and does not contain an activator dye. The irradiation of the activating light L2 causes the reporter dye of the fluorescent substance to be in an activated state capable of emitting fluorescence. The fluorescent material may include a reporter dye and an activator dye, where the activator dye activates the reporter dye when it receives the activating light L2. The fluorescent substance may be a fluorescent protein such as PA-GFP or Dronpa.

  The excitation light source 10b emits excitation light L1 that excites at least a part of the fluorescent material activated in the sample W. The fluorescent substance emits fluorescence or is inactivated when irradiated with the excitation light L1 in the activated state. When the activating light L2 is irradiated in a deactivated state (hereinafter, referred to as an inactivated state), the fluorescent substance is activated again.

  The activation light source 10a and the excitation light source 10b include, for example, a solid-state light source such as a laser light source, and each emit a laser beam having a wavelength according to the type of the fluorescent substance. The emission wavelength of the activation light source 10a and the emission wavelength of the excitation light source 10b are selected from, for example, about 405 nm, about 457 nm, about 488 nm, about 532 nm, about 561 nm, about 640 nm, about 647 nm, and the like. Here, the emission wavelength of the activation light source 10a is about 405 nm, and the emission wavelength of the excitation light source 10b is a wavelength selected from about 488 nm, about 561 nm, and about 647 nm.

  The shutter 11a is controlled by the control unit 42, and can switch between a state in which the activation light L2 from the activation light source 10a is transmitted and a state in which the activation light L2 is blocked. The shutter 11b is controlled by the control unit 42, and can switch between a state in which the excitation light L1 from the excitation light source 10b passes and a state in which the excitation light L1 is blocked.

  Further, the light source device 3 includes a mirror 12, a dichroic mirror 13, an acousto-optic element 14, and a lens 15. The mirror 12 is provided, for example, on the emission side of the excitation light source 10b. The excitation light L1 from the excitation light source 10b is reflected by the mirror 12 and enters the dichroic mirror 13.

  The dichroic mirror 13 is provided, for example, on the emission side of the activation light source 10a. The dichroic mirror 13 has a property of transmitting the activating light L2 and reflecting the excitation light L1. The activation light L2 transmitted through the dichroic mirror 13 and the excitation light L1 reflected by the dichroic mirror 13 enter the acousto-optic element 14 through the same optical path.

  The acousto-optic device 14 is, for example, an acousto-optic filter. The acousto-optic element 14 is controlled by the control unit 42 and can adjust each of the light intensity of the activation light L2 and the light intensity of the excitation light L1. Further, the acousto-optic element 14 is controlled by the control unit 42, and each of the activation light L2 and the excitation light L1 passes through the acousto-optic element 14 or is blocked by the acousto-optic element 14 or It is possible to switch between a light shielding state in which the intensity is reduced. For example, when the fluorescent substance contains a reporter dye and does not contain an activator dye, the control unit 42 controls the acousto-optic element 14 so that the sample W is irradiated with the activation light L2 and the excitation light L1 simultaneously. . When the fluorescent substance contains a reporter dye and an activator dye, the control unit 42 controls the acousto-optical element 14 such that the sample W is irradiated with the excitation light L1 after the irradiation with the activation light L2.

  The lens 15 is, for example, a coupler, and focuses the activation light L2 and the excitation light L1 from the acousto-optic element 14 on the light guide member 16. Note that the microscope device 1 may not include at least a part of the light source device 3. For example, the light source device 3 may be unitized, and may be provided in the microscope device 1 in a replaceable (attachable, removable) manner. For example, the light source device 3 may be attached to the microscope device 1 at the time of observation with the microscope device 1 or the like.

  The illumination optical system 4 irradiates an activation light L2 for activating a part of the fluorescent substance contained in the sample W and an excitation light L1 for exciting at least a part of the activated fluorescent substance. The illumination optical system 4 irradiates the sample W with the activating light L2 and the excitation light L1 from the light source device 3. The illumination optical system 4 includes a light guide member 16, a lens 17, a lens 18, a filter 19, a dichroic mirror 20, and an objective lens 21.

  The light guide member 16 is, for example, an optical fiber, and guides the activation light L2 and the excitation light L1 to the lens 17. In FIG. 1 and the like, the optical path from the emission end of the light guide member 16 to the sample W is indicated by a dotted line. The lens 17 is, for example, a collimator, and converts the activating light L2 and the excitation light L1 into parallel light. The lens 18 condenses, for example, the activation light L2 and the excitation light L1 at a position on the pupil plane of the objective lens 21. The filter 19 has, for example, a property of transmitting the activating light L2 and the excitation light L1, and blocking at least a part of light of another wavelength. The dichroic mirror 20 has a characteristic of reflecting the activating light L2 and the excitation light L1 and transmitting light in a wavelength band of fluorescence emitted from the fluorescent substance of the sample W. Light from the filter 19 is reflected by the dichroic mirror 20 and enters the objective lens 21. The sample W is arranged on the front focal plane of the objective lens 21 during observation.

  The sample W is irradiated with the activating light L2 and the excitation light L1 by the illumination optical system 4 as described above. The above-described illumination optical system 4 is an example, and can be appropriately changed. For example, a part of the illumination optical system 4 described above may be omitted. Further, the illumination optical system 4 may include at least a part of the light source device 3. Further, the illumination optical system 4 may include an aperture stop, an illumination field stop, and the like.

  The imaging optical system 5 forms an image of fluorescence from the fluorescent substance contained in the sample W. The imaging optical system 5 converts the primary image generated by the first optical system 5A from the first optical system 5A that forms the primary image of the fluorescent light emitted from the sample W into fluorescent light as a secondary image in the imaging unit 6. And a second optical system 5B that forms the image of The first optical system 5A includes an objective lens 21, a dichroic mirror 20, a filter 24, a lens 25, and an optical path switching member 26. The second optical system 5B includes a lens 27, a lens 28, and a cylindrical lens 61 as an astigmatism optical system.

  The imaging optical system 5 shares the objective lens 21 and the dichroic mirror 20 with the illumination optical system 4. In FIG. 1 and the like, the optical path between the sample W and the imaging unit 6 is indicated by a solid line. Further, the drive unit 50 moves the objective lens 21 in the optical axis direction of the objective lens 21, that is, in the Z direction in FIG.

  The fluorescence from the sample W enters the filter 24 through the objective lens 21 and the dichroic mirror 20. The filter 24 has a characteristic that light in a predetermined wavelength band of the light from the sample W selectively passes. The filter 24 blocks, for example, illumination light, external light, stray light, and the like reflected by the sample W. The filter 24 is unitized with, for example, the filter 19 and the dichroic mirror 20, and the filter unit 29 is provided exchangeably. The filter unit 29 is exchanged according to, for example, the wavelength of light emitted from the light source device 3 (eg, the wavelength of the activation light L2, the wavelength of the excitation light L1), the wavelength of the fluorescence emitted from the sample W, and the like. Alternatively, a single filter unit corresponding to a plurality of excitation and fluorescence wavelengths may be used.

  The light passing through the filter 24 enters the optical path switching member 26 via the lens 25. The light emitted from the lens 25 passes through the optical path switching member 26 and forms a primary image on the intermediate image plane 5b. The optical path switching member 26 is, for example, a prism, and is provided in the optical path of the imaging optical system 5 so as to be insertable and removable. The optical path switching member 26 is inserted into and removed from the optical path of the imaging optical system 5 by, for example, a driving unit (not shown) controlled by the control unit 42. The optical path switching member 26 guides the fluorescent light from the sample W to the optical path toward the imaging unit 6 by internal reflection when inserted into the optical path of the imaging optical system 5.

  The imaging unit 6 captures an image formed by the imaging optical system 5 (first observation optical system 5). The imaging unit 6 includes an imaging device 40 and a control unit 41. The image sensor 40 is, for example, a CMOS image sensor, but may be a CCD image sensor or the like. The imaging element 40 has, for example, a structure in which a plurality of pixels are two-dimensionally arranged, and a photoelectric conversion element such as a photodiode is arranged in each pixel. The imaging element 40 reads out, for example, electric charges accumulated in the photoelectric conversion element by a reading circuit. The image sensor 40 converts the read charges into digital data, and outputs digital data (eg, image data) in which pixel positions are associated with gradation values. The control unit 41 operates the image sensor 40 based on a control signal input from the control unit 42 of the control device 8 and outputs data of a captured image to the control device 8. Further, the control unit 41 outputs a charge accumulation period and a charge readout period to the control device 8.

  Note that the above-described imaging optical system 5 is an example, and can be appropriately changed. For example, a part of the above-described imaging optical system 5 may be omitted. Further, the imaging optical system 5 may include an aperture stop, a field stop, and the like.

  The microscope apparatus 1 according to the present embodiment includes a second observation optical system 30 used for setting an observation range and the like. The second observation optical system 30 includes an objective lens 21, a dichroic mirror 20, a filter 24, a lens 25, a mirror 31, a lens 32, a mirror 33, a lens 34, a lens 35, and a mirror in order from the sample W to the observer's viewpoint Vp. 36 and a lens 37. The observation optical system 30 shares the configuration from the objective lens 21 to the lens 25 with the imaging optical system 5.

  The light from the sample W enters the mirror 31 after passing through the lens 25 and in a state where the optical path switching member 26 is retracted from the optical path of the imaging optical system 5. The light reflected by the mirror 31 enters the mirror 33 via the lens 32, is reflected by the mirror 33, and then enters the mirror 36 via the lenses 34 and 35. The light reflected by the mirror 36 enters the viewpoint Vp via the lens 37. The second observation optical system 30 forms an intermediate image of the sample W in an optical path between the lens 35 and the lens 37, for example. The lens 37 is, for example, an eyepiece, and an observer can set an observation range by observing the intermediate image.

  The control device 8 controls each part of the microscope device 1 as a whole. The control device 8 includes a control unit 42 and the image processing unit 7. The control unit 42 controls the acousto-optic element 14 to transmit light from the light source device 3 to the acousto-optic device 14 based on signals indicating the charge accumulation period and the charge read period supplied from the control unit 41. And a control signal for switching between a light-blocking state and a light-blocking state for blocking light from the camera. The acousto-optic device 14 switches between a light transmitting state and a light blocking state based on the control signal. The control unit 42 controls the acousto-optic element 14 to control a period during which the sample W is irradiated with the activation light L2 and a period during which the sample W is not irradiated with the activation light L2. In addition, the control unit 42 controls the acousto-optical element 14 to control a period during which the sample W is irradiated with the excitation light L1 and a period during which the sample W is not irradiated with the excitation light L1. The control unit 42 controls the acousto-optic element 14 to control the light intensity of the activation light L2 and the light intensity of the excitation light L1 applied to the sample W. The control unit 42 controls the imaging unit 6 to cause the imaging device 40 to execute imaging.

  Note that, instead of the control unit 42, the control unit 41 controls the acousto-optic element 14 to switch between the light-shielding state and the light-transmitting state based on signals (information on imaging timing) indicating the charge accumulation period and the charge readout period. A signal may be supplied to control the acousto-optic element 14. The control unit 42 acquires data as an imaging result from the imaging unit 6. The image processing unit 7 performs image processing such as finding the position of the center of gravity of each image using the imaging result of the imaging unit 6. The control unit 42 causes the imaging unit 6 to perform imaging in a plurality of frame periods, and the image processing unit 7 generates one image using at least a part of the imaging results obtained in the plurality of frame periods.

  The control device 8 is communicably connected to each of the storage device 43 and the display device 44, for example. The display device 44 is, for example, a liquid crystal display. The display device 44 displays various images such as an image indicating various settings of the microscope device 1, an image captured by the imaging unit 6, and an image generated from the captured image. The control unit 42 controls the display device 44 and causes the display device 44 to display various images. For example, the control unit 42 supplies data of a super-resolution image such as a STORM image and a PALM image generated by the image processing unit 7 to the display device 44, and causes the display device 44 to display the image. For example, the microscope device 1 can display a super-resolution image of the sample W to be observed as a live video. The storage device 43 is, for example, a magnetic disk or an optical disk, and stores various data such as various setting data of the microscope device 1, data of an imaging result by the imaging unit 6, and data of an image generated by the image processing unit 7. The control unit 42 can supply, for example, the data of the super-resolution image stored in the storage device 43 to the display device 44 and cause the display device 44 to display the super-resolution image. The control unit 42 controls the storage device 43 and causes the storage device 43 to store various data.

  FIG. 2 is a diagram illustrating the imaging optical system 5 and the image sensor 40. Although the imaging optical system 5 in FIG. 1 is an optical system in which the optical axis 5a is bent by the optical path switching member 26, FIG. 2 conceptually illustrates the imaging optical system 5 as a straight optical system. In FIG. 2, the illustration of the configuration between the objective lens 21 and the cylindrical lens 61 is omitted. Here, a direction parallel to the optical axis 5a of the imaging optical system 5 is defined as a Z direction, and two directions perpendicular to the Z direction and perpendicular to each other are defined as an X direction and a Y direction. The X direction is, for example, the horizontal direction of the image sensor 40, and the Y direction is, for example, the vertical direction of the image sensor 40.

  The cylindrical lens 61 is an optical member having power (refractive power) only in one of the vertical direction and the horizontal direction. Here, the description will be made on the assumption that the cylindrical lens 61 has power in the XZ plane and has no power in the YZ plane. Note that the astigmatism optical system may use a toroidal lens having power in both the vertical and horizontal directions and having different powers in these two directions, instead of the cylindrical lens 61.

  FIG. 3 is a diagram showing an optical path of fluorescence from the position Z2 and an image of fluorescence in the sample W. First, the optical path in the XZ plane will be described. The fluorescent light from the fluorescent substance present at the position Z2 in the sample W enters the cylindrical lens 61 as parallel light. The fluorescent light travels toward the image sensor 40 while being collected by refraction at the cylindrical lens 61. The width Wx of the image of the fluorescent light in the X direction becomes minimum at a position Z4 corresponding to the focal point on the XZ plane. Further, the width Wx increases as the distance from the position Z4 to the opposite side (+ Z side) from the sample W increases.

  Next, an optical path in the YZ plane will be described. Fluorescence from a fluorescent substance present at the position Z2 in the sample W is incident on the cylindrical lens 61 as parallel light. This fluorescence enters the lens 28 almost without being refracted by the cylindrical lens 61. The fluorescent light travels toward the image sensor 40 while being collected by refraction of the lens 28. The width Wy of the fluorescent light image in the Y direction becomes minimum at a position Z6 corresponding to the focal point on the YZ plane. The width Wy increases as the distance from the position Z6 to the same side as the sample W (−Z side) increases.

  At a position Z5 between the positions Z4 and Z6, the width Wx of the fluorescent image in the X direction is equal to the width Wy of the fluorescent image in the X direction, and the fluorescent image is a perfect circle. Therefore, when the imaging element 40 is arranged at the position Z5, a perfect circle image is obtained as an image of the fluorescence from the fluorescent substance existing at the position Z2 in the sample W.

  FIG. 4 is a diagram showing an optical path of a fluorescent light from a fluorescent substance present at a position Z1 in the sample W and an image of the fluorescent light. The position Z1 is on the opposite side (−Z side) to the image sensor 40 with respect to the position Z2. On the XZ plane, the width Wx of the fluorescence image in the X direction becomes minimum at a position farther from the position Z4 to the sample W side (−Z side). On the other hand, on the YZ plane, the width Wy of the fluorescent light image in the Y direction becomes minimum at the position Z5. Therefore, when the imaging element 40 is arranged at the position Z5, an elliptical image having a long axis in the X direction is obtained as an image of the fluorescence from the fluorescent substance existing at the position Z1 in the sample W.

  FIG. 5 is a diagram showing an optical path of a fluorescent light from a fluorescent substance present at a position Z3 in the sample W and an image of the fluorescent light. The position Z3 is on the same side (+ Z side) as the image sensor 40 with respect to the position Z2. On the XZ plane, the width Wx of the fluorescent light image in the X direction becomes minimum at the position Z5. On the other hand, on the YZ plane, the width Wy of the fluorescent light image in the Y direction becomes minimum at a position away from the position Z6 on the opposite side (+ Z side) from the sample W. Therefore, when the imaging element 40 is arranged at the position Z5, an elliptical image having a long axis in the Y direction is obtained as an image of the fluorescence from the fluorescent substance present at the position Z3 in the sample W.

  As described above, the relationship between the width Wx and the width Wy of the fluorescence image, for example, the ratio changes according to the position of the sample W in the Z direction (the same direction as the optical axis 21a of the objective lens 21). Therefore, the image processing unit 7 performs, for example, elliptic Gaussian fitting, and specifies the width Wx and the width Wy in addition to the position of the center of gravity of the fluorescent light image. Thereby, the image processing unit 7 can specify the position in the XY plane of the fluorescent substance corresponding to the fluorescent image from the position of the center of gravity, and can specify the position in the Z direction based on the relationship between the width Wx and the width Wy. .

  FIG. 6 is a diagram illustrating an example of image processing by the image processing unit 7 at a predetermined imaging position. Here, the imaging position is the position of the front focus of the objective lens 21 on the sample W.

  In FIG. 6, reference numerals P1 to Pn represent imaging results obtained in a predetermined imaging frame period. Since the fluorescent substance activated at a probability corresponding to the intensity of the activation light L2 emits fluorescent light by irradiation with the excitation light L1, the number of fluorescent substances corresponding to the probability emits fluorescent light in one frame, and between frames. As can be seen, different fluorescent materials emit fluorescence.

  The image processing unit 7 calculates, for each of the captured images P1 to Pn, the position of the center of gravity of the fluorescent light image included in each image and the width in the XY direction. The image processing unit 7 calculates the XY position and the widths Wx and Wy of the center of gravity by performing, for example, elliptic Gaussian fitting on the distribution of pixel values in this area. The image processing unit 7 specifies the position of the center of gravity in the Z direction with reference to a table or the like stored in advance from the relationship between the calculated widths Wx and Wy. Accordingly, the image processing unit 7 calculates, for example, the center of gravity Q of the fluorescence image Im, and at least a part of the plurality of centers of gravity Q corresponding to the plurality of fluorescence images included in the plurality of captured images P1 to Pn. Is used to generate three-dimensional distribution information of the position of the center of gravity Q. Further, a three-dimensional super-resolution image SP1 is generated and displayed based on the three-dimensional distribution information. The super-resolution image SP1 is obtained by reconstructing the three-dimensional structure of the sample W. Note that the three-dimensional distribution information obtained at a predetermined imaging position is referred to as a layer.

  In the example of FIG. 6, the three-dimensional super-resolution image SP1 is shown as a three-dimensional image that allows a user to specify a viewing direction. Alternatively or additionally, a projection image SP2 on the XY plane and a projection image SP3 on the XZ plane may be displayed.

   FIG. 7 is an example of a data configuration 45 of the three-dimensional distribution information of the center of gravity position Q. In the example of FIG. 7, the XYZ coordinates of the center of gravity position Q are associated with the numbers for identifying the center of gravity position Q. Further, the brightness B and the widths Wx and Wy may be associated with each other. The three-dimensional distribution information according to the configuration is stored in the storage device 43.

  As described above with reference to FIG. 6, according to the microscope apparatus 1, even when the objective lens 21 is fixed in the optical axis direction, the structure of the sample W in the thickness direction (that is, the optical axis direction 21a of the objective lens) can be reconfigured. Can be. However, when the objective lens 21 is fixed in the optical axis direction 21a, the range in the thickness direction in which the structure can be reconstructed is the depth of field of the imaging optical system 5, the intensity of the fluorescence, It is limited by the S / N ratio of the element, the focal length of the cylindrical lens 61, and the like. When observing the sample W, the range in the thickness direction in which the structure can be reconstructed is about 300 nm. Therefore, in the present embodiment, the fluorescence image is captured by the imaging unit 6 while the objective lens 21 is moved in the optical axis direction 21a of the objective lens 21 by the driving unit 50, so that the structure of the sample W is widened in the thickness direction. Reconfigure.

  FIG. 8 is a schematic diagram of an operation sequence of the microscope device 1. FIG. 9 is a flowchart illustrating the imaging process of the microscope device 1. FIG. 10 is an example of a screen for inputting imaging conditions in the imaging processing.

  Making the imaging unit 6 perform imaging multiple times at a predetermined imaging position is defined as one imaging processing unit. In addition, as shown in FIG. 7, an image acquisition procedure at a plurality of predetermined imaging positions is referred to as a pass. Therefore, the path is, for example, that the objective lens 21 moves in the direction of the optical axis 5a of the objective lens 21 from the imaging position 1 to the imaging position 8, and performs imaging. Note that these terms are used for simplification of description, and do not limit the scope of rights.

  As shown in the flowchart of FIG. 9, first, an imaging condition is set in an imaging process (S100). In this case, the microscope device 1 displays a screen 100 as shown in FIG. The control device 8 uses the screen 100 to store the position of the lowermost surface of the image, the position of the uppermost surface of the image, the step size in the Z direction, the number of frames to be acquired at each image position, and the file name for storing the three-dimensional distribution information. Accept the input of.

  In the example of FIG. 10, the number of the imaging position and the number of frames corresponding thereto are manually input by the user. Alternatively, the number of frames for the imaging position may be automatically calculated based on a predetermined rule. For example, when the density of the stained intracellular tissue is higher in the lower layer of the cell and lower in the upper layer, it is preferable to increase the number of frames as the imaging position moves away from the cover glass 51.

Further, the imaging order in each pass may be specified. For example, in the initial setting, the sequential movement from the lower imaging position to the upper imaging position is selected as described above. Instead of this imaging order, it is also possible to select an imaging order such as random, etc., from top to bottom, even numbered imaging positions first, odd numbered imaging positions later.
For example, when moving to the imaging positions Z1 to Z8 shown in FIG. 8, the imaging order of Z1, Z8, Z2, Z7, Z3, Z6, and Z5 can be designated. With this imaging order, it is possible to obtain an image in which the outline of the sample is clear.

  When the imaging condition is input and the imaging start button is pressed, the control device 8 calculates an imaging position, that is, a driving amount by the drive unit 50 from the imaging condition (S102). In this case, it is preferable that the imaging position and / or other imaging conditions are set so that a part of the sample W in the thickness direction overlaps between adjacent layers. FIG. 8 shows an example of an operation sequence in which the number of layers is eight and the number of paths is one.

  The illumination optical system 4 irradiates the sample W1 with the activating light L2 and the excitation light L1 from the light source 3 (S104), and performs imaging by the imaging unit 6 (S106) a predetermined number of frames corresponding to the imaging position (S108: No). ).

  When the imaging of the predetermined number of frames is completed (S108: Yes), the drive unit 50 moves the objective lens 21 to the next imaging position (S109). Steps S104 to S109 are repeated for the calculated imaging position (S110: No).

  When the imaging at the final imaging position is completed (S110: Yes), the control unit 8 determines whether or not the final pass has been reached (S111). If the last pass has not been reached (S111: No), the process proceeds to the next pass, and steps S104 to S110 are repeated. When the last pass has been reached (S111: Yes), the imaging process ends.

  As described above, according to the present embodiment, when reconstructing the structure of a sample by a method such as STORM, the structure of the sample is reconstructed in a range wider than the range in the thickness direction limited by the focal length of the cylindrical lens and the like. can do. In particular, since the structure of the sample is reconstructed on the basis of the imaging results of the different number of frames for each imaging position, improvement in the image quality of the reconstructed image according to the shape and properties of the sample can be expected.

  During or after the imaging processing, the image processing unit 7 specifies the position of the center of gravity of the fluorescence in each imaging frame as described with reference to FIG. The image processing unit 7 further stores the three-dimensional distribution information of the position of the center of gravity specified in each imaging frame included in the imaging processing unit for each imaging position in the storage device 43 as a file having the data configuration illustrated in FIG.

  FIG. 11 is a schematic diagram illustrating a plurality of layers specified by imaging at a plurality of imaging positions. FIG. 11 shows a case where the number of layers is 3 and the number of paths is 1.

  The image processing unit 7 stores the center of gravity of the fluorescence specified for each path and each layer, that is, for each imaging processing unit, in the storage device 43 in association with information for identifying the path and the layer. The Z position of the center of gravity of each fluorescence in the sample space is determined based on the imaging position based on the driving amount of the driving unit 50 and the Z position of the center of gravity of the fluorescence in each layer (the Z position of the center of gravity calculated from the captured fluorescence image). And is specified using The imaging position is the position of the front focus of the objective lens 21 on the sample as described above, but the position of the front focus may be a position based on the cover glass 51.

  FIG. 12 is an example of a screen 150 for setting a display image of the reconstruction of the sample W. The screen 150 is displayed on the display device 44 and accepts a setting as to whether or not to display the path, the layer, or both of them visually.

  FIG. 12 shows an example in which display colors are used as a method for visually distinguishing. In this case, the path and the layer may be distinguished by different ones of the three attributes of the color, for example, the hue of the path and the brightness of the layer. This makes it possible to visually distinguish both the path and the layer.

  Instead of or in addition to the display colors, hatching or shapes, such as circles, triangles, crosses, etc., may be used so that they can be visually distinguished.

  An area 152 of the screen 150 is for selecting whether or not to perform color coding for each layer, and an area 154 is for selecting whether or not to perform color coding for each path. On the right side of the region 152, a state in which white layers are arranged in the vertical direction is schematically shown, and further on the right side, a state in which the layers are color-coded when color-coding is selected. Are represented. On the right side of the area 154, a state in which white outline paths are arranged in the left-right direction is schematically shown, and further on the right side, a state in which the respective paths are color-coded when the color-coding is selected. Are represented. A box 156 is for inputting a lower limit of a range in the Z direction to be displayed, and a box 157 is for inputting an upper limit. Further, a box 158 is for inputting a lower limit of a path to be displayed, and a box 159 is for inputting an upper limit.

  The screen 150 further displays information on the data configuration and the data range obtained in the photographing process. As the data configuration, the thickness of the layer, the number of layers, and the number of passes are displayed.

  FIG. 13 shows a display image CL1 color-coded for each layer by the setting of FIG. 12 using the three-dimensional distribution information of FIG. However, since colors cannot be used in the drawing, white, hatching, and black were used instead of colors.

  The image processing unit 7 calls the position of the center of gravity of the fluorescence stored in the storage device 43, and assigns a color based on the information for identifying the layer. The image processing unit 7 assigns white to the center of gravity of the fluorescent light of layer 1, assigns hatching to the center of gravity of the fluorescent light of layer 2, assigns black to the center of gravity of the fluorescent light of layer 3, and generates a display image CL1. It is displayed on the display device 44.

  As described above, according to the present embodiment, the image processing unit 7 visually indicates to which of the plurality of layers the barycentric position belongs, so that the drift of the objective lens 21 and the stage 2 between the layers (for example, (Appears as a position shift) can be easily recognized by the user.

  FIG. 14 is a schematic diagram showing the overlap of the sample in the thickness direction between the layers. As described above, it is preferable that the plurality of imaging positions are set such that a part of the three-dimensional distribution information generated as a result overlaps in the thickness direction of the sample. FIG. 14 shows that there is an overlap Lw between layer 1 and layer 2. Therefore, it is preferable that the setting screen 150 in FIG. 12 can be set to display the position of the center of gravity included in the overlap Lw manually or automatically.

  FIG. 15 is a schematic diagram illustrating a display image CL2 that displays the position of the center of gravity included in the overlap Lw. The image processing unit 7 specifies the barycentric position having the Z coordinate included in the overlap Lw from the three-dimensional distribution information of each layer. In the example illustrated in FIG. 15, the positions of the centers of gravity surrounded by the broken lines in Layer 1 and Layer 2 are specified as being included in the overlap Lw.

  The image processing unit 7 assigns white to the position of the center of gravity in the layer 1 and assigns hatching to the position of the center of gravity of the layer 2 to generate a display image CL2 and display it on the display device 44. Thereby, the degree of drift between the objective lens 21 and the stage 2 between the layers can be more easily recognized.

  FIG. 16 is a schematic diagram of another operation sequence of the microscope device 1. FIG. 17 is an example of a screen 110 for inputting imaging conditions in the imaging processing corresponding to FIG.

  In the example of FIG. 16, the microscope apparatus 1 repeats the pass a plurality of times. Specifically, ten passes from pass 1 to pass 10 are repeated.

  The number of passes is input by the user using the screen 110 of FIG. Further, similarly to the screen 100 of FIG. 10, the screen 110 includes a position of the lowermost surface of the imaging, a position of the uppermost surface of the imaging, a step size in the Z direction, an imaging order in each path, and a file name for storing the three-dimensional distribution information. Accept input. The screen 110 also accepts input of the total number of frames used for each layer.

  The imaging processing of FIGS. 16 and 17 may also be executed according to the flowchart of FIG. In this case, in step S102, by dividing the total number of frames input from the screen 110 by the number of passes, the number of frames acquired at each imaging position of each pass, that is, the number of frames of each imaging processing unit is calculated. For example, the number of frames used in the total of all passes for each imaging position is 10000, and when the number of passes is 10, the number of frames obtained at each imaging position is 1000. Note that the number of frames may be different for each imaging position as in the examples of FIGS.

  Thus, as shown in FIG. 16, the operation of sequentially moving the imaging positions 1 to 8 is defined as one pass, and this operation is repeated ten times. In other words, the camera moves to each imaging position by the number of times of the pass, and an image of the unit of the imaging processing is captured at each imaging position.

In the microscope apparatus using the Single-molecule Localization Microscopy method such as STORM and PALM, as described above, since the number of imaging frames is large, if imaging is performed by setting a single pass, as the imaging time becomes longer, The amount of the fluorescent substance may be greatly different between the layer at the first imaging position and the layer at the last imaging position, for example.
According to the present embodiment, since a plurality of passes can be set, a difference in the amount of fluorescent substance due to fading between layers can be reduced, and a difference in image quality between layers can be reduced.

  FIG. 18 is a schematic diagram illustrating a plurality of layers specified by imaging at a plurality of paths and a plurality of imaging positions. FIG. 18 shows a case where the number of layers is 3 and the number of paths is 2.

  FIG. 19 shows a display image CL3 color-coded for each layer using the path of FIG. However, for the same reason as in FIG. 13, white, hatching, and black were used instead of color.

  The image processing unit 7 calls the XYZ position of the fluorescent light image stored in the storage device 43 and assigns a color based on at least information for identifying the layer. The image processing unit 7 assigns white to the fluorescent image of layer 1 included in all paths, assigns hatching to the fluorescent image of layer 2 included in all paths, and assigns white to the fluorescent image of layer 3 included in all paths. A display image CL3 to which black is assigned is generated and displayed on the display device 44. In this case, it is preferable that the colors from Layer 1 to Layer 3 are shown as legends, and that no distinction is made for each path with a character string such as “all paths”.

  As described above, according to the present embodiment, the image processing unit 7 displays the three-dimensional position specified based on the imaging result of a plurality of layers so as to be visually distinguishable. The user can easily recognize how much drift (positional displacement) has occurred between the sample W and the objective lens caused by the difference between the two.

  FIG. 20 shows a display image CL4 color-coded for each path using the layer of FIG. However, white and black were used instead of color.

  The image processing unit 7 calls up the XYZ position of the fluorescent light image stored in the storage device 43, and assigns a color based on at least information for identifying a path. In the example of FIG. 12, the image processing unit 7 assigns white to the fluorescent images of all the layers included in the pass 1 and assigns black to the fluorescent images of all the layers included in the pass 2, and generates the display image CL4. , On the display device 44. In this case, it is preferable that the image processing unit 7 shows the color of pass 1 and the color of pass 2 as a legend, and indicates that the layers are not distinguished by a character string of “all layers” or the like.

  As described above, according to the present embodiment, the image processing unit 7 visually displays the three-dimensional position specified based on the imaging result of a plurality of paths, so that the sample W The user can easily recognize how much drift has occurred.

  In addition, when moving to the next imaging position after imaging all frames at each imaging position, the fluorescence fades toward the later imaging position in time, and sufficient image quality may not be obtained. There is. On the other hand, according to the embodiment using the sequence of FIG. 16, the number of frames used in each layer is distributed over a plurality of passes and imaging is performed, so that good image quality can be obtained even at an imaging position that is later in time. it can.

  In FIGS. 11, 15, 18 and 20, the three-dimensional distribution information is shown as "image", but this is for explanation. That is, the image processing unit 7 does not need to generate an image except when displaying it as a “display image” on the display device 44, and may use the data as shown in FIG.

  In the above-described embodiment, the image pickup position is moved by the drive unit 50 moving the objective lens 21 in the optical axis direction 5a. Alternatively, the driving unit 50 may move the imaging position by moving the stage 2 in the optical axis direction 5a of the objective lens 21.

  When two types of fluorescent substances are used, for example, an odd number of paths among plural paths is irradiated with excitation light having a wavelength for exciting one fluorescent substance, and an even number of paths is used for exciting other fluorescent substances. May be applied.

  As described above, the present invention has been described using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is apparent to those skilled in the art that various changes or improvements can be made to the above embodiment. It is apparent from the description of the appended claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

  The execution order of each processing such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before”, “before”. It should be noted that they can be realized in any order as long as the output of the previous process is not used in the subsequent process. Even if the operation flow in the claims, the specification, and the drawings is described using “first,” “second,” or the like for convenience, it means that it is essential to perform the operation in this order. Not something.

  DESCRIPTION OF SYMBOLS 1 Microscope device, 2 stages, 3 light source devices, 4 illumination optical systems, 5 imaging optical systems, 6 imaging units, 7 image processing units, 8 control devices, 10a activation light sources, 10b excitation light sources, 11a shutters, 11b shutters, 13 dichroic mirror, 14 acousto-optic element, 15 lens, 16 light guide member, 17 lens, 18 lens, 19 filter, 20 dichroic mirror, 21 objective lens, 24 filter, 25 lens, 26 optical path switching member, 28 lens, 29 filter Unit, 30 observation optical system, 31 mirror, 32 lens, 33 mirror, 34 lens, 35 lens, 36 mirror, 37 lens, 40 image pickup device, 41 control unit, 42 control unit, 43 storage device, 44 display device, 50 drive Part, 51 cover glass, 61 cylindrical glass 'S, 100 screen, 110 screen, 150 screen, 152 area, 154 area, 156 box, 157 box 158 box 159 box

Claims (1)

Activation light for activating a part of the fluorescent substance contained in the sample, and an illumination optical system for irradiating excitation light for exciting at least a part of the activated fluorescent substance,
An objective lens, having an astigmatism optical system that generates astigmatism for at least a part of the fluorescence from the fluorescent substance, and an imaging optical system that forms an image of the fluorescence,
An imaging unit that captures an image formed by the imaging optical system;
A driving unit that moves an imaging position on the sample along an optical axis direction of the objective lens,
And a control unit.
The control unit causes the image capturing unit to capture images at a plurality of frames at a first image capturing position and a second image capturing position different from the first image capturing position, respectively.
The microscope apparatus according to claim 1, wherein the control unit performs an imaging operation of performing imaging at the first imaging position and imaging at a second imaging position a plurality of times.

Images