JP5892424B2 - Image acquisition method, image acquisition device, and scanning microscope - Google Patents

Description

  The present invention relates to an image acquisition method, an image acquisition device, and a scanning microscope.

  A point scan type image acquisition device used in a scanning microscope or the like acquires a single image by discretely acquiring reflected light or fluorescence from an object while scanning the observation surface of the object with point-like illumination light. The point scan which comprises is implemented (for example, refer patent document 1).

JP 2004-212600 A

  In such a point scan type image acquisition apparatus, a time difference occurs between the image acquisition start time and the image acquisition end time. Therefore, since the acquisition time of each point in one image is different, there is a problem that it is not suitable for acquiring an image of an object that changes rapidly with respect to the scanning speed.

  The present invention has been made in view of such problems, and has an image acquisition method capable of acquiring an image in consideration of a time difference in point scanning even for a rapidly changing object, and the image acquisition method. An object is to provide an image acquisition device and a scanning microscope.

In order to solve the above problems, an image acquiring method according to the present invention, by obtaining each of the luminance values of a plurality of small regions in the observation plane, an image acquisition method of acquiring an image of the observation plane, the In the image, the luminance values of the plurality of minute regions are the luminance values acquired at different times, and two or more images are acquired, and the luminance values of the plurality of minute regions are obtained at at least two times. Obtaining and determining the brightness value of the micro area at a predetermined time based on the brightness values of the micro area acquired at two times, and using the brightness values of the plurality of micro areas at the predetermined time, An image at a predetermined time is generated.

により算出することが好ましい。但し、ｔａ≦ｔｘ≦ｔｂとする。 Also, in such an image acquisition method, the luminance value of the region at the coordinates (i, j) at time t is P _{i, j} (t), and the luminance values of the pixels acquired at time ta and time tb are P _{i. , j} (ta) and P _{i, j} (tb), the luminance value P _{i, j} (tx) at time tx, which is a predetermined time , is expressed by the following equation:

It is preferable to calculate by However, ta ≦ tx ≦ tb.

The image acquisition device according to the present invention includes an image acquisition unit that acquires an image of the observation surface by acquiring the luminance values of each of the plurality of minute regions on the observation surface, and an image of the observation surface at a predetermined time. have a, an image processing unit that generates, in this image, the luminance value of each of the plurality of micro-regions, the luminance values obtained at different times, the image acquisition unit acquires an image of two or more Then, the brightness values of each of the plurality of micro regions are acquired at least at two times, and the image processing unit is configured to perform predetermined processing based on the brightness values of the micro regions acquired at the two times acquired by the image acquisition unit . A luminance value of a minute area at a time is determined, and an image at a predetermined time on the observation surface is generated using the luminance values of the plurality of minute areas at a predetermined time .

により算出して、所定の時刻の画像を生成することが好ましい。但し、ｔａ≦ｔｘ≦ｔｂとする。 In such an image acquisition device, the image processing unit sets the luminance value of the region at the coordinates (i, j) at time t as P _{i, j} (t), and the pixels acquired at time ta and time tb. When the luminance values are P _{i, j} (ta) and P _{i, j} (tb), the luminance value P _{i, j} (tx) at time tx, which is a predetermined time , is expressed by the following equation:

It is preferable to generate an image at a predetermined time by calculating by the above . However, ta ≦ tx ≦ tb.

Further, the scanning microscope according to the present invention includes an illumination optical system that scans a sample using a scanning unit with illumination light emitted from a light source unit, a condensing optical system that guides light from the sample to a light detection unit, An image acquisition unit that acquires an image of the observation surface by acquiring the brightness value of each of a plurality of minute regions on the observation surface on the specimen corresponding to the scanning of the illumination light, and observation at a predetermined time possess an image processing unit that generates an image of the surface, the image acquisition unit acquires an image of two or more, to get the luminance value of each of the plurality of microscopic regions with at least two times, the image processing unit based on the luminance value of the acquired small area obtained two times were in the image acquiring unit determines the brightness value of the micro-region at a predetermined time, each of the luminance values plurality of microscopic regions at a predetermined time using, at a predetermined time of the observation plane And generating an image.

  When the image acquisition method according to the present invention, and the image acquisition apparatus and the scanning microscope having the image acquisition method are configured as described above, an image taking into account the time difference in the point scan is acquired even for a rapidly changing object. be able to.

It is explanatory drawing for demonstrating the structure of a scanning microscope. It is explanatory drawing for demonstrating the scanning method of an observation surface, Comprising: (a) shows the relationship between the pixel of an observation surface, and a scanning direction, (b) shows the pixel acquired for every time. It is explanatory drawing for demonstrating the relationship between the change of a sample, and the scanning time, (a) shows the change of the sample for every time, (b) is the state when the image of the changing sample is acquired. Indicates. It is explanatory drawing for demonstrating the relationship between the change of a sample, and the acquisition time of an image, Comprising: (a) shows the image acquired before changing, (b) shows the image acquired in the middle of change, (C) shows an image acquired after the change. It is explanatory drawing explaining the relationship between the change of a sample, an acquired image, and a correction | amendment image, Comprising: (a) shows the change of the brightness | luminance of a sample, (b) shows the luminance value of the pixel acquired by scanning, (c ) Shows the acquired image, and (d) shows the corrected image. The relationship between the luminance value acquired at the time t6 and the time t9 of the pixel p32 and the luminance value at the time t7 obtained from these values is shown. It is explanatory drawing explaining the change of the sample at the time of making the time resolution of a correction image high, and the relationship between an acquired image and a correction image, (a) shows the change of the brightness | luminance of a sample, (b) is the acquired image. (C) shows the corrected image. It is explanatory drawing explaining the structure of a control part. It is a flowchart for demonstrating the image acquisition method by a control part.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. First, the configuration of the scanning microscope will be described with reference to FIG. The scanning microscope 10 includes a light source device 20, a scanning optical system 30 that irradiates and scans a sample 60 placed on a stage with laser light (illumination light) emitted from the light source device 20, and a sample 60. And a detection unit 40 for detecting fluorescence (observation light) from the light source. In the following description, the optical axis direction of the scanning optical system 30 is defined as the z axis, and the directions orthogonal to each other in a plane orthogonal to the z axis are defined as the x axis and the y axis, respectively.

  In the scanning microscope 10, the light source device 20 includes a light source configured to be able to emit a plurality of laser beams having different wavelengths simultaneously or individually.

  The scanning optical system 30 includes a dichroic mirror 31, a scanning unit 32, a scan lens 33, a second objective lens 34, and an objective lens 35 in order from the light source device 20 side.

  The detection unit 40 is disposed on the side of the dichroic mirror 31, and is sequentially disposed from the dichroic mirror 31 side at a position substantially conjugate with the focal plane on the sample side of the condenser lens 41 and the objective lens 35. The light shielding plate 42 and the photodetector 43 are included. Here, the light shielding plate 42 is provided with a pinhole 42 a, and the pinhole 42 a is disposed so as to include the optical axis of the scanning optical system 30. For example, a PMT (Photo Multiplier Tube) is used as the photodetector 43.

  In the scanning microscope 10, the scanning unit 32 controls the position (coordinates in the scanning plane on the specimen 60) where the laser beam is scanned on the observation plane (scanning plane) of the specimen 60. A control unit 50 is provided that processes the detected value (the detected light intensity, which will be referred to as “brightness value” in the following description) to acquire an image of the observation surface of the specimen 60.

  In the scanning microscope 10, the laser light that is a substantially parallel light beam emitted from the light source device 20 passes through the dichroic mirror 31 and enters the scanning unit 32. The scanning unit 32 scans the laser light two-dimensionally in the direction orthogonal to the optical axis (the above-described x-axis direction and y-axis direction). A first deflection element that deflects in a predetermined direction (this direction is defined as an x-axis direction) in a plane orthogonal to the optical axis, and a deflection direction of the first deflection element in a plane orthogonal to the optical axis. It consists of two deflecting elements comprising a second deflecting element that deflects in an orthogonal direction (this direction is the y-axis direction). The laser light (substantially parallel light beam) emitted from the scanning unit 32 is imaged on the primary image plane I by the scan lens 33 and then passes through the second objective lens 34 to become a substantially parallel light beam again. Thus, the light is condensed on the focal plane of the objective lens 35 on the specimen 60. The laser light focused on the specimen 60 is a point image, and the diameter of the point image is determined by the numerical aperture (NA) of the objective lens 35.

  When the sample 60 is irradiated with the laser beam, fluorescence is generated from the sample 60, and this fluorescence becomes observation light and enters the objective lens 35. The objective lens 35 becomes a substantially parallel light beam, and the second objective lens. 34 forms an image on the primary image plane I. Further, the light is converted into a substantially parallel light beam by the scan lens 33 and incident on the scanning unit 32, descanned and emitted by the scanning unit 32, reflected by the dichroic mirror 31, and enters the detection unit 40. The light is condensed on the 42 pinholes 42a. In this detection unit 40, only light that has passed through the pinhole 42 a of the light shielding plate 42 reaches the photodetector 43 and is detected.

  As described above, the pinhole 42a of the light shielding plate 42 is conjugate with the point image of the laser beam condensed on the scanning surface (the focal plane of the objective lens 35) on the sample 60, and the irradiation region (objective) on the sample 60. Observation light (fluorescence) emitted from the focal plane of the lens 35 can pass through the pinhole 42a. On the other hand, most of the light emitted from other regions on the specimen 60 is not condensed on the pinhole 42a and cannot pass therethrough.

  As described above, the scanning optical system 30 is configured to scan the sample 60 with the illumination light emitted from the light source device 20 that is a light source unit, the scanning unit, and the light that guides the light from the sample 60 to the detection unit 40. It has the function of means. Then, the control unit 50 processes the optical signal detected by the photodetector 43 in synchronization with the scanning of the scanning unit 32, thereby using the scanning position of the laser beam on the sample 60 and the luminance value obtained from the optical signal. Thus, a two-dimensional image on the scanning surface (observation surface) of the sample 60 can be obtained. That is, the control unit 50 has a function as an image acquisition device having an image acquisition unit and an image processing unit. As a result, the scanning microscope 10 can obtain an image of the specimen 60 with high resolution.

  A method for acquiring an image of the observation surface (scanning surface) of the specimen 60 by performing point scanning with the control unit 50 serving as an image acquisition device in the scanning microscope 10 will be described. First, in order to simplify the explanation, as shown in FIG. 2A, the scanning plane is divided into three parts in the x-axis direction and the y-axis direction, respectively, and a 3 × 3 region is scanned with laser light. A case will be described in which a signal (luminance value that is the intensity of observation light) output from the light detector 43 is acquired for each time and an image of the observation surface (scanning surface) of the specimen 60 is formed. In the following description, each divided area is referred to as a “pixel”. Also, here, as shown in FIG. 2B, in the xy plane shown in FIG. 2, starting from the upper left pixel (p11), scanning in the x-axis direction and reaching the upper right pixel (p13), A case will be described in which the pixel is moved to the rightmost pixel in the middle stage in the y-axis direction, the scan in the x-axis direction is repeated, and the scan to the lower right pixel (p33) is performed. The horizontal axis shown in FIG. 2B represents the time at which a signal (luminance value) output from the photodetector 43 is acquired in each pixel.

  As shown in FIG. 3A, it is assumed that a specimen to be observed is arranged at three pixels in the middle row among 3 × 3 pixels on the scanning plane. Also, as shown in FIG. 3A, when scanning of this specimen is started at time T0, the state changes at time T3 (for example, as shown in FIG. 3A, the brightness of the specimen is changed). The value changes from 0 (expressed in white) to 100 (expressed in black)). Then, the image of the scanning plane obtained by the control unit 50 is as shown in FIG. 3B, and among the sample images, the upper pixel is acquired as an image having a luminance value of 0, but the middle and lower pixels. Is acquired as an image having a luminance value of 100. That is, when the relationship between the sample change and the scanning time is summarized, if the scanning of the scanning surface is completed before time T3, the state of the sample before the change as shown in FIG. When scanning is started after time T3, the state of the sample after the change as shown in FIG. 4C can be acquired. However, if the scan is performed during the change, the sample as shown in FIG. 4B is obtained. You will not be able to get the changes correctly.

  Therefore, the control unit 50 according to the present embodiment is configured to enable image acquisition of a rapidly changing sample by interpolating a difference in signal (luminance value) acquisition time for each pixel in one image. Yes. Specifically, the time lag of each pixel in one image is interpolated using two or more images acquired by the point scan type.

  FIG. 5A shows a change in the signal (luminance value) acquisition time of the middle row of pixels in which the sample exists (p21, p22, and p32, respectively) among the 3 × 3 pixels described above. ing. In FIG. 5, one image (signal, luminance value) acquisition on the scanning plane is called a frame (frame = f), and the signal (luminance value) acquisition time (in the above-mentioned middle column) in each frame ( Only time = t) is described. Further, the sample shown in FIG. 5 shows a case where the whole sample does not change at the same time (T3) as in FIG. 3 described above, but each part changes at a different time. Specifically, in the portion corresponding to the pixel p12, the luminance value changes from 0 to 100 at the time t5 (when the second image (signal, luminance value) is acquired) of the frame f2, and the luminance value at the time t13 of the frame f5. Has changed from 100 to 0. Similarly, the portion corresponding to pixel p22 changes at time t6 of frame f2 and time t14 of frame f5, and the portion corresponding to pixel p32 changes at time t7 of frame f3 and time t15 of frame f5.

  As shown in FIG. 5B, the luminance value of each pixel is acquired at each time of each frame. For example, the luminance value of the pixel p12 is acquired at time t1 of the frame f1, the luminance value of the pixel p22 is acquired at time t2, and the luminance value of the pixel p32 is acquired at time t3. The acquired image in FIG. 5C shows the acquired luminance values of the three pixels p12 to p32 for each frame.

The control unit 50 according to the present embodiment is an image at an arbitrary time between the images (signals and luminance values) of the two frames acquired from the images (luminance values for each pixel) of two adjacent frames. (This is referred to as a “corrected image”). Specifically, if the luminance value of the _i- th pixel in the x-axis direction and the j-th pixel in the y-axis direction (pixel of coordinates (i, j)) at time t is P _{i, j} (t) _, the luminance value at time tx The luminance value P _{i, j} (tx) is expressed by the following equation using the pixel luminance values P _{i, j} (ta) and P _{i, j} (tb) acquired at the time ta before and after the time tx and the time tb. Obtained by (1). However, ta ≦ tx ≦ tb (ta ≠ tb).

  FIG. 5D shows a corrected image interpolated at the first time of each frame. For example, the frame f3 shows a corrected image at time t7. As shown in FIG. 6, in the frame f3, the luminance value of the pixel p32 at time t7 is the luminance value 0 acquired at time t6 of the frame f2 and the luminance value 100 acquired at time t9 of the frame f3. From this, the luminance value 33 is obtained using the equation (1).

  Here, the time tx for obtaining the luminance value of each pixel can be selected at an arbitrary timing, and by dividing this time finely, it can be used to improve the time resolution of the acquired image. For example, as shown in FIG. 7C, when the luminance value interpolated by the above equation (1) is obtained at all times t1 to t18, the change of the sample can be acquired from the corrected image at each time. . FIG. 7A shows the luminance value of the sample, and FIG. 7B shows the images acquired in the frames f1 to f7.

  FIG. 8 shows the configuration of the control unit 50. The control unit receives a signal from the photodetector 43 and controls an operation of the scanning unit 32 to acquire an image, and a signal (luminance value) acquired from the photodetector 43 temporarily. A buffer memory 52 for storing the image, an image processing unit 53 for generating an interpolated image (corrected image) at each time from the acquired luminance value, and a storage unit 54 for storing the corrected image. The image acquisition unit 51 controls the scanning unit 32 based on a clock signal from an oscillator (not shown) to move the illumination light to the position on the scanning plane corresponding to each pixel in the order described above, and the timing thereof. Thus, the luminance value output from the photodetector 43 is acquired and stored in the buffer memory 52.

  As illustrated in FIG. 9, when the image acquisition unit 51 starts the sample image acquisition process, first, the luminance value of each pixel of the first frame is stored in the buffer memory 52 (step S100). The luminance value of each pixel is stored in association with the time when the luminance value is acquired. This time may be configured to store the time when each luminance value is acquired in association with the pixel, or the time when the acquisition of the image (signal, luminance value) of each frame starts and ends. May be obtained from these times, or when the time at which each pixel is acquired is known in advance, the time need not be stored. This time is not limited to an absolute time, and may be a relative time from a predetermined reference time (for example, the time when acquisition of a sample image (signal, luminance value) is started as described above).

  When the next frame is reached, the luminance value of the first pixel is acquired from the photodetector 43 and stored in the buffer memory (S110). Then, the image processing unit 53 reads the luminance value acquired in the previous frame from the buffer memory 52 in the pixel, and the interpolated luminance value from the previous acquisition time to the current acquisition time together with the luminance value acquired this time. Is calculated using the above equation (1) and stored in the storage unit 54 (step S120). The time to interpolate (the time to generate the corrected image) may be one time between two frames (the previous frame and the current frame) as shown in FIG. 5, or a plurality of times as shown in FIG. But it ’s okay. Then, it is determined whether or not the above-described processing has been completed for all the pixels in the current frame (step S130), and the above-described processing of step S110 and step S120 is repeated until the completion for all the pixels. . On the other hand, if it is determined that the processing has been completed for all the pixels in the current frame, it is determined whether or not the image acquisition is completed (step S140). Preparations for acquiring a frame image are made (step S150), and the process returns to step S110 described above. The buffer memory 52 is used cyclically so as to store the luminance values of the current frame and the previous frame.

  By performing the above processing by the image acquisition unit 51 and the image processing unit 53 of the control unit 50 according to the present embodiment, the image of the sample of at least two frames is acquired, so that the time between these two frames is obtained. A sample image (corrected image) interpolated at the time can be acquired, and a sample image considering a time difference in the point scan type can be acquired.

  In the above description, an example in which linear interpolation is performed using two images is shown. However, it is also possible to perform interpolation using a plurality of images (in this case, the interpolation formula is the above formula (1)). It is not limited and is interpolated using the least square method or Bezier curve). Further, in the examples so far, only an image at an arbitrary time between two images is estimated. From these images, an image before image acquisition and an image after image acquisition are estimated. Is also possible. For example, in the case of FIG. 7, by using the images of frames f1, f3, and f5, an image of frame f2 (for example, an image at time t4) is obtained by calculation, or from the images of frames f3, f4, and f5, It is also possible to obtain an image of the frame f6 (for example, an image at time t1 or time t16) by calculation.

  In the above description, the image processing unit 53 is configured to calculate a corrected image for each frame and store the corrected image in the storage unit 54. First, the image acquisition unit 51 generates images of a plurality of frames. It may be configured to acquire and calculate a corrected image at an arbitrary time using those images.

  Moreover, in the above description, although the case where the control part 50 which is an image acquisition apparatus which concerns on this embodiment was applied to the scanning microscope 10 was demonstrated, this image acquisition apparatus is limited to the image acquisition in a scanning microscope. The present invention can be applied to an electron microscope, a CT (Computed Tomograph) apparatus, a copy, a FAX, or the like.

DESCRIPTION OF SYMBOLS 10 Scanning microscope 20 Light source device (light source unit) 30 Scanning optical system 40 Detection unit 43 Photo detector 50 Control unit (image acquisition device) 51 Image acquisition unit 53 Image processing unit

Claims (5)

An image acquisition method for acquiring an image of the observation surface by acquiring a luminance value of each of a plurality of minute regions on the observation surface,
In the image, the luminance values of the plurality of minute regions are luminance values acquired at different times,
Obtaining two or more images, and obtaining brightness values of each of the plurality of microscopic regions at at least two times;
Based on the brightness value of the micro area acquired at the two times, determine the brightness value of the micro area at a predetermined time;
An image acquisition method comprising: generating an image of the observation surface at the predetermined time by using a luminance value of each of the plurality of minute regions at the predetermined time . The luminance value of the region at the coordinates (i, j) at time t is P _{i, j} (t), and the luminance values of the pixels acquired at time ta and time tb are P _{i, j} (ta) and P _{i. , j} (tb), the luminance value P _{i, j} (tx) at time tx, which is the predetermined time , is expressed by the following equation:

The image acquisition method according to claim 1 , wherein the image acquisition method is calculated by: However, ta ≦ tx ≦ tb. An image acquisition unit for acquiring an image of the observation surface by acquiring a luminance value of each of a plurality of micro regions on the observation surface;
An image processing unit that generates an image of the observation surface at a predetermined time, the possess,
In the image, the luminance values of the plurality of minute regions are luminance values acquired at different times,
The image acquisition unit acquires two or more images, and acquires the brightness value of each of the plurality of minute regions at at least two times,
The image processing unit
Based on the brightness value of the micro area acquired at the two times acquired by the image acquisition unit, the brightness value of the micro area at the predetermined time is determined,
An image acquisition device that generates an image of the observation surface at the predetermined time using luminance values of the plurality of minute regions at the predetermined time . The image processing unit sets the luminance value of the region at the coordinates (i, j) at time t as P _{i, j} (t), and sets the luminance values of the pixels acquired at time ta and time tb as P _{i, j.} (Ta) and P _{i, j} (tb), the luminance value P _{i, j} (tx) at time tx, which is the predetermined time , is expressed by the following equation:

The image acquisition apparatus according to claim 3 , wherein the image at the predetermined time is generated by calculation. However, ta ≦ tx ≦ tb. An illumination optical system that scans a sample using a scanning unit with illumination light emitted from a light source unit;
A condensing optical system for guiding light from the specimen to a light detection unit;
An image acquisition unit that acquires an image of the observation surface by acquiring the luminance value of each of a plurality of minute regions on the observation surface on the specimen corresponding to the scanning of the illumination light from the light detection unit ;
An image processing unit that generates an image of the observation surface of the predetermined time, the possess,
The image acquisition unit acquires two or more images, and acquires the brightness value of each of the plurality of minute regions at at least two times,
The image processing unit
Based on the brightness value of the micro area acquired at the two times acquired by the image acquisition unit, the brightness value of the micro area at the predetermined time is determined,
An image of the observation surface at the predetermined time is generated using the luminance value of each of the plurality of minute regions at the predetermined time .

Images