KR102655200B1 - Method and apparatus for multiplexted imaging through iterative separation of fluorophores - Google Patents

Abstract

The present disclosure provides a method and device for multiple molecular imaging through repetitive color separation. According to the present disclosure, even if the signals of both fluorescent substances are detected in the first fluorescence detection wavelength band, it is possible to separate the signals of the two images obtained through the two detection wavelength bands using iterative mutual information minimization. At this time, since the fluorescence emission intensities according to the emission wavelengths of the two fluorescent substances are different from each other, the signals of the two fluorescent substances appear at different intensities in the two images obtained in the two detection wavelength bands. Specifically, α, which minimizes the amount of mutual information, is calculated from IMG1 and IMG2, where the signals of the two fluorescent substances appear at different intensities, and through this, IMG1 _R1 and IMG2 _R1 are obtained. After that, if IMG1 _R1 and IMG2 _R1 are separated again using mutual information minimization, IMG1 _R2 and IMG2 _R2 can be obtained. Through the process of repeating α value inference and signal separation n times, the signals of the two fluorescent substances are gradually separated, and finally, IMG1 _Rn = F1 and IMG2 _Rn = F2, which are the signal separation of the two fluorescent substances, can be obtained. .

Description

Molecular multiple imaging method and device through repetitive color separation {METHOD AND APPARATUS FOR MULTIPLEXTED IMAGING THROUGH ITERATIVE SEPARATION OF FLUOROPHORES}

The present disclosure relates to a method and device for multiple molecular imaging through repetitive color separation.

Recently, immunotherapy drugs that produce anti-cancer effects by activating immune cells present in the patient's cancer tissue have been receiving a lot of attention. Immunoanticancer drugs have large differences in their anticancer effects depending on which immune cells are present in the patient's cancer tissue. In order to select the optimal anticancer agent for each patient or develop an immunotherapy agent with a new mechanism, there is a need to simultaneously image multiple immune biomarkers within the patient's cancer tissue. Many existing multi-marker simultaneous imaging technologies have several disadvantages, such as requiring expensive special equipment, complicated processes and slow imaging speeds, and sample destruction during the imaging process. It is not widely used to predict immunotherapy reactivity. Therefore, in order to recommend optimal immuno-anticancer drugs for each patient and develop new immuno-anticancer drugs, low-cost, high-efficiency, and intact multi-marker simultaneous imaging technology is needed.

The problem to be solved is to provide a molecular multiple imaging method and device through repeated color separation.

A method of operating an electronic device according to the present disclosure includes acquiring two images for two fluorescent substances each labeled with different biomolecules, and repeatedly calculating the amount of mutual information shared between the acquired images. While reducing, it may include separating images for each of the fluorescent substances from the acquired images.

An electronic device according to the present disclosure includes a memory, and a processor connected to the memory and configured to execute at least one instruction stored in the memory, wherein the processor includes two fluorescent substances each labeled with different biomolecules. For each image, it may be configured to acquire two images and separate images for each of the fluorescent substances from the acquired images while iteratively reducing the amount of mutual information shared between the acquired images.

According to the present disclosure, image separation performance can be improved. At this time, in the present disclosure, the strategy of selecting a specific fluorescent material and selecting a fluorescence detection wavelength band can be eliminated and alleviated through an iterative mutual information minimization strategy. In addition, in the present disclosure, the signals of the two fluorescent substances can be successfully separated through step-by-step α inference and gradual signal separation in two images in which the signals of the two fluorescent substances are detected at different intensities.

In addition, as a result of the present disclosure, it is possible to successfully separate signals of two fluorescent substances whose wavelengths at the maximum of the emission spectrum differ by less than 20 nm. In the present disclosure, through an iterative mutual information minimization strategy, it is possible to successfully separate signals of two fluorescent substances with similar emission spectra without optimization of the fluorescence detection wavelength band. In the present disclosure, by excluding the fluorescence detection wavelength band selection strategy, the range of the first fluorescence detection wavelength band can be set widely, increasing the signal-to-noise ratio, and thus obtaining high-quality signal separation images. Additionally, in the present disclosure, since there is no need to select a wavelength band specialized for fluorescent materials, hardware requirements are reduced and the selection of the emission filter can be freely performed.

1 is a schematic diagram illustrating a signal separation technique through iterative mutual information minimization according to the present disclosure.
2 is a block diagram of an electronic device according to various embodiments of the present disclosure.
Figure 3 is a schematic diagram to explain the advantages that can be obtained according to the present disclosure.
Figure 4 is a flowchart of a signal separation method through iterative mutual information minimization of an electronic device according to various embodiments of the present disclosure.
FIG. 5 is a flowchart showing steps for separating images for each fluorescent material from the acquired images of FIG. 4 .
FIGS. 6 and 7 are images for explaining the step of separating images for each fluorescent material from the acquired images of FIG. 4.
Figures 8, 9, and 10 are images for explaining the scope of application of the present disclosure.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily practice them. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part "includes" a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary. In addition, terms such as "unit", "unit", and "module" used in the specification refer to a unit that processes at least one function or operation, which is implemented as hardware, software, or a combination of hardware and software. It can be.

Although the operating entity may be omitted in the description, the method described in the present disclosure may be implemented in an electronic device, such as a device including a computing device and a fluorescence microscope.

Fluorescence imaging labels biomolecules inside a sample with various fluorophores, excites them with light, and then detects the light emitted from each fluorophore using an optical microscope to identify biomolecules inside the sample. It is a technique that can be observed indirectly. Fluorescent substances have different excitation and emission spectra due to their unique chemical structures, and always emit light with a longer wavelength than the light they absorb. The excitation and emission spectra of fluorescent materials generally have a wide width of about 100 nm within the visible light range (400 - 700 nm). In order to observe multiple biomolecules simultaneously in one sample, it is necessary to label multiple biomolecules with different fluorescent substances and then selectively obtain images of each fluorescent substance. To achieve this, it is necessary to use fluorescent materials whose excitation and emission spectra do not overlap. When four or more fluorescent materials are used simultaneously, the emission spectra overlap between the fluorescent materials due to the wide width of the emission spectrum and they become indistinguishable from each other. Therefore, in general, up to four biomolecules can be detected by exciting only one fluorescent substance per standard excitation wavelength (405, 488, 560, 633 nm).

Recently, the need to simultaneously image more biomolecules in a single sample has increased in the fields of medical diagnosis and research. However, existing fluorescence microscopy techniques have a limitation in that only up to four fluorescent substances can be used simultaneously at a time. To overcome these limitations, various technologies have been developed, and these technologies can be broadly divided into three categories.

First, multi-round staining involves imaging biomolecules inside a sample by labeling them with three or four fluorescent substances whose emission spectra do not overlap, and then deactivating the fluorescent substances through chemical treatment. Alternatively, fluorescent substances are removed from biomolecules. Afterwards, other biomolecules are labeled with the same three or four fluorescent substances and imaged. By repeating this method, it is possible to observe dozens of biomolecules simultaneously in one biological sample. However, it takes a long time because the fluorescent labeling and deactivation process must be repeated, and there is the inconvenience of having to register each image obtained during the repeated process. Additionally, due to matching problems, the distribution of multimolecules cannot be obtained in three dimensions. In addition, there is a problem of sample damage during the chemical treatment process.

Second, in the spectral imaging and signal unmixing technique, several biomolecules are individually labeled with several fluorescent substances with overlapping emission spectra, and then several fluorescent substances are excited simultaneously. Afterwards, images of the sample are obtained in several detection wavelength bands, and the obtained images are separated into images of each fluorescent material only, based on information about the relative intensity of each fluorescent material in each wavelength band. If the emission spectrum information of each fluorescent substance labeling various biomolecules is known, it is possible to separate (unmix) the images of each fluorescent substance based on the emission intensity for each wavelength of the fluorescent substance. However, to accurately calibrate the emission spectrum of a fluorescent material, expensive special equipment called a spectral detector is required. In addition, the intensity of each fluorescent material at each wavelength varies depending on the optical characteristics inside the microscope, the sensitivity of the camera at each wavelength, and the chemical composition of the sample. Accordingly, there is the inconvenience of having to measure the emission spectrum of each fluorescent substance separately for each microscope and each sample, making it very difficult to actually use it for tissue imaging.

Third, blind signal separation technique (Blind Unmixing) is a method of separating fluorescent material signals without knowing the emission spectrum of the fluorescent material. For this, Independent Component Analysis (ICA) or matrix analysis without negative values ( Non-negative matrix factorization (NMF) has been used. However, because millions of elements must be accurately inferred at the same time, the accuracy of signal separation is greatly reduced, and its use has been very limited. For example, to separate the signals of three fluorescent substances with overlapping emission spectra, more than 3,145,728 (=3 x 1024 x 1024) elements must be inferred simultaneously (based on 1024 x 1024 resolution). Additionally, existing ICA and NMF have a condition that the number of images (number of rows of the IMG matrix) required for separation must be greater than the number of fluorescent substances (number of rows of the F matrix). In other words, in order to simultaneously image eight fluorescent substances, more than nine images must be obtained, which requires expensive special equipment called a spectral detector.

In conclusion, existing multi-molecule simultaneous imaging techniques are difficult to practically utilize for research and diagnosis due to problems such as the inconvenience and complexity of the experimental and imaging process, signal separation inaccuracy, and the requirement for special equipment, so they are not actively used. To solve this problem, a new blind signal separation-based multimolecular simultaneous imaging technique (Process of ultra-multiplexed Imaging of biomoleCules viA the unmixing of the Signals of Spectrally Overlapping fluorophores; PICASSO) was developed. PICASSO uses three strategies (fluorescent material selection strategy, fluorescence detection wavelength band selection strategy, and α inference strategy by minimizing the amount of mutual information) to identify 8 fluorescent materials with much higher accuracy than existing blind signal separation techniques, with only 8 imaging times. It is a technique that can be separated at the same time. While the existing blind signal separation technique requires millions of elements to be inferred, which increases the mathematical complexity of signal separation and reduces accuracy, PICASSO uses a fluorescent material selection strategy and a fluorescence detection wavelength band selection strategy to determine the elements that need to be inferred [ It was simplified as shown in Equation 1].

As can be seen in [Equation 1] above, in PICASSO, through a selection strategy for fluorescent substances and fluorescence detection wavelength bands, the image (IMG1) obtained in the first detection wavelength band includes only the signal (F1) of the first fluorescent substance, so that the relationship of IMG1 = F1 is established. IMG2 can be expressed as the sum of F1 and F2 (IMG2 = F1 You can. In PICASSO, the value of α is accurately inferred through a strategy to minimize the amount of mutual information between IMG1 and IMG2. However, complete signal separation is achieved only when the relationship IMG1 = F1 is established, and in order to satisfy this prerequisite, a strategy for selecting fluorescent substances and fluorescence detection wavelength bands is essential. Accordingly, in order to perform imaging at the selected fluorescence detection wavelength, there is a constraint that requires a variety of optical filters optimized for the detection wavelength band or a spectral detector that can freely adjust the detection wavelength band.

Therefore, the present disclosure proposes a signal separation technique (iterative PICASSO; iPICASSO) through iterative mutual information minimization. This technique is a technique that allows signal separation without the IMG1 = F1 prerequisite through a strategy of repeatedly performing the process of minimizing the amount of mutual information.

1 is a schematic diagram illustrating a signal separation technique through iterative mutual information minimization according to the present disclosure.

Referring to Figure 1, the IMG1 = F1 precondition is solved through iterative mutual information minimization. Unlike the existing PICASSO, even if the signals of both fluorescent substances are detected in the first fluorescence detection wavelength band (blue dotted line), two images (IMG1, IMG2) obtained through the two detection wavelength bands (blue dotted line, red dotted line) using iterative mutual information minimization Signal separation is possible. At this time, as shown in (a) of Figure 1, the fluorescence emission intensity according to the emission wavelength of the two fluorescent substances (F1, F2) is different from each other, so in the two images obtained in the two detection wavelength bands, the signals of the two fluorescent substances appears at different intensities.

α, which minimizes the amount of mutual information, is calculated from IMG1 and IMG2, where the signals of the two fluorescent substances appear at different intensities, and through this, IMG1 _R1 and IMG2 _R1 are obtained. After that, if IMG1 _R1 and IMG2 _R1 are separated again using mutual information minimization, IMG1 _R2 and IMG2 _R2 can be obtained. Through the process of repeating α value inference and signal separation n times, the signals of the two fluorescent substances are gradually separated as shown in (b) of Figure 1, and the signals of the two fluorescent substances are finally separated, IMG1 _Rn = F1 and IMG2 _Rn = F2 can be obtained.

FIG. 2 is a block diagram of an electronic device 200 according to various embodiments of the present disclosure.

Referring to FIG. 2, the electronic device 200 according to various embodiments of the present disclosure includes at least one of a detector 210, an input module 220, an output module 230, a memory 240, and a processor 250. It can contain one. In some embodiments, at least one of the components of the electronic device 200 may be omitted, or one or more other components may be added to the electronic device 200.

The detector 210 can capture images of the sample. At this time, the detector 210 is installed at a predetermined location of the electronic device 200 and can capture an image. For example, the detector 210 includes at least one of a scientific complementary metal-oxide-semiconductor (sCMOS) camera, a photo multiplier tube (PMT), or other equipment that can measure the intensity of light and express it as an image. can do.

The input module 220 may receive a command or data to be used for at least one of the components of the electronic device 200 from outside the electronic device 200. At this time, the input module 220 may include at least one of an input device or a receiving device. For example, the input device may include at least one of a microphone, mouse, or keyboard. In some embodiments, the input device may include at least one of touch circuitry configured to detect a touch or a sensor circuit configured to measure the intensity of force generated by the touch. The receiving device may include at least one of a wireless receiving device or a wired receiving device.

The output module 230 may provide information to the outside of the electronic device 200. At this time, the output module 230 may include at least one of a display device or a transmission device. For example, the display device may include at least one of a display, a hologram device, or a projector. In some embodiments, the display device may be implemented as a touch screen by being assembled with at least one of the touch circuit or the sensor circuit of the input module 220. The transmission device may include at least one of a wireless transmission device or a wired transmission device.

According to one embodiment, the receiving device and the transmitting device may be integrated into one communication module. The communication module may support communication between the electronic device 200 and an external device (not shown). This communication module may include at least one of a wireless communication module or a wired communication module. At this time, the wireless communication module may be comprised of at least one of a wireless receiving device or a wireless transmitting device. Additionally, the wireless communication module may support at least one of a long-distance communication method or a short-distance communication method. The short-range communication method may include, for example, at least one of Bluetooth, WiFi direct, or infrared data association (IrDA). The wireless communication module may communicate in a long-distance communication manner through a network, and the network may include, for example, at least one of a cellular network, the Internet, or a computer network such as a local area network (LAN) or a wide area network (WAN). You can. Meanwhile, the wired communication module may be comprised of at least one of a wired receiving device or a wired transmitting device.

The memory 240 may store at least one of programs or data used by at least one of the components of the electronic device 200. For example, the memory 240 may include at least one of volatile memory and non-volatile memory.

The processor 250 may execute a program in the memory 240 to control at least one of the components of the electronic device 200 and perform data processing or calculation. The processor 250 can acquire two images (IMG1, IMG2) obtained through two detection wavelength bands. At this time, as shown in (a) of Figure 1, the fluorescence emission intensity according to the emission wavelength of the two fluorescent substances (F1, F2) is different from each other, so in the two images obtained in the two detection wavelength bands, the signals of the two fluorescent substances appears at different intensities. Then, the processor 250 calculates α, which minimizes the amount of mutual information, in IMG1 and IMG2, where the signals of the two fluorescent substances appear at different intensities, and obtains IMG1 _R1 and IMG2 _R1 through this. Afterwards, the processor 250 calculates α, which minimizes the amount of mutual information, from IMG1 _R1 and IMG2 _R1 , and obtains IMG1 _R2 and IMG2 _R2 through this. As the processor 250 repeats this process n times, the signals of the two fluorescent substances are gradually separated, as shown in (b) of FIG. 1. Finally, the processor 250 can obtain IMG1 _Rn = F1 and IMG2 _Rn = F2, which are signals separated from the two fluorescent substances.

Figure 3 is a schematic diagram to explain the advantages that can be obtained according to the present disclosure. Here, Figure 3(a) shows the operation procedure of the existing PICASSO, and Figure 3(b) shows the operation procedure of the present disclosure.

Referring to FIG. 3, compared to the existing PICASSO, the present disclosure can obtain the following advantages.

The first advantage is that selection of the fluorescence detection wavelength band is unnecessary.

Existing PICASSO has a prerequisite to set the first fluorescence detection wavelength band to include only F1, the signal of the first fluorescent substance (IMG1 = F1). Since IMG2 obtained in the second fluorescence detection wavelength band can be expressed as the sum of F1 and F2 (IMG2 = F1 )You can do it. However, there is a disadvantage that signal separation is not performed accurately if the condition IMG1 = F1 is not satisfied. Also, due to the above conditions, the wavelength band where the emission spectrum of the first fluorescent substance is maximum cannot be set as the first detection wavelength range. As a result, the signal-to-noise ratio of IMG1 deteriorates, thereby causing deterioration in image quality and inaccuracy in signal separation.

However, in the present disclosure, a signal of F2 is detected in IMG1 (IMG1 However, it has the advantage of being able to separate signals through an iterative mutual information minimization strategy. In addition, the signal-to-noise ratio of IMG1 is increased compared to the existing PICASSO, making it possible to obtain high-quality images with more accurate signal separation.

The second advantage is that the restrictive fluorescent material selection strategy is alleviated.

In order to set the fluorescence detection wavelength range so that the IMG1 = F1 condition is established, a strategy for selecting two fluorescent substances with similar emission spectra is required. In the existing PICASSO, two fluorescent materials whose maximum emission spectrum wavelengths differ by more than 20 nm are selected for each standard excitation laser. Due to this fluorescent material selection strategy, the range of the first fluorescent material detection wavelength band is set narrowly.

For example, the existing PICASSO selects two fluorescent materials (e.g., CF488A, ATTO514) whose wavelengths at the maximum emission spectrum differ by more than 20 nm, and sets the first fluorescence detection wavelength band where only the signal of the first fluorescent material appears. However, when using two fluorescent substances whose wavelengths at the maximum emission spectrum differ by less than 20 nm, it is very difficult to set the first fluorescence detection wavelength band in which only the signal of the first fluorescent substance appears, and the signal-to-noise ratio of IMG1 deteriorates, reducing image quality. This causes degradation and inaccuracy in signal separation. In the present disclosure, a signal of F2 is detected in IMG1 (IMG1 Even if this is the case, signal separation can be achieved through an iterative mutual information minimization strategy, so the fluorescent material selection strategy can be relaxed. In other words, signal separation is possible even when using two fluorescent materials whose wavelengths at the maximum emission spectrum differ by less than 20 nm. In addition, the signal-to-noise ratio of IMG1 is increased compared to the existing PICASSO, making it possible to obtain high-quality images with more accurate signal separation.

FIG. 4 is a flowchart of a signal separation method through iterative mutual information minimization of the electronic device 200 according to various embodiments of the present disclosure.

Referring to FIG. 4, the electronic device 200 may acquire two images for two fluorescent substances each labeled with different biomolecules in step 410. At this time, the processor 250 may respectively acquire the two images in different detection wavelength bands that are distinct from each other. Here, in each of the acquired images, both of the above fluorescent substances may appear. That is, the fluorescent materials each have different fluorescence emission intensities, and in the acquired images, the fluorescent materials may appear with different signal intensities. The processor 250 may define the two acquired images as two new variables, as shown in Equation 2 below.

Here, u1 and u2 may represent acquired images.

Next, the electronic device 200 may separate images for each fluorescent material from the acquired images while repeatedly reducing the amount of mutual information shared between the acquired images in step 420. To this end, the processor 250 may repeatedly perform an operation of acquiring two new images that are updated from the acquired images by applying a variable for minimizing the amount of mutual information calculated for the acquired images. According to one embodiment, this operation may be repeated a predetermined number of times, that is, N times. According to another embodiment, this operation may be repeated until the amount of mutual information between the acquired images becomes less than a reference value. When repetition of this operation is completed, the processor 250 can acquire the finally obtained images as images for each of the fluorescent substances. This will be described in more detail later with reference to FIGS. 5 and 6.

Here, the amount of mutual information is a value derived from information theory, and the amount of mutual information between two variables can refer to the total amount of information shared by the two variables. Therefore, the amount of mutual information between two random variables may be 0. Since a digital image is a discrete variable, the amount of mutual information between two images can be calculated as shown in [Equation 3] below.

_{Here , p} histogram).

FIG. 5 is a flowchart specifically illustrating the step 420 of separating images for each fluorescent material from the acquired images of FIG. 4 . FIGS. 6 and 7 are images for explaining the step of separating images for each fluorescent material from the acquired images of FIG. 4 (step 420).

Referring to FIG. 5, the electronic device 200 may optionally process the images acquired in step 521 into low-resolution images. According to one embodiment, step 521 may be omitted. According to another embodiment, in step 521, the processor 250 may process the acquired images into low-resolution images with a lower resolution than the acquired images, if necessary, as shown in Equation 4 below. As an example, the processor 250 may obtain a low-resolution image with a resolution k times lower by adding the values of adjacent k × k pixels within each acquired image. Through this, the computation speed in subsequent steps can be increased and the ability to respond to noise can be increased. As another example, the processor 250 may omit k=1 or use another image sampling method to process the acquired images into low-resolution images.

Here, u1 and u2 represent acquired images, v1 and v2 represent low-resolution images, and k can be defined as a factor applied to process the acquired images into low-resolution images, respectively.

The electronic device 200 may calculate variables from acquired images or low-resolution images using a loss function in step 523. According to one embodiment, if step 521 is omitted, the processor 250 may calculate variables from the acquired images. According to another embodiment, after step 521 is performed, the processor 250 may calculate variables from low-resolution images. At this time, the loss function can be defined as shown in [Equation 5] below. Additionally, the processor 250 can use a loss function to calculate variables from low-resolution images as shown in Equation 6 below. In step 523, the processor 250 may quantize the acquired images or low-resolution images, if necessary, before calculating the loss function.

here, and can represent a variable.

The electronic device 200 may apply variables to the acquired images in step 525 to obtain two new images updated from the acquired images. At this time, the processor 250 may obtain new updated images from the acquired images by applying a variable and an update rate (λ) to the acquired images, as shown in Equation 7 below. At this time, the update rate (λ) may have a value between 0 and 1. Images acquired in step 525 may be high-resolution images.

In the above description, steps 523 and 525 are performed simultaneously on the two acquired images, but are not limited thereto. That is, steps 523 and 525 may be sequentially performed on the acquired images. According to one embodiment, when step 521 is omitted, steps 523 and 525 may be sequentially performed on the acquired images. According to another embodiment, two variables ( , ) can be performed sequentially. Specifically, through step 521, low-resolution images are obtained, and from these images, one variable (e.g. ), and obtain an updated high-resolution image (e.g., u1) of one of the two images (e.g., v1) with this value. Then, this updated high-resolution image is passed through step 521 to obtain a low-resolution image (e.g. v1), and steps 523 and 525 are performed on this low-resolution image and one other image (e.g. v2) to obtain another variable ( ) and use it to update the high-resolution image (e.g., u2) of another image (e.g., v2). This process can be performed repeatedly and then proceed to step 527.

The electronic device 200 may determine whether to repeat previous steps in step 527. According to one embodiment, the processor 250 may determine whether the number of repetitions of steps 521 to 525 has reached a predetermined number, that is, N times. Here, if it is determined that the number of repetitions has not reached the set number, the processor 250 determines that the previous steps must be repeated, returns to step 521, and repeats steps 521 to 525. Meanwhile, if it is determined that the number of repetitions has reached a predetermined number, the processor 250 may determine that there is no need to repeat the previous steps and proceed to step 529. According to another embodiment, the processor 250 may determine whether the amount of mutual information between the images obtained in step 525 is less than or equal to a reference value. Here, if it is determined that the amount of mutual information between the acquired images exceeds the reference value, the processor 250 determines that the previous steps must be repeated, returns to step 521, and repeats steps 521 to 525. there is. Meanwhile, if it is determined that the amount of mutual information between the acquired images is less than the reference value, the processor 250 determines that there is no need to repeat the previous steps and may proceed to step 529.

Through this, as steps 521 to 525 are repeated, the signals of the fluorescent substances can be gradually separated, as shown in FIG. 6. As a result, the electronic device 200 can acquire the images finally obtained in step 529 as images for each of the fluorescent substances. At this time, the processor 250 acquires images with the minimum mutual information amount as the final solution, as shown in Equation 8 below, and these images may each represent different fluorescent substances.

According to the present disclosure, image separation performance can be improved. The existing PICASSO technology drastically reduced the number of elements that must be inferred for the signal separation technique through a strategy of selecting a specific fluorescent material and selecting a fluorescence detection wavelength band, and then accurately inferred them by minimizing the amount of mutual information. In the present disclosure, the strategy of selecting a specific fluorescent material and selecting a fluorescence detection wavelength band can be eliminated and alleviated through an iterative mutual information minimization strategy. In addition, in the present disclosure, the signals of the two fluorescent substances can be successfully separated through step-by-step α inference and gradual signal separation in two images in which the signals of the two fluorescent substances are detected at different intensities. Therefore, the present disclosure can be said to be novel compared to the existing PICASSO technology.

In addition, as a result of the present disclosure, as shown in FIG. 7, it is possible to successfully separate signals of two fluorescent substances whose wavelengths at the maximum emission spectrum differ by less than 20 nm, which is difficult to achieve in the existing PICASSO. In the present disclosure, through an iterative mutual information minimization strategy, it is possible to successfully separate signals of two fluorescent substances with similar emission spectra without optimization of the fluorescence detection wavelength band. In this disclosure, by excluding the fluorescence detection wavelength band selection strategy, the range of the first fluorescence detection wavelength band can be set widely, increasing the signal-to-noise ratio, and thus obtaining high-quality signal separation images compared to the existing PICASSO. Additionally, in the present disclosure, since there is no need to select a wavelength band specialized for fluorescent materials, hardware requirements are reduced and the selection of the emission filter can be freely performed. Therefore, the present disclosure can be said to be advanced compared to the existing PICASSO technology.

Figures 8, 9, and 10 are images for explaining the scope of application of the present disclosure.

The present disclosure is applicable to a wider range by removing and relaxing the prerequisites of the existing PICASSO for signal separation.

For example, biomolecules applicable to the present disclosure include cultured cells, animal tissue fragments, clinical samples, etc. As another example, the present disclosure enables simultaneous imaging of proteins labeled with multiple types of antibodies or mRNA labeled with DNA. As another example, the present disclosure is not limited to the type of sample or type of biomolecule, and can be used to image all biomolecules that can be labeled with a fluorescent substance in all types of samples in which a fluorescent substance can be used.

In order to observe biomolecules inside tissues with a fluorescence microscope, it is necessary to label specific biomolecules inside the tissues with a fluorescent substance. Generally, to observe proteins, an antibody with a fluorescent substance attached can be used, and when observing mRNA, an oligonucleotide with a fluorescent substance attached can be used. In the present disclosure, antibodies were used to observe proteins, and oligonucleotides were used to observe mRNA. However, the present disclosure is a technology for distinguishing fluorescent substances, and includes all labeling methods using fluorescent substances or all labels with fluorescent substances attached. Applicable to molecules.

The present disclosure can be implemented by a point-scanning confocal microscope equipped with a spectral detector, as shown in Figure 8, and as shown in Figure 9, a filter-type rotating disk ball. It can be implemented by spinning-disk confocal microscopy. In addition, as shown in Figure 10, the present disclosure is implemented with a filter-type point scan confocal microscope, a filter-type fluorescence microscope (widefield microscope), and any microscope that can selectively detect only light in a specific wavelength range. possible.

In the present disclosure, a total of 10 fluorescent substances can be imaged simultaneously by using eight fluorescent substances with overlapping emission spectra and two large Stokes shift fluorescent substances. The present disclosure is not limited to 10 fluorescent materials and can be used for all fluorescent materials. The present disclosure can separate signals of fluorescent proteins whose emission spectra overlap, or between a fluorescent protein and a fluorescent substance. This disclosure eliminates the existing PICASSO fluorescence detection wavelength selection strategy through α inference and gradual signal separation through iterative mutual information minimization. Therefore, it can be used more universally in spectral detectors and filter-type microscopes with adjustable detection wavelengths. The present disclosure can also be combined with repeated dyeing techniques and can be applied to all dyeing techniques using fluorescent substances. Additionally, the present disclosure can be applied to various imaging technologies, such as tissue transparency technology, super-resolution imaging technology, etc.

The embodiments of the present disclosure described above are not only implemented through devices and methods, but may also be implemented through programs that implement functions corresponding to the configurations of the embodiments of the present disclosure or recording media on which the programs are recorded.

Although the embodiments of the present disclosure have been described in detail above, the scope of the rights of the present disclosure is not limited thereto, and various modifications and improvements using the basic concept of the present disclosure defined in the following claims are also included in the scope of the rights of the present disclosure. belongs to

Claims (20)

In a method of operating an electronic device,
Acquiring two images for two fluorescent substances each labeled with different biomolecules; and
Separating images for each of the fluorescent substances from the acquired images while iteratively reducing the amount of mutual information shared between the acquired images.
Including,
The step of separating images for each of the fluorescent substances is,
Obtaining two new images updated from the acquired images by applying a variable for minimizing the amount of mutual information calculated for the acquired images.
Includes, whereby the acquired images are obtained as images for each of the fluorescent substances,
The step of acquiring new images updated from the acquired images is,
Processing each of the acquired images into low-resolution images with lower resolution than the acquired images as shown in [Equation i] below;
[Equation i]


Here, the u1 and the u2 represent the acquired images, the v1 and the v2 represent the low-resolution images, and the k represents a factor applied to respectively process the acquired images into the low-resolution images,
Calculating variables from the low-resolution images as shown in [Equation iii] below, using a loss function defined as [Equation ii] below; and
[Equation ii]


[Equation iii]


Here, the above and above represents the above variable,
Obtaining new updated images from the acquired images by applying the variable and update rate (λ) to the acquired images as shown in [Equation iv] below.
[Equation iv]


Including,
method.
According to claim 1,
The step of acquiring the images is,
Acquiring each of the two images at different detection wavelengths distinct from each other.
Including,
method.
According to claim 1,
In each of the acquired images,
Both of the above fluorescent substances appear,
method.
According to claim 3,
The fluorescent substances are,
each having different fluorescence emission intensities,
In the acquired images,
wherein the fluorescent substances appear at different signal intensities,
method.
delete According to claim 1,
The step of acquiring new images updated from the acquired images is,
Repeated a predetermined number of times,
When the repetition of the step of acquiring new images from the acquired images is completed, the acquired images are obtained as images for each of the fluorescent substances,
method.
According to claim 1,
The step of acquiring new images updated from the acquired images is,
This is repeated until the amount of mutual information between the acquired images becomes less than the reference value,
When the repetition of the step of acquiring new images from the acquired images is completed, the acquired images are obtained as images for each of the fluorescent substances,
method.
delete In electronic devices,
Memory; and
a processor coupled to the memory and configured to execute at least one instruction stored in the memory;
The processor,
For two fluorescent substances each labeled with a different biomolecule, two images are acquired,
configured to separate images for each of the fluorescent substances from the acquired images while iteratively reducing the amount of mutual information shared between the acquired images,
The processor,
configured to perform an operation of obtaining two new images updated from the acquired images by applying a variable for minimizing the amount of mutual information calculated for the acquired images,
As a result, the acquired images are obtained as images for each of the fluorescent substances,
When performing the operation, the processor
Each of the acquired images is processed into low-resolution images with lower resolution than the acquired images, as shown in [Equation i] below,
[Equation i]


Here, the u1 and the u2 represent the acquired images, the v1 and the v2 represent the low-resolution images, and the k represents a factor applied to respectively process the acquired images into the low-resolution images,
Using a loss function defined as in [Equation ii] below, variables are calculated from the low-resolution images as in [Equation iii] below,
[Equation ii]


[Equation iii]


Here, the above and above represents the above variable,
Configured to obtain new updated images from the acquired images by applying the variable and update rate (λ) to the acquired images as shown in [Equation iv] below,
[Equation iv]


Electronic devices.
According to clause 9,
The processor,
configured to each acquire said two images at different detection wavelength bands distinct from each other,
Electronic devices.
According to clause 9,
In each of the acquired images,
Both of the above fluorescent substances appear,
Electronic devices.
According to claim 11,
The fluorescent substances are,
each having different fluorescence emission intensities,
In the obtained images,
wherein the fluorescent substances appear at different signal intensities,
Electronic devices.
delete According to clause 9,
The processor,
Configured to repeat the operation a predetermined number of times,
When the operation is completed, the acquired images are obtained as images for each of the fluorescent substances,
Electronic devices.
According to clause 9,
The processor,
Configured to repeat the operation until the amount of mutual information between the acquired images becomes less than a reference value,
When the operation is completed, the acquired images are obtained as images for each of the fluorescent substances,
Electronic devices.
delete In a non-transitory computer-readable storage medium,
Acquiring two images for two fluorescent substances each labeled with different biomolecules; and
Separating images for each of the fluorescent substances from the acquired images while iteratively reducing the amount of mutual information shared between the acquired images.
Store one or more programs for executing a method including,
The step of separating images for each of the fluorescent substances is,
Obtaining two new images updated from the acquired images by applying a variable for minimizing the amount of mutual information calculated for the acquired images.
Includes, whereby the acquired images are obtained as images for each of the fluorescent substances,
The step of acquiring new images updated from the acquired images is,
Processing each of the acquired images into low-resolution images with lower resolution than the acquired images as shown in [Equation i] below;
[Equation i]


Here, the u1 and the u2 represent the acquired images, the v1 and the v2 represent the low-resolution images, and the k represents a factor applied to respectively process the acquired images into the low-resolution images,
Calculating variables from the low-resolution images as shown in [Equation iii] below, using a loss function defined as [Equation ii] below; and
[Equation ii]


[Equation iii]


Here, the above and above represents the above variable,
Obtaining new updated images from the acquired images by applying the variable and update rate (λ) to the acquired images as shown in [Equation iv] below.
[Equation iv]


Including,
Non-transitory computer-readable storage media.
According to claim 17,
The step of acquiring the images is,
Acquiring each of the two images at different detection wavelengths distinct from each other.
Including,
Non-transitory computer-readable storage media.
delete According to claim 17,
The step of acquiring new images updated from the acquired images is,
This is repeated a predetermined number of times or until the amount of mutual information between the acquired images falls below a reference value,
When the repetition of the step of acquiring new images from the acquired images is completed, the acquired images are obtained as images for each of the fluorescent substances,
Non-transitory computer-readable storage media.