KR102562741B1 - Method and apparatus for diagnosing malaria infection from red blood cells using artificial intelligence - Google Patents

Abstract

An apparatus for automatically diagnosing malaria infection according to an embodiment of the present disclosure includes staining a red blood cell sample to obtain a blood cell image for at least a portion of the red blood cell sample, identifying highly stained blood cell images excluding abnormal staining images among the acquired blood cell images, and , a processor for identifying malaria infection images among blood cell images identified through artificial intelligence, and classifying infected malaria species among malaria infection images; and displaying the species of infected malaria, displaying the malaria infection rate based on the identified malaria infection image, and displaying the patient's condition corresponding to the red blood cell sample according to a predetermined level standard based on the infected species and infection rate of malaria. Includes a display that shows

Description

Method and apparatus for diagnosing malaria infection from red blood cells using artificial intelligence using artificial intelligence

The present disclosure relates to a malaria infection diagnosis method and apparatus for precisely diagnosing malaria infection in a red blood cell sample through artificial intelligence and displaying the result for easy recognition by a user.

Malaria is one of the world's three leading infectious diseases and causes 400,000 deaths every year. In order to diagnose malaria infection, it is necessary to examine red blood cells under a microscope to determine whether or not they are infected with malaria. In addition, there is a problem that the test result for malaria infection diagnosis is different depending on the tester's skill level due to the difference in the skill level of the tester, and even if the tester is the same, the test result may be different depending on the tester's condition.

For rapid and accurate determination of malaria infection, rapid methods and devices that detect and display malaria infection rates and malaria species through image processing and artificial intelligence of at least a portion of the entire red blood cell sample are needed.

A method for automatically diagnosing malaria infection according to an embodiment of the present disclosure includes obtaining a blood cell image of at least a portion of a red blood cell sample by smearing, fixing, and staining at least a portion of the red blood cell sample. Also, the method for automatically diagnosing malaria infection according to an embodiment includes identifying blood cell images with high staining, excluding abnormal staining images, among the obtained blood cell images. A method for automatically diagnosing malaria infection according to an embodiment includes identifying a malaria infection image among blood cell images identified through artificial intelligence. A method for automatically diagnosing malaria infection according to an embodiment includes the step of distinguishing and displaying the infected species of malaria from the malaria infection image. The method for automatically diagnosing malaria infection according to an embodiment may further include displaying a malaria infection rate on a user output interface based on the identified malaria infection image, and collecting red blood cells according to a predetermined level standard based on the infected species of malaria and the infection rate. and displaying the patient's condition corresponding to on the user output interface.

An automatic diagnosis device for malaria infection according to an embodiment of the present disclosure obtains a blood cell image of at least a part of the red blood cell sample by smearing, fixing, and staining a red blood cell sample, and obtains a highly stained blood cell image excluding abnormal staining images among the acquired blood cell images. a processor for identifying a malaria infection image among the blood cell images identified through artificial intelligence, and classifying the species of malaria infected among the malaria infection images; and displaying the species of infected malaria, displaying the malaria infection rate based on the identified malaria infection image, and displaying the patient's condition corresponding to the red blood cell sample according to a predetermined level standard based on the infected species and infection rate of malaria. Includes a display that shows

According to the present disclosure, it is possible to quickly detect the malaria infection rate and malaria species of the entire red blood cell sample to be tested through image processing and artificial intelligence of the red blood cell sample, and display the infection rate and infected malaria species.

1 is a diagram illustrating a blood smear method according to a blood smear examination process.
2 is a flowchart of a method for diagnosing malaria infection according to an embodiment of the present disclosure.
3 is blood cell images according to the degree of staining according to an embodiment of the present disclosure.
4 is an example showing an image of red blood cells with unclear boundaries according to an embodiment of the present disclosure.
5A is a diagram illustrating acquisition of a plurality of blood cell images along the z-axis according to an embodiment of the present disclosure.
5B illustrates a plurality of blood cell images obtained along the z-axis according to an embodiment of the present disclosure.
6 is a graph of luminance values of multiple images of blood cells according to an embodiment of the present disclosure.
7 is a diagram illustrating a histogram of a plurality of images according to an embodiment of the present disclosure.
8 is a diagram illustrating identification of highly stained blood cell images excluding abnormal staining images among blood cell images according to an embodiment of the present disclosure.
9 is a user output interface showing red blood cell image classification according to staining evaluation according to an embodiment of the present disclosure.
10 is a structural diagram of ResNet-50 for automatically classifying malaria infection images according to an embodiment of the present disclosure.
11 is a block diagram showing an apparatus for diagnosing malaria infection according to an embodiment of the present disclosure.
12 is a structural diagram of a U-net used for cell segmentation according to an embodiment of the present disclosure.

Terms used in the present disclosure will be briefly described, and an embodiment of the present disclosure will be described in detail.

The terms used in the present disclosure have been selected from general terms that are currently widely used as much as possible while considering functions in an embodiment of the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. there is. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding embodiment of the present disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

When it is said that a certain part "includes" a certain element throughout the present disclosure, it means that it may further include other elements, not excluding other elements unless otherwise stated. In addition, terms such as "unit" and "module" described in this disclosure refer to a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. there is.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, an embodiment of the present disclosure may be implemented in many different forms and is not limited to the embodiment described herein. And in order to clearly describe an embodiment of the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the present disclosure.

A method and apparatus for automatically diagnosing malaria infection according to an embodiment of the present disclosure include a method for providing a user output interface so that a tester can easily determine whether or not there is malaria infection, infection rate, classification of malaria species, and patient condition according to infection rate; and It is a device. According to an embodiment of the present disclosure, an artificial intelligence-based automatic malaria infection diagnosis device performs a blood smear test, removes abnormal images from blood cell images to increase the accuracy of infection, and allows an examiner to easily check the diagnosis result. Provides a user output interface for

In one embodiment according to the present disclosure, in order to increase the accuracy of determining malaria infection in an image of a red blood cell by artificial intelligence, the blood cell image is divided into a normal stained image and an abnormal image according to a predetermined pattern - dark staining, poor staining color, partial staining, and cell After classification as falling, etc., only normal images are selected using ISP (Image Signal Processing) techniques and passed to an artificial intelligence classifier. The artificial intelligence classifier classifies the selected blood cell images into malaria-negative images and malaria-positive images, counts infected red blood cells, and calculates the malaria infection rate. When the malaria infection rate is derived, the automatic diagnosis device for malaria infection according to an embodiment of the present disclosure informs the examiner of the patient's current status - malaria infection according to the malaria symptom criteria of the Centers for Disease Control and Prevention (CDC). Whether and/or infection rate - can be provided.

In the blood smear test, which is mainly used for malaria testing, a method of loading a patient's red blood cells on a slide and smearing the blood is mainly used.

1 is a diagram illustrating a blood smear method according to a blood smear examination process.

The blood smear process is automated in the malaria diagnostic device. That is, a series of processes in which a sample is smeared, fixed, and stained does not require manual intervention by the user and is automatically performed by the malaria diagnosis device.

Referring to FIG. 1 , in the blood smear test process, first, in a state where a specimen T is placed on a slide S, another slide is brought into contact with the slide S on which the specimen T is placed so as to form a predetermined angle. Thereafter, when the operator slides the slide S on which the specimen T is placed while bringing the end of another slide into contact with the specimen T, the specimen T can be smeared while being spread on the slide S. In order to smear the sample T in a desired shape (eg, monolayer), it is necessary to appropriately adjust the angle between the slides and the sliding speed.

However, the blood smearing method according to FIG. 1 is merely an example of an existing method manually performed by a user, and in the malaria diagnosis device according to the present disclosure, the smearing process shown in FIG. 1 can be automatically performed in the diagnosis device. In the conventional blood smear test as shown in FIG. 1, the process of blood smear, staining, and microscopic observation is dependent on the manual work of the inspector, so that if the inspector is not an experienced inspector, the blood smear state may become non-uniform or the reaction conditions in the blood staining process It is difficult to perform a speculum due to contamination of the sample due to errors in the sample, and it is difficult to perform an accurate test if the examiner's skill is insufficient.

In addition, when diagnosing a patient sample in the early stage of malaria infection, a lot of time is inevitably required because the examiner must individually determine whether or not the red blood cell image is infected with malaria. Therefore, for early treatment, the examiner has no choice but to determine malaria infection by sampling in which only tens to hundreds of red blood cells are selected from among hundreds of thousands of red blood cells contained in the sample. It is also not easy to determine which species of malaria is infected by looking at the sample. Therefore, long-term training and experience are required for a tester to determine the infection rate and the infected malaria species while checking a red blood cell sample through a microscope. Therefore, there is a need for a method and apparatus for automatically diagnosing malaria infection that can quickly and accurately test hundreds of thousands of red blood cells included in a red blood cell sample to derive an accurate malaria infection rate and select which malaria species are infected. In order to achieve such rapid and accurate malaria diagnosis, it automatically extracts malaria-infected images in a short period of time while having constant staining, classifies malaria species, displays them on a display, which is a user output interface, and shows the degree of infection and the patient's condition. An automated user interface (UI) device is required. Throughout this disclosure 'red blood cell' will also be used simply as the term 'blood cell'.

2 is a flowchart of a method for diagnosing malaria infection according to an embodiment of the present disclosure.

In step 202, the apparatus for automatically diagnosing malaria infection stains the red blood cell sample to obtain a blood cell image of at least a part of the red blood cell sample. The processor that controls the entire automatic malaria diagnosis device can perform image processing and operations performed by artificial intelligence. The processor may be an artificial intelligence processor and may include a plurality of processors. In order to smear the red blood cell sample, the automatic diagnosis device for malaria infection may load the red blood cell sample on a slide and automatically smear the red blood cell sample through the sliding glass.

In step 204, the processor of the apparatus for automatically diagnosing malaria infection evaluates whether abnormal staining has occurred in the obtained blood cell image.

In one embodiment, in order to identify the blood cell image with high staining, the processor may remove regions with low staining in the sample of stained red blood cells. In one embodiment, excluding the abnormal staining image from the stained red blood cell image is a normal staining image from the red blood cell image, a staining image of a predetermined level or more intensity, a staining color defect image, a partially stained image, an image with unclear borders, or and classifying into cell drop images.

Referring to FIG. 3 to distinguish a blood cell image with high staining and an image with low staining according to an embodiment of the present disclosure.

3 is blood cell images according to the degree of staining according to an embodiment of the present disclosure.

Referring to FIG. 3 , 301 shows a blood cell image stained normally. On the other hand, 303 to 313 show images with low dyeability as an example. In the case of 303, it is an image strongly dyed with a dyeing intensity higher than a predetermined level. In the case of 305, it shows an image with poor dyeing color. 307 is a partially stained image in which only a part of the entire image is stained. 309 is a blood cell image with cell detachment. In the case of 311, it is a blood cell image showing an abnormal vacuole. 313 shows an image in which the boundary of the blood cell image is unclear.

Here, a method of controlling autofocus in order to obtain a properly stained blood cell image when a blood cell image includes an image with unclear boundaries is presented. A blood cell image with clear boundaries of the blood cell image may be acquired through autofocus according to an embodiment of the present disclosure. In order to acquire a blood cell image with clear boundaries, it is necessary to acquire a blood cell image through best focus when capturing an image. If the focus is not optimal, the boundary of the red blood cells is not clear, and a white border stands out inside the boundary.

4 is an example showing an image of red blood cells with unclear boundaries according to an embodiment of the present disclosure.

Referring to FIG. 4 , in the case of the 410 image, it can be seen that the left and right blood cell image focusing in the image is not constant due to the light source being turned. Usually, this phenomenon is often caused by a problem in the device itself for capturing images. In the case of the 420 image, it is an image that occurs when there is a problem with the smear or the blood itself during the sample prep process. It is unclear.

Reference is made to FIG. 5 to describe a method of acquiring a blood cell image with clear borders of the blood cell image.

5A is a diagram illustrating acquisition of a plurality of blood cell images along the Z-axis according to an embodiment of the present disclosure.

Referring to FIG. 5A , a camera 510 acquires an image while moving up and down along the z-axis in order to obtain an image of a dyed area fixed by being smeared on a slide glass 530 from an objective lens 520 . The dyed region may be identified more brightly through the light source module 540 for illuminating the image.

In order for the processor to obtain blood cell images, it is necessary to select an optimal lens to be used in the image spectrometer (not shown). Lenses for image spectroscopy optimize physical and digital image parameters so that optical magnification, including digital magnification, is possible over hundreds of times. According to one embodiment, the lens for image spectroscopy can magnify 200 times, 400 times, and 500 times, but is not limited thereto.

In order to obtain an optimal focus image with clear borders of the blood cell image, as shown in the right side of FIG. 5 , multiple red blood cell images are acquired in the z-axis direction when the horizontal axis of the cartridge 530 is the x-axis and the vertical axis is the y-axis. In other words, a plurality of images are acquired while changing the depth in the direction of the tomogram in which red blood cells are smeared and stacked (vertical direction of the slide). Since the thickness of red blood cells is about 5 μm, an optimal focus point can be determined by acquiring, for example, 10 images for each depth within 5 μm.

Auto focus data values used in auto focus use data values obtained by extracting edge components of each blood cell using a laplacian filter. This method has a disadvantage in that an image in which the edge value of the red blood cell image with unclear boundaries does not correspond to the optimal focus is determined as the optimal focus. However, this method has an advantage that the focus data changes linearly. For a more detailed description, reference is made to FIG. 6 .

5B illustrates a plurality of blood cell images obtained along the z-axis according to an embodiment of the present disclosure.

Referring to FIG. 5B , it can be seen that 10 blood cell images were acquired along the z-axis according to the method of FIG. 5A according to an embodiment of the present disclosure. 5B shows that the sixth image is an optimal focus image.

6 is a graph of luminance values of multiple images of blood cells according to an embodiment of the present disclosure.

In FIG. 6 , 610 and 620 are one image among a plurality of images when a plurality of images are obtained while changing a depth in the direction of a tomographic layer in which red blood cells are accumulated, according to an embodiment of the present disclosure. For example, when a plurality of images are acquired by changing from a low depth to a deep depth in step 10, 610 is a graph representing the luminance value of the 5th red blood cell image and 620 is a graph representing the luminance value of the 8th red blood cell image. Referring to image 610, 610 is a graph in which luminance values are acquired while following a specific line horizontally or vertically in a captured red blood cell image. When the red blood cell boundary line is encountered, the value becomes dark and the value drops downward. It can be seen that the boundary line of the 610 image is relatively darker than that of the 620 image. Accordingly, 620 is a graph corresponding to a red blood cell image with relatively optimal focus compared to 610 . In fact, it can be seen from FIG. 6 that the boundary of the red blood cells in the image 621 is clearer and clearer than in the image 611.

Accordingly, while obtaining a plurality of images of the red blood cells smeared on the cartridge in the vertical direction of the cartridge slide by this method, it is possible to identify an image corresponding to an optimal focus by converting the luminance value of each image into a graph.

In one embodiment, the higher the contrast (contrast) in each of the plurality of images, the image by optimal focus. This is because if the contrast is high, the border edges of the red blood cells are sharp. Accordingly, in order to obtain an image of a red blood cell with an optimal focus, a red blood cell image with clear boundaries may be obtained by intentionally increasing the contrast. In this case, an edge component of the red blood cell image may be extracted to obtain a red blood cell image with clear boundaries. In another embodiment, an image with an optimal focus may be identified by analyzing a histogram of the red blood cell image instead of using the edge of the red blood cell image. Referring to FIG. 7, a method of identifying an image with an optimal focus by analyzing the histogram will be described in more detail.

7 is a diagram illustrating a histogram of a plurality of images according to an embodiment of the present disclosure.

7 shows a histogram graph of a plurality of images captured for each depth of the red blood cell image. The histogram is a graph showing the cumulative number as a distribution diagram by counting the number of RGB values (0 to 255) for each of a plurality of images. In the 710 and 720 histogram graphs, the largest value on the y-axis is 255. In the 710 and 720 histogram graphs, the closest point to 255 is the area corresponding to the background. Next, a value close to 255 corresponds to the foreground. At this time, the bin value of the foreground in the 720 histogram is, for example, 95745, and the bin value of the foreground in the 710 histogram. With this 63596, the foreground bins of the 720 histogram are larger. In this case, the red blood cell image corresponding to the 720 histogram has a higher 'contrast' than the image corresponding to the 710, and thus can be selected as an image corresponding to the optimal focus. The histogram method for obtaining optimal focus data (blood cell image data having sharp boundaries) has a disadvantage in that the focus data changes nonlinearly, but has the advantage of being able to find an image similar to a human focus.

Therefore, it is possible to extract an image whose boundary is clear by optimal focus by improving an image with an unclear boundary by extracting optimal focus data using the histogram graph shown in FIG. 7 instead of the edge. In this case, when a plurality of images are detected, a plurality of images may be detected using a Laplacian filter, similarly to the edge method.

According to an embodiment of the present disclosure, a method using a Laplacian filter and an edge and a method using a histogram may be used together to obtain a red blood cell image with clear boundaries. In other words, a plurality of (eg, 10) blood cell images are acquired for each depth using a Laplacian filter to detect 10 focus images, and to find an image with the highest focus among the 10 images detected through a histogram graph , , and 10 images are extracted, 5 each up and down, based on the image with the highest focus, and used as red blood cell images.

So far, a method for obtaining an image with clear borders of red blood cells, not an image with unclear borders, has been described. Next, a more detailed method of identifying blood cell images with high staining characteristics, excluding abnormal images, among acquired blood cell images will be described with reference to FIG. 8 .

8 is a diagram illustrating identification of highly stained blood cell images excluding abnormal staining images among blood cell images according to an embodiment of the present disclosure.

Reference numeral 810 shows a blood cell image of at least a portion of a red blood cell sample obtained through an automatic diagnosis device for malaria infection according to an embodiment of the present disclosure.

820 is an image obtained by removing a background from a blood cell image obtained according to an embodiment of the present disclosure. For object (blood cell) analysis, it is necessary to remove the background from the blood cell image. To remove the background, use the following method. First, the brightness value of the image is obtained using the histogram of the acquired blood cell image. Based on the brightness value of the image obtained through a predetermined formula, all values higher than a predetermined level are determined as background values, and the corresponding areas are removed. For example, the blue gain for the brightness value of a pixel in an image is calculated by the following formula.

Blue gain = 256 / (background blue brightness value * 0.95) ... Equation 1

The blue gain obtained here is multiplied by the blue pixel value of RGB for each pixel, and if the value is 256 or more, it is determined as the background.

For example, if the image background blue brightness value is 215 and the blue pixel value of a specific image area is 210, the blue gain is derived as 1.2534, and since blue gain * 210 = 264.2 (>256), the corresponding area becomes the background area. As another example, if the blue pixel value of a specific image area is 180, the blue gain is still 1.2534, and blue gain * 180 = 225 (< 256), so it is determined as the red blood cell area rather than the background area.

By the above calculation, it is determined which area of the red blood cell image is the background area or the red blood cell area, and the area determined as the background area is removed.

830 is an image obtained by removing the border of the blood cell image from which the background image is removed in 820 according to an embodiment of the present disclosure. In the blood cell image, the border image of the blood cell is removed using a morphology closing technique. In order to use the morphological closing technique, the processor of the automatic diagnosis device for malaria infection first detects the edge of each blood cell image in the red blood cell image and applies the morphological closing technique.

840 is an image showing whether or not a blood cell image with high staining is determined by numerically evaluating the staining of the border-removed blood cell image. In one embodiment, the area of the image in the 840 image from which the border is removed is divided into a predetermined number of blocks (for example, 54 blocks), and RGB representative values are extracted from each block. Although there are various methods for selecting a representative value, a value with the highest frequency can be selected as an RGB representative value through a histogram analysis according to RGB values in each block. Calculate the dyeability value according to the selected representative RGB values. A method for calculating the dyeability value is as follows.

No stain = ((G-B)/(R-B))*5000 ... Equation 2

'No stain' quantifies how close the color is to yellow because the red blood cell image has a slightly darker yellow color than blood plasma, although it may be strong or weak depending on the hemoglobin level if the red blood cell image is not stained. One is the 'No stain' value. If No stain > 3120, it is judged as a staining error according to 'No stain'. The closer the No stain value according to Equation 2 is to 5000, the closer to yellow, the closer to 0, the red, and the closer to 0, the closer to magenta.

Similarly, the dyeing error is judged by the following judgment.

Over Methylene Blue = ((R/B)*512)*(R-B) ... Equation 3

Methylene Blue is a staining error determination formula when red blood cells are dyed blue, and it is a staining in which the blue component among RGB becomes stronger. If the result value according to Equation 3 is Over Methylene Blue < 12345, it is determined as a staining error by Over Methylene Blue.

Over Eosin = (R/G*512)*(R-G) ... Equation 4

Over Eosin stains red blood cells in red, and the red component of RGB becomes stronger. If (Over Eosin > 35000) & (Over Methylene Blue > 35000) by calculation of Equations 3 and 4, it is determined as a staining error due to Over Eosin. Also, if Over Eosin < 5900, it is judged as an unknown stain.

Over stain is a case of excessive staining of Methylene Blue and Eosin. If Over Eosin > 47000, it is judged as over stain and it is judged as a staining error.

In addition, if the red blood cell distribution (pixel) against the background in each block is 7.2% or less, this is called Low RBC (red blood cell) and is not included in the staining evaluation and is ignored. Likewise, if no red blood cell distribution is found against the background in each block, it is called No RBC (red blood cell) and is not included in the staining evaluation and is ignored.

Table 1 below is an image classification table for each red blood cell according to staining evaluation.

Dyeability evaluation classification normal normal absolute normal abnormal Over Methylene Blue Over Eosin Over stain No stain ignore No RBCs Low RBCs

Returning to FIG. 2, a method for diagnosing malaria infection will be further described.

In step 206, the processor of the apparatus for diagnosing malaria infection according to an embodiment of the present disclosure identifies a malaria infection image among blood cell images identified through artificial intelligence. Malaria infection images are identified through deep learning artificial intelligence.

In this case, cell boundaries of each blood cell image may be predicted based on a deep learning model in order to identify the malaria infection image. That is, boundary regions may be obtained for all red blood cell images through a deep learning model that predicts the cell boundary of each red blood cell image. To acquire the boundary region, the processor analyzes the initial input image including each red blood cell image and outputs a segment mask image divided into a dark black background and a white foreground representing each blood cell image. Through this segment masking, an output image in which the background and foreground are separated in the initial input image is generated. For such segment masking, a network model known as U-net may be modified and used.

12 is a structural diagram of a U-net used for cell segmentation according to an embodiment of the present disclosure.

An artificial neural network called U-net according to an embodiment of the present disclosure has an encoder-decoder structure, encodes a red blood cell (initial) input image tile through convolution layers, It is a network that decodes through a transposed convolution layer (also called deconvolution or upconvolution) to obtain segmentation masking as a final output. The U-net structure according to FIG. 12 may be used in the malaria infection diagnosis apparatus according to an embodiment of the present disclosure, or a modified U-net structure may be used in consideration of device performance and speed.

In one embodiment, an output image in which background and foreground are separated in an initial red blood cell input image is generated, and in the output image, white pixels connected to each other in the red blood cell image are grouped into one lump through connected component analysis. By determining the area of each red blood cell image, it is possible to easily obtain a boundary area for all red blood cell images.

Next, each red blood cell image can be cropped from the original image and passed to a classifier for subsequent recognition. The classifier analyzes each red blood cell image based on a deep learning model, and can finally classify the red blood cells by determining which type of malaria is infected. The shape and border shape of infected red blood cells can be different for each malaria species, and the deep learning model can classify malaria infection images from red blood cell images.

A convolutional neural network (CNN) may be used as a classifier for classifying malaria infection images. Among other things, according to one embodiment, it is possible to use artificial intelligence based on a structure known as ResNet-50.

10 is a structural diagram of ResNet-50 for automatically classifying malaria infection images according to an embodiment of the present disclosure.

ResNet-50 neural network, which can be called a deep learning algorithm to classify malaria infection images, image analysis is divided into 4 stages for proper operation, and ResNet-50 is implemented so that a total of 50 convolution layers are performed. It can be seen that the design Malaria infection images are classified from red blood cells images acquired by such a ResNet-50 neural network, and which malaria species are identified. For example, malaria is classified into P. falciparum, P. vivax, P. ovale, and P. malariae, each of which has a regional distribution. , show differences in microscopic findings, clinical features, and immunogenic potential.

Each malaria species can be classified by deep learning algorithm by ResNet-50 neural network. A processor (not shown) of the malaria infection diagnosis device analyzes the red blood cell image according to the ResNet-50 neural network. The analysis determines which malaria infection each red blood cell image belongs to.

In step 208, the processor of the apparatus for diagnosing malaria infection according to an embodiment of the present disclosure controls a user output interface to classify and display infected species of malaria among the malaria infection images. In the user output interface, among the malaria infection images, infected malaria species are classified and displayed.

Referring to FIG. 9 for a detailed description of an embodiment showing classification according to red blood cell images through a user output interface.

9 is a user output interface showing red blood cell image classification according to an embodiment of the present disclosure.

At 910, it is displayed that all red blood cells images are normal on the user output interface of the malaria infection diagnosis apparatus according to an embodiment of the present disclosure.

On the other hand, 911 indicates that an abnormal red blood cell image was detected. Referring to 911, 287 suspected abnormal red blood cell images (SUSPECTED RBCs) were detected among the red blood cells (TOTAL RBCs) to be tested, indicating that the infection rate is about 0.12%. In 911, abnormal red blood cells may mean malaria-infected red blood cells.

At 920, it is indicated that the red blood cell image is abnormal on the user output interface of the malaria infection diagnosis device according to an embodiment of the present disclosure. In 921, when an additional user input is made in the display state 920, which malaria species is infected may be displayed. Referring to 921, '0' is found in red blood cells infected with Parasite I, and 37 red blood cells infected with Parasite II are found. In 921, an image of red blood cells infected with Parasite II is displayed. Parasite I can correspond to any one of the malaria species, P. falciparum, P. vivax, P. ovale, P. malariae, and monkey fever. Parasite II is not a protozoan indicated in Parasite I, and among malaria species, P. falciparum, P. vivax, P. ovale, and P. malariae), and monkey fever protozoa.

930 shows an example of displaying the infection rate in the user output interface.

In one embodiment, referring to 930, the user sees the 'SUSPECTED' screen and confirms whether the patient is classified as a suspected malaria patient by the foreign substance or whether the patient is actually infected with malaria. Afterwards, the user can diagnose the patient as having malaria infection through CONFIRM input.

Referring again to 930, out of 200,106 red blood cells, there are 287 malaria-infected red blood cells, and the infection rate is 0.12%. Referring to 931, the patient's condition is displayed based on the infection rate. Referring to 931 according to an embodiment, it can be seen that the parasitemia level is displayed and the corresponding patient's condition is displayed in the 'Remarks' item. For example, a parasitemia level of 0.2% indicates that there are about 10,000 infected red blood cells per uL of red blood cells, and at this time, a specific symptom appears at a level above this level.

In step 210, a malaria infection rate may be displayed together on the user output interface based on the identified malaria infection image. In one embodiment, the infection rate may be displayed through a user output interface of the automatic malaria infection diagnosis device. For example, 930 of FIG. 9 shows an example of displaying an infection rate on a user output interface. The malaria infection rate is calculated by counting the number of malaria-infected blood cell images identified and the number of blood cell images to be tested, and the infection rate = the number of infected blood cell images/ the number of blood cell images to be tested.

In step 212, the user output interface of the automatic diagnosis device for malaria infection according to an embodiment of the present disclosure displays the state of the patient corresponding to the red blood cell sample according to a predetermined level based on the infected species of malaria and the infection rate derived by the processor. can be displayed.

An apparatus for automatically diagnosing malaria infection according to an embodiment of the present disclosure may provide a current condition of a patient to an examiner according to the malaria symptom criteria of the Centers for Disease Control and Prevention (CDC). 931 of FIG. 9 shows an example of such a display.

11 is a block diagram showing an apparatus for diagnosing malaria infection according to an embodiment of the present disclosure.

As shown in FIG. 11 , an apparatus for diagnosing malaria infection 2000 according to an embodiment of the present disclosure includes a cartridge loader unit 2100, a processor 2200, an image inspection unit 2300, and an image capture unit 2400. , a user output interface 2500, a user input interface 2600, and a memory 2700. Each component of the malaria infection diagnosis device 2000 is not essential, and each component may be added or subtracted according to the design idea of the manufacturer.

Hereinafter, the above components are examined in turn.

The cartridge loader unit 2100 is a unit for loading a red blood cell sample, and may be mounted so that the red blood cell sample is loaded into the cartridge, smeared, and examined by the image speculum unit 2300.

The processor 2200 controls overall operations of the malaria infection diagnosis apparatus 2000 . The processor 2200 executes programs stored in the memory 2700, thereby performing a cartridge loader unit 2100, an image inspection unit 2300, an image capture unit 2400, a user output interface 2500, and a user input interface 2600. , and memory 2700 can be controlled.

According to an embodiment of the present disclosure, the processor 2200 may include an artificial intelligence (AI) processor. The artificial intelligence (AI) processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), an existing general-purpose processor (eg CPU or application processor) or a graphics-only processor (eg GPU - Graphic Processing Unit) It may be manufactured as a part of and mounted in the malaria infection diagnosis device 2000. The processor 2200 may be an artificial intelligence (AI) processor capable of performing ResNet-50 according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the processor 2200 may classify the malaria species according to the red blood cell image, perform infection diagnosis according to the classification, and perform control operations according to the program in the malaria infection diagnosis apparatus 2000. The processor 2200 may perform the method of automatically diagnosing malaria infection described with reference to FIGS. 2 to 10 of the present disclosure.

Result values and result information according to malaria infection and/or malaria species classification may be stored in the memory 2700 .

The image speculum unit 2300 is an optical device such as a microscope, and refers to a device capable of enlarging and viewing an image of red blood cells. The image examination unit 2300 may be included with the malaria infection diagnosis device 2000 or may be a separately provided optional device. Through the image speculum unit 2300, the smeared red blood cell sample placed on the cartridge loader unit 2100 can be magnified and confirmed.

The image capture unit 2400 captures an image of a red blood cell specimen checked through the image speculum unit 2300 . The image capture unit 2400 usually consists of a precision camera. The camera of the image capture unit 2400 may be formed of a CMOS sensor, but is not limited thereto. The image including red blood cells and malaria-infected blood cells captured by the image capture unit 2400 is subjected to image processing-graphic processing by the processor 2200 for image classification.

The user output interface 2500 is for outputting an audio signal or a video signal, and may include a display unit 2510. Although not shown, the user output interface 2500 may optionally include a sound output unit according to a manufacturer's choice.

According to one embodiment of the present disclosure, the malaria infection diagnosis apparatus 2000 may display information related to the malaria infection diagnosis apparatus 2000 through the display unit 2510 . For example, a malaria infection image performed by the malaria infection diagnosis apparatus 2000 may be displayed on the display unit 2510 to determine which type of malaria it is, an infection rate, and a patient's condition.

The display unit 2510 and the touch pad may form a layer structure to form a touch screen. When the display unit 2510 is configured as a touch screen by forming a layer structure of a touch pad, the display unit 2510 may be used as an input device as well as an output device. The display unit 2510 includes a liquid crystal display, a thin film transistor-liquid crystal display, a light-emitting diode (LED), an organic light-emitting diode, At least one of a flexible display, a 3D display, and an electrophoretic display may be included. Also, depending on the implementation form of the malaria infection diagnosis apparatus 2000, two or more display units 2510 may be included.

According to an embodiment of the present disclosure, the user output interface 2500 may output malaria infection classification, infection rate, and patient condition results as well as various image examination-related information through the display unit 2510 . According to an embodiment of the present disclosure, the output interface 2500 may display a current power level, an operating mode (eg, image examination mode, malaria infection determination mode, sleep mode, etc.).

The user input interface 2600 is for receiving an input from a user. The user input interface 2600 includes a key pad, a dome switch, a touch pad (contact capacitance method, pressure resistive film method, infrared sensing method, surface ultrasonic conduction method, integral tension measurement method) , piezo effect method, etc.), a jog wheel, and a jog switch, but may be at least one, but is not limited thereto.

The user input interface 2600 may include a voice recognition module. For example, the malaria infection diagnosis apparatus 2000 may receive a voice signal, which is an analog signal, through a microphone, and convert the voice part into computer-readable text using an Automatic Speech Recognition (ASR) model. The malaria infection diagnosis apparatus 2000 may obtain the user's utterance intention by interpreting the converted text using a natural language understanding (NLU) model. Here, the ASR model or NLU model may be an artificial intelligence model. The artificial intelligence model can be processed by an artificial intelligence processor designed with a hardware structure specialized for the processing of artificial intelligence models. In this case, the processor 2200 may be a dedicated artificial intelligence processor. AI models can be created through learning. Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created. means burden. An artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weight values.

Linguistic understanding is a technology that recognizes and applies/processes human language/text, and includes natural language processing, machine translation, dialog system, question answering, and voice recognition. /Includes Speech Recognition/Synthesis, etc.

The memory 2700 may store programs for processing and control of the processor 2200, and input/output data (e.g., diagnosis information of the malaria infection diagnosis apparatus 2000, red blood cell images, or classification according to malaria species). information) may be stored. The memory 2700 may store an artificial intelligence model.

The memory 2700 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium. In addition, the control device 2000 may operate a web storage or cloud server that performs a storage function on the Internet.

The communication unit 2800 may include a short-distance communication unit 2810 and a long-distance communication unit 2820. The short-range wireless communication interface (2810) includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication interface (WLAN) communication unit, a Zigbee communication unit, and an infrared (IrDA) communication unit. , Infrared Data Association (WFD) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (Ultra Wideband) communication unit, Ant+ communication unit, etc. may be included, but is not limited thereto. The remote communication unit 2820 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the radio signal may include a voice call signal, a video call signal, or various types of data according to text/multimedia message transmission/reception. The remote communication unit 2820 may include a 3G module, 4G module, 5G module, LTE module, NB-IoT module, LTE-M module, etc., but is not limited thereto.

According to an embodiment of the present disclosure, the malaria infection diagnosis apparatus 2000 may communicate with an external server or other electric device through the communication unit 2800 and transmit/receive data. The communication unit 2800 may be included as an option for the purpose of selling the malaria infection diagnosis device 2000 or for price competitiveness, or may not be included when communication is not required.

The method according to an embodiment of the present disclosure may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the present disclosure, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

Some embodiments of the present disclosure may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media. In addition, some embodiments of the present disclosure may be implemented as a computer program or computer program product including instructions executable by a computer, such as a computer program executed by a computer.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary storage medium' only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium and temporary It does not discriminate if it is saved as . For example, a 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, a method according to various embodiments disclosed in this document may be loaded into a memory in a diagnosis device and executed by a processor of the diagnosis device. According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store or between two user devices (eg smartphones). It can be distributed (e.g., downloaded or uploaded) directly or online. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored on a device-readable storage medium such as a memory of a manufacturer's server, an application store server, or a relay server. It can be temporarily stored or created temporarily.

500; cartridge
2000; Malaria Infection Diagnosis Device
2100; cartridge loader
2200; processor
2300; image speculum
2400; image capture unit
2500; user output interface
2510; display
2600; user input interface
2700; Memory
2800; Ministry of Communications
2810; Near Field Communications Department
2820; telecommunications department

Claims (20)

obtaining a blood cell image of at least a portion of the red blood cell sample by staining the red blood cell sample;
identifying high-staining blood cell images excluding abnormal staining images from among the obtained blood cell images;
identifying a malaria infection image among the identified blood cell images through artificial intelligence;
classifying and displaying the infected malaria species among the malaria infection images;
displaying a malaria infection rate on a user output interface based on the identified malaria infection image; and
and displaying, on the user output interface, a condition of a patient corresponding to the red blood cell specimen according to a predetermined level criterion based on the infected species of malaria and the infection rate. According to claim 1,
The step of identifying the high staining blood cell image,
The method for automatically diagnosing malaria infection, comprising the step of removing low-staining areas from the stained red blood cell sample. According to claim 1,
Excluding the abnormal staining image from the stained red blood cell image is a normal staining image, a staining image of a predetermined level or more intensity, a staining color defect image, a partial staining image, an unclear border image, or a cell detachment image among the blood cell images. A method for automatically diagnosing malaria infection, comprising the step of classifying. According to claim 1,
The step of identifying blood cell images with high staining excluding abnormal staining images among the obtained blood cell images,
The method of automatically diagnosing malaria infection, comprising the step of controlling auto focus for image acquisition when the blood cell image includes an image with unclear boundaries, thereby acquiring a blood cell image with clear boundaries of the obtained blood cell image. According to claim 4,
Acquiring an image with clear boundaries of the obtained blood cell image by controlling the auto focus,
and obtaining a blood cell image with clear boundaries by increasing the contrast of the blood cell image. According to claim 5,
The step of obtaining a blood cell image with clear boundaries by increasing the contrast of the blood cell image,
The method of automatically diagnosing malaria infection comprising the step of acquiring the blood cell image with clear boundaries by extracting edge components of the obtained blood cell image using a Laplacian filter. According to claim 4,
Acquiring an image with clear boundaries of the obtained blood cell image by controlling the auto focus,
detecting a plurality of images of the blood cell image using a Laplacian filter;
detecting an image having an optimal contrast between a foreground and a background of the blood cell image by analyzing a histogram of the blood cell image for the plurality of detected images; and
and obtaining an image with clear borders of the blood cell image based on the image including the optimal contrast. According to claim 7,
The step of detecting a plurality of images of the blood cell image,
and acquiring the plurality of images sequentially in a vertical direction of the slide in a state where the red blood cell sample is smeared on the slide. According to claim 8,
Analyzing a histogram of the blood cell image for the plurality of detected images to detect an optimal contrast between a foreground and a background of the blood cell image,
Analyzing histograms of the plurality of images and detecting an image having the highest histogram value for the foreground of the blood cell image among the plurality of images as the blood cell image having the optimal contrast. Infection automatic diagnosis method. According to claim 1,
The step of identifying blood cell images with high staining excluding abnormal staining images among the obtained blood cell images,
removing a background image of the blood cell image;
removing an edge of the blood cell image from which the background image is removed; and
A method for automatically diagnosing malaria infection comprising the step of numerically evaluating the staining of the blood cell image from which the border is removed to determine whether the blood cell image has high staining. According to claim 10,
The step of removing the background image of the blood cell image,
determining a blue gain by obtaining a brightness value of a blue pixel of a background area;
obtaining a brightness value of a blue pixel among RGB values of each region of the blood cell image by using the histogram of the blood cell image;
determining each region of the blood cell image in which the product of the brightness value of the blue pixel and the blue gain is equal to or greater than a predetermined level as the background image; and
and removing the determined background image. According to claim 10,
In the step of removing the border of the blood cell image from which the background image is removed,
A method for automatically diagnosing malaria infection, comprising the step of removing a border of the blood cell image from which the background image has been removed using a morphological closing technique. According to claim 10,
The step of determining whether the blood cell image has high staining ability by numerically evaluating the stainability of the blood cell image from which the border is removed,
dividing the blood cell image from which the border is removed into a predetermined number of blocks;
extracting RGB representative values within the divided blocks;
evaluating the RGB representative value as a predetermined numerical value; and
and determining whether the borderless blood cell image is a high-staining blood cell image based on the evaluation. According to claim 13,
The step of extracting RGB representative values in the divided blocks
and extracting an RGB value having a maximum frequency as the RGB representative value through a histogram for each of the divided blocks. According to claim 1,
Displaying the malaria infection rate on the user output interface based on the identified malaria infection image,
The method of automatically diagnosing malaria infection according to claim 1, further comprising obtaining the infection rate by counting the number of identified malaria infection images and the number of blood cell images to be tested. According to claim 1,
The step of displaying the patient's condition corresponding to the red blood cell sample according to the predetermined level standard on the user output interface,
A method for automatically diagnosing malaria infection, characterized in that it comprises displaying a parasitemia level based on the infection rate. According to claim 1,
The step of identifying the malaria infection image,
A method for automatically diagnosing malaria infection, comprising predicting cell boundaries of each blood cell image based on deep learning. 18. The method of claim 17,
The step of predicting the cell boundary of each blood cell image,
and performing segment masking on an input image including the blood cell image to generate an output image in which a dark black background and a white foreground representing each blood cell image are separated. Methods for automatic diagnosis of malaria infection. According to claim 18,
The step of predicting the cell boundary of each blood cell image,
The method of automatically diagnosing malaria infection, characterized in that it comprises the step of determining the region of each blood cell image by grouping white pixels connected to each other in the output image through a connected component analysis. staining the red blood cell sample to obtain a blood cell image of at least a portion of the red blood cell sample;
Identifying blood cell images with high staining excluding abnormal staining images among the obtained blood cell images;
Identifying malaria infection images among the identified blood cell images through artificial intelligence, and
a processor for classifying the infected malaria species from among the malaria infection images; and
Indicate the species of infected malaria classified above,
Displaying the malaria infection rate based on the identified malaria infection image, and
and a display displaying a condition of a patient corresponding to the red blood cell sample according to a predetermined level criterion based on the infected species of malaria and the infection rate.

Images