TWI804204B - Apparatus and methods for providing machine learning to analyze a biological specimen - Google Patents

Abstract

A sampling device for providing machine learning to analyze a biological specimen, the device includes a transparent sample-carrying container, an image collecting element, a plurality of optical elements, and a control element. The transparent sample-carrying container is configured to carry the biological specimen and has a bearing plane; the image collecting element is disposed above the carrying plane to acquire an image of the biological specimen in real time; the plurality of optical elements are disposed at one of an upper position, a lower position, and a side position corresponding to the bearing plane; and the control element is electrically connected to the image collection element and the plurality of optical elements, wherein the plurality of optical elements comprises a plurality of groups of light-emitting units disposed at different positions, each group of light-emitting units is configured to be turned on according to a control command corresponding to each group from the control element, and the control commands are arranged according to a time sequence under the arrangement of the control element.

Description

Sampling method and device for providing machine learning interpretation-biological sample

The present invention relates to a sampling method and device for biological samples, in particular to a sampling method and device for providing machine learning to interpret biological samples by using optical images as analysis objects.

Automatic optical inspection is a common inspection method in the industry, and it has the obvious advantage of fast speed. Generally speaking, the automatic optical inspection system is suitable for various samples, such as wafers and panels, etc., and can maintain its inspection accuracy and efficiency. However, when it is used in biological samples, there is often a problem that a single optical illumination mode cannot be used effectively. The reason is that there are many types of cells or individuals in biological samples, not only are they different in structure and size, but even though they are of the same type of cells, there are often problems with individual differences. In addition, the transparency is usually high, resulting in low contrast, and in order to maintain life or originality. The saturation of water causes problems such as complex background.

Representative biological cells or individuals are parasites, such as worms or protozoa. Currently, the most widely used diagnostic method is to observe with an optical microscope, and it is necessary to find parasite eggs in complex environments such as feces or blood smears. Professional medical examination experience is time-consuming and labor-intensive to complete, even though this manual mode is still adopted today. Therefore, it is necessary to establish a sampling method that can provide computer learning and interpretation, and provide more abundant information to greatly reduce the misjudgment rate.

Referring to FIG. 1, US Patent Application Publication No. US2012/0120221 discloses a body fluid analysis system 10, including: a biological sample carrying container 11, an image acquisition component 12, and an image The processing device 13 is used to collect the image 17 of the somatic cells of the target worm in the body fluid.

The images 17 of various types of parasite cells in the body fluid acquired by the autofocus photography used in the prior art shown in FIG. 1 are not clear, resulting in inaccurate identification. This invention divides the space into N layers with different depths along the vertical direction. The image data 17 of each layer 15 is captured synchronously through the control of the computer system, and the digitized image data of each layer is then stacked by software program processing. The method uses a computer system combined with optical photography to perform composite image processing and stacking, and a computer program is used to identify and count tiny cells. Generally, the continuous illumination light sources 16 are arranged in different positions to maintain uniform illumination.

In the conventional sampling method for machine learning interpretation, after image sampling, the computer can expand the number of sample images for identification through simulation to provide machine learning. However, the characteristics of biological cells or individuals are not necessarily symmetrical, and the contours and features of each part of the surface of the organism are not easy to fully display under the same or uniform illumination. If images obtained under the same illumination conditions are used To virtualize sample images for identification, for organisms (such as insect eggs) with complex and asymmetric surface contours and features, the identification effect of machine learning is quite limited.

Therefore, how to avoid the disadvantages of the above devices is a technical problem to be solved.

The present invention proposes a sampling method and device for machine learning to interpret a biological sample. It aims to provide a method that can capture a large number of different optical features of a single sample, and simultaneously create different projection angles, Illumination of wavefront, spectrum, and polarization, together with image acquisition of different optical focal planes, can obtain a large amount of oriented information of a single sample, and complex Combine images or multiple images to provide information to artificial intelligence software for machine learning.

According to an embodiment of the present invention, a sampling device for providing machine learning to interpret a biological sample is proposed. The device includes a transparent sample holding container, an image collection element, a plurality of optical elements, and a control element. The transparent sample holding container is configured to carry the biological sample, and has a carrying plane; the image collection element is arranged above the carrying plane to obtain an image of the biological sample in real time; the plurality of optical elements are arranged opposite at one of an upper position, a lower position, and a side position of the carrying plane; and the control element, electrically connected to the image collection element and the plurality of optical elements, wherein the plurality of optical elements are arranged on Multiple groups of light-emitting units at different positions, each group of light-emitting units is configured to be turned on according to the control instructions corresponding to each group from the control element, and these control instructions are arranged according to a time sequence under the arrangement of the control element Turn on at least one of the plurality of groups of light emitting units.

According to another embodiment of the present invention, a sampling method for providing machine learning to interpret a biological sample is provided, comprising the following steps: providing a sample holding container with a carrying plane for carrying the biological sample; spaced apart from the carrying plane An image collection element is arranged at a distance to acquire an image of the biological sample in real time; a plurality of optical elements are provided, wherein the plurality of optical elements form a plurality of sets of light-emitting units located at different positions to project a plurality of light sources to the biological sample; and A control element is electrically connected to the image collection element and the plurality of optical elements, so as to control the plurality of light sources to be composed of different groups of light emitting units at different times.

According to another embodiment of the present invention, a sampling method for providing machine learning to interpret a biological sample is provided, comprising the following steps: providing a sample holding container with a carrying plane for carrying the biological sample; spaced apart from the carrying plane An image collection element is arranged at a distance to obtain an image of the biological sample in real time; multiple optical elements are provided to provide the biological The sample projects a plurality of light sources; a control element is used to electrically connect the image collection element and the plurality of optical elements, and a plurality of control commands are issued in a sequence; and the plurality of optical elements are activated in response to the plurality of control commands.

The sampling device and method proposed by the present invention for providing machine learning to interpret a biological sample are suitable for analysis in the biotechnology industry and have industrial applicability.

10: Body fluid analysis system

11: Biological sample carrying container

12: Image acquisition components

13: Image processing device

15: layers

16: Lighting source

17: The image of the somatic cells of the target worm

100/200/300: sampling device

110: sample holding container

115: bearing plane

120: Image collection component

121: Lens

122: Photographic device

130: Control element

160/260/360: Optics

161-169/261-269/261-369: Lighting unit

171/173/175: biological samples

501-505/511-515/610/620/630/640/650: light emitting unit group

D: distance

This case can be better understood through the detailed description of the following drawings: Figure 1 is a schematic diagram of a conventional sampling device used to obtain images of biological samples; Figure 2 is the sampling device used to provide machine learning to interpret biological samples in the present invention A schematic diagram of an embodiment of the method and device; FIG. 3 is a schematic diagram of another embodiment of a sampling method and device for providing machine learning to interpret biological samples according to the present invention; FIG. 4 shows a schematic diagram for providing machine learning to interpret biological samples according to the present invention A schematic diagram of another embodiment of the sample sampling method and device; FIG. 5-1 shows a schematic diagram of an embodiment of arranging multiple sets of light-emitting units at different positions according to the present invention; FIG. 5-2 shows the configuration proposed according to the present invention A schematic diagram of another embodiment of multiple groups of light emitting units at different positions; FIG. 6 shows a schematic diagram of another embodiment of disposing multiple groups of light emitting units at different positions according to the present invention; FIG. 7 shows a calculation using a convolutional neural network according to the present invention A schematic diagram of an embodiment of the method for image recognition and determination; FIG. 8A shows a schematic diagram of the output accuracy rate after CNN learning provided by an image captured according to the previous technology:

FIG. 8B shows a schematic diagram of the output accuracy rate after CNN learning is provided from images captured according to an embodiment of the present invention.

The present invention will be fully understood by the description of the following examples, so that those skilled in the art can complete it, but the implementation of the present invention cannot be limited by the following examples.

Please refer to FIG. 2 , which shows an embodiment of a sampling device and method for machine learning interpretation of biological samples according to the present invention. As shown in the figure, the sampling device 100 includes a sample holding container 110 , an image collection component 120 , a plurality of optical components 160 , and a control component 130 . The sample holding container 110 is configured to hold biological samples 171 / 173 / 175 and has a carrying plane 115 . In order to provide illumination from different angles and avoid shadows in the edge area, the sample holding container 110 is preferably made of a transparent material. Biological samples 171/173/175 may be blood cells, lymphocytes, parasite eggs, or other organisms or cells. The biological samples 171/173/175 in the sample holding container 110 usually exist in light-permeable body fluids or tissue fluids, and can also be feces or light-transparent tissue sections, and these biological samples 171/173/ The body fluid, interstitial fluid, feces or tissue slices at 175 are accommodated in the sample holding container 110 .

The image collection unit 120 is usually disposed at a certain distance D above the carrying plane 115 to acquire images of the biological samples 171/173/175 in real time (not shown). According to an embodiment, the image collection element 120 is composed of a lens 121 and a photographing device 122, and the focal depth of photography can be arranged near the carrying plane 115, so that all the biological samples in the sample holding container 110 171/173/175 images can be acquired synchronously. The photographing device 122 can be, but not limited to, a camera or video recorder equipped with a charge-coupled device (CCD). With the cooperation of a shutter (not shown), it can set The captured images are converted into electronic signals.

These electronic signals carrying image information can be sent to the control unit 130 or other computer equipment for processing subsequent image recognition. Please refer to FIG. 7 , which shows the basic steps of a typical convolutional neural network (CNN) calculation method for image recognition and judgment. After the digital image data containing the image of the biological sample 171 is input, features, such as the right local contour of the image of the biological sample 171 , are extracted through a convolution step. In the following pooling step, the extracted features are sampled, and after repeated pooling sampling, the information is flattened into a one-dimensional vector and then fully connected layer classification, and the final judgment is output. The output identification decision may be that several specific numbers of eggs are identified.

Returning to the embodiment shown in Figure 2, since the effectiveness of the process shown in Figure 7 depends on whether the content of the input image information is sufficient, so that the function of the neural network-like calculation can be correctly and effectively played, the sampling device of the present invention 100 is configured with one or more optical elements 160 above the carrying plane 115 . Two optical elements 160 are shown, but not limited thereto. The optical element 160 includes multiple groups of light emitting units 161 - 169 arranged in different positions, which can respectively provide illumination from different angles and positions. According to an embodiment of the present invention, each group of the light emitting units 161-169 can be configured to provide a plurality of light sources of different wavelengths, the different wavelengths fall within the range of visible light such as 350-1100 nm. According to another embodiment, each group of the light emitting units 161 - 169 can be configured to provide a plurality of light sources with different polarizations, and the polarized light sources can be filtered to reduce image interference, or to highlight the image. Alternatively, the user can select only one optical element 160, such as the upper left of the figure The optical element 160, and the light emitting units 161/163/165/167/169 are respectively set as a group.

These light sources provide illumination with various projection angles, wavefronts, spectra, and polarizations, so that the features used to distinguish and identify biological samples 171/173/175 can be fully displayed, which is conducive to the exertion of CNN computing functions. achieve precise identification.

The control element 130 is electrically connected to the image collection element 120 and the plurality of optical elements 160 , and each group of light emitting units is configured to be turned on and off sequentially according to control commands (not shown) corresponding to each group from the control element 160 . For example, the optical element 160 includes 5 groups of light-emitting units 161-169 in total, wherein the light-emitting units 161/162 are the first group, the light-emitting units 163/164 are the second group, the light-emitting units 165/166 are the third group, and the light-emitting units 167/168 is the fourth group, the light-emitting unit 169 is the fifth group, and there are five kinds of control instructions, and the appearance of each instruction indicates that the corresponding group of light-emitting units is activated. According to an embodiment, each control command appears independently, so each group of light-emitting units is also turned on individually within a specific period of time. According to another embodiment, different control commands can also appear synchronously, so multiple groups of light-emitting units are turned on synchronously within a specific period of time.

In order to effectively control the lighting period of each group of light emitting units 161-169 to match the shutter rhythm of the image collection unit 120, these control commands can be arranged by the control unit 130 to respectively turn on the multiple groups of light emitting units according to a time sequence. at least one of them. The time sequence includes a plurality of time periods, and the time length of each time period can be set as between 200 to 2000 microseconds, or other suitable time lengths. Therefore, each of the groups of light emitting units 161 - 169 of the plurality of groups of light emitting units can be sequentially turned on in each of the time periods. Since the moving speed of general organisms is not fast, dozens or hundreds of samples can be taken within one second, and photography samples can be taken at a higher shutter speed under suitable lighting conditions. The large amount of biological image information obtained in this way under different lighting conditions is more helpful to provide samples required for in-depth machine learning and improve the recognition ability of machine learning interpretation.

Please refer to FIG. 3 , which shows another embodiment of the sampling device and method for machine learning interpretation of biological samples according to the present invention. As shown in the figure, the sampling device 200 includes a sample holding container 110 , an image collection component 120 , a plurality of optical components 260 , and a control component 130 . In this embodiment, the sample holding container 110 should have light transmission, preferably a transparent sample holding container 110 , so as to facilitate the lighting effect of the optical element 260 .

Different from the arrangement in the embodiment of FIG. 2 , the plurality of optical elements 260 in FIG. 3 are arranged at a lower position relative to the bearing plane 115 . Two optical elements 260 are shown, but not limited thereto. The optical element 260 includes multiple groups of light emitting units 261-269 arranged in different positions, which can respectively provide illumination from different angles and positions. The selection of the number and grouping of the optical elements 260 and the light emitting units 261-269 has been described above, and will not be repeated here.

Similarly, these light sources provide illumination with various projection angles, wavefronts, spectra, and polarizations, so that the features used to distinguish and identify biological samples 171/173/175 can be fully displayed, which is beneficial to convolutional neural networks. The road calculation function is used to achieve the effect of precise identification.

Please refer to FIG. 4 , which shows another embodiment of the sampling device and method for machine learning interpretation of biological samples according to the present invention. As shown in the figure, the sampling device 300 includes a sample holding container 110 , an image collection component 120 , a plurality of optical components 360 , and a control component 130 . In this embodiment, at least the sidewall of the sample holding container 110 should have light transmission, preferably a transparent sample holding container 110 , so as to facilitate the lighting effect of the optical element 360 .

Different from the arrangement in the embodiments of FIG. 2 and FIG. 3 , the plurality of optical elements 360 in FIG. 4 are arranged at a side position relative to the carrying plane 115 . Two optical elements 360 are shown in the figure, but it is not limited thereto. For example, more groups of optical photos can be provided with reference to the configuration concept of FIG. bright. The optical element 360 includes multiple groups of light emitting units 361-369 arranged in different positions, which can respectively provide illumination from different angles and positions. The selection of the number and grouping of the optical element 360 and the light emitting units 361-369 has been described above, and will not be repeated here.

Figures 5-1 and 5-2 respectively provide schematic diagrams of different embodiments of disposing multiple groups of light emitting units at different positions, wherein the five groups of light emitting units 501-505 shown in Figure 5-1 are arranged with one group of light emitting units 501 in the center and The other four sets of light-emitting units 502-505 are arranged in a ring around them, and the arrangement shown in Figure 5-2 is to arrange the ring-shaped outlines of each set of light-emitting units 511-515 with different outer diameters in a concentric manner. Users can also It is within the scope of the concept of the present invention to develop different numbers of groups or configurations of different outlines according to needs.

6 shows a schematic diagram of yet another embodiment in which multiple groups of light emitting units are arranged at different positions according to the present invention. In the figure, six groups of light emitting units 610-660 surround different projection positions of the sample holding container 110 to provide illumination from different angles. Each group of light emitting units in Fig. 5-1, 5-2 and Fig. 6 can also include a plurality of light sources with different wavelengths or polarizations, so that the images of the biological samples 171/173/175 captured by the image collection element 120 can be To fully identify its biological characteristics.

For example, some organisms have flagella at one end of their tail and no flagella in other parts, and this organism may swim in any direction, so only when the direction of the projected light is sufficient to show the flagella characteristics of this organism, The images obtained are sufficient to identify the organism. The sampling device provided by the present invention for providing machine learning to interpret a biological sample uses multiple sets of light-emitting units arranged at different positions to obtain multiple light sources from different positions or even different wavelengths or polarizations in a short time according to a time sequence A large number of images obtained under irradiation, so that the features used to distinguish and identify the images of biological samples 171/173/175 can be fully displayed, which is beneficial Accurate identification is achieved through the use of CNN computing functions.

Please refer to Figures 8A and 8B, which respectively provide the interpretation accuracy of several outputs after CNN learning based on the images captured by the prior art and the images captured according to an embodiment of the present invention (six different lighting modes) And provide a schematic diagram of the correct rate of interpretation after CNN learning. The so-called specified discrimination accuracy rate is the accuracy percentage of comparing the number of eggs of a certain type obtained through computer interpretation in the image with the known number of eggs of this type in the sample.

The average value of the specified discrimination accuracy shown in Figure 8A is actually 69.93%, and the control boundary threshold calculated based on the data shown by the dotted line in the figure falls near 60%, while the average specified discrimination accuracy shown in Figure 8B The value is actually 99.94%, and the control boundary threshold shown by the dotted line in the figure falls at 99.6%. Comparing the accuracy rate data and statistical results of the two, it is not difficult to find that the image information obtained by the sampling method and device for providing machine learning to interpret a biological sample according to the present invention can make a breakthrough in the accuracy rate of interpretation after CNN learning. It can be said that it is a major innovation of technology to improve the interpretation ability of machine learning to a very reliable state.

Although this case discloses the above with a preferred embodiment, it is not used to limit the scope of this case. Any changes and modifications made by those who are familiar with this technology without departing from the spirit and scope of this case should fall within the scope of this case.

100: sampling device

110: sample holding container

115: bearing plane

120: Image collection component

121: Lens

122: Photographic device

130: Control element

160/260/360: Optics

161-169: Lighting unit

171/173/175: biological samples

D: distance

Claims (10)

A sampling device for providing machine learning to interpret a biological sample, comprising: a transparent sample carrying container configured to carry the biological sample and having a transparent carrying plane; an image collection element disposed above the transparent carrying plane , to obtain an image of the biological sample in real time; a plurality of optical elements are arranged at one of an upper position, a lower position, and a side position relative to the transparent bearing plane; and a control element is electrically connected In the image collection element and the plurality of optical elements, wherein: the plurality of optical elements include multiple groups of light emitting units arranged at different positions, wherein each group of each group of light emitting units includes multiple light sources providing different polarizations, and is configured to It is turned on according to the control commands corresponding to the groups from the control element; and the control commands are arranged by the control element to respectively turn on at least one of the plurality of groups of light-emitting units according to a time sequence. The device according to claim 1, wherein the machine learning adopts a convolutional neural network computing method, and the convolutional neural network computing method includes a convolution, a pooling, a flattening and a fully connected layer Algorithm. The device according to claim 1, wherein the multiple sets of light emitting units include at least 5 sets of light emitting units, and the image collection element acquires the image of the biological sample according to the timing, and each set of the multiple sets of light emitting units includes a Multiple light sources of different wavelengths, and the different wavelengths fall within a range of 350-1100 nm. The device as claimed in claim 1, wherein the time sequence includes a plurality of time periods, the time length of each time period is between 200 and 2000 microseconds, and each of the groups of light-emitting units of the plurality of groups of light-emitting units is sequentially in each of the time periods enabled in . A sampling method for providing machine learning to interpret a biological sample, comprising the following steps: providing a sample holding container with a transparent carrying plane for carrying the biological sample; disposing an image collection at a distance from the transparent carrying plane A component is used to acquire an image of the biological sample in real time; multiple optical components are provided, wherein the multiple optical components form multiple groups of light-emitting units located at different positions to project multiple light sources to the biological sample, wherein each of the multiple groups of light-emitting units The set includes a plurality of light sources providing different polarizations; and a control element is electrically connected to the image collection element and the plurality of optical elements so as to control the plurality of light sources to be composed of different sets of light emitting units at different times. The method as claimed in claim 5, wherein the sample holding container is transparent, the control element issues a plurality of control commands in a time sequence, and the method further includes: activating the plurality of groups of light emitting units in response to the plurality of control commands. A sampling method for providing machine learning to interpret a biological sample, comprising the following steps: providing a sample holding container with a transparent carrying plane for carrying the biological sample; disposing an image collection at a distance from the transparent carrying plane The component is used to obtain an image of the biological sample in real time; multiple optical components are provided to project multiple light sources to the biological sample to provide multiple light sources with different polarizations; a control component is electrically connected to the image collection component and the multiple optical components components, and issue multiple control commands in a time sequence; and activate the multiple optical components in response to the multiple control commands. The method as described in claim 7, wherein the machine learning adopts a convolutional neural network calculation method, and the convolutional neural network calculation method includes a convolution, a pooling, a flattening and a fully connected layer Algorithm. The method as claimed in item 7, wherein the step of providing a plurality of optical elements to project a plurality of light sources to the biological sample further comprises: arranging the plurality of optical elements into multiple groups of light emitting units located at different positions; and making the multiple groups of light emitting units Each group of cells includes a plurality of light sources providing different wavelengths, and the different wavelengths fall within a range of 350-1100 nm. The method as described in claim 9, wherein the time sequence includes a plurality of time periods, the time length of each time period is between 200 and 2000 microseconds, and each of the groups of light-emitting units of the plurality of groups of light-emitting units is sequentially in each of the time periods enabled individually.

Images