JP7617717B2 - Image Processing Method - Google Patents

Description

This invention relates to image processing of images captured, for example, of pathological specimens, and in particular to technology that supports the creation of learning data required for image processing using machine learning.

In recent years, with the development of imaging technology that can easily convert entire pathological tissues into image data, digitized pathological tissue images are being accumulated. By applying image processing and machine learning to these pathological tissue images, it is now possible to analyze and quantitatively evaluate the state of cells and structures of interest.

To perform effective analysis using the results of machine learning, it is necessary to collect a large amount of high-quality training image data that represents typical examples of cells and structures, and to correctly teach the condition of each piece of training image data. This task requires an expert such as a pathologist to visually observe each image and assign a label according to its condition, which places a heavy burden on the worker.

For example, the technology described in Patent Document 1 uses a deep learning algorithm with a neural network structure to determine whether or not a tissue image contains a tumor. In this technology, deep learning is performed using training data (learning data) created from images of tissue that contain tumors and images that do not contain tumors, and a trained model is constructed that has the function of determining whether or not a tumor is present by learning the differences in the morphological characteristics of tissue depending on whether or not a tumor is present. The determination of the presence or absence of a tumor that serves as the basis for the training data is to be performed by a pathologist.

JP 2019-148473 A

Such human judgments can also include various factors that lead to erroneous judgments. First, the tissues, cells, structures, etc. that are the subject of judgment vary greatly from specimen to specimen, and quantitative and standardized judgment criteria for them have not been established. There is also variation in the staining properties of specimens. This poses the problem that judgments must be made relative to the judge's knowledge and experience. Furthermore, judgment results can vary even for the same person depending on the judge's personal skills, physical condition, degree of fatigue, and the influence of other images viewed prior to making the judgment.

For these reasons, there is a demand for technology that can automatically or semi-automatically organize and present collected images so that users (judges) can efficiently judge the state of cells and structures contained in images of pathological tissue specimens using stable criteria. However, the above-mentioned conventional technologies do not yet meet this demand.

This invention has been made in consideration of the above problems, and aims to provide a technology that can effectively support the user's teaching work for machine learning by displaying and outputting multiple learning images, which are images of tissue specimens, in an appropriate display format.

In order to achieve the above-mentioned object, the image processing method of the present invention includes the steps of: extracting features of objects in each of a plurality of learning images obtained by capturing an image of a tissue specimen through image processing; selecting a plurality of images from the learning images based on the extracted features and a predetermined selection criterion, and constructing an image set including the selected plurality of learning images; outputting to a display device a display image in which a plurality of the learning images included in the image set are arranged on the same screen; and accepting instruction input from a user for each of the displayed learning images and associating the learning image with the result of the instruction input corresponding thereto. The image processing method creates a plurality of image sets each having a different combination of the learning images, switches between and displays a plurality of the display images corresponding to each of the plurality of image sets, accepting the instruction input each time, and selecting the learning images while changing the selection criterion, thereby including one learning image in a plurality of image sets , and causing each of the learning images to be displayed on the display device a plurality of times while accepting the instruction input each time .

In an invention configured in this way, multiple learning images selected according to a predetermined selection criterion are displayed on one screen, allowing the user to compare the multiple images and perform instruction input. It is easier to obtain a more accurate judgment result when other learning images to compare with are also displayed, rather than judging a single learning image or multiple learning images presented in a random order against one's own knowledge.

Furthermore, in this invention, the same learning image is displayed multiple times while changing the combination with other learning images. As mentioned above, there is variation in user judgment, and even if it is the same learning image, the evaluation may fluctuate depending on the combination with other images displayed at the same time. By displaying the same learning image multiple times while changing the combination and accepting instruction input, it is possible to statistically converge such fluctuations.

In this way, by displaying each of the multiple learning images that are to receive evaluation input multiple times in combination with other learning images, and in different combinations, the user is able to make a relative evaluation by actually comparing the images, rather than relying solely on memory, and erroneous teaching due to fluctuations in judgment can be reduced.

As described above, the present invention can effectively support a user's instruction work for machine learning by displaying each of a plurality of training images that are to receive instruction input multiple times in combination with other training images, and in different combinations.

1 is a flowchart showing a first embodiment of an image processing method according to the present invention. FIG. 13 is a schematic diagram showing an example of classification and arrangement of learning images from the viewpoint of similarity. 10 is a flowchart illustrating a second embodiment of an image processing method according to the present invention. 11A and 11B are diagrams illustrating examples of display aspects of a display image. FIG. 13 is a diagram illustrating an example of a combination of learning images included in one image set. 10A and 10B are diagrams illustrating examples of instruction input by a user.

A specific embodiment of the image processing method according to the present invention will be described below. This image processing method is intended to support the creation of training data, which is necessary for analyzing tissue specimen images using a machine learning model, such as a deep learning model. More specifically, the creation of training data requires the user to label (teach) each of the collected training images (hereinafter, sometimes simply referred to as "images"); however, this image processing method displays the training images in an appropriate manner, allowing the user to perform the teaching work efficiently while reducing fluctuations in judgment.

To improve the accuracy of machine learning, it is necessary to collect a large number of training images and provide appropriate training for each of them. However, there are no absolute established standards for judgment, such as the nuclear atypia used to identify tumor cells, and judgments must be made relative to one another between images. In this case, it is virtually impossible to present all of the training images collected when there are a large number of images, and it is also difficult for users to compare a large number of images showing various morphologies and make an accurate evaluation, which is also a very time-consuming task.

For this reason, in practice, a portion of the many learning images is selected and presented to the user, and the user is asked to evaluate them based on a relative comparison among them. In this case, even if a relative evaluation between the displayed images is possible, there may be cases where the evaluation cannot be said to be correct in relation to other images that are not included in the display. The embodiment of the present invention described below addresses this problem.

First Embodiment
1 is a flow chart showing a first embodiment of an image processing method according to the present invention. In this process, the collection of learning images in step S101 and the instruction input in step S106 are basically performed by a user, but the other process steps can be executed by a general computer device. In this case, it is assumed that these processes are executed by a computer device equipped with input devices such as a mouse and keyboard as a user interface, and an output device such as a display device.

First, a large number of learning images are collected (step S101). The learning images may be collected by imaging various tissue specimens, or images of previously captured tissue specimens that are appropriate for the purpose may be obtained from a library.

It is assumed that the specimens captured as learning images are derived from a tissue or organ and that they have (or do not have) a disease. This is a reasonable assumption given the purpose of collecting tissue as a specimen. Furthermore, the specimens are assumed to have been stained in a manner suitable for visual observation of cells, such as hematoxylin-eosin staining (HE staining). In the following, the word "tissue" can be read as "organ" as necessary, unless otherwise specified.

The collected learning images are subjected to appropriate image processing to extract features (step S102), and an index value relating to similarity is derived based on the features (step S103). Image processing can include, for example, binarization of images, edge detection, and extraction of areas with specific colors or brightness. Areas with specific characteristics may also be extracted using a region segmentation algorithm previously constructed using machine learning. For example, various information that quantitatively represents the tissue specimen, such as the number (or density), shape, and shade of cells or nuclei, can be used as the index value.

In addition, the staining may vary for each stained specimen. Such differences in staining may be used as an index value, for example, the intensity of the red or blue component. Furthermore, the amount of secondary components in the image may be used as an index value. Here, "secondary components" refers to structures contained in the tissue image that are not directly evaluated in the analysis. For example, blood vessels in the tissue, artifacts (such as artificial scratches caused during specimen preparation), and areas where no tissue exists (or is in a state equivalent thereto) are secondary components. The amount of these components contained in the image, for example, the area they occupy, may be used as an index value. In addition, when there are images that have already been labeled, such as images obtained from an existing library, it is also possible to treat this information as an index of similarity. For example, it can be said that images with the same label have a relatively high similarity, and images with different labels have a lower similarity. In addition, when there are labels that express the characteristics of the image in terms of a degree such as high, medium, or low, they can be used as an index of similarity.

When the difference in index values between multiple training images is relatively small, the similarity between them can be said to be relatively high. On the other hand, when the difference in index values is large, the similarity can be said to be low. In this way, the index values can be used to evaluate the relative similarity between multiple training images.

Figure 2 is a schematic diagram showing an example of classification and arrangement of training images from the perspective of similarity. Note that this diagram is for the purpose of explaining the principle, and such classification and arrangement are not required in actual image processing. Assume that the collected training images contain tissue types A, B, ... and disease types a, b, c, d, e, ... (of which, diseases a to c are assumed to be diseases that occur in tissue A, and diseases d and e are assumed to be diseases that occur in tissue B). For tissues that do not contain diseases, in other words, normal tissues, the "normal" state can be treated as one type of disease.

For each combination of tissue and disease, multiple training images are collected. Each training image is associated with information indicating which tissue and disease the image corresponds to, and an index value calculated from the characteristics of the image. In the following description, in order to distinguish between the training images, each training image is assigned a symbol as shown in the figure. For example, training images corresponding to tissue A and disease a are assigned symbols Aa1, Aa2, .... The numbers 1, 2, ... are consecutive numbers assigned in order of magnitude of the index value (ascending or descending). Similarly, training images corresponding to tissue A and disease b are assigned symbols Ab1, Ab2, ..., and training images corresponding to tissue B and disease d are assigned symbols Ba1, Ba2, ....

Returning to FIG. 1, next, an image set is constructed consisting of a plurality of images selected from the collected learning images (step S104). As described below, a plurality of images included in one image set are arranged on one screen and displayed. Therefore, the user can only simultaneously view the learning images included in one image set.

However, in this embodiment, the image selection criteria used when constructing the image sets are varied so that one learning image is included in multiple image sets in duplicate. Therefore, when switching between image sets, the combination of learning images displayed changes sequentially, and one learning image is displayed multiple times in different combinations with other images.

There can be various criteria for selecting training images to construct an image set. For example, an image set can be constructed by collecting training images corresponding to the same tissue or the same disease. In this case, it is possible to combine images with high similarity, i.e., small differences in index values, and to combine images with low similarity, i.e., large differences in index values.

It is also possible to combine training images that differ in at least one of the types of tissue and the type of disease. In this way, even between specimens of different origins, the features that appear in the images may be similar. By displaying them simultaneously and making it possible to compare them, it is possible to reduce misjudgments between them.

When multiple index values are derived for each learning image, each of these can be an index of similarity. For example, similarity can be defined as an index value for cell density, a similarity value for stainability, a similarity value for the amount of secondary components, etc. Each of these similarities can be a selection criterion when constructing an image set. In other words, it is possible to construct an image set that is combined with a focus on differences in cell density, an image set that is combined with a focus on differences in stainability, etc. Thus, one learning image can be included in various image sets.

A display image is generated from one of the image sets thus constructed and displayed on the display device (step S105). The display image is a set of multiple learning images included in the image set arranged on the same screen.

Then, instruction input from the user is accepted for each of the training images thus displayed (step S106). The user compares the displayed training images to determine the state of the tissue or the cells and structures contained therein, and inputs the result as instruction input via an input device. The input result is associated with each training image as a label based on the user's judgment.

The condition of the tissue must be judged comprehensively from various perspectives, including cell density and shape, the presence or absence of irregular nuclear edges, the presence or absence of multinuclei, and the presence or absence of apoptosis. Therefore, the user in this case is required to be a pathologist, researcher, or other person with specialized knowledge and experience.

Steps S105 and S106 are repeated while changing the image set being displayed until instruction input is completed for all image sets (step S107). Then, based on the results of the instruction input, the instruction results for each learning image, i.e., the label to be assigned to that learning image, are determined (step S108).

Because one training image is included in multiple image sets, multiple instruction inputs are made to that training image. On the other hand, the user's judgment is a relative one based on a comparison between images, so the results of multiple instruction inputs to one training image will not necessarily be the same due to the influence of other training images displayed at the same time. In particular, if the morphological differences between the training images displayed at the same time are not clear, it can be difficult to judge their condition. Also, apparent differences due to differences in stainability can lead to misjudgment of differences based on the original judgment criteria.

If the results of multiple teaching inputs are all the same, then that teaching input can be regarded as being highly accurate and used as the label for the learning image as is. On the other hand, for learning images where the results of multiple teaching inputs are different, the teaching result can be determined statistically from those input results. For example, the most frequently inputted result among the multiple teaching inputs can be used as the final teaching result. The results of multiple teaching inputs can also be averaged to determine the final teaching result.

In this manner, in this embodiment, an image set is constructed consisting of multiple learning images selected from a large number of collected learning images, and these learning images are simultaneously displayed on the screen to accept instruction input from the user. At this time, instruction input is received for all learning images while switching between the images displayed by changing the image set. One learning image is included in multiple image sets, and therefore receives multiple instruction inputs. If the input results do not match, the final instruction result is determined by statistical processing.

By doing as described above under the condition that the number of learning images that can be displayed at one time is limited, this embodiment makes it possible to accurately label each learning image while suppressing fluctuations in judgment that can occur when there are few images to compare. For the user providing the teaching input, the workload is greatly reduced because it is sufficient to make a judgment while comparing images that are displayed at the same time (in other words, without considering images that are not currently displayed). In this way, this embodiment makes it possible to support the user in performing efficient and accurate teaching tasks.

By providing the learning images labeled in this way as training data to an appropriate learning model and performing machine learning, it is possible to construct a trained model that can accurately determine the tissue state even for unknown tissue specimen images. By performing learning including specimen images related to various tissues and diseases, a highly versatile learning model that can handle a variety of tissues and diseases can be created.

Second Embodiment
In the first embodiment described above, display images corresponding to an image set preconfigured based on the characteristics of each of the collected learning images are displayed in sequence, and instruction input for each learning image is accepted. The second embodiment described next has the same basic concept of processing, but differs from the above embodiment in that an image set is dynamically configured in accordance with the progress of the instruction input.

Figure 3 is a flowchart showing a second embodiment of the image processing method according to the present invention. The processing contents of steps S201 to S203 are the same as the processing contents of steps S101 to S103 in the first embodiment.

In step S204, one image set is constructed based on appropriate selection criteria. The method of constructing the image set may be the same as in the first embodiment. Then, a display image corresponding to the image set is created and displayed (step S205), and instruction input is accepted for the learning images contained therein (step S206). The basic concept of these is also the same as in the first embodiment.

The processes of steps S204 to S206 are repeated until instruction input has been performed for all learning images (step S207). In the following step S208, it is determined whether the result of the instruction input given to the currently displayed learning image has converged. "Convergence" here means that the variation in multiple instruction inputs to one learning image has become sufficiently small. The reason for adding this determination will be described later.

The processing of steps S204 to S206 is repeated until convergence is achieved for the teaching results for all learning images (YES in both steps S207 and S208). If convergence is not achieved (NO in step S208), in step S204, a new image set is constructed taking into account the results of the teaching input up to that point in addition to the same selection criteria as in the first embodiment. For example, the selection criteria are adjusted so that learning images with large fluctuations in the results of multiple teaching inputs are included more frequently in subsequent image sets, and learning images with small fluctuations are included less frequently.

In this way, images that are difficult to judge are displayed in combination with various other images in sequence, allowing the user to evaluate from multiple perspectives. This reduces fluctuations in the teaching results and increases their accuracy, i.e., it is possible to converge the teaching results. On the other hand, for images with little fluctuation and close to convergence, the frequency of their display can be reduced or they can be excluded from the display targets, thereby reducing the effort and time required for the teaching work.

In addition, initially, the image may be paired with a learning image that is easy to see the difference from other learning images, i.e., with a low similarity, but as the teaching input progresses, a combination with a learning image with a higher similarity may be adopted. By first comparing images with relatively large differences, and then comparing images with smaller differences, it is possible to increase the accuracy of the teaching results even in situations where a relative judgment is made between the displayed images.

In this way, by dynamically constructing an image set based on past teaching input results, in this embodiment, it is possible to reduce fluctuations in the teaching results for each learning image and converge to a certain teaching result. The converged teaching result is determined as the teaching result to be associated with each learning image (step S209). The converged teaching result has a high degree of accuracy. Therefore, it is possible to improve the accuracy of the trained model that is constructed as a result of machine learning using this as training data.

Convergence can be determined, for example, when the number of learning images for which the teaching results change with repeated teaching input falls below a certain number. It is desirable to eliminate fluctuations in the teaching results for all learning images, but there may be peculiar images for which the results do not converge no matter what. Since attempting to converge for such images may take a significant amount of time, it may be determined that overall convergence has occurred when convergence has been observed for the majority of the images. Alternatively, an upper limit may be set for the number of repetitions from step S208 to S204, and convergence may be forcibly determined when the number of repetitions reaches the upper limit.

<Screen display example>
4A and 4B are diagrams showing examples of display modes of display images. Fig. 4A shows an example in which all of a plurality of learning images (six images in this case) included in one image set are tiled in a window Wa. In addition, a group of buttons Ba, such as a button for switching the display to the previous image set or the next image set and a button for changing the display magnification, are displayed on the screen. Each learning image is displayed at the same magnification, and when the display magnification is changed, the magnification of all images changes in the same way.

In FIG. 4(b), some of the learning images in the same image set are displayed in a window Wb, and the remaining images can be displayed by moving these images by scrolling horizontally. On the other hand, in FIG. 4(c), the displayed images in a window Wc are moved by scrolling vertically. In these display modes, button groups Bb and Bc similar to those described above are arranged in appropriate positions. Of course, if it is possible to arrange all of the learning images in the same image set in a row vertically or horizontally on the same screen while ensuring sufficient visibility, this is preferable.

Figure 5 is a diagram illustrating an example of a combination of learning images included in one image set. In the example shown in Figure 5 (a), learning images Aa1, Aa2, and Aa3 corresponding to the same tissue and the same disease are arranged in order of index value related to similarity. In this type of combination and display mode, images with relatively high similarity are arranged adjacent to each other, making it easy to compare them.

The example shown in FIG. 5(b) is a schematic diagram showing the change in the display image accompanying the switching of the image set. First, the learning images Aa1 and Aa5, which correspond to the same tissue and the same disease but have a relatively small degree of similarity, that is, it is easy to judge whether they are different or not, are displayed side by side. Here, image Aa5 corresponds to the "first learning image" of the present invention, and the image set at this time corresponds to the "first image set" of the present invention. Then, by switching the image set thereafter, the learning images Aa1 and Aa2, which have a higher degree of similarity and are therefore more difficult to judge, are displayed side by side. Here, image Aa2 corresponds to the "second learning image" of the present invention, and the image set at this time corresponds to the "second image set" of the present invention. This display mode is particularly useful in the second embodiment in which the subsequent image set is dynamically set based on the previous teaching result. In this case, information that allows the previously input teaching result to be known may be added to each learning image.

Figure 5(c) shows an example of a combination of images with different types of tissue and disease. Here, images Aa1 and Ab2, which are of the same tissue A but have different disease types, are combined with image Bd1, which corresponds to tissue B. By including such combinations in the image set, it is ultimately possible to obtain a trained model that can distinguish between specimens that are of different origins but similar in appearance.

Figure 6 is a schematic diagram showing an example of instruction input by a user. Figure 6(a) shows an example using a drop-down list. When one of the displayed images is clicked, candidate labels to be assigned to that image are displayed as a drop-down list. In this example, the level of nuclear atypia (high, medium, low) can be selected as the label. By making such input for each image displayed in an image set, labels can be assigned to all learning images.

Figure 6(b) shows an example of rearrangement of the image order. The image numbers shown in Figure 2 are based on specific index values and may not necessarily match the similarity when an expert comprehensively evaluates the images from various perspectives. As shown by the dashed line in Figure 6(b), if an image can be dragged to swap its order with other images, it will be possible to display the images in an order that corresponds to their original similarity.

Figure 6(c) shows an example of instruction input for setting a "boundary" for images rearranged in this way. Between images rearranged according to similarity, it is possible to set a boundary that represents the difference in the state of each tissue. For example, when there is a mixture of images that do not contain lesions (normal) and images that contain lesions, if they are rearranged according to similarity, it is expected that a boundary will exist that distinguishes between images of normal tissue and images that contain lesions. Figure 6(c) shows an operation in which the user specifies such a boundary as instruction input.

＜Other＞
The present invention is not limited to the above-mentioned embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, in the description of the above-mentioned embodiment, the image selection criteria for constructing an image set, the combination of images, the type of labels, the display mode, and the like are merely examples for the purpose of explanation, and are not limited to the above-mentioned embodiment. For example, in the example of teaching input in FIG. 6, the display example in FIG. 4(b) is cited, but the same can be applied to the display examples shown in FIG. 4(a) and FIG. 4(c).

As described above by way of example of specific embodiments, in the image processing method according to the present invention, the similarity between learning images determined based on, for example, extracted features or the results of instruction input can be used as one of the selection criteria. By comparing images with high and low similarity while changing the combinations, the user can perform more accurate instruction input.

In this case, for example, a first image set including a learning image and a first learning image having a relatively small similarity thereto, and a second image set including the learning image and a second learning image having a larger similarity thereto than the first learning image, may be created, and the display image corresponding to the second image set may be displayed after the display image corresponding to the first image set. With this configuration, images with low similarity, and therefore relatively easy to judge whether they are similar or not, are compared before comparing images with higher similarity, so that it is expected that the relative comparison between the displayed images will result in highly accurate teaching input.

For example, the image set may be dynamically configured according to the results of previous instruction input. The multiple learning images may include images that are expected to have a clear tissue state and little fluctuation in the instruction input results, and images that are not so different from other learning images that the user may have difficulty in making a judgment. When including learning images in multiple image sets, it is desirable to display images that are difficult to judge in combination with various other images to enable a multifaceted judgment, but it is wasteful to have instruction input be performed multiple times for images with a clear state. By reflecting the results of past instruction input in the combination of learning images in subsequent image sets, it is possible to respond to such situations and perform the instruction work efficiently.

For example, the image processing method according to the present invention may further include a step of performing machine learning on the learning model based on a plurality of learning images and associated teaching inputs. The trained model thus constructed using teaching input results with little fluctuation has the function of accurately determining the state of tissue contained in an unknown specimen image.

For example, the specimen image may be an image obtained by bright-field imaging of a tissue specimen stained with hematoxylin and eosin (HE). HE staining can distinguish between cell nuclei and cytoplasm. Furthermore, an image obtained by bright-field imaging of this specimen has good compatibility with visual observation by a user.

This invention effectively supports the user's work to build a trained model for automatically determining tissue conditions from images, and in particular can improve the efficiency and accuracy of teaching work on training images.

Aa1, Aa2, ... Learning images Ba, Bb, Bc Button group Wa, Wb, Wc Window

Claims (6)

extracting features of objects in each of a plurality of learning images of a tissue specimen by image processing;
selecting a plurality of images from among the learning images based on the extracted features and a predetermined selection criterion, and constructing an image set including the selected plurality of learning images;
outputting, to a display device, a display image in which a plurality of the learning images included in the image set are arranged on the same screen;
receiving a user instruction input for each of the displayed learning images and associating the learning image with a corresponding result of the instruction input;
forming a plurality of image sets each having a different combination of the learning images, switching between and displaying a plurality of the display images corresponding to each of the plurality of image sets, and receiving the instruction input each time;
An image processing method in which the learning images are selected while changing the selection criteria, thereby including one learning image in multiple image sets , and displaying each of the learning images multiple times on the display device while accepting the instruction input each time . The image processing method according to claim 1, wherein the similarity between the learning images calculated based on the features or the results of the instruction input is one of the selection criteria. creating a first image set including one of the training images and a first training image, and a second image set including the one of the training images and a second training image, wherein a similarity of the second training image to the one of the training images is greater than that of the first training image;
3. The image processing method according to claim 2, wherein the display image corresponding to the second image set is displayed after the display image corresponding to the first image set. An image processing method according to any one of claims 1 to 3, in which the image set is dynamically constructed according to the results of the previous instruction input. The image processing method according to any one of claims 1 to 4, further comprising a step of performing machine learning on a learning model based on a plurality of the learning images and the associated teaching input. 6. The image processing method according to claim 1, wherein the learning images are images obtained by bright-field imaging of the tissue specimen stained with hematoxylin and eosin.

Images