CN112132232A - Method, system and server for classification and labeling of medical images - Google Patents

Abstract

The invention discloses a medical image classification and labeling method, system and server, and belongs to the technical field of medical images. The method includes: obtaining an original medical image and sending it to a manual labeling terminal; obtaining a manual labeling image returned by the manual labeling terminal and obtaining the pixel boundary of the labeling area through an edge extraction algorithm; The image is processed to obtain a feature map whose size is the size of the original medical image and the depth is the number of annotation types; at the pixel boundary, the feature map is compared with the manual annotation map; The pixel is corrected; otherwise, multiple pixels around the pixel on the feature map are selected, and the category with the largest number of pixels is used as the category of the pixel; the corrected feature map is used as the network annotation map. output.

Description

Method, system and server for classification and labeling of medical images

technical field

The invention belongs to the technical field of medical image processing, and in particular relates to a method, system and server for classifying and labeling medical images.

Background technique

Advanced medical imaging technologies such as ultrasound imaging and magnetic resonance imaging technology generate a large number of 2D and 3D medical images for medical diagnosis. These images contain various information such as pathological tissues and organs, which require medical staff to combine professional knowledge. judge. However, compared with conventional images, medical imaging has the characteristics of poor contrast and high noise, resulting in time-consuming and error-prone diagnosis. By segmenting and labeling medical images in advance, the target area can be quantitatively separated from the background area, which can not only reduce the workload and cost of medical experts in the process of disease diagnosis, but also reduce the error rate of manual processes, thereby improving Diagnostic efficiency and accuracy.

At present, traditional image segmentation methods based on threshold and edge detection are mostly used for segmentation of medical images. These methods are easily affected by noise, and it is difficult to ensure the continuity and closure of segmentation edges. Artificial intelligence algorithms represented by neural networks have been maturely applied in image processing fields such as autonomous driving, and have become an emerging technology to solve medical image processing problems.

SUMMARY OF THE INVENTION

In order to solve the above problems, the embodiments of the present invention provide a method, system, and server for classifying and labeling medical images. The technical solution is as follows:

On the one hand, an embodiment of the present invention provides a method for classifying and labeling medical images, the method comprising:

S101: Obtain the original medical image and send it to the manual labeling terminal;

S102: Obtain the manual annotation map returned by the manual annotation terminal and obtain the pixel boundary of the marked area through an edge extraction algorithm, where the manual annotation map is obtained by manually labeling the target area on the original medical image by the manual annotation terminal according to a predetermined strategy;

S103: Process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map whose size is the size of the original medical image and whose depth is the number of annotation types, where the point with the largest pixel value in the depth direction of the feature map is located. The layer number is the marked category number;

S104: at the pixel boundary, compare the feature map with the manual annotation map; if the pixel point is consistent with the results on the manual annotation map and the feature map, do not correct the pixel point; otherwise, select the pixel point on the feature map around the pixel point. For multiple pixels, the category with the largest number of pixels is used as the category of the pixel;

S105: Output the feature map processed in step S104 as the network annotation map.

Among them, at the manual labeling terminal, the edge contour of the target area is manually drawn, the area enclosed by the edge contour is filled by the connected graph algorithm, and the filled area is manually classified and marked. correspond.

Specifically, the labeling method for manually labeling the terminal in the embodiment of the present invention is:

S201: Manually select a brush of a specific color and select a thick line type, different colors represent different tissues and correspond to corresponding labeling classifications;

S202: Manually use a brush to outline the rough outline of the target area;

S203: through the connected graph algorithm, mark the pixels located in the outline as the brush color to fill the area enclosed by the edge outline;

S204: Manually select the thin line type, and repair the edge of the area;

S205: If there is still an area to be marked, repeat steps S201-S204, if not, obtain a manual marked map.

Further, the classification and labeling method provided by the present invention further includes: each original medical image has a unique data identification code; in step S101, the original medical image and the corresponding data identification code are obtained and sent to the manual labeling terminal together; In step S102, the manual annotation diagram and the corresponding data identification code are obtained, and in step S105, the network annotation diagram and the corresponding data identification code are output; the generation method of the data identification code is:

S301: the original medical image X obtains the vector Y through the linear projection matrix W matrix operation;

Among them, the size of the original medical image X is: the width is w, the height is 3h; the size of the linear projection matrix W is 1*w; the size of the vector Y is 1*3h; the linear projection matrix W is calculated according to the following formula:

S302: Round down each element of the vector Y to the nearest integer, so that each element of the Y vector is between [0, 255];

S303: Map the Y vector to a specific digital identification code ID through a hash function; if the specific digital identification code ID is the same as the existing image ID, add a suffix identifier to the specific digital identification code ID and the existing image ID To distinguish; the specific digital identification code ID with or without the suffix identifier is the data identification code.

Wherein, step S103 specifically includes:

卷积核和

最大池化提取原始医学图像的特征信息，得到初始特征图；S401: Pass multiple

convolution kernel and

Maximum pooling extracts the feature information of the original medical image to obtain the initial feature map;

卷积核卷积处理；再依次通过双线性插值方法与等尺寸卷积将系列尺寸特征图还原到初始特征图大小，得到还原特征图；S402: Downsample the initial feature map to a range of sizes through mean pooling, and pass

Convolution kernel convolution processing; then through the bilinear interpolation method and equal-size convolution in turn, the series size feature map is restored to the initial feature map size, and the restored feature map is obtained;

S403: splicing the restored feature map and the initial feature map to obtain a spliced feature map;

S404: The stitched feature map is restored to the size of the original medical image through the bilinear interpolation method and equal-size convolution in turn, and then a feature map whose size is the size of the original medical image and the depth is the number of annotation types is obtained through the softmax algorithm.

Preferably, the classification and labeling method provided by the present invention further includes: selecting an original medical image with a smaller error between the feature map and the manual labeling map as a high-quality sample, using the high-quality sample as the input of the fully convolutional neural network model, and using the original medical image as the input of the full convolutional neural network model. The network annotation map corresponding to the image is used as the output of the fully convolutional neural network model, and the fully convolutional neural network model is optimized for training.

Specifically, in step S104, if the proportion of pixels whose results are inconsistent between the feature map and the manual annotation map in the pixel boundary is less than a predetermined threshold, it is considered that the error between the feature map and the manual annotation map is small, and the corresponding original medical image is High-quality samples; the optimization training method includes: using the original marked medical images as a verification set, and performing k-fold cross-validation on the high-quality samples.

Specifically, in step S104: the number of the plurality of pixel points is 15-25.

On the other hand, an embodiment of the present invention also provides a system for classifying and labeling medical images, the system comprising:

User: used to upload the original medical image to the server, and can display the network annotation map;

Central database: used to acquire and store the original medical images uploaded by users, distribute the original medical images to the server, store the network annotation map and send the network annotation map to the user;

Server: used to distribute the original medical image to the manual labeling terminal, process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map whose size is the size of the original image and the depth is the number of labeling types, and obtain the manual labeling map And through the edge extraction algorithm, the artificial annotation map is processed to obtain the pixel boundary of the labeled area; at the pixel boundary, the feature map is corrected by comparing with the artificial annotation map to obtain the network annotation map, and the network annotation map is uploaded to the central database;

Manual labeling terminal: used to display the original medical image and manual labeling map, manually label the original medical image to obtain the manual labeling map, and upload the manual labeling map to the server.

In another aspect, an embodiment of the present invention also provides a server, including:

Data transceiver module: used to receive the original medical image sent by the central database, send the original medical image to the manual labeling terminal, and send the network labeling map to the central database;

Image processing module: It is used to process the artificially labeled image through the edge extraction algorithm to obtain the pixel boundary of the labeled area;

Fully convolutional neural network module: used to process the original medical image through the pre-trained fully convolutional neural network model to obtain a feature map whose size is the size of the original medical image and whose depth is the number of annotation types. The depth direction of the feature map is The layer number of the point with the largest pixel value is the labeled category number;

Correction module: It is used to compare the feature map with the artificial annotation map at the pixel boundary; if the result of the pixel point on the artificial annotation map and the feature map is consistent, the pixel point will not be corrected; otherwise, the pixel on the feature map will be selected. For multiple pixels around the point, the category with the largest number of pixels is used as the category of the pixel; the corrected feature map is used as the network annotation map.

Wherein, the fully convolutional neural network module provided by the embodiment of the present invention includes:

Feature map extraction unit: used to extract the feature information of the original medical image through multiple convolutions and pooling to obtain the initial feature map;

卷积核卷积处理，再依次通过双线性插值方法与等尺寸卷积将系列尺寸特征图还原到初始特征图大小，得到还原特征图；Multi-scale convolution pooling unit: used to downsample the initial feature map to a range of sizes through average pooling, through

Convolution kernel convolution processing, and then through the bilinear interpolation method and equal-size convolution to restore the series size feature map to the initial feature map size to obtain the restored feature map;

Splicing unit: used to splicing the restored feature map and the original feature map to obtain a spliced feature map;

Upsampling unit: It is used to restore the stitched feature map to the original medical image size through the bilinear interpolation method and equal-size convolution in turn, and then use the softmax algorithm to obtain a feature map whose size is the size of the original medical image and the depth is the number of annotation types. .

In another aspect, an embodiment of the present invention also provides a manual labeling terminal, including: an original image display module: used to display the original medical image and be able to manually label it. When manually labeling, a rough outline of the target area is first outlined with a thick line. outline, and then use thin lines to repair the edge of the area;

Control module: used to process the labeled original medical images to obtain manual labeled maps;

Manual annotation map display module: used to display manual annotation map;

Selection module: used to select brush color and line thickness, different colors represent different tissues and correspond to corresponding label classifications;

Communication module: used to receive original medical images and upload manual annotations.

The beneficial effects brought by the technical solutions provided by the embodiments of the present invention are: the present invention constructs a medical image segmentation and labeling method and system based on a fully convolutional neural network, which can realize the end-to-end mapping relationship between the original medical image pixels and the network labeled image pixels. , the tensorization operation greatly improves the processing speed of the image; compared with the traditional image processing algorithm, the neural network labeling method used in the present invention does not require artificial design of image processing operators, and can be used in the case of using an excellent training data set. It is used for labeling medical images of different types and sizes; by synthesizing the output labeling results and manual labeling results of the convolutional neural network, the weights of the neural network are optimized to improve the results of image segmentation and labeling, and realize positive feedback in the image processing process. , and continuously improve the efficiency and accuracy of medical imaging diagnosis.

Description of drawings

1 is a flowchart of a method for classifying and labeling medical images provided by an embodiment of the present invention;

2 is a flowchart of a method for manually labeling a terminal provided by an embodiment of the present invention;

3 is an operation interface diagram of a manual labeling terminal provided by an embodiment of the present invention;

4 is an operation flowchart of a manual labeling terminal provided by an embodiment of the present invention;

5 is a flowchart of a method for generating a data identification code provided by an embodiment of the present invention;

Fig. 6 is the concrete flow chart of step S103;

Fig. 7 is the specific processing mode of step S103;

8 is a schematic block diagram of a system for classifying and labeling medical images provided by an embodiment of the present invention;

9 is a schematic block diagram of a server provided by an embodiment of the present invention;

10 is a schematic block diagram of a fully convolutional neural network module provided by an embodiment of the present invention;

FIG. 11 is a schematic block diagram of a terminal for manual annotation provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

Example 1

Referring to FIG. 1, Embodiment 1 discloses a method for classifying and labeling medical images, the method includes:

S101: Obtain the original medical image and send it to the manual labeling terminal, and also obtain the data identification code.

S102: Obtain the manual labeling map (and the corresponding data identification code) returned by the manual labeling terminal, and obtain the pixel boundary of the labeling area through an edge extraction algorithm; wherein, the target area of the manual labeling map on the original medical image is manually labelled by the terminal. It is obtained by classification and labeling according to a predetermined strategy; in this embodiment, only the outline of the target area needs to be labelled (that is, rough labeling), the work intensity is low, and the labeling speed is very fast. Among them, the pixel boundary is the nearby area where the labeling result can be wrong.

S103: Through the pre-trained fully convolutional neural network model, the original medical image is processed to obtain a feature map whose size is the size of the original medical image and the depth is the number of annotation types. Among them, the number of layers (corresponding to N in Figure 7) where the point with the largest pixel value in the depth direction of the feature map is located is the labeled category number.

S104: At the pixel boundary, compare the feature map with the manual annotation map (pixel points). If the result of the pixel point on the artificial annotation map and the feature map is consistent, the pixel point will not be corrected. On the contrary, select multiple (eg 20) pixels around the pixel on the feature map, and use the category with the largest number of pixels as the category of the pixel to correct the corresponding pixel in the feature map. This step realizes the correction of the feature map, which can not only improve the accuracy of the network annotation map, but also be used for feedback optimization training. Specifically, among the 20 pixels around the marked different pixels, there are 8 pixels whose category numbers are category 1, 5 pixels whose category numbers are category 2, and 3 pixels whose category numbers are category 3, The category number of 2 pixels is category 4 and the category number of 2 pixels is category 4, then the category with the largest number is category 1, and the category numbers marked with different pixels are category 1. Specifically, in step S104: the number of the plurality of pixels is 15-25, specifically 20; the selection of the number of pixels is selected according to the complexity of the image, the more complex the image, the greater the number.

S105: Output the corrected feature map (and the corresponding data identification code) as the network annotation map; of course, the manual annotation map may also be output together.

Among them, at the manual labeling terminal, the edge contour of the target area is manually drawn, and the area enclosed by the edge contour is filled by the connected graph algorithm (specifically, the same color as the edge contour can be used), and the filled area is manually classified and marked ( Color can be directly used for classification, or it can be filled and then defined). The manual labeling map corresponds to the category labeled by the feature map. When manually labeling, a certain strategy should be followed (such as the definition according to the actual situation, the color of cancer tissue, the color of diseased tissue) , the color of normal tissue, etc.) to ensure that the manual annotation map corresponds to the category marked by the feature map.

Specifically, referring to FIGS. 2-4 , a labeling method for manually labeling a terminal in the embodiment of the present invention is:

S201: Manually select a brush of a specific color and select a thick line type. Different colors represent different tissues and correspond to corresponding labeling classifications.

S202: Manually use a brush to outline the rough outline of the target area (for example, use a thick dotted line to outline), and click OK.

S203: Using a connected graph algorithm, mark the pixels located in the outline as the brush color to fill the area enclosed by the edge outline.

S204: Manually select the thin line type, and repair the edge of the area to make the marked area as correct as possible.

S205: If there is still an area to be marked, repeat steps S201-S204 (different parts use corresponding colors), if not, get a manual marked map, click the next one to mark the next one.

S206: If there is an image that needs to be modified, click the previous image to review it.

S207: After all images are marked, packaged and sent to the server.

Wherein, the marking process of the present invention can be implemented on a conventional PC, preferably on a touch screen device to facilitate outline drawing.

Specifically, the operator logs in to the account assigned by the server to obtain the original medical image. The manual labeling terminal has the functions of displaying, switching, and storing medical images. After completing the annotation of a medical image, it is temporarily stored in the manual annotation terminal, and the next image is annotated. After completing the rough labeling of a group of, for example, 20 images, the manual labeling terminal sends the labeling data back to the server, and the server performs the next step of processing. Similarly, the manual labeling terminal can also work online. For example, every time the user completes the rough labeling of a medical image, the terminal returns the crudely labelled image to the server, and downloads a new medical image from the server for labeling.

Further, the classification and labeling method provided by the present invention further includes: each original medical image has a unique data identification code; in step S101, the original medical image and the corresponding data identification code are obtained and sent to the manual labeling terminal together; In step S102, the manual annotation map and the corresponding data identification code are obtained, and in step S105, the network annotation map and the corresponding data identification code are output.

Specifically, after the central database generates the data identification code, the original medical image and the data identification code are sent to the server, and the server randomly distributes the medical image to be labeled (original medical image) to the manual labeling terminal. After the manual labeling terminal receives the data, the operator manually labels it roughly, and sends the crude labeling map back to the server. The server processes the rough-labeled manual annotation map, extracts the accurate pixel edges of the coarse-labeled features, and combines with the fully convolutional neural network model to achieve pixel-level segmentation and multi-classification of the target tissue or organ. After the image annotation is completed, each server sends the marked image (network annotation map) back to the central database, and then the central database returns it to the user. The central database can generate a unique data identification code for each original medical image uploaded by the user to establish storage and indexing, and the data identification code is transmitted to the server together with the corresponding image. After the server completes the processing of the medical image, the image is sent back to the central database, and the central database establishes a one-to-one correspondence between the marked medical image and the original medical image according to the data identification code and stores it. Users can extract original medical images, manual annotation maps and network annotation maps from the central database according to their needs.

Specifically, referring to Fig. 5, the method for generating the data identification code in the embodiment of the present invention is:

S301: the original medical image X obtains the vector Y through the linear projection matrix W matrix operation;

Among them, the size of the original medical image X is: the width is w, the height is 3h; the size of the linear projection matrix W is 1*w; the size of the vector Y is 1*3h; the linear projection matrix W is calculated according to the following formula:

。

.

S302: Each element of the vector Y is rounded down to the nearest integer, so that each element of the Y vector is between [0, 255].

S303: Map the Y vector to a specific digital identification code ID through a hash function; if the specific digital identification code ID is the same as the existing image ID, add a suffix identifier to the specific digital identification code ID and the existing image ID To distinguish; the specific digital identification code ID with (same as the existing image ID) or without (not the same as the existing image ID) suffix identifier is the data identification code.

6 and 7, step S103 specifically includes:

S401: Extract the feature information of the original medical image (specifically, 512*512*3) through multiple convolutions and pooling, while reducing the size of the image, deepen the depth of the feature map, thereby extracting the deep layers contained in the original image. feature information to obtain the initial feature map.

卷积核卷积处理（结果分别为8*8*256、4*4*256、2*2*256和1*1*256）。再依次通过双线性插值方法与等尺寸卷积将系列尺寸特征图还原到初始特征图大小，得到还原特征图。S402: Downsample the initial feature map to a range of sizes (such as downsampling by 8*8*256, 4*4*256, 2*2*256, and 1*1*256) through mean pooling, and pass

Convolution kernel convolution processing (results are 8*8*256, 4*4*256, 2*2*256 and 1*1*256 respectively). Then, the series size feature map is restored to the original feature map size by bilinear interpolation method and equal-size convolution in turn, and the restored feature map is obtained.

S403: Splicing the restored feature map and the initial feature map to obtain a spliced feature map, which is consistent with the conventional technology.

S404: Restore the stitched feature map to the size of the original medical image through the bilinear interpolation method and equal-size convolution in turn, and then use the softmax algorithm to obtain a feature map whose size is the size of the original medical image and the depth is the number of annotation types (512*512 *N, where N is the number of annotation types).

卷积核和

最大池化提取原始医学图像的特征信息，得到初始特征图。具体一次为卷积3*3*64、最大池化2*2、卷积3*3*256、最大池化2*2、卷积3*3*512。Specifically, step S401 includes: through multiple

convolution kernel and

Maximum pooling extracts the feature information of the original medical image and obtains the initial feature map. The specific one is convolution 3*3*64, max pooling 2*2, convolution 3*3*256, max pooling 2*2, and convolution 3*3*512.

Preferably, the classification and labeling method provided by the present invention further includes: selecting an original medical image with a smaller error between the feature map and the manual labeling map as a high-quality sample, using the high-quality sample as the input of the fully convolutional neural network model, and using the original medical image as the input of the full convolutional neural network model. The network annotation map corresponding to the image is used as the output of the fully convolutional neural network model, and the fully convolutional neural network model is optimized and trained. The optimization training is a conventional technique in the field, and detailed description is omitted in this embodiment.

Specifically, in step S104, if the proportion of pixels whose results are inconsistent between the feature map and the manual annotation map in the pixel boundary (pixel set) (the proportion of the number of pixels) is less than a predetermined threshold (design according to actual needs, specific can be 1%), it is considered that the error between the feature map and the manual annotation map is small, and the corresponding original medical image is a high-quality sample. The optimized training method includes: using the original annotated medical images (stored in the central database for training) as the validation set, performing k-fold cross-validation on high-quality samples, and improving the output of the fully convolutional neural network model and the actual image annotation Average intersection ratio of graphs.

Example 2

Referring to FIG. 8 , Embodiment 2 discloses a system for classifying and labeling medical images, and the system includes:

User: used to upload the original medical images to the server, and can display the network annotation map.

Central database: used to acquire and store the original medical images uploaded by users, distribute the original medical images to the server, store the network annotation map and send the network annotation map to the user.

Server: used to distribute the original medical image to the manual labeling terminal, process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map whose size is the size of the original image and the depth is the number of labeling types, and obtain the manual labeling map And through the edge extraction algorithm, the artificial annotation map is processed to obtain the pixel boundary of the annotation area; at the pixel boundary, the feature map is corrected by comparing with the manual annotation map (for details, please refer to step S104) to obtain the network annotation map, and the network annotation map is uploaded. to the central database.

Manual labeling terminal: used to display the original medical image and manual labeling map, manually label the original medical image to obtain the manual labeling map, and upload the manual labeling map to the server.

Among them, users such as hospitals, medical research institutes, medical companies, etc., after obtaining medical images, upload the medical images to the central database.

Among them, the central database can connect one or more servers according to the actual number of medical images, and automatically distribute the original medical images to different servers for processing. When the number of medical images to be processed is small, the server can be combined with the central database. After receiving the original medical image, the server automatically distributes the image to one or more manual labeling terminals for rough labeling. The central database can be set up in the local area network, cloud storage, etc. of the user unit, and users can upload data through wireless network, Ethernet, USB and other communication methods. The central database sets a dedicated account and password for each user, and assigns permissions such as read, write, and change to ensure data security. After acquiring the medical images uploaded by the user, the central database generates a unique digital identification code for each image, binds it with the corresponding user, and distributes the original medical image and the digital identification code to the server.

Among them, the server may adopt various computing platforms such as physical server, cloud server, personal computer, etc. The server assigns an account number and password to each manual labeling terminal, and after receiving the original medical image and digital identification code, distributes it to the manual labeling terminal, and the manual labeling unit performs rough labeling.

Among them, the manual labeling terminal can run on Windows, Android, IOS and other operating systems, and can perform cross-platform data synchronization. After the operator completes the rough annotation of the original image, the manual annotation map can be sent back to the server.

The manual labeling terminal sends the manual labeling map back to the server; the server transmits the processed network labeling map back to the central database for users to download. K, M, N, etc. in FIG. 8 refer to unknown quantities. In practical applications, servers can be increased or decreased according to the actual number of images, and each server runs the same instructions and a fully convolutional neural network model. Here, the central database and the server refer to corresponding functional units. In practical applications, the central database and the server can also be integrated into one. Manual labeling terminals can be computers, laptops, tablet computers, mobile phones, etc., and can run on Windows, Android, IOS and other operating systems, and the number of them can be increased or decreased according to actual applications.

Example 3

Referring to FIG. 9, Embodiment 3 discloses a server, including:

Data transceiver module: used to receive and store the original medical images sent by the central database, send the original medical images to the manual labeling terminal, and send the network labeling map to the central database. is a common structure.

Image processing module: It is used to process the artificially labeled map through the edge extraction algorithm to obtain the pixel boundary of the labeled area.

Fully convolutional neural network module: used to process the original medical image through the pre-trained fully convolutional neural network model to obtain a feature map whose size is the size of the original medical image and whose depth is the number of annotation types, and the pixel value in the depth direction of the feature map. The layer number where the largest point is located is the labeled category number.

Correction module: It is used to compare the feature map with the artificial annotation map at the pixel boundary; if the result of the pixel point on the artificial annotation map and the feature map is consistent, the pixel point will not be corrected; otherwise, the pixel on the feature map will be selected. For multiple pixels around the point, the category with the largest number of pixels is used as the category of the pixel to correct the corresponding pixel in the feature map; the corrected (after step S104) feature map is used as Network annotation diagram.

Storage module: used to store data.

Further, the server also includes a feedback module for selecting an original medical image with a smaller error between the feature map and the manual annotation map as a high-quality sample, using the high-quality sample as the input of the full convolutional neural network model, and using the original medical image corresponding to the high-quality sample. The network annotation map is used as the output of the fully convolutional neural network model, and the fully convolutional neural network model is optimized for training.

10, the fully convolutional neural network module provided by the embodiment of the present invention includes:

卷积核和

最大池化提取原始医学图像的特征信息，得到初始特征图。Feature map extraction unit: used to extract the feature information of the original medical image through multiple convolutions and pooling to obtain the initial feature map;

convolution kernel and

Maximum pooling extracts the feature information of the original medical image and obtains the initial feature map.

卷积核卷积处理，再依次通过双线性插值方法与等尺寸卷积将系列尺寸特征图还原到初始特征图大小，得到还原特征图。Multi-scale convolution pooling unit: used to downsample the initial feature map to a range of sizes through average pooling, through

Convolution kernel convolution processing, and then through the bilinear interpolation method and equal-size convolution to restore the series size feature map to the initial feature map size to obtain the restored feature map.

Splicing unit: used to splicing the restored feature map and the original feature map to obtain a spliced feature map;

Upsampling unit: It is used to restore the stitched feature map to the original medical image size through the bilinear interpolation method and equal-size convolution in turn, and then use the softmax algorithm to obtain a feature map whose size is the size of the original medical image and the depth is the number of annotation types. .

Example 4

Referring to FIG. 11 , Embodiment 4 discloses a manual labeling terminal, including: an original image display module: used to display the original medical image and be able to manually label it. When manually labeling, a rough outline of the target area is first outlined with a thick line. Then use a thin line to repair the edge of the area.

Control module: It is used to process the labeled original medical images to obtain manual labeling images, and at the same time control the entire manual labeling terminal, such as outputting display data to the original image display module and the manual labeling image display module.

Manual annotation map display module: used to display manual annotation map.

Selection module: used to select the brush color and line thickness. Different colors represent different organizations and correspond to corresponding label classifications. Specifically, multiple line types (at least two) and multiple colors (defined according to a predetermined strategy) can be selected.

Communication module: used to receive original medical images and upload manual annotations.

Specifically, the original image display module, the manual annotation image display module and the selection module can be implemented by the same touch display screen.

Specifically, the original image display module generates a blank canvas with the same size as the original image, the selection module can select brushes of different colors and line types of different thicknesses, and the selection module can also include a confirmation button, a next button, a previous button and a finish button. After the operator completes the rough annotation of the original image and clicks the confirmation button, the annotation will be saved to the annotation map cache area and the filled annotation map will be displayed on the manual annotation map display module; the operator clicks the The next button can jump to the next unmarked original image for annotation; the previous button can review the image that has been marked before. If the image is modified during review, the modified result will be saved to the marked image cache Area. When the original image display area is blank, it indicates that the server has not assigned tasks at present, and the completion button is clicked to complete the labeling.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. A method for classification labeling of medical images, the method comprising:

s101: acquiring an original medical image and sending the original medical image to a manual labeling terminal;

s102: acquiring an artificial annotation figure returned by an artificial annotation terminal and obtaining a pixel boundary of an annotation region through an edge extraction algorithm, wherein the artificial annotation figure is obtained by artificially classifying and annotating a target region of the artificial annotation terminal on an original medical image according to a preset strategy;

s103: processing an original medical image through a pre-trained full convolution neural network model to obtain a feature map with the size of the original medical image and the depth of the feature map as the number of labeled types, wherein the sequence number of the layer number of a point with the largest pixel value in the depth direction of the feature map is a labeled class number;

s104: comparing the feature map with the artificial labeling map at the pixel boundary; if the results of the pixel points on the artificial annotation graph and the characteristic graph are consistent, the pixel points are not corrected; otherwise, selecting a plurality of pixel points around the pixel point on the characteristic diagram, and taking the category with the largest number in the plurality of pixel points as the category of the pixel point;

s105: and outputting the feature graph processed in the step S104 as a network annotation graph.

2. The medical image classification labeling method according to claim 1, wherein, at the manual labeling terminal, the edge contour of the target region is manually drawn, the region surrounded by the edge contour is filled through a connected graph algorithm, the filled region is manually classified and labeled, and the manual labeling graph corresponds to the classification of the feature graph labeling.

3. The classification labeling method of medical images according to claim 2, wherein the labeling method of the manual labeling terminal is as follows:

s201: manually selecting a painting brush with a specific color and selecting a thick line type, wherein different colors represent different tissues and correspond to corresponding label classifications;

s202: manually drawing a rough outline of the target area by using a painting brush;

s203: marking pixel points in the outline as the color of a painting brush by a connected graph algorithm so as to fill the area enclosed by the edge outline;

s204: manually selecting a thin line type, and repairing the edge of the area;

s205: if the regions are still to be labeled, the steps S201-S204 are repeated, and if not, a manual labeling diagram is obtained.

4. The method for classification labeling of medical images according to claim 1, further comprising: each original medical image has a unique data identification code; in step S101, an original medical image and a corresponding data identification code are obtained and sent to a manual labeling terminal together; in step S102, acquiring a manual annotation graph and a corresponding data identification code, and in step S105, outputting a network annotation graph and a corresponding data identification code;

the generation method of the data identification code comprises the following steps:

s301: obtaining a vector Y of the original medical image X through linear projection matrix W matrix operation;

wherein the size of the original medical image X is: the width is w, and the height is 3 h; the size of the linear projection matrix W is 1 × W; the size of the vector Y is 1 x 3 h; the linear projection matrix W is calculated according to the following formula:

s302: each element of the vector Y is rounded down to the nearest integer, such that each element of the Y vector is between [0,255 ];

s303: mapping the Y vector into a specific digital identification code ID through a hash function; if the specific digital identification code ID is the same as the existing image ID, distinguishing the specific digital identification code ID and the existing image ID by adding a suffix identifier; the specific digital identification code ID with or without a suffix identifier is the data identification code.

5. The method for classifying and labeling medical images according to claim 1, wherein the step S103 specifically comprises:

s401: by a plurality of

Convolution kernel sum

Extracting feature information of the original medical image in a maximum pooling manner to obtain an initial feature map;

s402: downsampling an initial feature map to a series of sizes by mean pooling

Performing convolution kernel convolution processing; reducing the series size characteristic diagram to the size of the initial characteristic diagram through a bilinear interpolation method and equal-size convolution to obtain a reduced characteristic diagram;

s403: splicing the restored characteristic diagram and the initial characteristic diagram to obtain a spliced characteristic diagram;

s404: and restoring the spliced feature map to the size of the original medical image through a bilinear interpolation method and equal-size convolution in sequence, and obtaining the feature map with the size of the original medical image and the depth of the marked type number through a softmax algorithm.

6. The method for classification labeling of medical images according to claim 1, further comprising: and selecting an original medical image with a small error between the characteristic graph and the artificial annotation graph as a high-quality sample, taking the high-quality sample as the input of the full convolution neural network model, taking the network annotation graph corresponding to the original medical image as the output of the full convolution neural network model, and performing optimization training on the full convolution neural network model.

7. The method for classification labeling of medical images according to claim 6,

in step S104, if the ratio of the pixel points of the feature map and the artificial labeling map with inconsistent results in the pixel boundary is smaller than the predetermined threshold, the error between the feature map and the artificial labeling map is considered to be small, and the corresponding original medical image is a high-quality sample;

the optimization training method comprises the following steps: and taking the original labeled medical image as a verification set, and performing k-fold cross verification on the high-quality sample.

8. A system for classification labeling of medical images, the system comprising:

the user: the system is used for uploading the original medical image to a server and displaying a network annotation graph;

a central database: the system comprises a server, a network annotation graph and a user, wherein the server is used for acquiring and storing an original medical image uploaded by the user, distributing the original medical image to the server, storing the network annotation graph and sending the network annotation graph to the user;

a server: the system comprises a data processing module, an edge extraction algorithm module, a data processing module and a data processing module, wherein the data processing module is used for distributing an original medical image to an artificial labeling terminal, processing the original medical image through a pre-trained full convolution neural network model to obtain a feature map with the size of the original image and the depth of the feature map as the number of labeling types, acquiring an artificial labeling map and processing the artificial labeling map through the edge extraction algorithm to obtain a pixel boundary of a labeling area; at the pixel boundary, correcting the characteristic graph by comparing with the artificial labeled graph to obtain a network labeled graph, and uploading the network labeled graph to a central database;

manual labeling of the terminal: the system is used for displaying the original medical image and the artificial annotation drawing, carrying out artificial annotation on the original medical image to obtain the artificial annotation drawing, and uploading the artificial annotation drawing to the server.

9. A server, comprising:

a data receiving and transmitting module: the system comprises a central database, a manual annotation terminal and a network annotation terminal, wherein the central database is used for receiving an original medical image sent by the central database, sending the original medical image to the manual annotation terminal and sending a network annotation drawing to the central database;

an image processing module: the pixel boundary of the labeling area is obtained by processing the artificial labeling graph through an edge extraction algorithm;

a full convolution neural network module: the method comprises the steps that a pre-trained full convolution neural network model is used for processing an original medical image to obtain a feature map with the size of the original medical image and the depth of the feature map as the number of labeled types, and the serial number of the layer number where a point with the largest pixel value in the depth direction of the feature map is located is a labeled class number;

a correction module: the characteristic graph and the artificial labeling graph are compared at the boundary of the pixel; if the results of the pixel points on the artificial annotation graph and the characteristic graph are consistent, the pixel points are not corrected; otherwise, selecting a plurality of pixel points around the pixel point on the characteristic diagram, and taking the category with the largest number in the plurality of pixel points as the category of the pixel point; and taking the corrected feature diagram as a network annotation diagram.

10. The server of claim 9, wherein the full convolutional neural network module comprises:

a feature map extraction unit: the method comprises the steps of extracting characteristic information of an original medical image through multiple convolution and pooling to obtain an initial characteristic map;

a multi-scale convolution pooling unit: for down-sampling the initial feature map to a range of sizes by averaging pooling

Performing convolution kernel convolution processing, and reducing the series size characteristic diagram to the size of the initial characteristic diagram through a bilinear interpolation method and equal-size convolution in sequence to obtain a reduced characteristic diagram;

splicing unit: the method comprises the steps of splicing a reduction characteristic diagram and an initial characteristic diagram to obtain a spliced characteristic diagram;

an up-sampling unit: and the feature map with the size of the original medical image and the depth of the marked type number is obtained through a softmax algorithm.

Images