CN118212459A - A method, device and medium for predicting esophageal structure, tumor contour and stage - Google Patents

Abstract

The invention discloses a method, a device and a medium for predicting esophageal structure, tumor contour and stage, wherein the method comprises the following steps: extracting features of the original ultrasonic endoscope picture by using the model to obtain full-size features; carrying out structure and node processing on the full-size characteristics to obtain a three-layer esophagus structure and control points of tumor growth contours; generating a cubic B-spline curve of esophagus and tumor according to the control points; obtaining a tumor staging result by calculating the relative distance between the three B-spline curves; and forming a prediction result by the cubic B-spline curve and the tumor stage result. The invention provides a method, a device and a medium for predicting esophageal structures and tumor contours and stage, which can accurately represent the predicted results of the esophageal wall structures and the tumor contours by generating three times of B-spline curves, can ensure the interpretability of the tumor stage results by calculating the relative distance between the curves, and can solve the problem of low accuracy of tumor positioning and stage results.

Description

A method, device and medium for predicting esophageal structure, tumor contour and stage

Technical Field

The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for predicting tumor staging probability based on esophageal structure.

Background technique

Gastric cancer, colorectal cancer and esophageal cancer are among the most common cancers in the world. Early detection of the disease is crucial to the cure rate. Ultrasound endoscopic examination has special advantages in distinguishing tumor T1-T4 stages, or the more detailed T1a and T1b stages, and is an important method for early detection of digestive tract malignant diseases. At the same time, artificial intelligence technology can assist medical staff in more accurately identifying early cancers; artificial intelligence-assisted endoscopic examination can improve screening efficiency, assist medical staff in making more accurate diagnoses, solve the problems of early screening and early diagnosis, and create a new situation in the medical field.

However, artificial intelligence medical technology still has many problems in the processing and diagnosis of ultrasound endoscopic images. On the basis of using the neural network diagnostic model to select and extract the diagnosis-related areas, the convolutional neural network (CNN) is directly used to obtain the classification prediction results, resulting in a lack of good interpretability of the classification prediction results. The imaging quality of ultrasound endoscopic images is unstable, and the existing methods fail to effectively use anatomical knowledge for tumor identification and staging differentiation, resulting in low accuracy of tumor localization and staging results.

Summary of the invention

The present invention provides a method, device and medium for predicting tumor staging probability based on esophageal structure, so as to solve the problem of low accuracy of tumor positioning and staging results.

In order to solve the above problems, the present invention provides a method for predicting esophageal structure and tumor contour and staging, comprising:

Obtain original endoscopic ultrasound images;

Using a preset model to extract features from the original ultrasound endoscopy image to obtain full-size features;

Use the preset model to perform structural decomposition and node reconstruction on the full-size feature to obtain control points of the three-layer esophageal structure and tumor growth contour;

Generating cubic B-spline curves of the esophagus and the tumor according to the control points by means of spline interpolation;

The tumor staging result is obtained by calculating the relative distance between the cubic B-spline curves;

The cubic B-spline curve and the tumor staging result form a prediction result of the esophageal structure, tumor contour and tumor staging;

Among them, the preset model is a deep learning model composed of different devices and modules.

The present invention can remove redundant images by extracting features from original endoscopic ultrasound images, thereby effectively ensuring the data accuracy of endoscopic ultrasound images and reducing the complexity of data processing; by constructing control points of the three-layer esophageal structure and tumor growth contour, the characteristics of tumor growth can be fully acquired, which is conducive to the effective evaluation of the esophageal disease state; by generating a cubic B-spline curve in a spline interpolation manner, discrete point data can be connected into a curve to accurately represent the shape of the esophagus and the tumor; because the relative distance between the curves can represent the degree of tissue invasion between different membrane layers, the tumor staging result can be obtained quickly and conveniently.

Compared with the prior art, the present invention can improve the accurate positioning of the model on the region of interest by introducing a model for image processing, and provide a cubic B-spline curve that conforms to the anatomical structure through multi-layer polygon generation technology, which can accurately represent the esophageal wall structure contour prediction and tumor contour prediction results, and by calculating the relative distance between the curves, the degree of tissue invasion can be obtained, ensuring the interpretability of the tumor staging results, and can solve the problem of low accuracy of tumor positioning and staging results.

As a preferred solution, the preset model is used to perform structural decomposition and node reconstruction on the full-size feature to obtain the control points of the three-layer esophageal structure and tumor growth contour, specifically:

Using the structure detection module of the preset model, the full-size features are dimensionally transformed and split to obtain global feature tokens;

The global feature token is combined with the preset esophageal and tumor control point features to obtain the control points of the three-layer esophageal structure through coordinate transformation;

By performing feature mapping on the preset tumor features based on different radii, the control points of the tumor's outward and inward growth contours are obtained.

This preferred solution predicts the control points of the contour lines in the two dimensions of tumor outward growth and inward growth to obtain the control points of the tumor outward and inward growth contour lines, which can fully obtain the characteristics of tumor growth and is conducive to the effective evaluation of the esophageal disease status.

As a preferred solution, the structure detection module of the preset model is used to perform dimension transformation and splitting on the full-size features to obtain global feature tokens, specifically:

Using the structure detection module of the preset model, performing dimension transformation on the full-size feature to obtain a first token;

constructing a learnable second token according to the number of esophageal structure control points that need to be predicted;

Concatenate the first token and the second token to obtain a third token;

The third token is input into the Transformer module of the preset model and token splitting is performed to obtain the global feature token.

The steps of dimension transformation, token construction, token splicing and token splitting of the full-size features in this preferred solution are the process of organizing and collecting the feature information in the full-size features, so that the obtained global feature tokens have good invariance and intuitive representation while having global features, which helps to provide a solid data foundation for the later esophageal structure and tumor prediction.

As a preferred solution, the global feature token is combined with the preset esophageal and tumor control point features to obtain the control points of the three-layer esophageal structure through coordinate transformation, specifically:

Using a first preset formula, calculating the esophagus and tumor control point features according to the global feature tokens;

Mapping the esophagus and tumor control point features using a linear transformation layer, and converting the mapping results into polar coordinates to obtain node coordinates of the mucosal layer contour line;

According to the control point features, the relative radii of the submucosal layer and the adventitia layer are calculated using the linear transformation layer, and the actual radii of the submucosal layer and the adventitia layer are calculated according to the node coordinates and the relative radii;

According to the actual radius and the angle of the node coordinates relative to the center of the esophagus, the control points corresponding to the control point features are converted into a form represented by Cartesian coordinates to obtain the control points of the three-layer esophagus structure.

The relative radius of the submucosal layer and the adventitia layer calculated by this preferred solution has certain limitations and cannot accurately represent the distance between different mucosal layers. Since the relative radius represents the relative displacement, the actual radius is calculated based on the relative displacement and combined with the node coordinates, thereby accurately representing the displacement of the submucosal layer and the adventitia layer relative to the contour line of the mucosal layer.

As a preferred solution, by performing feature mapping on the preset tumor features based on different radii, the control points of the tumor's outward and inward growth contours are obtained, specifically:

define tumor characteristics;

Mapping the tumor feature on a first radius scale and a second radius scale respectively to obtain a first relative radius and a second relative radius;

The first relative radius and the second relative radius are used in sequence to calculate with the radius of the node coordinate relative to the center of the esophagus, to obtain the actual radii of the outward and inward growth contours of the tumor respectively;

According to the actual radius of the tumor's outward and inward growth contours and the angle of the mucosal layer, the coordinates of the node positions are transformed to obtain the control points of the tumor's outward and inward growth contours.

This preferred solution is the construction process of the control points of the tumor growth contour. The conversion relationship between Cartesian coordinates and polar coordinates is referenced in the processing of control points. The advantages of Cartesian coordinates in terms of intuitive expression, concise description and accurate description can be utilized to make the calculation method of the tumor growth shape relatively simple, thereby being able to clearly describe the control points of the tumor growth contour.

As a preferred solution, a cubic B-spline curve of the esophagus and the tumor is generated according to the control points by spline interpolation, specifically:

Using a cubic B-spline interpolation method to process the control points of the three-layer esophageal structure and tumor growth contour respectively, to obtain a parameterized esophageal structure contour line and a tumor contour line;

On the esophageal structure contour line and the tumor contour line, calculating the normal vector of each curve point to obtain a unit vector of the normal line;

The cubic B-spline curves of the esophagus and the tumor are constructed by the esophagus structure contour line, the tumor contour line, and the unit vector of the normal line.

This preferred solution connects discrete point data into a curve using control points as basic elements, thereby obtaining a curve with high precision, convergence and accurate fitting as the structural contour line, and by combining the normal vector, a complete cubic B-spline curve is obtained, which can be used to accurately represent the shape of the esophagus and tumors.

As a preferred solution, the tumor staging result is obtained by calculating the relative distance between the cubic B-spline curves, specifically:

According to the first contour line of the tumor outward growth and the second contour line of the submucosal layer in the cubic B-spline curve, constructing the Euclidean distance between all points on the curve to obtain a distance matrix;

By indexing, performing distance calculation and normal vector judgment from the first contour line to the second contour line according to the distance matrix, obtaining the maximum distance from each point on the first contour line to the nearest point on the second contour line, and using the maximum distance as a relative distance value set;

In the relative distance value set, the tumor staging result is obtained according to the numerical value of the relative distance between the tumor outward growth contour line and the submucosal layer, and the numerical value of the relative distance between the tumor outward growth contour line and the adventitia layer.

This preferred solution can accurately represent the distances between all points on the curve by constructing a distance matrix. By means of indexing, points on different curves can be associated with each other, so as to obtain the maximum value of the distance from each point on the first contour line to the nearest point on the second contour line. Moreover, the relative distances in the relative distance value set can be positive or negative, so by judging the size of the value, the degree of tissue invasion between different membrane layers can be known, thereby obtaining the tumor staging result.

As a preferred solution, a preset model is used to extract features from the original ultrasound endoscopy image to obtain full-size features, specifically:

Using the effective area detector of the preset model, identifying and cutting the effective area boundary box of the original ultrasound endoscopy image to obtain a valid ultrasound endoscopy image;

The feature encoder and feature decoder of the preset model are used to perform feature extraction and upsampling on the effective ultrasound endoscopy image to obtain the full-size feature.

This preferred solution can remove redundant images by identifying and cropping the effective area boundary box, effectively ensuring the data accuracy of the ultrasound endoscopic image while reducing the complexity of data processing. Through feature extraction and upsampling, tumor-related features can be extracted from the effective ultrasound endoscopic image, so that features such as tumor size, shape and density are included in the full-size features, which can better describe the detailed features of the original ultrasound endoscopic image.

As a preferred solution, the preset model is a deep learning model composed of different devices and modules, and also includes:

By measuring the consistency between the segmentation mask predicted by the preset model and the actual segmentation mask, a DICE loss function is obtained;

By measuring the error between the predicted nodes and the actual nodes of the preset model, a polygon loss function is obtained;

A classification loss function is obtained by quantifying the error between the T stage label predicted by the preset model and the actual T stage label;

Using a preset training set and masks of the tumor and the esophageal wall, the preset model is trained according to the DICE loss function, the polygon loss function and the classification loss function to obtain an optimized preset model.

In this preferred solution, the DICE loss function can enhance the accuracy of recognition and positioning of specific targets, the polygonal loss function can improve the prediction accuracy of esophageal and tumor contour nodes, and the classification loss function can improve the prediction accuracy of tumor staging results. Therefore, the loss functions established from different dimensions can significantly improve the optimization of the model and enhance the prediction accuracy of the preset model.

As a preferred solution, the training set is specifically:

Obtain training data from esophageal cancer tumor cases;

Transforming the polygonal contour points of the mucosal layer, the submucosal layer and the adventitia layer in the training data to obtain first transition data;

generating second transition data including contour point data, a segmentation bitmap mask and a staging label of the ultrasound endoscopy image according to the first transition data;

Data enhancement is performed on the second transition data to obtain the training set.

The present invention also provides a prediction device for esophageal structure and tumor contour and staging, comprising a data module, a feature module, a node module, a curve module, a staging module and an integration module;

Wherein, the data module is used to obtain the original ultrasound endoscopy image;

The feature module is used to extract features from the original ultrasound endoscopy image using a preset model to obtain full-size features;

The node module is used to perform structural decomposition and node reconstruction on the full-size feature using the preset model to obtain control points of the three-layer esophageal structure and tumor growth contour;

The curve module is used to generate a cubic B-spline curve of the esophagus and the tumor according to the control points by means of spline interpolation;

The staging module is used to obtain a tumor staging result by calculating the relative distance between the cubic B-spline curves;

The integrated module is used to construct the prediction results of esophageal structure, tumor contour and tumor staging based on the cubic B-spline curve and the tumor staging result;

Among them, the preset model is a deep learning model composed of different devices and modules.

As a preferred solution, the node module includes a global unit, a transformation unit and a mapping unit;

The global unit is used to use the structure detection module of the preset model to perform dimension transformation and splitting on the full-size feature to obtain a global feature token;

The transformation unit is used to combine the global feature token with the preset esophageal and tumor control point features to obtain the control points of the three-layer esophageal structure through coordinate transformation;

The mapping unit is used to perform feature mapping on preset tumor features based on different radii. As a preferred solution, the global unit includes a dimension subunit, a construction subunit, a splicing subunit and a splitting subunit;

The dimension subunit is used to perform dimension transformation on the full-size feature using the structure detection module of the preset model to obtain a first token;

The construction subunit is used to construct a learnable second token according to the number of esophageal structure control points that need to be predicted;

The splicing subunit is used to splice the first token and the second token to obtain a third token;

The splitting subunit is used to input the third token into the Transformer module of the preset model and perform token splitting to obtain the global feature token.

As a preferred solution, the transformation unit includes a feature subunit, a coordinate subunit, a radius subunit and a control subunit;

Wherein, the feature subunit is used to calculate the esophageal and tumor control point features according to the global feature token using a first preset formula;

The coordinate subunit is used to map the esophagus and tumor control point features using a linear transformation layer, and convert the mapping results into polar coordinates to obtain the node coordinates of the mucosal layer contour line;

The radius subunit is used to calculate the relative radius of the submucosal layer and the adventitia layer respectively according to the control point features using the linear transformation layer, and calculate the actual radius of the submucosal layer and the adventitia layer according to the node coordinates and the relative radius;

The control subunit is used to convert the control points corresponding to the control point features into a form of Cartesian coordinates according to the actual radius and the angle of the node coordinates relative to the center of the esophagus, so as to obtain the control points of the three-layer esophageal structure.

As a preferred solution, the mapping unit includes a definition subunit, a scale subunit, a growth subunit and a transformation subunit;

Wherein, the definition subunit is used to define tumor characteristics;

The scale subunit is used to map the tumor feature on the first radius scale and the second radius scale respectively to obtain a first relative radius and a second relative radius;

The growth subunit is used to sequentially use the first relative radius and the second relative radius to calculate with the radius of the node coordinate relative to the center of the esophagus, to obtain the actual radius of the tumor's outward and inward growth contour lines respectively;

The conversion subunit is used to perform coordinate conversion of the node position according to the actual radius of the tumor's outward and inward growth contour line and the angle of the mucosal layer to obtain the control points of the tumor's outward and inward growth contour line.

As a preferred solution, the curve module includes a parameter unit, a vector unit and an integration unit;

The parameter unit is used to process the control points of the three-layer esophageal structure and tumor growth contour respectively using a cubic B-spline interpolation method to obtain a parameterized esophageal structure contour line and a tumor contour line;

The vector unit is used to calculate the normal vector of each curve point on the esophageal structure contour line and the tumor contour line to obtain the unit vector of the normal line;

The integration unit is used to construct a cubic B-spline curve of the esophagus and the tumor using the esophageal structure contour line and the tumor contour line, and the unit vector of the normal line.

As a preferred solution, the staging module includes a distance unit, an index unit and a comparison unit;

The distance unit is used to construct the Euclidean distances between all points on the curve according to the first contour line of the tumor growing outward and the second contour line of the submucosal layer in the cubic B-spline curve to obtain a distance matrix;

The index unit is used to perform distance calculation and normal vector judgment from the first contour line to the second contour line according to the distance matrix by means of index, obtain the maximum distance from each point on the first contour line to the nearest point on the second contour line, and use the maximum distance as a relative distance value set;

The comparison unit is used to obtain the tumor staging result according to the numerical value of the relative distance between the tumor's outward growth contour and the submucosal layer, and the numerical value of the relative distance between the tumor's outward growth contour and the outer membrane layer in the relative distance value set.

As a preferred solution, the feature module includes a recognition unit and an extraction unit;

Wherein, the recognition unit is used to use the effective area detector of the preset model to recognize and cut the effective area boundary box of the original ultrasound endoscopy image to obtain a valid ultrasound endoscopy image;

The extraction unit is used to extract and upsample the effective ultrasonic endoscopic image using the feature encoder and feature decoder of the preset model to obtain the full-size feature.

As a preferred solution, the integrated module further includes a DICE loss unit, a polygon loss unit, a classification loss unit and a training unit;

The DICE loss unit is used to obtain a DICE loss function by measuring the consistency between the segmentation mask predicted by the preset model and the true segmentation mask;

The polygon loss unit is used to obtain a polygon loss function by measuring the error between the predicted node and the actual node of the preset model;

The classification loss unit is used to obtain a classification loss function by quantifying the error between the T staging label predicted by the preset model and the actual T staging label;

The training unit is used to use a preset training set and a mask of the tumor and the esophageal wall to train the preset model according to the DICE loss function, the polygon loss function and the classification loss function to obtain an optimized preset model.

As a preferred solution, the training set is specifically:

Obtain training data from esophageal cancer tumor cases;

Transforming the polygonal contour points of the mucosal layer, the submucosal layer and the adventitia layer in the training data to obtain first transition data;

generating second transition data including contour point data, a segmentation bitmap mask and a staging label of the ultrasound endoscopy image according to the first transition data;

Data enhancement is performed on the second transition data to obtain the training set.

The present invention also provides a storage medium having a computer program stored thereon. The computer program is called and executed by a computer to implement the above-mentioned method for predicting esophageal structure, tumor contour and staging.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for predicting esophageal structure, tumor contour and stage provided by an embodiment of the present invention;

FIG2 is an anatomical diagram of the esophagus provided in an embodiment of the present invention;

FIG3 is a diagram showing the T staging structure of esophageal cancer provided by an embodiment of the present invention;

FIG4 is a schematic diagram of a unit normal vector provided by an embodiment of the present invention;

FIG5 is a schematic diagram of a prediction process provided by an embodiment of the present invention;

FIG6 is a schematic diagram of the structure of a device for predicting esophageal structure, tumor contour and staging provided by an embodiment of the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

In the description of the present application, it should be understood that the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first", "second" and "third" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "several" means two or more.

In the description of this application, it should be noted that mask means mask; yolo is the abbreviation of You Only Look Once, which means an object detection algorithm based on deep neural network; mask rcnn is the abbreviation of Mask Region-based Convolutional Neural Network, which means an instance segmentation algorithm.

The method for predicting esophageal structure, tumor contour and staging described in the embodiment of the present invention is mainly used in situations where it is necessary to predict the esophageal structure and tumor contour, as well as the tumor staging, based on ultrasound endoscopic images.

Embodiment 1:

Please refer to FIG1 . An embodiment of the present invention provides a method for predicting esophageal structure, tumor contour and staging, including S1 to S6. The specific implementation steps are as follows:

S1. Obtain the original endoscopic ultrasound image.

Step S1 of the embodiment of the present invention is specifically:

An original endoscopic ultrasound image I is obtained; wherein the original endoscopic ultrasound image I refers to an image obtained by combining endoscopy and ultrasound to examine the digestive tract.

To apply the embodiment of the present invention, please refer to Figures 2 and 3. Figure 2 is an anatomical structure diagram of the esophagus provided in the embodiment of the present invention, which shows the structure of the esophagus observable under an ultrasonic endoscope, which can be simplified to a three-layer tubular cavity structure composed of a mucosal layer, a submucosal layer, and an adventitia layer, which are nested layer by layer; the formation of tumors usually starts in the esophageal mucosal layer and can grow in the cavity and outside the cavity.

FIG3 is a T staging structure diagram of esophageal cancer provided by an embodiment of the present invention, which shows the T staging division method of esophageal cancer. The T staging of a tumor is mainly determined according to the degree of its growth outside the cavity. For example, a T1a-level tumor is located between the mucosal layer and the submucosal layer, a T1b-level tumor has broken through the submucosal layer and grown outward, a T2-level tumor further extends to the deep layer of the esophageal wall, a T3-level tumor penetrates the outer membrane layer, and a T4-level tumor has penetrated the outer membrane layer and invaded external organs. Using an ultrasonic endoscope, it is possible to identify the T1a, T1b, T2 stages of a tumor and whether it is greater than or equal to the T3 stage.

S2. Use the preset model to extract features from the original ultrasound endoscopy image to obtain full-size features.

In step S2 of the embodiment of the present invention, S2 includes S2.1 to S2.2; wherein S2.1 is a process of obtaining a valid ultrasonic endoscopic image I _valid , and S2.2 is a process of constructing a full-size feature, specifically:

S2.1. Inputting the original ultrasound endoscopy image I into the effective area detector of the preset model to identify the effective area bounding box, thereby obtaining the bounding box of the effective area;

The bounding box of the valid area is cropped to obtain a valid ultrasound endoscopy image I _valid .

Among them, the effective area detector can be implemented using target detection algorithms such as yolo or mask rcnn.

In this embodiment S2.1, by identifying and clipping the effective area boundary box, redundant images can be removed, effectively ensuring the data accuracy of the ultrasound endoscopic image while reducing the complexity of data processing. Through feature extraction and upsampling, tumor-related features can be extracted from the effective ultrasound endoscopic image, so that features such as the size, shape, and density of the tumor are included in the full-size features, which can better describe the detailed features of the original ultrasound endoscopic image.

S2.2, first input the valid ultrasound endoscopy image I _valid into the feature encoder of the preset model for feature extraction, and the input matrix is [B, C, H, W]; wherein [B, C, H, W] can take C = 3, H = W = 256;

Then, the feature decoder of the preset model is used to upsample the extracted features to obtain the full-size features of [B, h, H, W]; wherein [B, h, H, W] can take h = 64.

In addition, the feature encoder and feature decoder constitute the backbone network using the U-net network structure; the feature encoder can choose convolutional neural network CNN or Transformer as the feature extractor, and the feature decoder adopts the U-Net structure.

It should be noted that the preset model is a deep learning model composed of an effective area detector, a feature encoder, a feature decoder and a structure detection module.

S3. Use the preset model to perform structural decomposition and node reconstruction on the full-size features to obtain the control points of the three-layer esophageal structure and tumor growth contour.

In step S3 of the embodiment of the present invention, S3 includes S3.1 to S3.4; wherein S3.1 is a process of constructing a global feature token, S3.2 is a process of defining the features of the esophagus and the tumor control points, S3.3 is a process of constructing the control points of the three-layer esophagus structure, and S3.4 is a process of constructing the control points of the outward and inward growth contours of the tumor, specifically:

S3.1. Use the structure detection module of the preset model to transform the full-size features of [B, h, H, W] into features of dimensions [B, HW, h]. Get the first token /> And for the first token/> Embed the corresponding coordinate position into positional embedding, where B represents the batch size, h represents the feature dimension, and H and W represent the height and width of the source matrix.

Construct a learnable second token based on the number of esophageal structure control points that need to be predicted Among them,/> The dimension is [B, n _s , h], n _s represents the number of esophageal structure control points that need to be predicted;

The first token and the second token are concatenated to obtain a third token token ^{0} ; wherein the dimension of the third token token ^{0} is [B, _ns +HW, h];

The third token token ^{0} is input into the L-layer Transformer module in the preset model to obtain token ^{L} , and token ^{L} is re-split to obtain the global feature token ( and/> ); where /> The dimension is [B, _ns , h], which means that the model will output ns esophageal/tumor control point features.

The steps of dimension transformation, token construction, token splicing and token splitting of the full-size features in S3.1 of this embodiment are the process of organizing and collecting the feature information in the full-size features, so that the obtained global feature tokens have good invariance and intuitive representation while having global features, which helps to provide a solid data foundation for the later esophageal structure and tumor prediction.

In addition, this embodiment 1 introduces a Transformer-based multi-layer polygon generation technology to explicitly represent organs that conform to the anatomical structure. Through this representation method, the polygonal curves of the esophageal wall and tumor area are more consistent with the anatomical structure, which can effectively improve the structural accuracy and interpretability of the segmentation results, and provide medical staff with a more reliable basis for interpreting image data.

S3.2. Order Using the first preset formula, the esophageal and tumor control point features F _structure are calculated based on the global feature tokens; wherein the dimension of F _structure is [B, _ns , h];

Among them, the characteristics of esophagus and tumor control points are:

F _structure = softmax(Q _s ×K _f ^T )×V _f

S3.3. For the mucosal layer, submucosal layer and adventitia layer of the esophageal lumen, it is necessary to predict several nodes (control points) at their respective levels. These nodes will determine the shape of the cubic B-spline curve, thereby describing the contour of the esophageal structure. Therefore, for each esophageal control point, it is necessary to predict the node coordinates of the mucosal layer (expressed in Cartesian coordinates), the relative radius of the submucosal layer and the adventitia layer (the radius difference between the membrane layer distances). The specific steps are as follows:

Assume that the feature of the current esophageal structure is F _structure , and the dimension is [B, _ns , h], where B is the batch size, _ns is the number of esophageal structure control points to be predicted, and h is the feature dimension;

A linear transformation layer is used to map the esophageal and tumor control point features F _structure of the n _s control nodes to two dimensions (2D), that is, P _mucosa =W _p *F _structure ; and the mapping result is converted into polar coordinates to obtain the node coordinates of the mucosal layer contour line (r _mucosa ,θ _mucosa ); wherein W _p is a weight matrix with a dimension of [2, h], and the dimension corresponding to P _mucosa is [B, n _s , 2], representing the 2D coordinates of the control points of the n _s mucosal layer contour line; and r _mucosa and θ _mucosa represent the radius and angle of the node relative to the center of the esophagus, respectively;

According to the control point features, two linear transformation layers are used to calculate the relative radius of the submucosal layer dR _submucosa and the relative radius of the adventitia dR _out , that is, dR _submucosa = relu(W _dR1 *F _structure ) and dR _out = relu(W _dR2 *F _structure ), and the relu activation function is used to ensure that the predicted radius is a positive number, where W _dR1 and W _dR2 are weight matrices with dimensions [1, h], and the dimensions corresponding to dR _submucosa and dR _out are [B, _ns , 1], which represent the relative radii of the submucosal layer and the adventitia corresponding to the n _s control nodes;

The actual radius R _submucosa of the submucosal layer and the actual radius R _{out of} the adventitia layer are calculated according to the node coordinates and the relative radius, that is, R _submucosa = r _mucosa + dR _submucosa and R _out = R _submucosa + dR _out ;

According to the actual radius and the angle of the node coordinates relative to the esophageal center, the control points corresponding to the control point features are converted from polar coordinates to Cartesian coordinates to obtain the control points of the three-layer esophageal structure (P _mucosa , P _submucosa and P _out ).

The relative radius of the submucosal layer and the adventitia layer calculated in this embodiment has certain limitations and cannot accurately represent the distance between different layers of the mucosa. Therefore, on the basis of the relative radius, the actual radius is calculated in combination with the node coordinates, so that the degree of displacement of the contour line of the submucosal layer and the adventitia layer relative to the mucosal layer can be accurately represented.

S3.4. In order to simulate the growth of the tumor, it is necessary to predict the outward and inward growth of the tumor in the esophageal mucosal layer, that is, it is necessary to obtain the contour lines of the tumor in all directions, including S3.41 to S3.44, specifically:

S3.41. Define tumor characteristics F _tumor = F _structure ;

S3.42. Predict the radius difference of the tumor's outward growth contour relative to the mucosal layer, i.e. the relative radius of outward growth Specifically:

The features of F _tumor are mapped to the relative radius (first relative radius) through a linear transformation layer, that is, And the relu activation function is used to ensure the non-negativity of the relative radius; is a weight matrix with dimension [1, h];

Using the first relative radius and the radius of the node coordinate relative to the center of the esophagus, r _mucosa , the actual radius of the tumor outward growth contour is calculated. That is/> Furthermore, the angle corresponding to the actual radius of the tumor's outward growth contour is equal to the angle θ _mucosa of the mucosal layer, i.e./>

S3.43. To ensure that the contour of the tumor's inward growth is always inside the mucosal layer, it is necessary to calculate the relative radius of the tumor inside. Right now:

The features of F _tumor are mapped to the base radius (the second relative radius) through a linear transformation layer, that is, And the relu activation function is used to ensure the non-negativity of the relative radius; is a weight matrix with dimension [1, h];

Using the second relative radius and the radius T _mucosa of the node coordinate relative to the center of the esophagus, the actual radius of the tumor outward growth contour is calculated. That is/> Furthermore, it is ensured that the actual radius is greater than 0, and the angle corresponding to the actual radius of the tumor outward growth contour is equal to the angle θ _mucosa of the mucosal layer, that is,

S3.44. According to the actual radius of the tumor's outward and inward growth contours and the angle of the mucosal layer, the coordinates of the node positions are transformed to obtain the control points of the tumor's outward and inward growth contours ( and/> ).

This embodiment S3.4 is the construction process of the control points of the tumor growth contour line. The conversion relationship between Cartesian coordinates and polar coordinates is referenced in the processing of control points. The advantages of Cartesian coordinates in terms of intuitive expression, concise description and accurate description can be utilized to make the calculation method of the tumor growth shape relatively simple, thereby being able to clearly describe the control points of the tumor growth contour line.

From an overall perspective, in this embodiment S3, by predicting the control points of the contour lines in the two dimensions of tumor outward growth and inward growth, the control points of the tumor outward and inward growth contour lines are obtained, which can fully obtain the characteristics of tumor growth and is conducive to the effective evaluation of the esophageal disease status.

S4. Generate cubic B-spline curves of the esophagus and the tumor according to the control points by means of spline interpolation.

In step S4 of the embodiment of the present invention, S4 includes S4.1 to S4.3; wherein S4.1 is a process of constructing the esophageal structure contour line, S4.2 is a process of constructing the tumor contour line, and S4.3 is a process of constructing a cubic B-spline curve, specifically:

S4.1. For each layer of the esophageal structure (including the mucosal layer, submucosal layer, and adventitia layer), there are corresponding control point coordinates. That is, the control points of the three-layer esophageal structure and the control points of the outward and inward growth contours of the tumor; wherein _Pi = ( _xi , _yi ), (i = 1, 2, ..., _ns ) represents the Cartesian coordinates of the ith control point;

The control points of the three-layer esophageal structure are processed using the cubic B-spline interpolation method to obtain the parameterized esophageal structure contour line S(t) = [x(t), y(t)]; the function S(t) represents the continuous contour line generated by the control points. If t uniformly samples s points in [0,1], the number of nodes of each cubic B-spline curve is n _spline = n _s *s; thus, the cubic B-spline curves S _mucosa , S _submucosa and S _out are obtained, namely the esophageal structure contour line;

The cubic B-spline curve is a piecewise function defined by a set of control points. Let _Ai (i = 1, 2, 3, 4) be the four basis functions of the cubic B-spline, used for interpolation and fitting curves, then the mathematical definition of the curve S(t) is:

S(t ₎ ＝ _{A1P1 + A2P2 + A3P3 + A4P4}

For parameters A ₁ , A ₂ , A ₃ and A ₄ we have:

Wherein, S(t) is a point on the curve with parameter t∈[0,1], and P ₁ , P ₂ , P ₃ and P ₄ are control points of the cubic B-spline.

S4.2. As described in S4.1, the control points of the tumor growth contour are processed using the cubic B-spline interpolation method (including outward growth and inward growth) to obtain a cubic B-spline curve. and/> The tumor outline.

S4.3. Calculate the normal vector of each curve point on the esophageal structure contour line and the tumor contour line to obtain the unit vector of the normal line;

The cubic B-spline curves of the esophagus and the tumor are constructed by the esophageal structure contour line, the tumor contour line, and the unit vector of the normal line.

The specific process of obtaining the unit vector of the normal is as follows:

The tangent vector can be derived from the first derivative of the curve, that is, V = dy/dx;

The unit normal vector of a curve can be calculated by rotating the tangent vector 90 degrees counterclockwise; that is, in two dimensions, by flipping the x and y coordinates of the vector and changing the sign of one of the components, i.e.:

For parameters and/> have:

Therefore, the normal vector is normalized based on the L2 norm to obtain the unit vector of the normal line.

To apply the embodiment of the present invention, please refer to FIG. 4 , which is a schematic diagram of a unit normal vector provided by the embodiment of the present invention, and shows the relationship between the unit vector of the normal line and the anatomical structure of the esophagus.

In this embodiment S4, by using control points as basic elements, discrete point data are connected into a curve, thereby obtaining a curve with high precision, convergence and accurate fitting as a structural contour line, and by combining normal vectors, a complete cubic B-spline curve is obtained, which can be used to accurately represent the shape of the esophagus and tumors.

S5. The tumor staging result is obtained by calculating the relative distance between the cubic B-spline curves.

In step S5 of the embodiment of the present invention, S5 includes S5.1 to S5.2; S5.1 is a process of constructing a tumor staging result, and S5.2 is a process of returning model data, specifically:

S5.1. The T stage of esophageal cancer is inferred based on the tissue structure invaded by the tumor. Therefore, the outward growth contour of the tumor can be calculated. The relative distances D _submucosa and D _out from the submucosa (S _submucosa ) and adventitia (S _out ) contours determine the T stage. Includes S5.11 to S5.18, specifically:

S5.11. Calculate the Euclidean distances between all points on curve A (the first contour line of tumor outward growth) and curve B (the second contour line of the submucosal layer) in the cubic B-spline curve to obtain a distance matrix;

S5.12. Find the distance and index of each point on curve A to the nearest point on curve B;

S5.13, according to the index of the nearest point, take the corresponding point from curve B and treat it as the point corresponding to the corresponding point in curve A;

S5.14, obtaining the normal vector of curve B at the nearest point;

S5.15. Calculate the direction vector from each point on curve A to the nearest point on curve B, and determine whether it is consistent with the normal vector at the nearest point on curve B;

S5.16. If it is determined that the point of curve A is inside curve B, the distance is taken as a negative number.

S5.17. Take the maximum value of the distance from each point on curve A to the nearest point on curve B as the measure of relative distance, and obtain a relative distance value set.

S5.18. In the relative distance value set, the relative distance between the tumor's outward growth contour and the submucosal layer is recorded as d1, and the relative distance between the tumor's outward growth contour and the adventitia layer is recorded as d2;

According to the numerical value of the relative distance between the outward growth contour of the tumor and the submucosal layer, and the numerical value of the relative distance between the outward growth contour of the tumor and the outer membrane layer, the tumor staging result, that is, the T stage of esophageal cancer, is obtained, which is specifically:

1) If d1 < 0, the T stage is T1a, which means that when the tumor's outward growth contour is inside the mucosal layer, esophageal cancer is diagnosed as T1a.

2) If d1 ≥ 0, and d2 < 0, and (|d1|)/(|d1|+|d2|) < threshold_ratio, the T stage is T1b, which means that when the tumor's outward growth contour is inside the submucosal layer but has broken through the mucosal layer, esophageal cancer is diagnosed as T1b.

3) If d1≥0, and d2＜0, and (|d1|)/(|d1|+|d2|)≥threshold_ratio, the T stage is T2, indicating that the tumor's outward growth outline has broken through the esophageal muscle layer but has not yet spread to adjacent lymph nodes or other structures. Esophageal cancer is diagnosed as T2.

4) If d2 ≥ 0, the T stage is T3 or T4, which means that when the tumor's outward growth contour has broken through the outer membrane layer, esophageal cancer is diagnosed as T3\4.

Among them, threshold_ratio represents the thickness of the submucosal layer, and the empirical value range is 0.05 to 0.2.

This embodiment S5.1 can accurately represent the distances between all points on the curve by constructing a distance matrix. By means of indexing, points on different curves can be associated with each other, so as to obtain the maximum distance from each point on the first contour line to the nearest point on the second contour line. Moreover, the relative distances in the relative distance value set can be positive or negative, so by judging the size of the value, the degree of tissue invasion between different membrane layers can be known, thereby obtaining the tumor staging result.

S5.2. After the above calculations, the model returns the cubic B-spline curves S _mucosa , S _submucosa , S _out , and/> Return the differentiable mask Y _diff-mask of each structure; return the relative distance D _submucosa and D _out between the tumor and the submucosal layer and the adventitia layer, as well as the T stage classification result/>

To apply the embodiment of the present invention, please refer to FIG. 5 , which is a schematic diagram of a prediction process, which shows the general process of data prediction in this embodiment, specifically:

1. Ultrasound endoscopic image: obtain the original ultrasound endoscopic image I.

2. Valid region detection: Use a valid region detector to perform valid region detection on the original ultrasound endoscopic image I to obtain a valid ultrasound endoscopic image I _valid .

3. Backbone network: Use the backbone network composed of feature encoder and feature decoder to process the ultrasound endoscopy image I _valid to obtain full-size features.

4. Structural detection module: The structural detection module is used to process the full-size features to obtain the esophageal structure and tumor contour, that is, the cubic B-spline curve of the esophagus and the tumor.

6. Rule-based T staging discrimination module: Use the structure detection module to process the cubic B-spline curve to obtain the T staging prediction result, that is, the tumor staging result.

In this embodiment S5, the traditional deep learning segmentation model cannot directly obtain the contour lines of each tissue structure, and requires a lot of post-processing and large amount of calculation; or chooses to use a deep learning classification model, which lacks interpretability. After obtaining the contour lines of each tissue structure, this embodiment S5 only needs rules to determine the T stage, and has strong clinical interpretability.

S6, the prediction results of esophageal structure, tumor contour and tumor stage are constructed by cubic B-spline curve and tumor stage results;

Among them, the preset model is a deep learning model composed of different devices and modules.

In step S6 of the embodiment of the present invention, S6 includes S6.1 to S6.5; wherein S6.1 is a process of constructing a prediction result, S6.2 is a process of constructing a mask of a tumor and an esophageal wall, S6.3 is a process of constructing a training set, S6.4 is a process of constructing a loss function, and S6.5 is a process of model training, specifically:

S6.1. The prediction results of esophageal structure, tumor contour and tumor stage are constructed by cubic B-spline curve and tumor stage results.

This embodiment S6.1 can achieve multi-task output within one training, including the prediction of esophageal structure, tumor area and tumor T stage; this comprehensive technological innovation provides medical staff with more comprehensive and integrated diagnostic information, greatly improving the efficiency of ultrasound endoscopic image processing and diagnosis.

S6.2. First, create a blank differentiable image in the diffvg library Set the width to W, the height to H, and initialize all pixel values to 0;

Then, a function y=M(S) is defined to generate a corresponding mask image; wherein S represents a closed cubic B-spline curve, and y represents a mask of a two-dimensional plane bitmap obtained after drawing a polygon;

Secondly, draw the mask of the tumor and esophageal wall, including S6.21 to S6.24, specifically:

S6.21. Tumor mask: using cubic B-spline contour of the tumor and/> Generate tumor mask Y _tumor , that is/>

S6.22. Esophageal mucosa layer mask: The esophageal mucosa layer mask Y _mucosa is generated using the cubic B-spline contour of the esophageal mucosa layer, ie, Y _mucosa =M(S _mucosa ).

S6.23. Esophageal submucosal mask: The esophageal mucosal layer and the cubic B-spline contour line of the submucosal layer are used to generate the esophageal submucosal mask Y _submucosa , ie, Y _submucosa =M(P _submucosa )\M(S _mucosa ).

S6.24. Esophageal submucosal layer mask: Generate the esophageal adventitial layer mask Y _out according to the cubic B-spline contours of the esophageal adventitial layer and the submucosal layer, ie, Y _out =M(S _out )\M(S _submucosa )\M(S _mucosa ).

Among them, "\" represents the difference of sets, that is,

Finally, the four masks in S6.21 to S6.24 are combined to generate a 4-channel bitmap mask: Y _diff-mask = [Y _tumor , Y _mucosa , Y _submucosa , Y _out ], that is, the mask of the tumor and the esophageal wall is obtained.

All operations in S6.2 of this embodiment are differential operations and can generate differential images.

S6.3, obtaining training data from esophageal cancer tumor cases;

Using the shapely library of python, the polygonal contour points of the mucosa layer, the submucosal layer and the adventitia layer are converted into shapely.geometry.polygon.LinearRing objects, and the tumor area is converted into a shapely.geometry.polygon.Polygon object to obtain the first transition data; and in the process of conversion, the legitimacy of the contour nodes is checked using the function of the shapely library, and the required number of nodes is calculated using the interpolate function of the shapely library;

Generate second transition data including contour point data S _contour , segmentation bitmap mask Y _mask and staging label Y _label of the ultrasound endoscopy image according to the first transition data;

Performing data enhancement operations such as brightness contrast enhancement, rotation, horizontal flipping, vertical flipping, and zooming in and out on the second transition data to obtain a training set; wherein, when performing data enhancement operations such as rotation and other geometric deformations, it is necessary to perform synchronous operations on the Y _contour and the Y _mask to maintain data consistency;

The second transition data specifically includes:

1) Contour point data: The contour point data S _contour is the polygon object that has passed the legality check and is recalculated using the interpolate function, that is, S _contour = {S _tumor , S _structure }.

2) Segmentation bitmap mask: Use the opencv library to convert the four contours into a 4-channel bitmap. The conversion logic is consistent with the above "Use the diffvg library to generate a differentiable mask" and save it as the Y _mask .

3) T staging label: The T staging label of each tumor in the ultrasound endoscopy image is Y _label , which includes four staging labels: T1a, T1b, T2, and T3\4.

S6.4. Calculate three loss functions, including S6.41 to S6.43, specifically:

S6.41. Use dice loss to measure the consistency between the segmentation mask predicted by the preset model and the actual segmentation mask. Perform dice loss calculation on the masks of the four channels to obtain the DICE loss function. The dice coefficient is a commonly used indicator to measure the similarity between two samples. Suppose the predicted segmentation mask is The actual segmentation mask is Y and the dice coefficient is:

The closer the dice value is to 1, the more consistent the prediction is with the actual result. Therefore, the goal is to minimize 1-dice, that is, diceloss is defined as:

dice _loss = 1-dice

In this embodiment S6.41, Dice Loss can well reflect the matching degree between the predicted contour mask and the true contour mask.

S6.42. Use polygon loss poly loss to measure the error between the predicted node (the contour line is obtained by cubic B-spline interpolation) and the actual node, and obtain the polygon loss function; the polygon loss function is a loss function based on the mean square error (MSE).

Assume the predicted node is The actual contour node is S, then the poly loss is defined as:

Where _nspline represents the number of nodes of the cubic B-spline curve of the esophageal structure to be predicted, _Si and are the coordinates of the i-th actual node and predicted node respectively.

S6.43. Use the class loss function based on the T stage label of esophageal cancer tumors to handle the four-classification problem, that is, the T stage label of esophageal cancer can be one of T1a, T1b, T2 and T3\4; define a multi-classification version of the distance margin loss to quantify the error between the T stage label predicted by the model and the actual T stage label, and set the goal to find the model parameters that minimize the Loss loss function to obtain the classification loss function;

Among them, threshold is a given margin value, specifically:

The loss function is:

Loss = α ₁ *dice _loss + α ₂ *poly _loss + α ₃ *class-loss

Among them, α ₁ , α ₂ and α ₃ are the weight coefficients of dice loss, poly loss and class loss respectively.

In this embodiment S6.4, the DICE loss function can enhance the accuracy of recognition and positioning of specific targets, the polygon loss function can improve the prediction accuracy of esophageal and tumor contour nodes, and the classification loss function can improve the prediction accuracy of tumor staging results. Therefore, the loss functions established from different dimensions can significantly improve the optimization of the model and enhance the prediction accuracy of the preset model.

S6.5, set the batch size to 32, the epoch to 300, the initial learning rate to 10^-4, and reduce it to 1/10 every 100 epochs;

By optimizing the DICE loss function, polygon loss function, and classification loss function, a graphics processing unit (GPU) is used to update the parameters of the preset model according to the training set and the mask of the tumor and the esophageal wall to obtain an optimized preset model;

Moreover, the average dice of all masks obtained in the end is 0.76, and the average classification accuracy is 0.87.

In this embodiment S6.5, dice loss is used to optimize the contour line through differentiable rendering technology; compared with the traditional method of using only L2 loss or L1 loss for node optimization, the combination of L2 loss and dice loss can obtain a more accurate contour line, thereby improving the prediction ability of the preset model, so as to obtain more accurate prediction results of esophageal structure, tumor contour and stage in the future;

In addition, in the traditional mask conversion process, the open source computer vision library OpenCV is used to convert the node into a mask, but this conversion method does not have a gradient. Since the gradient is an important indicator for parameter evaluation and update, it can represent the direction and rate of change of the target function at a certain point. Therefore, the results obtained by the traditional mask conversion method cannot be used for neural network training; while the first embodiment of this invention generates a differentiable mask through the diffvg library, which can be used for neural network training, and uses dice loss as the loss function, which can make the similarity between the predicted segmentation result and the actual segmentation result higher and higher, so the preset model will be effectively trained to make the prediction results of the esophageal structure, tumor contour and staging more accurate.

Overall, this embodiment has the following beneficial effects:

This embodiment can remove redundant images by extracting features from the original ultrasound endoscopy images, thereby effectively ensuring the data accuracy of the ultrasound endoscopy images and reducing the complexity of data processing; by constructing control points of the three-layer esophageal structure and the tumor growth contour, the characteristics of tumor growth can be fully obtained, which is conducive to the effective evaluation of the esophageal disease state; by generating a cubic B-spline curve by spline interpolation, discrete point data can be connected into a curve to accurately represent the shape of the esophagus and the tumor; because the relative distance between the curves can represent the degree of tissue invasion between different membrane layers, the tumor staging result can be obtained quickly and conveniently.

Moreover, this embodiment successfully improves the accurate positioning of the model for the region of interest by introducing a positioning model for effective region detection, and overcomes the problem of relatively poor quality of the original data image. Secondly, based on the multi-layer polygon generation technology of Transformer, the organ structure is explicitly modeled and the anatomical prior knowledge is injected, so that this embodiment can generate esophageal wall structure contour prediction and tumor contour prediction results that are more in line with the anatomical structure, effectively solving the shortcomings of the existing technology in terms of structural accuracy and interpretability. In addition, the introduction of differentiable rendering technology successfully converts polygonal curves into bitmap masks with gradients, and the accuracy of the contours can be further improved by using dice loss while using L2 for node distance constraints. At the same time, due to the use of explicit cubic B-spline curves, while being able to accurately describe the esophageal and tumor structures, the T staging conclusion can be drawn based on their relative distance and simple rules. Such rules are also in line with the understanding of the diagnostic process by clinical medical staff, and can comprehensively improve the accurate segmentation and classification performance of ultrasound endoscopic images, providing a more reliable and interpretable basis for tumor localization and staging.

Embodiment 2:

Please refer to FIG6 , an embodiment of the present invention provides a prediction device for esophageal structure, tumor contour and staging, including a data module 10 , a feature module 20 , a node module 30 , a curve module 40 , a staging module 50 and an integration module 60 ;

Wherein, the data module 10 is used to obtain the original ultrasound endoscopy image;

A feature module 20 is used to extract features from the original ultrasound endoscopy image using a preset model to obtain full-size features;

A node module 30 is used to perform structural decomposition and node reconstruction on the full-size features using a preset model to obtain control points of the three-layer esophageal structure and tumor growth contour;

A curve module 40, for generating a cubic B-spline curve of the esophagus and the tumor according to the control points by means of spline interpolation;

The staging module 50 is used to obtain the tumor staging result by calculating the relative distance between the cubic B-spline curves;

An integration module 60, for forming a prediction result of esophageal structure, tumor contour and tumor stage from a cubic B-spline curve and the tumor stage result;

Among them, the preset model is a deep learning model composed of different devices and modules.

In one embodiment, the data module 10 is specifically:

An original endoscopic ultrasound image I is obtained; wherein the original endoscopic ultrasound image I refers to an image obtained by combining endoscopy and ultrasound to examine the digestive tract.

To apply the embodiment of the present invention, please refer to Figures 2 and 3. Figure 2 is an anatomical structure diagram of the esophagus provided in the embodiment of the present invention, which shows the structure of the esophagus observable under an ultrasonic endoscope, which can be simplified to a three-layer tubular cavity structure composed of a mucosal layer, a submucosal layer, and an adventitia layer, which are nested layer by layer; the formation of tumors usually starts in the esophageal mucosal layer and can grow in the cavity and outside the cavity.

FIG3 is a T staging structure diagram of esophageal cancer provided by an embodiment of the present invention, which shows the T staging division method of esophageal cancer. The T staging of a tumor is mainly determined according to the degree of its growth outside the cavity. For example, a T1a-level tumor is located between the mucosal layer and the submucosal layer, a T1b-level tumor has broken through the submucosal layer and grown outward, a T2-level tumor further extends to the deep layer of the esophageal wall, a T3-level tumor penetrates the outer membrane layer, and a T4-level tumor has penetrated the outer membrane layer and invaded external organs. Using an ultrasonic endoscope, it is possible to identify the T1a, T1b, T2 stages of a tumor and whether it is greater than or equal to the T3 stage.

In one embodiment, the feature module 20 includes an identification unit and an extraction unit; wherein the identification unit is a process of obtaining a valid ultrasound endoscopy image I _valid , and the extraction unit is a process of constructing a full-size feature, specifically:

The recognition unit is used to input the original ultrasound endoscopy image I into the effective area detector of the preset model to identify the effective area boundary box, so as to obtain the bounding box of the effective area;

The recognition unit is further used to clip a bounding box of a valid area to obtain a valid ultrasound endoscopy image I _valid .

Among them, the effective area detector can be implemented using target detection algorithms such as yolo or mask rcnn.

The recognition unit of this embodiment can remove redundant images by identifying and clipping the effective area boundary box, effectively ensuring the data accuracy of the ultrasound endoscopic image while reducing the complexity of data processing. Through feature extraction and upsampling, tumor-related features can be extracted from the effective ultrasound endoscopic image, so that features such as the size, shape and density of the tumor are included in the full-size features, which can better describe the detailed features of the original ultrasound endoscopic image.

The extraction unit is used to first input the valid ultrasound endoscopy image I _valid into the feature encoder of the preset model for feature extraction, and the input matrix is [B, C, H, W]; wherein [B, C, H, W] can take C=3, H=W=256;

Then, the feature decoder of the preset model is used to upsample the extracted features to obtain the full-size features of [B, h, H, W]; wherein [B, h, H, W] can take h = 64.

In addition, the feature encoder and feature decoder constitute the backbone network using the U-net network structure; the feature encoder can choose convolutional neural network CNN or Transformer as the feature extractor, and the feature decoder adopts the U-Net structure.

It should be noted that the preset model is a deep learning model composed of an effective area detector, a feature encoder, a feature decoder and a structure detection module.

In one embodiment, the node module 30 includes a dimension subunit, a construction subunit, a splicing subunit, a splitting subunit, a feature subunit, a coordinate and radius subunit, a control subunit, and a synthesis unit;

Among them, the dimension subunit, construction subunit, splicing subunit and splitting subunit are the processes of constructing global feature tokens, the feature subunit is the process of defining the features of the esophagus and tumor control points, the coordinate and radius subunit and the control subunit are the processes of constructing the control points of the three-layer esophageal structure, and the comprehensive unit is the process of constructing the control points of the tumor's outward and inward growth contours, specifically:

The dimension subunit is used to transform the full-size features of [B, h, H, W] into the dimensions of [B, HW, h] using the structure detection module of the preset model. Get the first token /> And for the first token/> Embed the corresponding coordinate position into positional embedding, where B represents the batch size, h represents the feature dimension, and H and W represent the height and width of the source matrix.

A construction subunit is used to construct a learnable second token according to the number of esophageal structure control points that need to be predicted Among them,/> The dimension is [B, n _s , h], n _s represents the number of esophageal structure control points that need to be predicted;

A concatenation subunit, configured to concatenate the first token and the second token to obtain a third token token ^{0} ; wherein the dimension of the third token token ^{0} is [B, _ns +HW, h];

The splitting subunit is used to input the third token token ^{0} into the L-layer Transformer module in the preset model to obtain token ^{L} , and re-split token ^{L} to obtain the global feature token ( and/> ); where /> The dimension is [B, _ns , h], which means that the model will output n _s esophageal/tumor control point features.

In this embodiment, the steps of dimension subunit, construction subunit, splicing subunit and splitting subunit performing dimension transformation, token construction, token splicing and token splitting on the full-size feature are the process of organizing and collecting the feature information in the full-size feature, so that the obtained global feature token has good invariance and intuitive representation while having global features, which helps to provide a solid data foundation for the later esophageal structure and tumor prediction.

In addition, this second embodiment introduces a Transformer-based multi-layer polygon generation technology to explicitly represent organs that conform to the anatomical structure. Through this representation method, the polygonal curves of the esophageal wall and tumor area are more consistent with the anatomical structure, which can effectively improve the structural accuracy and interpretability of the segmentation results, and provide medical staff with a more reliable basis for interpreting image data.

Characteristic subunit, used to make Using the first preset formula, the esophageal and tumor control point features F _structure are calculated based on the global feature tokens; wherein the dimension of F _structure is [B, _ns , h];

Among them, the characteristics of esophagus and tumor control points are:

F _structure = softmax(Q _s ×K _f ^T )×V _f

The coordinate and radius subunit is used for the mucosal layer, submucosal layer and adventitia layer of the esophageal lumen. It is necessary to predict several nodes (control points) at their respective levels. These nodes will determine the shape of the cubic B-spline curve, thereby describing the contour of the esophageal structure. Therefore, for each esophageal control point, it is necessary to predict the node coordinates of the mucosal layer (expressed in Cartesian coordinates), the relative radius of the submucosal layer and the adventitia layer (the radius difference between the membrane layer distances). The specific steps are as follows:

Assume that the feature of the current esophageal structure is F _structure , and the dimension is [B, _ns , h], where B is the batch size, _ns is the number of esophageal structure control points to be predicted, and h is the feature dimension;

A linear transformation layer is used to map the esophageal and tumor control point features F _structure of the n _s control nodes to two dimensions (2D), that is, P _mucosa =W _p *F _structure ; and the mapping result is converted into polar coordinates to obtain the node coordinates of the mucosal layer contour line (r _mucosa ,θ _mucosa ); wherein W _p is a weight matrix with a dimension of [2, h], and the dimension corresponding to P _mucosa is [B, n _s , 2], representing the 2D coordinates of the control points of the n _s mucosal layer contour line; and r _mucosa and θ _mucosa represent the radius and angle of the node relative to the center of the esophagus, respectively;

According to the control point features, two linear transformation layers are used to calculate the relative radius of the submucosal layer dR _submucosa and the relative radius of the adventitia dR _out , that is, dR _submucosa = relu(W _dR1 *F _structure ) and dR _out = relu(W _dR2 *F _structure ), and the relu activation function is used to ensure that the predicted radius is a positive number, where W _dR1 and W _dR2 are weight matrices with dimensions [1, h], and the dimensions corresponding to dR _submucosa and dR _out are [B, _ns , 1], which represent the relative radii of the submucosal layer and the adventitia corresponding to the n _s control nodes;

The actual radius R _submucosa of the submucosal layer and the actual radius R _{out of} the adventitia layer are calculated according to the node coordinates and the relative radius, that is, R _submucosa = r _mucosa + dR _submucosa and R _out = R _submucosa + dR _out ;

The control subunit is used to convert the control points corresponding to the control point features from polar coordinates to Cartesian coordinates according to the actual radius and the angle of the node coordinates relative to the esophageal center, and obtain the control points (P _mucosa , P _submucosa and P _out ) of the three-layer esophageal structure.

The relative radius of the submucosal layer and the adventitia layer calculated in this embodiment has certain limitations and cannot accurately represent the distance between different layers of the mucosa. Therefore, on the basis of the relative radius, the actual radius is calculated in combination with the node coordinates, so that the degree of displacement of the contour line of the submucosal layer and the adventitia layer relative to the mucosal layer can be accurately represented.

The comprehensive unit is used to simulate the growth of the tumor. It is necessary to predict the outward and inward growth of the tumor in the esophageal mucosa layer, that is, it is necessary to obtain the contour lines of the tumor in all directions; the comprehensive unit includes a definition subunit, a scale subunit, a growth subunit and a transformation subunit, specifically:

Define subunits, also used to define tumor features F _tumor = F _structure ;

The scale subunit is used to predict the radius difference of the tumor's outward growth contour relative to the mucosal layer, that is, the relative radius of outward growth Specifically:

The features of F _tumor are mapped to the relative radius (first relative radius) through a linear transformation layer, that is, And the relu activation function is used to ensure the non-negativity of the relative radius; is a weight matrix with dimension [1, h];

Using the first relative radius and the radius of the node coordinate relative to the center of the esophagus, r _mucosa , the actual radius of the tumor outward growth contour is calculated. That is/> Furthermore, the angle corresponding to the actual radius of the tumor's outward growth contour is equal to the angle θ _mucosa of the mucosal layer, i.e./>

The growth subunit is used to ensure that the contour of the tumor's inward growth is always inside the mucosal layer. It is necessary to calculate the relative radius of the tumor inside. Right now:

The features of F _tumor are mapped to the base radius (the second relative radius) through a linear transformation layer, that is, And the relu activation function is used to ensure the non-negativity of the relative radius; is a weight matrix with dimension [1, h];

Using the second relative radius and the radius of the node coordinate relative to the center of the esophagus, r _mucosa , the actual radius of the tumor outward growth contour is calculated. That is/> Furthermore, it is ensured that the actual radius is greater than 0, and the angle corresponding to the actual radius of the tumor outward growth contour is equal to the angle θ _mucosa of the mucosal layer, that is,

The conversion subunit is used to convert the coordinates of the node position according to the actual radius of the tumor's outward and inward growth contours and the angle of the mucosal layer to obtain the control points of the tumor's outward and inward growth contours ( and ).

The comprehensive unit of this embodiment is the construction process of the control points of the tumor growth contour. The conversion relationship between Cartesian coordinates and polar coordinates is referenced in the processing of control points. The advantages of Cartesian coordinates in terms of intuitive expression, concise description and accurate description can be utilized to make the calculation method of the tumor growth shape relatively simple, thereby being able to clearly describe the control points of the tumor growth contour.

From an overall perspective, the node module 30 of this embodiment predicts the control points of the contour lines in the two dimensions of tumor outward growth and inward growth, and obtains the control points of the tumor outward and inward growth contour lines, which can fully obtain the characteristics of tumor growth and is conducive to the effective evaluation of the esophageal disease status.

In one embodiment, the curve module 40 includes a parameter unit, a parameter deformation unit, a vector unit and an integration unit; wherein the parameter unit is a process of constructing the esophageal structure contour line, the parameter deformation unit is a process of constructing the tumor contour line, and the vector unit and the integration unit are processes of constructing a cubic B-spline curve, specifically:

Parameter unit, used for each layer of the esophageal structure (including mucosal layer, submucosal layer and adventitia layer), there are corresponding control point coordinates That is, the control points of the three-layer esophageal structure and the control points of the outward and inward growth contours of the tumor; wherein _Pi = ( _xi , _yi ), (i = 1, 2, ..., _ns ) represents the Cartesian coordinates of the ith control point;

The parameter unit is also used to process the control points of the three-layer esophageal structure using a cubic B-spline interpolation method to obtain a parameterized esophageal structure contour line S(t)=[x(t), y(t)]; the function S(t) represents a continuous contour line generated by the control points, and if t uniformly samples s points in [0, 1], the number of nodes of each cubic B-spline curve is n _splie = _ns *s; thereby, cubic B-spline curves S _mucosa , S _submucosa and S _out are obtained, namely, the esophageal structure contour line;

The cubic B-spline curve is a piecewise function defined by a set of control points. Let _Ai (i = 1, 2, 3, 4) be the four basis functions of the cubic B-spline, used for interpolation and fitting curves, then the mathematical definition of the curve S(t) is:

S(t ₎ ＝ _{A1P1 + A2P2 + A3P3 + A4P4}

For parameters A ₁ , A ₂ , A ₃ and A ₄ we have:

Wherein, S(t) is a point on the curve with parameter t∈[0,1], and P ₁ , P ₂ , P ₃ and P ₄ are control points of the cubic B-spline.

A parameter deformation unit is used to process the control points of the tumor growth contour (including outward growth and inward growth) using a cubic B-spline interpolation method as described in the parameter unit to obtain a cubic B-spline curve and/> The tumor outline.

A vector unit is used to calculate the normal vector of each curve point on the esophageal structure contour line and the tumor contour line to obtain the unit vector of the normal line;

An integrated unit is used to construct cubic B-spline curves of the esophagus and the tumor from the esophageal structure contour and the tumor contour, as well as the unit vectors of the normal line.

The specific process of obtaining the unit vector of the normal is as follows:

The tangent vector can be derived from the first derivative of the curve, that is, V = dy/dx;

The unit normal vector of a curve can be calculated by rotating the tangent vector 90 degrees counterclockwise; that is, in two dimensions, by flipping the x and y coordinates of the vector and changing the sign of one of the components, i.e.:

For parameters and/> have:

Therefore, the normal vector is normalized based on the L2 norm to obtain the unit vector of the normal line.

To apply the embodiment of the present invention, please refer to FIG. 4 , which is a schematic diagram of a unit normal vector provided by the embodiment of the present invention, and shows the relationship between the unit vector of the normal line and the anatomical structure of the esophagus.

In this embodiment S4, by using control points as basic elements, discrete point data are connected into a curve, thereby obtaining a curve with high precision, convergence and accurate fitting as a structural contour line, and by combining normal vectors, a complete cubic B-spline curve is obtained, which can be used to accurately represent the shape of the esophagus and tumors.

In one embodiment, the staging module 50 includes a staging unit and a return unit; the staging unit is a process of constructing tumor staging results, and the return unit is a process of returning model data, specifically:

The T stage for esophageal cancer is based on the tissue structure invaded by the tumor, so the tumor can be calculated by calculating the outward growth contour of the tumor. The relative distances D _submucosa and D _out from the submucosa (S _submucosa ) and adventitia (S _out ) contours determine the T stage. Specifically:

1. Calculate the Euclidean distance between all points on curve A (the first contour line of tumor outward growth) and curve B (the second contour line of the submucosal layer) in the cubic B-spline curve to obtain a distance matrix;

2. Find the distance and index of each point on curve A to the nearest point on curve B;

3. According to the index of the nearest point, take the corresponding point from curve B and treat it as the point corresponding to the corresponding point in curve A;

4. Get the normal vector of curve B at the nearest point;

5. Calculate the direction vector from each point on curve A to the nearest point on curve B, and determine whether it is consistent with the normal vector at the nearest point on curve B:

6. If the point of curve A is judged to be inside curve B, then the distance is taken as a negative number.

7. Take the maximum value of the distance from each point on curve A to the nearest point on curve B as the measure of relative distance to obtain a relative distance value set.

8. In the relative distance value set, the relative distance between the tumor's outward growth contour and the submucosal layer is recorded as d1, and the relative distance between the tumor's outward growth contour and the adventitia layer is recorded as d2;

9. According to the numerical value of the relative distance between the tumor's outward growth contour and the submucosal layer, and the numerical value of the relative distance between the tumor's outward growth contour and the outer membrane layer, the tumor staging result, that is, the T stage of esophageal cancer, is obtained, which is specifically:

1) If d1 < 0, the T stage is T1a, which means that when the tumor's outward growth contour is inside the mucosal layer, esophageal cancer is diagnosed as T1a.

2) If d1 ≥ 0, and d2 < 0, and (|d1|)/(|d1|+|d2|) < threshold_ratio, the T stage is T1b, which means that when the tumor's outward growth contour is inside the submucosal layer but has broken through the mucosal layer, esophageal cancer is diagnosed as T1b.

3) If d1≥0, and d2＜0, and (|d1|)/(|d1|+|d2|)≥threshold_ratio, the T stage is T2, indicating that the tumor's outward growth outline has broken through the esophageal muscle layer but has not yet spread to adjacent lymph nodes or other structures. Esophageal cancer is diagnosed as T2.

4) If d2 ≥ 0, the T stage is T3 or T4, which means that when the tumor's outward growth contour has broken through the outer membrane layer, esophageal cancer is diagnosed as T3\4.

Among them, threshold_ratio represents the thickness of the submucosal layer, and the empirical value range is 0.05 to 0.2.

The staging unit of this embodiment can accurately represent the distances between all points on the curve by constructing a distance matrix. By means of indexing, points on different curves can be associated with each other, so as to obtain the maximum value of the distance from each point on the first contour line to the nearest point on the second contour line. Moreover, the relative distances in the relative distance value set can be positive or negative, so by judging the value, the degree of tissue invasion between different membrane layers can be known, and then the tumor staging result can be obtained.

The return unit is used to make the model return the cubic B-spline curves S _mucosa , S _submucosa , S _out , of all structures after the above calculations. and/> Return the differentiable mask Y _diff-mask of each structure; return the relative distance D _submucosa and D _out between the tumor and the submucosal layer and the adventitia layer, as well as the T stage classification result/>

To apply the embodiment of the present invention, please refer to FIG. 5 , which is a schematic diagram of a prediction process, which shows the general process of data prediction in this embodiment, specifically:

1. Ultrasound endoscopic image: obtain the original ultrasound endoscopic image I.

2. Valid region detection: Use a valid region detector to perform valid region detection on the original ultrasound endoscopic image I to obtain a valid ultrasound endoscopic image I _valid .

3. Backbone network: Use the backbone network composed of feature encoder and feature decoder to process the ultrasound endoscopy image I _valid to obtain full-size features.

4. Structural detection module: The structural detection module is used to process the full-size features to obtain the esophageal structure and tumor contour, that is, the cubic B-spline curve of the esophagus and the tumor.

6. Rule-based T staging discrimination module: Use the structure detection module to process the cubic B-spline curve to obtain the T staging prediction result, that is, the tumor staging result.

In the staging module 50 of the present embodiment, the traditional deep learning segmentation model cannot directly obtain the contours of each tissue structure, and requires a lot of post-processing and high computational complexity; or the deep learning classification model is used, which lacks interpretability. The staging module 50 of the present embodiment only needs rules to determine the T stage after obtaining the contours of each tissue structure, and has strong clinical interpretability.

In one embodiment, the integration module 60 includes S6.1 to S6.3; wherein the combination unit is a process of constructing a prediction result, the mask unit is a process of constructing a mask of a tumor and an esophageal wall, the training set unit is a process of constructing a training set, the calculation unit is a process of constructing a loss function, and the model training unit is a process of model training, specifically:

The combined unit is used to construct the prediction results of esophageal structure, tumor contour and tumor stage from the cubic B-spline curve and the tumor stage result.

The combined unit of this embodiment can achieve multi-task output within one training, including prediction of esophageal structure, tumor area and tumor T stage; this comprehensive technical innovation provides medical staff with more comprehensive and integrated diagnostic information, greatly improving the efficiency of ultrasound endoscopic image processing and diagnosis.

The mask unit is used to first create a blank differentiable image in the diffvg library Set the width to W, the height to H, and initialize all pixel values to 0;

Then, a function y=M(S) is defined to generate a corresponding mask image; wherein S represents a closed cubic B-spline curve, and y represents a mask of a two-dimensional plane bitmap obtained after drawing a polygon;

Secondly, draw the mask of the tumor and the esophageal wall, including the first unit, the second unit, the third unit and the fourth unit, specifically:

Unit 1, for tumor masking: using cubic B-spline contours of the tumor and/> Generate tumor mask Y _tumor , that is/>

The second unit is used for esophageal mucosa layer mask: using the cubic B-spline contour line of the esophageal mucosa layer, the esophageal mucosa layer mask Y _mucosa is generated, that is, Y _mucosa =M(S _mucosa ).

The third unit is used for the esophageal submucosal mask: the esophageal submucosal mask Y _submucosa is generated using the cubic B-spline contours of the esophageal mucosal layer and the submucosal layer, ie, Y _submucosa =M(P _submucosa )\M(S _mucosa ).

The fourth unit is used for esophageal submucosal layer mask: according to the cubic B-spline contours of the esophageal adventitia layer and the submucosal layer, the esophageal adventitia layer mask Y _out is generated, that is, Y _out =M(S _out )\M(S _submucosa )\M(S _mucosa ).

Among them, "\" represents the difference of sets, that is,

Finally, the four masks in S6.21 to S6.24 are combined to generate a 4-channel bitmap mask: Y _diff-mask = [Y _tumor , Y _mucosa , Y _submucosa , Y _out ], that is, the mask of the tumor and the esophageal wall is obtained.

All operations of the mask unit in this embodiment are differential operations, and can generate differential images.

A training set unit, used for obtaining training data from esophageal cancer tumor cases;

The training set unit is also used to use the shapely library of python to convert the polygon contour points of the mucosa layer, the submucosal layer and the adventitia layer into shapely.geometry.polygon.LinearRing objects, and convert the tumor area into a shapely.geometry.polygon.Polygon object to obtain the first transition data; and, in the process of conversion, the function of the shapely library is used to check the legitimacy of the contour nodes, and the interpolate function of the shapely library is used to calculate the required number of nodes;

The training set unit is further used to generate second transition data including contour point data S _contour , segmentation bitmap mask Y _mask and staging label Y _label of the ultrasonic endoscopic image according to the first transition data;

The training set unit is further used to perform data enhancement operations such as brightness contrast enhancement, rotation, horizontal flipping, vertical flipping, and zooming in and out on the second transition data to obtain a training set; wherein, when performing data enhancement operations such as rotation and other geometric deformations, it is necessary to perform synchronous operations on the Y _contour and the Y _mask to maintain data consistency;

The second transition data specifically includes:

1) Contour point data: The contour point data S _contour is the polygon object that has passed the legality check and is recalculated using the interpolate function, that is, S _contour = {S _tumor , S _structure }.

2) Segmentation bitmap mask: Use the opencv library to convert the four contours into a 4-channel bitmap. The conversion logic is consistent with the above "Use the diffvg library to generate a differentiable mask" and save it as the Y _mask .

3) T staging label: The T staging label of each tumor in the ultrasound endoscopy image is Y _label , which includes four staging labels: T1a, T1b, T2, and T3\4.

The calculation unit is used to calculate three loss functions, including DICE loss unit, polygon loss unit and classification loss unit, specifically:

The DICE loss unit is used to use dice loss to measure the consistency between the segmentation mask predicted by the preset model and the actual segmentation mask. The dice loss is calculated for the masks of the four channels to obtain the DICE loss function. The dice coefficient is a commonly used indicator to measure the similarity between two samples. Suppose the predicted segmentation mask is The actual segmentation mask is Y and the dice coefficient is:

The closer the dice value is to 1, the more consistent the prediction is with the actual result. Therefore, the goal is to minimize 1-dice, that is, diceloss is defined as:

dice _loss = 1-dice

In the DICE loss unit of this embodiment, Dice Loss can well reflect the matching degree between the predicted contour mask and the real contour mask.

The polygon loss unit is used to measure the error between the predicted node (the contour line is obtained by cubic B-spline interpolation) and the actual node using polygon loss poly loss to obtain a polygon loss function; wherein the polygon loss function is a loss function based on the mean square error (MSE).

Assume the predicted node is The actual contour node is S, then the poly loss is defined as:

Where _nspline represents the number of nodes of the cubic B-spline curve of the esophageal structure to be predicted, _Si and are the coordinates of the i-th actual node and predicted node respectively.

The classification loss unit is used to process the four-classification problem using a class loss function based on the T stage label of esophageal cancer tumors, that is, the T stage label of esophageal cancer can be one of T1a, T1b, T2 and T3\4; the error between the T stage label predicted by the model and the actual T stage label is quantified by defining a multi-classification version of the distance margin loss, and the goal is set to find the model parameters that minimize the Loss loss function to obtain the classification loss function;

Among them, threshold is a given margin value, specifically:

The loss function is:

Loss = α ₁ *dice _loss + α ₂ *poly _loss + α ₃ *class_loss

Among them, α ₁ , α ₂ and α ₃ are the weight coefficients of dice loss, poly loss and class loss respectively.

In the calculation unit of this embodiment, the DICE loss function can enhance the accuracy of recognition and positioning of specific targets, the polygonal loss function can improve the prediction accuracy of esophageal and tumor contour nodes, and the classification loss function can improve the prediction accuracy of tumor staging results. Therefore, the loss functions established from different dimensions can significantly improve the optimization of the model and enhance the prediction accuracy of the preset model.

Model training unit, used to set batch size to 32, epoch to 300, initial learning rate to 10^-4, and reduce it to 1/10 every 100 epochs;

The model training unit is further used to update the parameters of the preset model according to the training set and the mask of the tumor and the esophageal wall by optimizing the DICE loss function, the polygon loss function and the classification loss function, so as to obtain an optimized preset model;

Moreover, the average dice of all masks obtained in the end is 0.76, and the average classification accuracy is 0.87.

The model training unit of this embodiment uses dice loss to optimize the contour line through differentiable rendering technology; compared with the traditional method of using only L2 loss or L1 loss for node optimization, the combination of L2 loss and dice loss can obtain more accurate contour lines, thereby improving the prediction ability of the preset model, so as to obtain more accurate prediction results of esophageal structure, tumor contour and stage in the future;

In addition, in the traditional mask conversion process, the open source computer vision library OpenCV is used to convert the node into a mask, but this conversion method does not have a gradient. Since the gradient is an important indicator for parameter evaluation and update, it can represent the direction and rate of change of the target function at a certain point. Therefore, the results obtained by the traditional mask conversion method cannot be used for neural network training; while the second embodiment of this invention generates a differentiable mask through the diffvg library, which can be used for neural network training, and uses dice loss as the loss function, which can make the similarity between the predicted segmentation result and the actual segmentation result higher and higher, so the preset model will be effectively trained to make the prediction results of the esophageal structure, tumor contour and staging more accurate.

Overall, this embodiment has the following beneficial effects:

This embodiment can remove redundant images by extracting features from the original ultrasound endoscopy images, thereby effectively ensuring the data accuracy of the ultrasound endoscopy images and reducing the complexity of data processing; by constructing control points of the three-layer esophageal structure and the tumor growth contour, the characteristics of tumor growth can be fully obtained, which is conducive to the effective evaluation of the esophageal disease state; by generating a cubic B-spline curve by spline interpolation, discrete point data can be connected into a curve to accurately represent the shape of the esophagus and the tumor; because the relative distance between the curves can represent the degree of tissue invasion between different membrane layers, the tumor staging result can be obtained quickly and conveniently.

Moreover, this embodiment successfully improves the accurate positioning of the model for the region of interest by introducing a positioning model for effective region detection, and overcomes the problem of relatively poor quality of the original data image. Secondly, based on the multi-layer polygon generation technology of Transformer, the organ structure is explicitly modeled and the anatomical prior knowledge is injected, so that this embodiment can generate esophageal wall structure contour prediction and tumor contour prediction results that are more in line with the anatomical structure, effectively solving the shortcomings of the existing technology in terms of structural accuracy and interpretability. In addition, the introduction of differentiable rendering technology successfully converts polygonal curves into bitmap masks with gradients, and the accuracy of the contours can be further improved by using dice loss while using L2 for node distance constraints. At the same time, due to the use of explicit cubic B-spline curves, while being able to accurately describe the esophageal and tumor structures, the T staging conclusion can be drawn based on their relative distance and simple rules. Such rules are also in line with the understanding of the diagnostic process by clinical medical staff, and can comprehensively improve the accurate segmentation and classification performance of ultrasound endoscopic images, providing a more reliable and interpretable basis for tumor localization and staging.

Embodiment three:

An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for predicting the esophageal structure, tumor contour and stage;

Wherein, the method for predicting esophageal structure and tumor contour and staging, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

The above are preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims (10)

1. A method for predicting esophageal structure and tumor contour and stage, comprising:

acquiring an original ultrasonic endoscope picture;

extracting features of the original ultrasonic endoscope picture by using a preset model to obtain full-size features;

Performing structural splitting and node reconstruction on the full-size features by using the preset model to obtain a three-layer esophageal structure and control points of tumor growth contours;

Generating a cubic B-spline curve of the esophagus and the tumor according to the control point in a spline interpolation mode;

obtaining a tumor staging result by calculating the relative distance between the cubic B-spline curves;

Forming a predicted result of an esophagus structure, a tumor contour line and tumor stage by the cubic B-spline curve and the tumor stage result;

The preset model is a deep learning model formed by different devices and modules.

2. The method for predicting esophageal structure and tumor contour and stage according to claim 1, wherein the method for performing structural splitting and node reconstruction on the full-size features by using the preset model is characterized in that control points of three-layer esophageal structure and tumor growth contour are obtained, and specifically:

Using a structure detection module of the preset model to perform dimension transformation and splitting on the full-size features to obtain a global feature token;

Combining the global feature token with preset esophagus and tumor control point features, and obtaining a control point of a three-layer esophagus structure through a coordinate transformation form;

And obtaining control points of the tumor outward and inward growth contour lines by performing feature mapping on preset tumor features on the basis of different radiuses.

3. The method for predicting esophageal structure and tumor contour and stage according to claim 2, wherein the global feature token is combined with preset esophageal and tumor control point features to obtain the control points of the three-layer esophageal structure through a coordinate transformation form, specifically comprising the following steps:

Calculating to obtain the features of the esophagus and the tumor control points according to the global feature token by using a first preset formula;

mapping the features of the esophagus and the tumor control points by using a linear transformation layer, and converting a mapping result into a polar coordinate form to obtain node coordinates of a mucosal layer contour line;

According to the control point characteristics, calculating the relative radius of the submucosa and the adventitia respectively by using the linear transformation layer, and calculating the actual radius of the submucosa and the adventitia according to the node coordinates and the relative radius;

And converting the control points corresponding to the control point characteristics into a form of Cartesian coordinate representation according to the actual radius and the angle of the node coordinates relative to the center of the esophagus to obtain the control points of the three-layer esophagus structure.

4. A method for predicting esophageal structure and tumor contour and stage as defined in claim 3, wherein the control points of tumor outward and inward growth contours are obtained by feature mapping of preset tumor features on the basis of different radii, specifically:

Defining tumor characteristics;

Mapping the tumor features on a first radius scale and a second radius scale respectively to obtain a first relative radius and a second relative radius;

sequentially using a first relative radius and a second relative radius, and calculating the radius of the node coordinates relative to the center of the esophagus to obtain the actual radius of the tumor outward and inward growth contour lines respectively;

And carrying out coordinate transformation of the node positions according to the actual radius of the tumor outward and inward growth contour lines and the angles of the mucous membrane layers to obtain the control points of the tumor outward and inward growth contour lines.

5. The method for predicting esophageal structure and tumor contour and stage according to claim 1, wherein a cubic B-spline curve of esophagus and tumor is generated according to the control point by means of spline interpolation, specifically:

processing control points of the three-layer esophagus structure and the tumor growth contour by using a three-time B-spline interpolation method respectively to obtain a parameterized esophagus structure contour line and a parameterized tumor contour line;

Calculating the normal vector of each curve point on the esophageal structure contour line and the tumor contour line to obtain a unit vector of a normal;

and forming a cubic B-spline curve of the esophagus and the tumor by the esophageal structure contour line, the tumor contour line and the unit vector of the normal.

6. The method for predicting esophageal structure and tumor contour and stage according to claim 1, wherein the tumor stage result is obtained by calculating the relative distance between the cubic B-spline curves, specifically:

constructing Euclidean distances between all points on the curve according to a first contour line of tumor growth outwards in the cubic B-spline curve and a second contour line of submucosal layer to obtain a distance matrix;

Performing distance calculation and normal vector judgment on the first contour line to the second contour line according to the distance matrix in an index mode to obtain a maximum value of distances from each point on the first contour line to the nearest point on the second contour line, and taking the maximum value of distances as a relative distance value set;

and in the numerical value set of the relative distance, acquiring a tumor stage result according to the numerical value of the relative distance between the outline of the tumor outward growth and the submucosa and the numerical value of the relative distance between the outline of the tumor outward growth and the adventitia layer.

7. The method for predicting esophageal structure and tumor contour and stage according to claim 1, wherein the feature extraction is performed on the original ultrasonic endoscopic picture by using a preset model to obtain full-size features, specifically:

identifying and cutting an effective area boundary box of the original ultrasonic endoscope picture by using an effective area detector of the preset model to obtain an effective ultrasonic endoscope picture;

And performing feature extraction and up-sampling on the effective ultrasonic endoscope picture by using a feature encoder and a feature decoder of the preset model to obtain the full-size feature.

8. The method for predicting esophageal structure and tumor contour and stage of claim 1, wherein said pre-set model is a deep learning model composed of different devices and modules, further comprising:

obtaining a DICE loss function by measuring the consistency between the segmentation mask predicted by the preset model and the real segmentation mask;

Obtaining a polygon loss function by measuring the error between a predicted node and an actual node of the preset model;

obtaining a classification loss function by quantifying the error between the T stage label predicted by the preset model and the actual T stage label;

and training the preset model according to the DICE loss function, the polygonal loss function and the classification loss function by using a preset training set and a mask of tumor and food tube wall to obtain an optimized preset model.

9. The device for predicting the esophagus structure, the tumor outline and the stage is characterized by comprising a data module, a characteristic module, a node module, a curve module, a stage module and an integration module;

the data module is used for acquiring an original ultrasonic endoscope picture;

the feature module is used for carrying out feature extraction on the original ultrasonic endoscope picture by using a preset model to obtain full-size features;

The node module is used for carrying out structural splitting and node reconstruction on the full-size characteristics by using the preset model to obtain a three-layer esophagus structure and a control point of a tumor growth contour;

the curve module is used for generating a cubic B-spline curve of the esophagus and the tumor according to the control point in a spline interpolation mode;

The staging module is used for obtaining a tumor staging result by calculating the relative distance between the cubic B-spline curves;

the integration module is used for forming a predicted result of an esophagus structure, a tumor contour line and tumor stage by the cubic B-spline curve and the tumor stage result;

The preset model is a deep learning model formed by different devices and modules.

10. A storage medium, characterized in that the storage medium has stored thereon a computer program, which is called and executed by a computer to implement a method for predicting esophageal structure and tumor contour and stage according to any one of the preceding claims 1-8.