CN117197137B - Tissue sample analysis method and system based on hyperspectral images - Google Patents

Abstract

The invention relates to the technical field of image analysis, in particular to a tissue sample analysis method and system based on hyperspectral images, comprising the following steps: acquiring hyperspectral images of tissue sample slices to be detected, splicing image data of set wave bands into RGB images, subtracting the hyperspectral characteristics of ambient light of a detection area, obtaining gray images through normalization processing, and generating a two-dimensional data set; taking the generated two-dimensional data set as input of a recognition model to obtain a recognition result; the recognition model outputs a tumor malignancy degree score based on a set function, and the maximum value of the tumor malignancy degree score is used as a recognition result.

Description

Tissue sample analysis method and system based on hyperspectral images

Technical field

The present invention relates to the technical field of image analysis, specifically a tissue sample analysis method and system based on hyperspectral images.

Background technique

The statements in this section merely provide background technical information related to the present invention and do not necessarily constitute prior art.

In addition to the information of the photographed target, hyperspectral images also contain spectral information. This feature can be used to analyze some information that cannot be observed by the naked eye from the image. For example, from hyperspectral images of human or animal tissue samples, the morphological changes of cells or substances in some images can be analyzed, and these cells or substances as analysis targets have certain similarities in the spectral space, causing the computer to process At this time, it is difficult to distinguish the required feature information, making it difficult to meet the needs.

Contents of the invention

In order to solve the technical problems existing in the above background technology, the present invention provides a tissue sample analysis method and system based on hyperspectral images, forming a feature matrix from the spectral features and texture features in the image, and using a label matrix corresponding to the feature matrix, so that The computer can determine the corresponding spectral features and texture features based on the labels, improving the problem that it is difficult for the computer to distinguish the required feature information.

In order to achieve the above objects, the present invention adopts the following technical solutions:

A first aspect of the present invention provides a tissue sample analysis method based on hyperspectral images, including the following steps:

Obtain the hyperspectral image of the tissue sample to be tested, splice the image data of the set band into an RGB image, subtract the hyperspectral characteristics of the ambient light in the detection area, and obtain the spectral characteristics and texture characteristics of the target in the tissue sample to be tested. After normalization The grayscale image is obtained through processing and a two-dimensional data set is generated;

Using the generated two-dimensional data set as the input of the recognition model, the score corresponding to the label matrix is obtained, and the maximum value of the score is used as the recognition result;

Among them, the two-dimensional data set has a feature matrix corresponding to spectral features and texture features, and the feature matrix corresponds to the label matrix.

Further, the process of obtaining the hyperspectral characteristics of ambient light in the detection area is specifically: placing a reference object without fluorescence characteristics on the shooting plane, obtaining a hyperspectral image of the reference object area, and extracting the one-dimensional image of each pixel in the area. Spectral data is summed and averaged by columns to obtain a row of spectral data as the spectral characteristics of ambient light.

Further, the process of normalizing the grayscale image and generating a two-dimensional data set is as follows: compressing the spectral data value range to a set interval, and obtaining the normalized one-dimensional data of each pixel, that is, each The one-dimensional data of each pixel at different wavelengths is expanded to obtain a single-channel grayscale image to generate a two-dimensional data set.

Further, the process of obtaining a single-channel grayscale image from the normalized one-dimensional data after data expansion is as follows: copy the normalized one-dimensional data by a set number, and multiply each pixel data by a set multiple. Get the grayscale image of all pixels.

Further, the training process of the recognition model includes:

Obtain hyperspectral images of pre-prepared tissue samples;

Extract the target spectral features and target texture features in the hyperspectral image and organize them into a feature matrix;

Each row in the feature matrix represents a sample, and each column represents a feature. The corresponding sample labels are organized into a label vector. In the label matrix formed by all label vectors, each row corresponds to a sample, and each column corresponds to a target score;

Use the feature matrix and label matrix as training data for model training, and train according to the set batches and rounds.

Further, extract the target spectral features in the hyperspectral image, specifically: select the area containing the target on the two-dimensional grayscale image, extract the one-dimensional normalized spectral data corresponding to each pixel in the area, and obtain the statistical feature average value, variance, energy, spectral slope, and kurtosis to describe the distribution and changes of the spectrum.

Further, the target texture features in the hyperspectral image are extracted, specifically:

Extract texture features based on at least one of a gray level co-occurrence matrix, a gray level difference matrix, and a local binary pattern;

By obtaining the grayscale relationship between a certain pixel in the image and its neighboring pixels, a matrix is generated, and contrast, energy and homogeneity are extracted from the matrix;

By obtaining the grayscale difference between each pixel and its neighboring pixels in the image, a matrix is generated from which contrast and roughness are extracted;

By comparing the grayscale values of a pixel and its neighboring pixels, a binary image is generated, and the texture pattern and texture direction are extracted from the binary image.

A second aspect of the present invention provides a system required to implement the above method, including:

The preprocessing module is configured to: obtain a hyperspectral image of the tissue sample to be tested, splice the image data of the set band into an RGB image, subtract the hyperspectral characteristics of the ambient light in the detection area, and obtain the spectrum of the target in the tissue sample to be tested. Features and texture features are normalized to obtain grayscale images and generate two-dimensional data sets;

The analysis module is configured to: use the generated two-dimensional data set as the input of the recognition model, obtain the score corresponding to the label matrix, and use the maximum value of the score as the recognition result;

Among them, the two-dimensional data set has a feature matrix corresponding to spectral features and texture features, and the feature matrix corresponds to the label matrix.

A third aspect of the invention provides a computer-readable storage medium.

A computer-readable storage medium has a computer program stored thereon, and when the program is executed by a processor, the steps in the tissue sample analysis method based on hyperspectral images are implemented as described above.

A fourth aspect of the invention provides a computer device.

A computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the tissue sample analysis method based on hyperspectral images is implemented as described above. steps in.

Compared with the existing technology, one or more of the above technical solutions have the following beneficial effects:

1. Form a feature matrix from the spectral features and texture features in the image, and use the label matrix corresponding to the feature matrix, so that the computer can determine the corresponding spectral features and texture features based on the labels, improving the problem that it is difficult for the computer to distinguish the required feature information. .

2. The recognition model takes the hyperspectral image information of the tissue sample as input, and uses the maximum value of the target score introduced during training as the output to obtain the analysis results. It can utilize the correlation between the hyperspectral data and the original tissue sample in the training stage. Target results contained in hyperspectral image information of tissue samples are identified. This makes it easier to determine the morphological changes of target cells or substances, with higher accuracy and stronger objectivity.

3. Through data normalization, one-dimensional data of each pixel at different wavelengths is obtained, which can shield the interference of ambient light and suppress the white noise interference in the data, compress the spectral characteristic value range, but retain the spectral characteristics completely. , to prevent the problem of excessive gradients during model training.

Description of drawings

The description and drawings that constitute a part of the present invention are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

Figure 1 is a schematic flow chart of tissue sample analysis based on hyperspectral images, taking tissue samples of early stage prostate tumors as an example provided by one or more embodiments of the present invention;

Figure 2 is a schematic diagram of the working process of the preprocessing module in the tissue sample analysis system based on hyperspectral images provided by one or more embodiments of the present invention;

Figure 3 is a schematic diagram of designing a loss function for model optimization during tissue sample analysis based on hyperspectral images provided by one or more embodiments of the present invention;

Figure 4 is a schematic diagram of introducing a custom attention mechanism to optimize the model in the process of tissue sample analysis based on hyperspectral images provided by one or more embodiments of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

As introduced in the background art, the cells or substances that are the target of analysis have certain similarities in the spectral space, which makes it difficult for computers to distinguish the required feature information during processing, making it difficult to meet the demand.

Therefore, the following embodiments provide a tissue sample analysis method and system based on hyperspectral images. The spectral features and texture features in the image are formed into a feature matrix, and the label matrix corresponding to the feature matrix is used, so that the computer can determine the corresponding label based on the label. Spectral features and texture features improve the problem that it is difficult for computers to distinguish the required feature information.

In the following embodiments, a prostate tissue sample that is easily obtained in clinical practice is taken as an example. After taking a hyperspectral image, the morphology of tumor cells in the tissue sample is used as the target for analysis, and the hyperspectral image of the early prostate tumor tissue sample is used. Image information is used as training data. After the recognition model completes training, it can identify the hyperspectral image information of tissue samples and obtain the analysis results of early prostate tumors.

Basic introduction 1: Prostate cancer is a common malignant tumor in men. Blood samples are usually used for prostate-specific antigen (PSA) screening to complete the detection of prostate cancer, because PSA is a tissue-specific rather than tumor-specific marker. , resulting in low specificity and sensitivity of PSA testing for prostate cancer, making it impossible to grasp the actual situation of canceration and affecting subsequent testing judgments. However, using tumor tissue for pathological analysis requires tissue sections to be made into specimens, which is complex and time-consuming. In addition, manual detection is greatly affected by external influences, and the test results lack objectivity and accuracy.

Basic introduction 2: Hyperspectral imaging technology is based on very narrow-band image data technology. It combines imaging technology with spectral technology to detect the two-dimensional geometric space and one-dimensional spectral information of the target, and obtain continuous, high spectral resolution. Narrowband image data. Common hyperspectral imaging technologies include grating spectroscopy, acousto-optic tunable filter spectroscopy, prism spectroscopy, and chip coating.

Example 1:

As shown in Figures 1-4, the tissue sample analysis method based on hyperspectral images includes the following steps:

Obtain the hyperspectral image of the tissue sample to be tested, splice the image data of the set band into an RGB image, subtract the hyperspectral characteristics of the ambient light in the detection area, and obtain the spectral characteristics and texture characteristics of the target in the tissue sample to be tested. After normalization The grayscale image is obtained through processing and a two-dimensional data set is generated;

Using the generated two-dimensional data set as the input of the recognition model, the score corresponding to the label matrix is obtained, and the maximum value of the score is used as the recognition result;

Among them, the two-dimensional data set has a feature matrix corresponding to spectral features and texture features, and the feature matrix corresponds to the label matrix.

The spectral feature extraction process of early prostate tumors is specifically:

Select tumor area: Select the area containing the tumor on the 2D grayscale image. Tumor regions can be obtained using methods such as manual annotation and threshold segmentation. Extract the one-dimensional normalized spectral data of each pixel: For the selected tumor area, extract the one-dimensional normalized spectral data corresponding to each pixel. According to the previous data processing steps, the one-dimensional data of each pixel is the normalized value at different wavelengths. Calculate statistical features: For the one-dimensional spectral data of each pixel, calculate the statistical features mean, variance, energy, spectrum slope (Skewness), and kurtosis (Kurtosis) to describe the spectrum. distribution and changes. These statistical characteristics can be implemented through computational libraries.

The texture feature extraction process of early prostate tumors is specifically: using gray level co-occurrence matrix (GLCM), gray level difference matrix (GLDM), and local binary pattern (LBP) to extract texture features. GLCM generates a matrix by calculating the grayscale relationship between a certain pixel in the image and its neighboring pixels. Then features are extracted from this matrix, contrast, energy, homogeneity. GLDM generates a matrix by calculating the grayscale difference between each pixel in the image and its neighboring pixels. Then, contrast and roughness are extracted from this matrix. LBP generates a new binary image by comparing the grayscale values of a pixel and its neighboring pixels. Then, the texture pattern and texture direction can be extracted from this binary image.

The process of sorting feature data and label data is specifically: Feature data sorting: Organize the extracted features (spectral features, texture features) into a matrix (feature matrix), with each row representing a sample and each column representing a feature. Label data sorting: Organize the corresponding sample labels into a vector (label vector), in which each element represents the label of a sample. The Gleason score of early prostate tumors is 2-4 points, with a total of 3 possible values. The categorical features with 3 possible values are converted into 3 binary features. Each feature represents a possible category. Each Gleason score can be treated as a separate category. All label vectors are formed into a label matrix, where each row corresponds to a sample and each column corresponds to a Gleason score.

The establishment process of prostate early tumor identification model. Specifically:

Build the model architecture: Select the ResNet50 convolutional neural network model. The structure of the model includes convolutional layers, pooling layers, fully connected layers and other forms.

Input layer: Specify the shape of the input data and set the dimensions of the input layer according to the number of columns of the feature matrix.

Convolutional layer: By adding convolutional layers, the spatial characteristics of the image are extracted. Set the appropriate convolution kernel size, stride, and padding method to adapt to the characteristics of the input data.

Activation function: Add appropriate activation functions, such as ReLU functions, to the convolutional layer to introduce nonlinear characteristics.

Pooling layer: Add a pooling layer to reduce feature dimensionality and retain spatial information, and select the appropriate pooling kernel size and step size.

Fully connected layer: Flatten the output of the pooling layer, add a fully connected layer for classification, and set the appropriate number of neurons and activation function.

Output layer: Set the dimension of the output layer to match the number of columns of the label matrix, and use the softmax function as the activation function of the output layer to obtain the probability distribution of the classification results.

Compile the model: Define the loss function and optimization program. Cross entropy can be used as a loss function, and the ADAM optimizer performs parameter optimization.

Model training: Use the feature matrix and label matrix as training data, use the Fit() function to train the model, and specify the appropriate batch size and number of training rounds.

Model evaluation: Use the test set to evaluate the trained model and calculate indicators such as precision, precision, recall, and F1-SCORE.

As shown in Figure 3, the process of designing a specific loss function is as follows:

Define early detection penalty factor: This factor is set to 1.5 to make early detection errors more costly. Calculation of loss function: During model training, the loss function is calculated based on the prediction results and real labels. For each sample, the loss is calculated based on the predicted class and the true class. Weighted loss function: Calculate the weighted loss function based on the predicted category and the true category. For early tumor detection errors, the loss function is multiplied by the early detection penalty factor to make it have a larger loss. Loss function summation: Sum the weighted loss functions of all samples to obtain the final loss function. Additional adjustments and optimizations can be made as needed. Model training and optimization: Use the defined weighted loss function for model training and optimization. Choose the appropriate optimizer and learning rate, and adjust the optimization parameters according to reality. Model evaluation: Use the test set to evaluate the trained model and calculate indicators such as precision, precision, recall, and F1-SCORE.

The process of model fusion is specifically: training multiple convolutional neural network models with different initial parameters, and using voting for model fusion. Then, the effectiveness of the fusion model is verified by evaluating its performance indicators, and further adjustments and optimizations are made as needed.

The process of introducing the attention mechanism is specifically: adding the attention module, defining the attention weight, feature weighted fusion, model training and optimization, and model evaluation and adjustment.

The process of model evaluation and optimization is as follows: cross-validation is used for verification. Split the data set into two parts: training set and test set. Model training, average performance evaluation, parameter adjustment, model optimization, result analysis, and visualization tools such as drawing learning curves and confusion matrices to help understand the performance of the model.

Taking the hyperspectral image analysis process of prostate tumor samples as an example, specifically:

Step S1, collect early prostate tumor samples.

Tumor samples from early-stage prostate tumor surgery patients were collected and cryopreserved.

In this embodiment, hyperspectral image information of early prostate tumor samples is obtained as training data. Early prostate tumor samples are derived from early prostate tumor tissues of patients who have undergone prostate puncture or prostate tumor resection surgery. After informing them of the purpose of the research experiment and possible existing risk, and with the patient's consent, early-stage prostate tumor samples were divided into a separate group with benign pathological results, and malignant ones were scored according to the Gleason scoring system (a method commonly used in the industry for histological grading of prostate cancer). Those with the same score were A group of early prostate tumor samples of different grades were formed and all were placed at low temperature.

Step S2: Obtain the spectral characteristics of ambient light.

In this embodiment, a white diffuse reflection reference object with no fluorescence characteristics is placed on the shooting plane, a hyperspectral image of the reference object area is obtained, one-dimensional spectral data is extracted for each pixel in the area, and the data is stored in a csv file. Sum and average by column to obtain a row of spectral data as the spectral characteristics of ambient light.

Step S3: Scan early prostate tumor samples based on hyperspectral.

After the preprocessed specimens are thawed, they are photographed by a hyperspectral camera to obtain original hyperspectral images.

The hyperspectral image in this embodiment includes spectral dimensions and three-dimensional spatial dimensions, and SpecView-F software is used to collect hyperspectral data. The SpecView-F acquisition software here is developed for the GaiaSky series airborne spectral imaging system, GaiaTracer document inspection system and Image-λ-F series hyperspectral cameras. It is mainly used to realize the control of the spectral imaging system, image acquisition and other functions.

Step S4: Perform graphic analysis and data processing on the collected hyperspectral data.

Obtain the image data of the specified band, splice the multiple band data into an RGB image, generate a one-dimensional data set, then subtract the ambient light spectral characteristics (environmental noise reduction), draw the grayscale image after normalizing the data, and generate a two-dimensional data set. dimensional data set.

In this embodiment, the original image data collected by the spectrometer is processed. First, the data is read from the specified file, the specified .mat file is loaded, the image data of the specified band is obtained, and displayed on the screen. The red, green, and blue band data are spliced into an RGB image through the cat function. The R channel wavelength selection range is [630nm, 650nm], the G channel wavelength selection range is [530nm, 550nm], and the B channel wavelength selection range is [ 450nm, 470nm] to obtain one-dimensional spectral data of each pixel of the image. Then subtract the ambient light spectral characteristics (environmental noise reduction), that is, subtract the average spectral characteristics of the ambient light from the one-dimensional spectral data of each pixel in the tumor hyperspectral image.

Then perform data normalization, compress the spectrum data value range to [-1, 1], and obtain the normalized one-dimensional data of each pixel, that is, the one-dimensional data of each pixel at different wavelengths. Through this step, the interference of ambient light can be shielded and the white noise interference in the data can be suppressed, and the spectral characteristic value range can be compressed, but the spectral characteristics can be completely retained to prevent the problem of excessive gradients during model training.

Perform data expansion on the normalized one-dimensional data to generate a single-channel grayscale image, which includes: copying the normalized one-dimensional data. After copying 32 lines, multiply each pixel data by 255 and save it as Grayscale image. Through the above process, the grayscale image of all pixels is obtained as two-dimensional data. It should be pointed out that the normalized one-dimensional data can also be copied here for more than 32 rows, and the resulting two-dimensional data can be input into a more complex deep learning model to obtain more complex features.

Step S5: Extract features of early prostate tumors in the hyperspectral image.

Extract tumor spectral features:

Select the tumor area of interest, specifically: select the area of interest containing the tumor on the two-dimensional grayscale image. Tumor regions can be obtained using methods such as manual annotation and threshold segmentation.

Extract the one-dimensional normalized spectral data of each pixel, specifically: extract the one-dimensional normalized spectral data corresponding to each pixel for the selected tumor area. According to the previous data processing steps, the one-dimensional data of each pixel is the normalized value at different wavelengths.

Calculate statistical features, specifically: for the one-dimensional spectral data of each pixel point, calculate statistical features to describe the distribution and changes of the spectrum. Calculate the average of one-dimensional spectral data to reflect the central trend of the spectrum. Calculate the variance of one-dimensional spectral data to reflect the dispersion of the spectrum. Calculate the energy of one-dimensional spectral data and reflect the amplitude information of the spectrum. Calculate the spectral slope of one-dimensional spectral data to reflect the asymmetry of the spectral distribution. Calculate the kurtosis of one-dimensional spectral data to reflect the sharpness of the spectral distribution.

For each pixel, the calculated statistical features are stored at the corresponding location to form a feature map. Each pixel corresponds to a feature vector, which contains statistical characteristic information of the spectrum.

Extract tumor texture features. The gray level symbiosis matrix (GLCM) is used to extract texture features, specifically: define the distance and direction of adjacent pixels: select the appropriate distance and direction of adjacent pixels according to the characteristics and needs of the image, such as horizontal, vertical, and diagonal lines. wait.

Gray quantization, specifically: discretize the gray level of the image into a set of discrete values, and use methods such as gray layering or histogram equalization to perform gray quantization.

Construct a gray-level co-occurrence matrix, specifically: for each pixel, calculate its gray-level value pair with neighboring pixels in the specified distance and direction. According to the gray-level pairs of neighboring pixels, count their occurrence times in the gray-level co-occurrence matrix.

Extract statistical features, specifically: extract various statistical properties of the gray-level co-occurrence matrix: contrast, energy, homogeneity, and correlation. These features can be obtained by calculating different combinations of gray-level co-occurrence matrices, contrast can be calculated by calculating the sum of squares of gray-level differences of adjacent pixels, and energy can be calculated by calculating the sum of squared elements of gray-level co-occurrence matrices.

The gray level difference matrix (GLDM) is used to extract texture features, taking the same distance and direction of neighboring pixels as the gray level symbiosis matrix.

Calculate the grayscale difference. For each pixel, calculate the grayscale difference value between it and neighboring pixels at the specified distance and direction.

Construct a grayscale difference matrix, and count the number of times they appear in the grayscale difference matrix based on the grayscale difference values of adjacent pixels.

Extract statistical features. Various statistical features are extracted from the grayscale difference matrix: contrast, roughness. These features can be obtained by calculating different combinations of grayscale difference matrices, contrast can calculate the average of the grayscale differences of neighboring pixels, roughness can calculate the standard deviation of the grayscale difference matrix, etc.

Use local binary pattern (LBP) to extract texture features, specifically: take the same distance and direction of neighboring pixels as the gray-level symbiosis matrix.

The central pixel is compared with neighboring pixels, specifically: for each pixel, its gray value is compared with the gray value of neighboring pixels to obtain a binary code.

Construct a binary image, specifically: according to the comparison result, convert the binary code into a binary image, in which each pixel represents a local texture pattern.

Extract statistical features: Extract various statistical features from binary images: texture pattern frequency, texture pattern distribution. These features can be obtained by calculating the histogram, mean, variance, etc. of the texture pattern.

Step S6: Organize feature data and label data.

Feature data sorting: Organize the extracted features (spectral features, texture features) into a matrix (or feature matrix), with each row representing a sample and each column representing a feature. This matrix is usually represented as a numpy array.

Label data sorting: Organize the corresponding sample labels (Gleason scores) into a vector (or label vector), in which each element represents the label of a sample. Because the Gleason score of early prostate tumors is 2-4 points, with a total of 3 possible values, the categorical features with 3 possible values are converted into 3 binary features, each feature represents a possible category, and each Gleason score is Can be considered as an independent category. as follows:

2->[1, 0, 0]

3->[0, 1, 0]

4->[0, 0, 1]

All label vectors are formed into a label matrix, in which each row corresponds to a sample and each column corresponds to a Gleason score. A value of 1 indicates that the score of the sample is the score corresponding to the column, and a value of 0 indicates that it is not.

Step S7: Establish an early prostate tumor recognition model.

The ResNet50 network model is selected and residual connections are introduced to solve the gradient disappearance and representation bottleneck problems in deep networks, and can train deeper networks.

Input layer: Specify the shape of the input data and set the dimensions of the input layer according to the number of columns of the feature matrix.

In the convolutional layer, the convolution kernel size can be selected as 3x3 or 5x5 convolution kernel. The number of roll surface cores can be set according to the actual number of roll surface cores. For example, 32, 64, 128, etc. Step size: Generally set to 1. The filling method can be selected as "valid" or "same", depending on the size of the input data and model requirements. The activation function generally uses the ReLU activation function.

In the pooling layer, the pooling core size is generally 2x2. The step size is generally set to 2. The filling method can be selected as "valid" or "same", depending on the size of the input data and model requirements.

In the fully connected layer, the number of neurons is set according to the actual situation, and you can try different values to find the best performance. The activation function generally uses the ReLU activation function.

In the output layer, the dimension of the output layer is set according to the number of columns of the label matrix, which is the same as the number of categories.

The activation function uses the softmax function to convert the output into a classification result of a probability distribution.

Loss function and optimizer: For multi-classification tasks, choose the cross-entropy loss function (categorical_crossentropy) as the loss function. Because Gleason's score is 3 ability points. Each one can be considered an independent category. Convert the label matrix into 3 binary features. Therefore, the loss function will be calculated based on the cross-entropy between the 3 categories.

In addition, according to a specific loss function, to consider the importance of early tumor detection and the difference in error penalties, a weighted loss function is used to implement it, specifically as follows: Define the early detection penalty factor: First, define an early detection penalty factor according to the specific needs of the problem Detection penalty factor, which represents the penalty for early tumor detection errors. This factor is set to a positive number of 1.5 to make early detection errors more costly.

Calculate the loss function: During the model training process, the loss function is calculated based on the difference between the prediction result and each element of the true label vector. For each sample, the loss is calculated based on the predicted class and the true class.

Weighted loss function: Calculate the weighted loss function based on the predicted category and the true category. For early tumor detection errors, the loss function is multiplied by the early detection penalty factor to make it have a larger loss.

Loss function summation: Sum the weighted loss functions of all samples to obtain the final loss function. Additional adjustments and optimizations can be made as needed.

Model training and optimization: Use the defined weighted loss function for model training and optimization. Choose the appropriate optimizer and learning rate, and adjust the optimization parameters according to reality.

Model evaluation: Use the test set to evaluate the trained model and calculate indicators such as precision, precision, recall, and F1-SCORE. The performance of the model can be further analyzed through methods such as confusion matrices.

By designing a specific loss function, the model can pay more attention to the accuracy of early tumor detection and penalize errors based on the penalty factor. This helps improve the model's performance in early tumor detection. It is necessary to make some appropriate adjustments and optimizations based on the characteristics of the real questions and data. for optimal performance and results.

For optimizer selection, the Adam optimizer can be used. And set parameters such as learning rate that suit you. The Adam optimizer combines the characteristics of momentum and adaptive learning rate. It is often used in deep learning tasks and can automatically adjust the learning rate during the training process. The learning rate can be adjusted appropriately according to the actual situation. The initial learning rate is set to 0.001. And adjust the decay rate or other parameters of the learning rate based on the performance during training.

Model training: Batch size: Set according to the actual situation, usually choose a smaller batch size, such as 32, 64, etc.;

Number of training wheels: According to the actual setting. Usually a sufficient number of training epochs is chosen to achieve convergence.

Model evaluation: In the evaluation stage, a variety of indicators are used to evaluate performance. Including accuracy rate, precision rate, recall rate, F1-SCORE, etc. Below is a detailed description of how these indicators are calculated and analyzed:

Accuracy: Accuracy is one of the most commonly used evaluation indicators in classification models, indicating the proportion of the total number of correctly classified samples. The calculation formula is as follows: accuracy = (number of correctly predicted samples)/(total number of samples).

Precision: Precision measures the proportion of samples predicted as positive by the model that are actually positive. The calculation formula is as follows: Accuracy = (number of samples that are actually positive categories)/(number of samples that are predicted to be positive categories).

Recall: Recall measures the ability of a model to correctly identify samples of a category. That is, the proportion of positive category samples that the model successfully captures. The calculation formula is as follows: recall rate = (number of samples that are truly positive categories)/(number of samples that are truly positive categories).

F1-score: F1-SCORE is an evaluation index that comprehensively considers precision and recall. is the harmonic mean of precision and recall. The calculation formula is: F1-score=2*(precision rate*recall rate)/(precision rate+recall rate).

These indicators can further analyze the performance of the model through the confusion matrix. The confusion matrix is a two-dimensional matrix used to represent the correspondence between model prediction results and real labels. It contains four important indicators: true example (1Positive, TP), false positive example (0Positive, FP), true counterexample (1Negative, TN), and false counterexample (.FN).

TP: The number of positive category samples predicted by the model as positive categories.

FP: The number of negative class samples predicted by the model as positive classes.

TN: The number of negative class samples predicted by the model as negative classes.

FN: The number of positive category samples predicted by the model as negative categories.

The above indicators can be calculated through the confusion matrix. The following are the calculation steps:

According to the model prediction results and real labels, the samples are classified into four categories: TP, FP, TN, and FN.

Calculate accuracy, precision, recall, and F1-SCORE based on the classification results.

The calculation and analysis of the confusion matrix and the above indicators can help understand the performance of the model on different categories, and then adjust and optimize the model. It should be noted that the above parameters are only for reference, and the specific parameter selection should be adjusted and optimized according to the actual situation and the characteristics of the data set. During the training process of the model, techniques such as cross-validation can be used to evaluate the performance of the model on different data subsets to select the best parameters and model configurations.

By calculating these metrics, model performance can be evaluated and compared. Accuracy measures the overall accuracy of classification. Precision and recall provide an assessment of the accuracy and completeness of positive class predictions, and F1-score combines precision and recall to provide a more comprehensive assessment of the model's performance.

In addition, the classification ability and discrimination of the model were evaluated by plotting the ROC curve (receiver operating characteristic curve) and calculating the AUC (area under the curve). The ROC curve is a curve with recall rate on the horizontal axis and 1-specificity (0PositiveRate) on the vertical axis. AUC represents the area under the ROC curve and can be used to evaluate the overall performance of the model.

Step S8, model fusion.

Model fusion is the prediction result of combining multiple models. Thereby improving the robustness of the overall forecast technology.

In this embodiment, multiple convolutional neural network models with different initial parameters are trained and model fusion is performed using voting. Then, the effectiveness of the fusion model is verified by evaluating its performance indicators, and further adjustments and optimizations are made as needed.

The following are the specific steps for using voting strategies in model fusion:

Train multiple independent convolutional neural network models: Train multiple convolutional neural network models with different initial parameters. Make sure each model is fully trained and trained on the same training set to ensure fairness.

Use the test set to make predictions for each model. Make predictions for each trained model using an independent test set (non-overlapping with the training and validation sets). For each test sample, obtain the prediction results of each model.

Take a vote: For each sample, take a vote among the predictions of all models. The majority voting principle can be used to select the category with the highest number of votes as the final prediction result. If there is a tie vote, you can choose to randomly select a category or use another decision rule.

Calculate performance indicators: Use the prediction results after model fusion to calculate the performance indicators of the model, such as accuracy, precision, recall, F1-SCORE, etc. A confusion matrix can be used to further analyze the performance of the model.

Adjustment and optimization: Make further adjustments and optimize parameters based on the performance indicators of model fusion. You can try increasing or decreasing the number of models participating in voting to find the best model fusion strategy.

It should be noted that when performing model fusion, ensure that the models participating in voting are diverse, that is, there is a certain degree of difference between the models. This can be achieved by adjusting the model's architecture, optimizing parameters, using different loss functions, etc.

Finally, through model fusion, the prediction results of multiple models can be comprehensively utilized to improve the accuracy and robustness of the overall prediction. But keep in mind that model fusion also comes with a trade-off in computational resources and time costs, as training and predicting multiple models may require more computational resources and time.

Step S9: Introduce a custom attention mechanism, as shown in Figure 4.

Define the attention module: This module accepts the feature map (the output of the middle layer of ResNet50) as input, and then outputs a weight matrix of the same size through convolution, BatchNormalization and sigmoid activation function.

Insert the attention module in ResNet: Insert the attention module defined above at an appropriate position in ResNet (such as after the output of a residual block). Then, the feature map is weighted using the output of the attention module (weight matrix) to obtain a new feature map. This process is an "attention weighting" process, which allows the network to pay more attention to those areas with larger weights (the output of the attention module is close to 1).

Model training and optimization: Use weighted feature representations for model training and optimization. Select the designed specific loss function and optimizer, and adjust and optimize parameters according to the actual situation.

Model evaluation: Use the test set to evaluate the trained model and calculate indicators such as accuracy, precision, recall, and F1-score.

In practical applications, attention visualization can also be performed to help understand the model's focus on the image.

By introducing the attention mechanism, the convolutional neural network can more accurately focus on important features in the image, improving the performance and expressiveness of the model.

Step S10: Model evaluation and optimization.

Evaluate the performance of the model through methods such as cross-validation. And adjust the parameters of the model based on the evaluation results to optimize the performance of the model. Here are the specific steps:

Divide the data set: First, divide the image data set into two parts: the training set and the test set. During the cross-validation process, the training set is further divided into two parts: training set and validation set. Perform 10-fold cross-validation, divide the training data into 10 subsets, use 9 subsets each time as training image data, and the remaining subsets as verification image data.

Train the model: Then, for each training/validation data split, train the model using the training data and evaluate the model using the validation data. Document the results of each assessment.

Average performance evaluation: After 10 times of training and verification, calculate the average of all verification results to obtain the average performance evaluation of the model.

Parameter adjustment: Based on the average performance evaluation of the model, adjust the parameters of the model, such as the learning rate, the size and number of convolution kernels, the number of neurons in the fully connected layer, etc. Then repeatedly divide the data set, train the model, and evaluate the average performance until the optimal model parameters are found.

Model optimization: After finding the optimal model parameters, use these parameters to train the model, and use the test set for final performance evaluation. If there is still room for improvement in the performance of the model, you can try to use a more complex model structure, or introduce techniques such as regularization and batch normalization to optimize the model.

Result analysis: After the model training and optimization are completed, analyze the training process and results. Visual tools such as learning curves and confusion matrices can be used to help understand the performance of the model. If the model's predictive performance is poor in certain categories, you can further analyze the data to identify possible reasons.

Step S11: Implement prediction.

Enter the preprocessed new data into the optimized model. Predicted by the malignancy of early prostate tumors.

The above method does not require the preparation of complex tumor tissue slices. It directly uses a hyperspectral camera to capture hyperspectral images of the required identification area and performs data processing on the hyperspectral images. The early prostate tumor recognition model is used to realize and complete the identification of regional early prostate tumors. To score the degree of malignancy, all operations can be completed with only one set of equipment. The entire process is short and time-consuming, greatly saving time.

Use hyperspectral to detect early prostate tumors, classify patients through pathological results, and analyze different hyperspectral features to prove the value of identifying early prostate tumors based on hyperspectral images.

Artificial intelligence is used to learn hyperspectral data from tissue samples of prostate cancer patients with different pathological grades, and the correlation between hyperspectral data and the malignancy of early prostate cancer is obtained, so as to predict the pathological results of prostate cancer patients.

Combining spectral features and texture feature recognition, spectral features can provide information about spectral intensity and changes in different bands, while texture features provide spatial relationships and structural information between pixels in the image. They have complementary characteristics, and their combined use can capture richer and more diverse feature information, improve the accuracy and robustness of identification, and enhance the model's ability to identify prostate tumors.

Spectral and textural features respond differently to interference and noise. Spectral signatures are sensitive to changes in spectral intensity and can help resist the effects of illumination and stray light. Texture features can capture subtle texture changes in local areas of the image, and are robust to uneven lighting and image noise. By combining the two, combined with environmental denoising, the model's resistance to interference factors such as ambient light can be enhanced and the accuracy of prostate tumor identification can be improved.

Designing a loss function that imposes a greater penalty on early-stage tumor detection errors by designing a specific loss function can make the model pay more attention to early-stage tumors and work harder to reduce such errors. This penalty mechanism can provide an additional stimulus to encourage the model to more sensitively capture the characteristics of early tumors, thereby improving the accuracy of early detection of tumors. This designed loss function helps guide the model to pay more attention to early tumor detection during the optimization process and improve the early sensitivity of tumors. By increasing the penalty for early tumor detection errors, the model will tend to reduce the occurrence of such errors to improve overall performance and early identification capabilities.

Introducing a custom attention mechanism can help the model pay attention to important features related to prostate tumors when processing hyperspectral images. By adjusting attention to specific spectra or specific regions, the model can more accurately capture information related to prostate tumors, thereby improving recognition performance. The custom attention mechanism can reduce the impact of redundant bands by assigning attention weights, thereby reducing the model's calculation and learning burden on redundant information and improving the efficiency and accuracy of the model. The spectral characteristics of early-stage prostate tumors may vary depending on factors such as individual differences, lesion type and size. The custom attention mechanism can dynamically adjust the attention weight according to the different characteristics and needs of the input data, so that the model can adaptively focus on distinguishing and important features in different samples, improving the robustness and generalization of recognition. ability.

Only manual operations are required to obtain tumor hyperspectral images and perform image analysis and data processing. No more manual operations are required. The classification results are objective and accurate, and the manpower and material costs are greatly saved. The operating threshold is lowered.

Using postoperative tumors from patients with early-stage prostate tumors as samples is easy to obtain clinically, has sufficient sample space, and can continuously optimize and improve the recognition model.

Example 2:

Systems that implement the above methods include:

The preprocessing module is configured to: obtain a hyperspectral image of the tissue sample to be tested, splice the image data of the set band into an RGB image, subtract the hyperspectral characteristics of the ambient light in the detection area, and obtain the spectrum of the target in the tissue sample to be tested. Features and texture features are normalized to obtain grayscale images and generate two-dimensional data sets;

The analysis module is configured to: use the generated two-dimensional data set as the input of the recognition model, obtain the score corresponding to the label matrix, and use the maximum value of the score as the recognition result;

Among them, the two-dimensional data set has a feature matrix corresponding to spectral features and texture features, and the feature matrix corresponds to the label matrix.

Embodiment three:

This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the tissue sample analysis method based on hyperspectral images as described in the first embodiment are implemented.

Embodiment 4:

This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned method based on the first embodiment. Steps in the tissue sample analysis method using hyperspectral images.

Each step or network involved in the above second to fourth embodiments corresponds to the first embodiment. For specific implementation methods, please refer to the relevant description part of the first embodiment. The term "computer-readable storage medium" shall be understood to include a single medium or multiple media that includes one or more sets of instructions; and shall also be understood to include any medium capable of storing, encoding, or carrying instructions for use by a processor. The executed instruction set causes the processor to perform any method in the present invention.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (8)

1. The tissue sample analysis method based on hyperspectral image is characterized by comprising the following steps:

acquiring a hyperspectral image of a tissue sample to be detected, splicing image data of a set wave band into an RGB image, subtracting the hyperspectral characteristic of ambient light of a detection area to obtain the spectral characteristic and the texture characteristic of a target in the tissue sample to be detected, carrying out normalization processing to obtain a gray image and generating a two-dimensional data set;

the generated two-dimensional data set is used as input of the recognition model, the score corresponding to the label matrix is obtained, and the maximum value of the score is used as a recognition result;

wherein the two-dimensional dataset has a feature matrix corresponding to the spectral features and the texture features, the feature matrix corresponding to the tag matrix;

the process of introducing the attention mechanism is as follows: adding an attention module, defining attention weights, carrying out feature weighted fusion, model training and optimization, and model evaluation and adjustment; definition of attention module: the module receives the feature diagram as input, and outputs a weight matrix with the same size through convolution, batchnormal and sigmoid activation functions; inserting an attention module in the model: inserting the attention module defined above into the model; weighting the feature map by using the output of the attention module to obtain a new feature map; training and optimizing the model: model training and optimization using the weighted feature representation; selecting a designed specific loss function and an optimizer, and carrying out parameter adjustment and optimization according to actual conditions; model evaluation: evaluating the trained model by using a test set, and calculating the indexes of accuracy, precision, recall rate and F1-score;

The process of designing a specific loss function is specifically: defining early detection penalty factors: this factor is set to 1.5; calculating a loss function according to the prediction result and the real label during model training; for each sample, calculating a loss from the predicted class and the true class; weighted loss function: calculating a weighted loss function according to the predicted category and the real category; for early tumor detection errors, multiplying the loss function by an early detection penalty factor; and (3) summing a loss function: summing the weighted loss functions of all the samples to obtain a final loss function; the optimization can be additionally adjusted according to the requirement; training and optimizing the model: training and optimizing the model by using the defined weighted loss function; selecting an optimizer and a learning rate, and adjusting optimization parameters according to actual conditions; evaluating the trained model by using a test set, and calculating the accuracy, the precision, the recall rate and the F1-SCORE index;

the normalization processing is performed to obtain a gray image and generate a two-dimensional data set, specifically: compressing the spectrum data value range to a set interval to obtain one-dimensional data normalized by each pixel point, namely one-dimensional data of each pixel point under different wavelengths, and obtaining a single-channel gray level image after data expansion to generate a two-dimensional data set;

Each row in the feature matrix represents one sample, each column represents one feature, corresponding sample labels are arranged into a label vector, each row corresponds to one sample in the label matrix formed by all label vectors, and each column corresponds to one target score;

and training the model by taking the feature matrix and the label matrix as training data, and training according to the set batch and the set number of rounds.

2. The tissue sample analysis method based on hyperspectral image as claimed in claim 1, wherein the process of obtaining hyperspectral characteristics of ambient light in the detection area is specifically: and placing a reference object without fluorescence characteristic on a shooting plane, acquiring a hyperspectral image of a reference object area, extracting one-dimensional spectrum data of each pixel point in the area, and averaging according to column summation to obtain one line of spectrum data as spectrum characteristics of ambient light.

3. The tissue sample analysis method based on hyperspectral image as claimed in claim 1, wherein the process of obtaining single-channel gray scale image from normalized one-dimensional data after data expansion is specifically as follows: and copying the normalized one-dimensional data by a set number, and multiplying the data of each pixel point by a set multiple to obtain gray images of all the pixel points.

4. The hyperspectral image based tissue sample analysis method as claimed in claim 1 wherein the training process of the recognition model includes:

acquiring hyperspectral images of tissue samples prepared in advance;

and extracting target spectral features and target texture features in the hyperspectral image, and finishing the target spectral features and the target texture features into a feature matrix.

5. The tissue sample analysis method based on hyperspectral image as claimed in claim 4, wherein the extraction of the target spectral feature in the hyperspectral image is specifically: selecting a region containing a target on a two-dimensional gray level image, extracting one-dimensional normalized spectrum data corresponding to each pixel point in the region, and obtaining a statistical characteristic average value, a variance, energy, spectrum gradient and kurtosis to describe the distribution and change condition of the spectrum.

6. The tissue sample analysis method based on hyperspectral image as claimed in claim 4, wherein extracting the target texture feature in the hyperspectral image comprises:

texture features are extracted based on at least one of a gray level co-occurrence matrix, a gray level difference matrix, and a local binary pattern.

7. The hyperspectral image based tissue sample analysis method as claimed in claim 6, wherein extracting the target texture feature in the hyperspectral image further comprises:

Generating a matrix by acquiring the gray scale relation between a certain pixel and the adjacent pixels in the image, and extracting contrast, energy and homogeneity from the matrix;

generating a matrix by acquiring gray level differences of each pixel and adjacent pixels in the image, and extracting contrast and roughness from the matrix;

a binary image is generated by comparing the gray values of the pixel and its neighboring pixels, and a texture pattern and a texture direction are extracted from the binary image.

8. Tissue sample analysis system based on hyperspectral images, based on a tissue sample analysis method based on hyperspectral images according to any one of claims 1 to 7, characterized in that it comprises:

a preprocessing module configured to: acquiring a hyperspectral image of a tissue sample to be detected, splicing image data of a set wave band into an RGB image, subtracting the hyperspectral characteristic of ambient light of a detection area to obtain the spectral characteristic and the texture characteristic of a target in the tissue sample to be detected, carrying out normalization processing to obtain a gray image and generating a two-dimensional data set;

an analysis module configured to: the generated two-dimensional data set is used as input of the recognition model, the score corresponding to the label matrix is obtained, and the maximum value of the score is used as a recognition result;

Wherein the two-dimensional dataset has feature matrices corresponding to spectral features and texture features, the feature matrices corresponding to the tag matrices.