Disclosure of Invention
In order to solve at least one or more of the technical problems mentioned above, the present application proposes, in various aspects, a solution for predicting a prostate puncture tendency.
In a first aspect, the present application provides an apparatus for predicting a tendency to prostate puncture, comprising a processor, and a memory having stored therein program instructions for predicting a tendency to prostate puncture, which when executed by the processor, cause the apparatus to obtain basic information of a subject and text information of a prostate image, input the basic information and the text information of the prostate image into a natural language processing model for a first tendency to prostate puncture prediction to obtain a first prediction result regarding a tendency to prostate puncture, obtain the prostate image of the subject, input the prostate image of the subject into an image processing model for a second tendency to prostate puncture prediction to obtain a second prediction result regarding a tendency to prostate puncture prediction, and obtain a final prediction result regarding a tendency to prostate puncture prediction based on the first prediction result and the second prediction result.
In one embodiment, wherein the subject's underlying information includes at least age information and/or prostate specific antigen examination information.
In another embodiment, the text information of the prostate image includes at least diameter information and/or abnormality signal information of a prostate lesion.
In yet another embodiment, wherein the image processing model includes an attention module and a residual module, and the program instructions, when executed by the processor, cause the apparatus to further perform the operations of inputting a prostate image of the subject to the image processing model and sequentially performing an attention operation and a residual operation on the prostate image using the attention module and the residual module in the image processing model to extract a prostate feature for the second prostate puncture trend prediction.
In yet another embodiment, wherein the program instructions, when executed by the processor, cause the apparatus to further perform operations of performing an attention operation on the prostate image using the attention module in the image processing model to obtain attention features and performing a residual operation on the attention features using the residual module in the image processing model to extract prostate features for the second prostate puncture propensity prediction.
In yet another embodiment, wherein the attention module comprises a location attention module and a channel attention module, and the program instructions, when executed by the processor, cause the apparatus to further perform the operations of performing a location attention operation and a channel attention operation on the prostate image sequentially using the location attention module and the channel attention module to obtain a location attention feature and a channel attention feature.
In yet another embodiment, wherein the residual module, the position attention module, and the channel attention module each comprise multiple sets.
In yet another embodiment, wherein the program instructions, when executed by the processor, cause the apparatus to further perform operations of sequentially performing a position attention operation, a channel attention operation, and a residual operation on the prostate image using each set of position attention module, channel attention module, and residual module to obtain each set of intermediate features, and sequentially performing a next position attention operation, channel attention operation, and residual operation on each set of intermediate features using a next set of position attention module, channel attention module, and residual module until a final set of intermediate features is obtained to extract the prostate feature.
In yet another embodiment, wherein the program instructions, when executed by the processor, cause the apparatus to further perform the operation of stitching the first and second predictors to obtain the final predictor of predicted prostate penetration propensity.
In a second aspect, the present application provides a computer-readable storage medium having stored thereon computer-readable instructions for predicting a prostate puncture propensity, which when executed by one or more processors, perform operations performed by an apparatus as described in the various embodiments of the first aspect.
By the scheme for predicting the prostate puncture tendency provided above, the embodiment of the application predicts the first prostate puncture tendency by processing the basic information of the tested person and the text information of the prostate image through the natural language processing model so as to obtain a first prediction result. And processing the prostate image of the tested person through the image processing model to predict a second prostate puncture tendency so as to obtain a second predicted result, and further obtaining a final predicted result by combining the first predicted result and the second predicted result. Based on the method, the defect that the natural language processing model cannot fully utilize image information is overcome by adding the image processing model to process the prostate image, and the accuracy of predicting the prostate puncture tendency is greatly improved by combining the respective prediction results of the natural language processing model and the image processing model. Further, the image processing model of the embodiment of the application comprises an attention module and a residual error module, wherein the attention module can also comprise a position attention module and a channel attention module, and attention operation and residual error operation are sequentially carried out on the prostate image by the attention module and the residual error module to extract the prostate characteristics, so that the prostate information is fully extracted, and the accuracy of the second prediction result is improved.
Detailed Description
The technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the embodiments described in this specification are only some embodiments of the application provided for the purpose of facilitating a clear understanding of the solution and meeting legal requirements, and not all embodiments of the application may be implemented. All other embodiments, which can be made by those skilled in the art without making any inventive effort, are intended to be within the scope of the present application based on the embodiments disclosed herein.
Fig. 1 is an exemplary block diagram illustrating an apparatus 100 for predicting a prostate puncture propensity according to an embodiment of the present application. As shown in fig. 1, the apparatus 100 may include a processor 101 and a memory 102. The processor 101 may include, for example, a general-purpose processor ("CPU") or a special-purpose graphics processor ("GPU"), and the memory 102 stores program instructions executable on the processor. In some embodiments, the aforementioned Memory 102 may include, but is not limited to, a resistive Random Access Memory RRAM (Resistive Random Access Memory), a dynamic Random Access Memory DRAM (Dynamic Random Access Memory), a Static Random Access Memory SRAM (Static Random-Access Memory), an enhanced dynamic Random Access Memory EDRAM (ENHANCED DYNAMIC Random Access Memory).
Further, the memory 102 may store program instructions for predicting a prostate puncture tendency, which when executed by the processor, cause the apparatus 100 to obtain basic information of a subject and text information of a prostate image, and input the basic information and the text information of the prostate image into a natural language processing model for first prostate puncture tendency prediction to obtain a first prediction result regarding the predicted prostate puncture tendency. And then, acquiring the prostate image of the tested person, inputting the prostate image of the tested person into an image processing model for second prostate puncture tendency prediction so as to obtain a second prediction result related to the predicted prostate puncture tendency, and further obtaining a final prediction result for predicting the prostate puncture tendency based on the first prediction result and the second prediction result.
In some embodiments, the basic information of the subject may include, but is not limited to, age information and/or prostate specific antigen examination information (e.g., tPSA/fPSA), for example, disease history information of the subject, and the like. The prostate image of the subject may be, for example, mpMRI images, and the corresponding text information describes the lesion information existing in mpMRI images through a text language. In one implementation scenario, the text information of the prostate image may include, but is not limited to, diameter information of the prostate lesion, wherein the diameter information includes three information of length×width×height of the prostate lesion, such as 58mm×40mm×51mm, and/or abnormal signal information including signal information such as T2wI, DWI, TIWI and ADC. When there is a prostate lesion in the image, by describing lesion diameter information and/or abnormal signal information, for example, the diffuse abnormal signal of the prostate is represented by a T2wI low signal, a DWI high signal and an ADC low signal, and the visible abnormal patch signals of the pubic bone on the right side and the ischium on the left side are represented by a T2WI high signal, a TIWI low signal, a DWI high signal and an ADC signal which are reduced, so as to form text information of the prostate image.
Based on the obtained basic information of the subject and the text information of the prostate image, the program instructions, when executed by the processor, cause the apparatus 100 to further perform inputting the basic information and the text information of the prostate image into the natural language processing model for performing the first prostate puncture tendency prediction, so as to obtain a first prediction result related to the predicted prostate puncture tendency. In one implementation scenario, the natural language processing model may be, for example, a natural language processing model based on a Transformer architecture, such as GPT. In this scenario, the natural language processing model performs overall analysis on the text content according to the input basic information of the subject and the text information of the prostate image, that is, performs the first prostate puncture tendency prediction, and finally outputs the first prediction result regarding the predicted prostate puncture tendency. The first prediction result includes puncture or no puncture. In some embodiments, the natural language processing model may also output PI-RADS scores.
According to the foregoing, the natural language processing model is only sensitive to text content, so that the more abundant feature information contained in the prostate image cannot be obtained. Thus, in the embodiment of the application, the image processing model is added to process the prostate image, and the prostate image of the subject is input to the image processing model to perform the second prostate puncture tendency prediction, so as to obtain the second prediction result related to the predicted prostate puncture tendency. In one embodiment, the image processing model may be, for example, a convolutional neural network model, and the image processing model may include an attention module and a residual module, which when executed by the processor, cause the apparatus to further perform the operations of inputting a prostate image of the subject to the image processing model, and extracting a prostate feature using the attention module and the residual module in sequence on the prostate image in the image processing model to perform the second prostate puncture propensity prediction.
Specifically, in one implementation scenario, the second prostate puncture propensity prediction may be performed by performing an attention operation on the prostate image using an attention module in the image processing model to obtain an attention feature, and then performing a residual operation on the attention feature using a residual module in the image processing model to extract the prostate feature. In some embodiments, the aforementioned attention module may include a position attention module and a channel attention module, and the position attention feature and the channel attention feature may be obtained by sequentially performing a position attention operation and a channel attention operation on the prostate image using the position attention module and the channel attention module. In the embodiments of the present application, the foregoing may be understood in turn as obtaining a location attention feature by performing a location attention operation on a prostate image using a location attention module, and then obtaining a channel attention feature by performing a channel attention operation on the location attention feature using a channel attention module. Further, a residual operation is performed on the channel attention feature using a residual module to extract a prostate feature for a second prostate puncture propensity prediction to obtain a second prediction result.
In some embodiments, the residual module, the position attention module, and the channel attention module each comprise multiple sets. In one embodiment, the apparatus further performs a position attention operation, a channel attention operation, and a residual operation on the prostate image sequentially using each set of the position attention module, the channel attention module, and the residual module to obtain each set of intermediate features, and then sequentially performs a next position attention operation, a channel attention operation, and a residual operation on each set of intermediate features using the next set of the position attention module, the channel attention module, and the residual module until a final set of intermediate features is obtained to extract the prostate features. That is, after the position attention operation, the channel attention operation and the residual operation are performed by the set of residual modules, the position attention module and the channel attention module, a set of intermediate features is obtained, and the next set of position attention operation, the channel attention operation and the residual operation are performed on the set of intermediate features again until the last set of intermediate features is obtained, that is, the prostate feature. The operation of extracting the prostate feature will be described in detail later with reference to fig. 4.
Based on the obtained first and second prediction results, the apparatus further obtains a final prediction result for predicting the tendency of prostate puncture by stitching the first and second prediction results. In some embodiments, the final prediction is whether a prostate puncture is required, e.g., including 0 or 1,1 indicates that a prostate puncture is required, and 0 indicates that a prostate puncture is not required.
As can be seen from the above description, in the embodiment of the present application, the basic information of the tested person and the text information of the prostate image are processed by the natural language processing model to obtain the first prediction result, and the second prediction result based on the prostate image is combined with the image processing model to make up for the defect that the natural language processing model cannot fully utilize the image information, so as to improve the accuracy of predicting the prostate puncture tendency. Further, the embodiment of the application extracts the global information of the image through a plurality of groups of position attention operation and channel attention operation, and adopts residual error fitting global information, so that the obtained prostate characteristic information is richer and more complete, thereby improving the precision of the second prediction result.
Fig. 2 is an exemplary schematic diagram illustrating an ensemble for predicting a prostate puncture tendency according to an embodiment of the present application. As shown in fig. 2, basic information of a subject and text information 201 of a prostate image are input to a natural language processing model 202 to perform first prostate puncture tendency prediction, and a first prediction result 203 concerning the predicted prostate puncture tendency is obtained. In one implementation scenario, the aforementioned subject's underlying information may include age information, prostate specific antigen exam information (e.g., tPSA/fPSA), or disease history information, etc. The text information of the prostate image may include mpMRI diameter information and abnormality signal information on the image about the lesion, etc. In one implementation scenario, the aforementioned natural language processing model may be, for example, GPT. After the foregoing information is input to the natural language processing model for analysis, a first prediction result can be directly obtained, including, for example, whether puncture is required and PI-RADS scoring.
As further shown in the figure, the prostate image 204 of the subject is input to an image processing model 205 to perform a second prostate puncture tendency prediction, and a second prediction result 206 concerning the predicted prostate puncture tendency is obtained. In one implementation scenario, the aforementioned image processing model may be, for example, a convolutional neural network model, and the image processing model may include an attention module 207 and a residual module 208. In this scenario, the attention and residual operations are performed sequentially via the attention module 207 and the residual module 208 to extract the prostate features to achieve a second prostate puncture propensity prediction. In some embodiments, the aforementioned attention module 207 may include a location attention module 209 and a channel attention module 210. In an implementation scenario, the location attention module 209 performs a location attention operation on the prostate image to obtain a location attention feature, the channel attention module 210 performs a channel attention operation on the location attention feature to obtain a channel attention feature, and the residual module 208 performs a residual operation to extract the prostate feature. The second prediction result 206 may then be output by concatenating a full concatenated layer and the softmax function.
After the first prediction result 203 and the second prediction result 206 are obtained, the first prediction result 203 and the second prediction result 206 are spliced to form a vector. Similarly, the final prediction 211 can be obtained later by concatenating a full concatenated layer with a softmax function. Wherein the final prediction is whether a prostate puncture is required, e.g. comprising 0 or 1.
Fig. 3 is an exemplary diagram illustrating a natural language processing model processing basic information and text information of a prostate image according to an embodiment of the present application. As shown on the left side of fig. 3 is an input of a natural language processing model 202, which can input age information of a subject, prostate specific antigen examination information (e.g., tPSA/fPSA), disease history information, diameter information of a prostate lesion, abnormality signal information, and the like. By way of example, the subject has an age of 89, the text information of the prostate image is a prostate enlargement of about 58mm x 40mm x 51mm, a diffuse abnormality of the left prostate, a low T2wI signal, a high DWI signal, a low ADC signal, an enhanced scan showing a clear enhancement of early lesions, a maximum lesion diameter of about 3.1cm, incomplete left prostate capsule, multiple nodules visible in the transition zone, which may be a benign prostatic hyperplasia or a manifestation of prostate cancer, a visible abnormal patch of the right pubic bone, left ischial bones, a reduced DWI signal, an ADC signal, which may be indicative of bone metastasis, a reduced seminal capsule volume, a reduced T2WI signal, a general filling of the bladder wall, no abnormal in the cavity, a non-visible enlargement of the pelvic wall and groin, and no clear abnormal signal in the composed bones.
As shown on the right side of fig. 3, the output end of the natural language processing model is shown, and based on the foregoing information, the natural language processing model performs a first prediction of prostate puncture tendency, so as to obtain a first prediction result, that is, a Pi-RADS score of 5 points is output to the subject according to a Pi-RADS score criterion, and it is proposed that a highly suspected prostate cancer may exist, and bone metastasis may exist. Further, the age and imaging performance of the combined patient, the subject is considered to be at high risk of developing prostate cancer and bone metastases. While biopsies can provide a more accurate diagnosis, it is recommended that no prostate puncture be performed, given the age of the patient and the potential complications of the biopsy procedure.
Fig. 4 is an exemplary schematic diagram illustrating processing of a prostate image by an image processing model according to an embodiment of the present application. As shown in fig. 4, the image processing model may include a location attention module 209, a channel attention module 210, and a residual module 208. Further, the location attention module 209, the channel attention module 210, and the residual module 208 may include multiple groups, such as five groups shown in the figure by way of example. From the foregoing, it can be seen that after each group of residual modules, position attention modules and channel attention modules perform position attention operations, channel attention operations and residual operations, corresponding intermediate features of each group are obtained. Next, the next set of position attention operations, channel attention operations and residual operations are again performed on the respective intermediate features until the last set of intermediate features is obtained.
For example, the intermediate feature 301 is obtained after a first set of position attention operations, channel attention operations, and residual operations are performed on the prostate image 204 in sequence. Intermediate feature 302 is obtained after performing a position attention operation, a channel attention operation, and a residual operation on this intermediate feature 301 in this order using the second set. Similarly, intermediate features 303 and 304 and the last set of intermediate features, namely prostate features 305, may also be obtained. As previously described, the second prediction result 206 may be output by concatenating a full concatenated layer and a softmax function. Further, the second prediction result 206 and the first prediction result 203 are spliced, and a final prediction result 211 can be obtained.
Fig. 5 is an exemplary schematic diagram illustrating a location attention module and a channel attention module according to an embodiment of the present application. Fig. 5 (a) shows a position attention module, and fig. 5 (b) shows a channel attention module. It should be understood that, for both the location attention module and the channel attention module, three feature vectors are obtained by performing an attention operation on the above-mentioned prostate image or intermediate feature, wherein the associated weights are determined according to two of the feature vectors, and the weighted feature vectors are obtained by multiplying the other feature vector. Further, the attention feature can be obtained by adding the feature vector with increased weight to the original prostate image or the intermediate feature.
For example, for the location attention module, it may first convolve 501 and 502 the above-mentioned prostate image 204 or intermediate feature, and then perform a location attention operation to obtain three location feature vectors, such as location feature vector 503, location feature vector 504 and location feature vector 505. Then, a position-related weight 506 is determined according to the matrix multiplication of the position feature vector 504 and the position feature vector 505, and then the matrix multiplication is performed with the position feature vector 501 to obtain a feature vector 507 with increased position weight, and the feature vector is restored to a three-dimensional feature 508. Finally, the position attention feature 509 is obtained by adding the original prostate image or intermediate features.
For the channel attention module, it may be laid out in two dimensions for the location attention feature 509 to perform the channel attention operation to obtain three channel feature vectors, such as channel feature vector 510, channel feature vector 511, and channel feature vector 512. Then, the channel correlation weight 513 is determined according to the matrix multiplication of the channel feature vector 510 and the channel feature vector 511, and the matrix multiplication is performed on the channel correlation weight 513 and the channel feature vector 512 to obtain a feature vector 514 with increased channel weight, and then the three-dimensional feature 515 is restored. Finally, the channel attention feature 516 is obtained by adding the original position attention feature 509. Intermediate or final prostate features may be obtained by subsequently performing a residual operation via a residual module.
Based on the foregoing description, the embodiment of the application compensates the defect that the natural language processing model cannot process the image by adding the image processing model, predicts the second prostate puncture tendency to obtain a second prediction result, and combines the first prediction result of the natural language processing model, thereby improving the accuracy of predicting the prostate puncture tendency. In addition, the embodiment of the application further extracts global information of the image by arranging the position attention module, the channel attention module and the residual error module, so that the accuracy of the second prediction result is improved.
Fig. 6 is an exemplary block diagram illustrating an apparatus 600 for predicting a prostate puncture propensity according to an embodiment of the present application. It is to be appreciated that the device implementing aspects of the present application may be a single device (e.g., a computing device) or a multi-function device including various peripheral devices.
As shown in fig. 6, the apparatus of the present application may include a central processing unit or central processing unit ("CPU") 611, which may be a general purpose CPU, a special purpose CPU, or other information processing and program running execution unit. Further, the device 600 may also include a mass memory 612 and a read only memory ("ROM") 613, wherein the mass memory 612 may be configured to store various types of data, including various basic information with the subject and text information of the prostate image, various features, algorithm data, intermediate results, and various programs required to operate the device 600. ROM 613 may be configured to store data and instructions required to power up self-test of device 600, initialization of functional modules in the system, drivers for basic input/output of the system, and boot the operating system.
Optionally, the device 600 may also include other hardware platforms or components, such as a tensor processing unit ("TPU") 614, a graphics processing unit ("GPU") 615, a field programmable gate array ("FPGA") 616, and a machine learning unit ("MLU") 617, as shown. It will be appreciated that while various hardware platforms or components are shown in device 600, this is by way of example only and not limitation, and that one of skill in the art may add or remove corresponding hardware as desired. For example, device 600 may include only a CPU, associated memory device, and interface device to implement the method of the present application for predicting a prostate puncture propensity.
In some embodiments, to facilitate the transfer and interaction of data with external networks, device 600 of the present application further comprises a communication interface 618, whereby communication interface 618 may be coupled to local area network/wireless local area network ("LAN/WLAN") 605, and thereby local server 606 or Internet ("Internet") 607. Alternatively or additionally, device 600 of the present application may also be directly connected to the Internet or a cellular network via communication interface 618 based on wireless communication technology, such as 3 rd generation ("3G"), 4 th generation ("4G"), or 5th generation ("5G") wireless communication technology. In some application scenarios, the device 600 of the present application may also access the server 608 and database 609 of the external network as needed to obtain various known algorithms, data and modules, and may remotely store various data, such as various types of data or instructions for presenting basic information of a subject and text information of a prostate image, various features, etc.
Peripheral devices of device 600 may include a display 602, an input 603, and a data transfer interface 604. In one embodiment, display device 602 may include, for example, one or more speakers and/or one or more visual displays configured for voice prompts and/or visual image displays of the present application for predicting a prostate puncture propensity. The input device 603 may include other input buttons or controls, such as a keyboard, mouse, microphone, gesture-capture camera, etc., configured to receive input of audio data and/or user instructions. The data transfer interface 604 may include, for example, a serial interface, a parallel interface, or a universal serial bus interface ("USB"), a small computer system interface ("SCSI"), serial ATA, fireWire ("FireWire"), PCI Express, and high definition multimedia interface ("HDMI"), etc., configured for data transfer and interaction with other devices or systems. In accordance with aspects of the present application, the data transmission interface 604 may receive images of the prostate acquired from the MRI acquisition device and transmit data or results, including images of the prostate or various other types, to the device 600.
The above-described CPU 611, mass memory 612, ROM 613, TPU 614, GPU 615, FPGA 616, MLU 617, and communication interface 618 of the device 600 of the present application may be connected to each other via a bus 619, and data interaction with peripheral devices is achieved via the bus. In one embodiment, the CPU 611 may control other hardware components in the device 600 and its peripherals via the bus 619.
An apparatus for predicting a prostate puncture propensity that may be used to implement the present application is described above in connection with fig. 6. It is to be understood that the device structure or architecture herein is merely exemplary and that the implementation and implementation entities of the present application are not limited thereto, but that changes may be made without departing from the spirit of the present application.
Those skilled in the art will also appreciate from the foregoing description, taken in conjunction with the accompanying drawings, that embodiments of the present application may also be implemented in software programs. The present application thus also provides a computer readable storage medium having stored thereon computer readable instructions for predicting a prostate penetration propensity which, when executed by one or more processors, may be used to implement the method for predicting a prostate penetration propensity of the present application described in connection with figure 1.
It should be noted that although the operations of the method of the present application are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in that particular order or that all of the illustrated operations be performed in order to achieve desirable results. Rather, the steps depicted in the flowcharts may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
It should be understood that when the terms "first," "second," "third," and "fourth," etc. are used in the claims, the specification and the drawings of the present application, they are used merely to distinguish between different objects, and not to describe a particular order. The terms "comprises" and "comprising" when used in the specification and claims of the present application are taken to specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification and claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the term "and/or" as used in the present specification and claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Although the embodiments of the present application are described above, the descriptions are merely examples for facilitating understanding of the present application, and are not intended to limit the scope and application of the present application. Any person skilled in the art can make any modification and variation in form and detail without departing from the spirit and scope of the present disclosure, but the scope of the present disclosure is defined by the appended claims.