KR102216689B1 - Method and system for visualizing classification result of deep neural network for prediction of disease prognosis through time series medical data - Google Patents

Abstract

Disclosed is a method and system for visualizing classification results of deep neural networks for predicting disease prognosis using time series medical data. A computer-implemented method, comprising the steps of: generating sequence data by representing medical data of a patient in the form of a sequence that is time-series data; Predicting a disease of the patient from the sequence data through a disease prediction model using a recurrent neural network (RNN) and an attention network; And visualizing an attention weight of the sequence data for disease prediction of the patient using a gradient-weighted class activation map (Grad-CAM).

Description

A method and system for visualizing classification results of deep neural networks for predicting disease prognosis through time series medical data {METHOD AND SYSTEM FOR VISUALIZING CLASSIFICATION RESULT OF DEEP NEURAL NETWORK FOR PREDICTION OF DISEASE PROGNOSIS THROUGH TIME SERIES MEDICAL DATA}

The description below relates to a technology that predicts disease prognosis and provides prediction results.

Accurate prediction of high-risk diseases such as cardiovascular and cerebrovascular diseases is a very important issue to prevent death in the medical field. Most existing methods use primarily pathological, radiological, and clinical information to provide predictive results.

For example, Korean Patent Application Publication No. 10-2014-0098561 (published on August 08, 2014) predicts the risk of a user's disease on the basis of a single nucleotide polymorphism (SNP) combination related to the disease to be analyzed. The technology is disclosed.

Although pathological and radiological measurements provide competitive precision, obtaining and analyzing data for prediction requires a lot of time and money.

An alternative is to use an electronic health record (EHR), a medical record that includes the patient's diagnosis and medication records. EHR data contains time series information and is measured by medical professionals, which is useful for predicting disease prognosis.

Recent advances in deep learning have achieved remarkable success in various fields such as computer vision, natural language processing, and audio signal processing, while many studies in the medical field are predicting various diseases from existing medical data and EHR data. We are introducing a deep learning-based approach for this.

However, even if the deep learning-based method shows promising results, it is not easy to investigate which factors have a significant influence on the disease state due to their black box characteristics.

We provide an EHR-based disease prediction model, a new disease prediction method based on RNN (Recurrent Neural Network) and attention mechanism.

It provides a technology that can visualize the relative influence of EHR data at a specific point in time to be classified as a specific type of disease in the form of a linear agreement on the time axis.

A computer-implemented method, comprising the steps of: generating sequence data by representing medical data of a patient in the form of a sequence that is time-series data; Predicting a disease of the patient from the sequence data through a disease prediction model using a recurrent neural network (RNN) and an attention network; And visualizing an attention weight of the sequence data for disease prediction of the patient using a gradient-weighted class activation map (Grad-CAM).

According to one aspect, the generating step includes generating sequence data for each information type of information included in diagnosis records and information included in prescription records as the medical data, and then predicting the disease. It may include embedding the model to be input.

According to another aspect, the generating may include the step of integrating the medical data into an embedding vector as an input of bidirectional gated recurrent units (GRU).

According to another aspect, the generating may include encoding the medical data into an embedding vector including a time interval between consecutive medical events.

According to another aspect, the disease prediction model may use an attention mechanism to learn the importance of time information included in the medical data for disease prediction.

According to another aspect, the disease prediction model may calculate an attention weight for the sequence data by a time-distributed softmax layer of the attention network and then reflect it in the output of the RNN. have.

According to another aspect, the predicting may include merging attention weights of the sequence data and performing class labeling according to disease prediction through a fully connected layer.

According to another aspect, the visualizing may include calculating a gradient of a softmax output of a class with respect to the attention weight; Obtaining a time average gradient by performing one-dimensional GAP (Global Average Pooling) on the time dimension; And calculating a Grad-CAM of the attention weight by using the time average gradient.

According to another aspect, in the visualizing step, the Grad-CAM of the attention weight for the class according to the disease prediction may be visualized through a heat map.

According to another aspect, the step of visualizing may provide a heat map of the Grad-CAM indicating a relative influence on a class according to disease prediction through a time domain for the visit time of the patient.

It provides a computer-readable recording medium, characterized in that the program for executing the method in a computer is recorded.

A computer system comprising at least one processor implemented to execute a computer-readable instruction, wherein the at least one processor generates sequence data by expressing patient medical data in a sequence form that is time-series data A data processing unit; And an attention weight of the sequence data for predicting the patient's disease using a gradient-weighted class activation map (GD-CAM) and predicting the patient's disease from the sequence data through a disease prediction model using an RNN and an attention network. It provides a computer system including a visualization unit for visualizing.

According to embodiments of the present invention, it is possible to provide a new disease prediction model based on RNN capable of predicting high-risk diseases as a prediction model using EHR data of a patient and an attention network.

According to embodiments of the present invention, more detailed information may be provided by visualizing the relative influence of EHR data at a specific time point on classification as a specific type of disease in the form of a linear sum on a time axis.

1 is a block diagram illustrating an example of an internal configuration of a computer system according to an embodiment of the present invention.
2 is a diagram illustrating an example of components that may be included in a processor of a computer system according to an embodiment of the present invention.
3 is a flowchart illustrating an example of a disease prognosis prediction method that can be performed by a computer system according to an embodiment of the present invention.
4 shows the structure of an EHR-based attention network (EHAN) according to an embodiment of the present invention.
5 illustrates an example of an attention structure used in an EHR-based attention network (EHAN) according to an embodiment of the present invention.
6 is a table showing examples of disease codes classified as high-risk diseases according to an embodiment of the present invention.
7 illustrates an example of an interface screen visualizing Grad-CAM for a diagnostic feature in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to technology for predicting disease prognosis and providing predictive results.

Examples including those specifically disclosed in this specification can visualize the classification result of a deep neural network for predicting the prognosis of a specific disease through learning of time series medical data, through which efficiency, accuracy, speed, and cost reduction It achieves considerable advantages in terms of the back.

1 is a block diagram illustrating an example of an internal configuration of a computer system according to an embodiment of the present invention. For example, the disease prognosis prediction system according to embodiments of the present invention may be implemented through the computer system 100 of FIG. 1. As shown in Fig. 1, the computer system 100 is a component for executing the disease prognosis prediction method, a processor 110, a memory 120, a permanent storage device 130, a bus 140, an input/output interface ( 150) and a network interface 160.

Processor 110 may include or be part of any device capable of processing a sequence of instructions as a component for predicting a high risk disease from a patient medical record. The processor 110 may include, for example, a processor and/or a digital processor in a computer processor, mobile device, or other electronic device. The processor 110 may be included, for example, a server computing device, a server computer, a series of server computers, a server farm, a cloud computer, a content platform, and the like. The processor 110 may be connected to the memory 120 through the bus 140.

Memory 120 may include volatile memory, permanent, virtual or other memory for storing information used by or output by computer system 100. The memory 120 may include, for example, random access memory (RAM) and/or dynamic RAM (DRAM). Memory 120 may be used to store any information, such as state information of computer system 100. The memory 120 may also be used to store instructions of the computer system 100, including instructions for predicting disease prognosis, for example. Computer system 100 may include one or more processors 110 as needed or where appropriate.

Bus 140 may include a communication infrastructure that enables interaction between various components of computer system 100. Bus 140 may carry data between components of computer system 100, for example, between processor 110 and memory 120, for example. Bus 140 may include wireless and/or wired communication media between components of computer system 100 and may include parallel, serial or other topological arrangements.

Persistent storage device 130 is a component such as a memory or other persistent storage device as used by computer system 100 to store data for a predetermined extended period of time (e.g., compared to memory 120). Can include. Persistent storage device 130 may include nonvolatile main memory as used by processor 110 in computer system 100. The persistent storage device 130 may include, for example, a flash memory, a hard disk, an optical disk, or other computer-readable medium.

The input/output interface 150 may include interfaces to a keyboard, mouse, voice command input, display, or other input or output device. Configuration commands and/or inputs for predicting disease prognosis may be received through the input/output interface 150.

The network interface 160 may include one or more interfaces to networks such as a local area network or the Internet. The network interface 160 may include interfaces for wired or wireless connections. Configuration commands and/or input for predicting disease prognosis may be received through the network interface 160.

Further, in other embodiments, the computer system 100 may include more components than the components of FIG. 1. However, there is no need to clearly show most of the prior art components. For example, the computer system 100 may be implemented to include at least some of the input/output devices connected to the input/output interface 150 described above, or a transceiver, a global positioning system (GPS) module, a camera, various sensors, Other components such as a database may be further included.

The present invention proposes an EHR History-based prediction using Attention Network (EHAN) as a new disease prediction model based on an RNN and an attention mechanism.

In the present specification, cardiovascular disease and cerebrovascular disease are described as representative examples of high-risk diseases that cause death, but are not limited thereto. In addition, although diagnosis records determined by a doctor and prescription records for treatment are described as representative examples of patient medical records, these are not limited thereto.

2 is a diagram showing an example of components that can be included in a processor of a computer system according to an embodiment of the present invention, and FIG. 3 is a disease prognosis that can be performed by a computer system according to an embodiment of the present invention. It is a flowchart showing an example of a prediction method.

As shown in FIG. 2, the processor 110 may include a data processing unit 210 and a classification and visualization unit 220. Components of the processor 110 may be expressions of different functions performed by the processor 110 according to a control command provided by at least one program code. For example, the data processing unit 210 may be used as a functional expression that operates to control the computer system 100 so that the processor 110 pre-processes medical data. The processor 110 and components of the processor 110 may perform steps S310 to S330 included in the disease prognosis prediction method of FIG. 3. For example, the processor 110 and the components of the processor 110 may be implemented to execute an instruction according to the code of the operating system included in the memory 120 and the at least one program code described above. Here, at least one program code may correspond to a code of a program implemented to process the disease prognosis prediction method.

The disease prognosis prediction method may not occur in the order shown, and some of the steps may be omitted or an additional process may be further included.

In step S310, the processor 110 may load a program code stored in a program file for a method for predicting disease prognosis into the memory 120. For example, a program file for a method for predicting disease prognosis may be stored in the permanent storage device 130 described with reference to FIG. 1, and the processor 110 may be configured from a program file stored in the permanent storage device 130 through a bus. Computer system 110 may be controlled so that program code is loaded into memory 120. At this time, each of the data processing unit 210 and the classification and visualization unit 220 included in the processor 110 and the processor 110 executes a command of a corresponding part of the program code loaded in the memory 120 to perform a subsequent step. It may be different functional expressions of the processor 110 for executing the elements S320 to S330. In order to execute steps S320 to S330, the processor 110 and components of the processor 110 may directly process an operation according to a control command or control the computer system 100.

In step S320, the data processing unit 210 may pre-process medical data including time series information as a medical record of a patient in a sequence format. In other words, the data processing unit 210 may generate sequence data for a corresponding patient by expressing the patient's medical record as a feature in the form of a sequence that is time-series data.

As electronic medical data (EHR data) collected in actual hospitals to predict the prognosis of high-risk diseases (cardiovascular disease and cerebrovascular disease), patients without vascular disease, patients with cardiovascular disease, and cerebrovascular disease Medical records of patients who are present are available.

As electronic medical data, a code for classifying a patient's diseases and symptoms, for example, a disease classification code expressed as ICD-10, the 10th revision of the international statistical classification, can be used. A total of 6,667 ICD-10 codes are used as medical data, and it is possible to use all of these codes or to selectively use some codes that appear more than a certain number of times in patients. In addition to the disease classification code, the patient's medical data may include various additional information such as time information (diagnosis date, prescription date, visit time interval, etc.), drug code, and visit type.

For example, patient medical records may include diagnostic records and prescription records, and in addition to biometric information such as blood pressure and cholesterol, as well as date and patient status information, and patient clinical information such as age or gender may also be used. . Hereinafter, a specific embodiment is described using diagnostic records and prescription records as examples of patient medical records, but is not limited thereto, and all patient information included in the electronic medical record or derived from the electronic medical record can be used. It is natural that there is.

The diagnosis record may include a patient identification number (pid), a diagnosis date, a disease classification code (code), and a visit type (type), and the prescription record may include a patient identification number (pid), a prescription date, and May include drug code and visit type. The type of visit is a type of visit by a patient to a hospital, and can be classified into an outpatient, an inpatient, and an emergency patient.

The data processing unit 210 may generate sequence data for each information type of information included in the patient's medical record, and then embed information to be input into a disease prediction model. For feature representation, various types of medical data can be connected to inputs of bidirectional gated recurrent units (GRUs) and integrated into event embedding vectors. In particular, time intervals between consecutive medical events can be explicitly encoded in an embedding vector to solve the limitation of RNNs that can only process regular time interval data. In other words, an additional feature can be expressed that represents the time interval between the diagnosis date or the prescription date, for example the time interval between the first and second diagnosis is 30 weeks, and the second and third diagnosis are If the time interval is 7 weeks, sequence data can be generated from the recording by recording as (30, 7).

In step S330, the classification and visualization unit 210 predicts high-risk diseases from the sequence-type medical data through the EHR-based attention network (EHAN) model and visualizes the relative influence of the medical data on the predicted values of high-risk diseases. I can. At this time, the classification and visualization unit 210 visualizes how much the medical data at a specific point in time influences the prediction of a specific disease through a gradient-weighted class activation map (GRAD-CAM), thereby providing a more clear and detailed interpretation result. can do.

The EHR-based attention network (EHAN) model according to the present invention includes (1) a two-way GRU for automatic sequential modeling, (2) an attention mechanism for improving long-term dependency modeling, and (3) an RNN for visualizing class-specific attention-weights. Based gradient-weighted class activation mapping (Grad-CAM) is included.

The EHR-based attention network (EHAN) model according to the present invention may include a module for classification as well as a module for feature expression. At this time, the EHR-based Attention Network (EHAN) model uses an attention mechanism to improve predictive ability by focusing on important medical events that have a great influence on distinguishing disease states. In addition, the EHR-based Attention Network (EHAN) model can incorporate Grad-CAM to explicitly provide the importance score of medical events related to disease classification.

RNN is a neural network frequently used for modeling variable length sequential data. The RNN contains a circular link of hidden units that process one input at a time and implicitly store past information. Recent RNNs use more sophisticated hidden units that act as some kind of memory cell such as long short term memory (LSTM) and GRU. In addition, the performance of the RNN can be further improved by applying an attention mechanism designed to more effectively learn long-term dependence by weighting a specific point in time. In this case, the attention may be implemented in various forms according to the architecture of the neural network.

Meanwhile, CAM (Class Activation Mapping) is a deep learning technology designed for effective visualization of convolutional neural networks (CNN). Using CAM, you can visualize predicted class scores by highlighting areas of identification at a given high-level feature. CAM may cause degradation of classification performance due to network limitations in that the classification structure uses GAP (Global Average Pooling) for feature maps of a specific layer. In contrast, Grad-CAM is an improved CAM, and it is possible to remove the constraints due to the classification architecture by applying GAP to the gradient of the layer of interest.

The EHR-based Attention Network (EHAN) model according to the present invention is based on an RNN and an attention mechanism, and an end-to-end for highlighting important events using Grad-CAM while predicting high-risk diseases from EHR sequence data. It is implemented as an end-to-end) model.

4 shows the structure of an EHR-based attention network (EHAN) according to an embodiment of the present invention.

The disease prognosis prediction method by the EHR-based attention network (EHAN) may be composed of an input representation (pre-processing) step, a feature extraction step, a classification and a visualization step.

In the first input presentation step, the given diagnostic record 401 and prescription record 402 are converted into word vector representations.

In the feature extraction step, the word vector may be transferred to the bidirectional GRU by an attention mechanism and merged into an aggregate feature.

The extracted feature is classified as a softmax binary vector in the classification step, and time-related information may be obtained through the Grad-CAM-based visualization step.

4, the EHR-based attention network (EHAN) uses information of four types (pid, date, code, type) for each of the patient's diagnosis record 401 and prescription record 402 as input, A two-dimensional softmax vector indicating whether the patient is a high-risk group is output as a result.

In the data processing step S320, the diagnostic record 401 and the prescription record 402 are encoded as binary word vectors for input representation. The encoded binary word vector is a classification and visualization unit 210, a feature extraction module (RNN and attention module) 410, a classification module (multiple fully connected layer) 420, and a visualization module (Grad-CAM). ) 430 is processed by the Neural Network Architecture.

Input expression

For convenience of explanation, the input expression for a single patient is described.

와

로 표시할 수 있다. 여기에서,

는 의료 기록 x의 어휘 목록(vocabulary)을 나타낸다. 예를 들어, 어휘 목록 사이즈가 |c|인 특정 의료 기록 x를 사용하면 i번째 방문 시 의료 코드를 가진 이진 벡터

로 나타낼 수 있으며, 여기서 j번째 좌표에서 각각의 값은

가 해당되는 방문에서 문서화됨을 나타낸다.Use disease progression modeling (DPM) to represent the sequence of patient medical records. Assuming that the number of visits is k, the i-th (i=1,2,…,k) time stamp and medical record are respectively

Wow

Can be marked with From here,

Represents the vocabulary of medical record x. For example, using a specific medical record x with a vocabulary list size of |c|, a binary vector with medical code at visit i

Can be expressed as, where each value at the j-th coordinate is

Is documented at the applicable visit.

Below we consider four types of medical records: disease code d _c , visit type d _t (of diagnostic record), drug code m _c , and visit type m _t (of medication record). The type of visit is a type of visit by a patient to a hospital, and can be classified into an outpatient, an inpatient, and an emergency patient.

The disease code d _c may be expressed as a vector (1 or 0) corresponding to all codes that can be diagnosed by date. For example, if there are 100 disease codes, it may be expressed as a vector of size 100.

EHR base Attention network( EHAN )

For visualization using Grad-CAM, we need an appropriate feature representation that can be reverted to the original input domain. The EHR-based attention network (EHAN) includes a feature extraction module 410, a classification module 420, and a visualization module 430.

-Embedding (feature extraction module 410)

In this embodiment, a soft-embedding technique defined as in Equation 1 is used.

[Equation 1]

과

은 주어진 입력

의 임베딩 벡터와 학습할 임베딩 행렬을 의미한다. m은 임베딩 차원의 크기이다.here,

and

Is the given input

It means the embedding vector of and the embedding matrix to be learned. m is the size of the embedding dimension.

에 대해 해당 임베딩 벡터

은 수학식 2와 같이 정의된다.Each type of medical record covered by four types of medical records (d _c , d _t , m _c , m _t )

For the corresponding embedding vector

Is defined as in Equation 2.

[Equation 2]

의 각 임베딩 차원의 숫자는 각각

,

,

,

이다.

The number of each embedding dimension of is

,

,

,

to be.

로 계산할 수 있다.The number of each embedding dimension is determined as the root of the square of the corresponding vocabulary size. For example, when the vocabulary size is 100, the number of embeddings is

Can be calculated as

Since the disease code d _c and the diagnosis type d _t are derived from the same timestamp, two embedding vectors can be connected as shown in Equation 3.

[Equation 3]

은 진단 기록의 임베딩 벡터이고,

는 진단 기록의 i번째 방문과 i-1번째 방문 사이의 시간 간격을 의미한다.here,

Is the embedding vector of the diagnostic record,

Is the time interval between visit i and visit i-1 of the diagnostic record.

는 수학식 4와 같이 정의될 수 있다.

May be defined as in Equation 4.

[Equation 4]

는 i번째 방문일의 타임스탬프를 나타낸다.here,

Represents the timestamp of the i-th visit date.

을 얻을 수 있다.In the same way, the embedding vector of the prescription record

Can be obtained.

-Feature extraction (feature extraction module 410)

Two bidirectional GRU layers and an attention mechanism are used for each sequential modeling of the embedding vector of the diagnostic record (diagnostic feature) and the embedding vector of the prescription record (prescription feature).

와

를 방문 시점(time point)

로 사용하고 수학식 5와 수학식 6의 가중치 행렬 H_d 및 H_m을 반환한다.Bidirectional GRU layers G _d and G _m are embedding vectors

Wow

The time point

And return the weight matrices H _d and H _m of Equations 5 and 6.

[Equation 5]

[Equation 6]

Here, n _d and n _m denote the number of GRU units of each layer, and k denote the number of visits.

The attention mechanism can be applied to the RNN outputs H _d and H _m to learn the importance of the time position of a given input x.

5 illustrates an example of an attention structure used in an EHR-based attention network (EHAN) according to an embodiment of the present invention.

In order to learn attention weights for each output of the bidirectional GRU layer, two softmax layers consisting of a fully connected layer and softmax activation, respectively, are used.

In this embodiment, a kind of self-attention mechanism is used. The attention weight vector is calculated by a time-distributed softmax layer, and the RNN output is multiplied by the attention weight.

와

은 수학식 7과 수학식 8의 어텐션 가중치 행렬

와

을 구성하는데 사용된다.Learnable weight matrix

Wow

Is the attention weight matrix of Equations 7 and 8

Wow

Is used to construct.

[Equation 7]

[Equation 8]

와

는 수학식 9, 수학식 10와 같이 정의될 수 있다.Attention weight vector

Wow

May be defined as in Equations 9 and 10.

[Equation 9]

[Equation 10]

는 소프트맥스 활성화 함수를 나타내고,

은 행렬을 취하여 대각선 벡터를 생성하는 함수를 나타내고,

와

의 i번째 요소는 각각의 i번째 방문에 대한 질병 코드 d_c와 의약품 코드m_c에 대한 어텐션 가중치를 나타낸다.here,

Denotes the softmax activation function,

Represents a function that takes a matrix and produces a diagonal vector,

Wow

The i-th element of is the attention weight for disease code d _c and drug code m _c for each i-th visit.

C _d and C _{m, which} are attention-weighted representations of the weight matrices H _d and H _m , can be obtained through Equations 11 and 12.

[Equation 11]

[Equation 12]

와

은 n_d 차원 벡터와 n_m 차원 벡터를 나타내고,

은 요소별 곱셈(element-wise multiplication)을 나타내고,

은 외적 연산자(outer product operator)를 나타낸다.here,

Wow

Represents an n _d dimensional vector and an n _m dimensional vector,

Represents element-wise multiplication,

Denotes the outer product operator.

-Classification (classification module 420)

The EHR-based Attention Network (EHAN) can predict disease labels using two fully connected layers by merging the diagnostic and prescription features extracted in the previous step. For example, for class labeling, a disease code group determined by a medical professional as shown in the table of FIG. 6 may be used.

와

로 평면화(flattened)될 수 있으며, 이 두 벡터는 통합 표현(integrated representation)

으로 연관시킬 수 있다. 그리고, 중간 출력 O를 수학식 13을 통해 계산할 수 있다.Attention weight matrices C _d and C _m are vectors

Wow

Can be flattened, and these two vectors are integrated representation

Can be associated with Then, the intermediate output O can be calculated through Equation 13.

[Equation 13]

은 로지스틱 시그모이드 함수(logistic sigmoid function)를 나타내고,

은 학습 가능한 가중치 행렬을 나타내고,

는 바이어스 벡터를 나타내고,

는 중간 출력 O의 차원을 나타낸다.here,

Denotes a logistic sigmoid function,

Denotes the learnable weight matrix,

Denotes the bias vector,

Represents the dimension of the intermediate output O.

를 출력할 수 있다.The second fully connected layer outputs the final softmax as shown in Equation 14

Can be printed.

[Equation 14]

는 학습 가능한 가중치 행렬을 나타내고,

는 바이어스 벡터를 나타낸다. 손실 함수에는 이진 크로스-엔트로피가 사용될 수 있다.here,

Denotes a learnable weight matrix,

Denotes the bias vector. Binary cross-entropy can be used for the loss function.

-Grad-CAM (gradient-weighted class activation map) (visualization module 430)

를 Grad-CAM의 목표 피처로 간주한다.In the present invention, it is possible to visualize which point in the medical record has the greatest influence on the classification result. Thus, indicating the highest-level representation containing time information

Is regarded as the target feature of Grad-CAM.

를 수학식 15를 통해 계산할 수 있다.First, the gradient of the softmax output of class c for the attention weight matrix C _d , which is a diagnostic feature.

Can be calculated through Equation 15.

[Equation 15]

는 클래스 c의 최종 소프트맥스 출력을 나타낸다.here,

Denotes the final softmax output of class c.

를 얻을 수 있다.Time average gradient through Equation 16 by performing a one-dimensional GAP on the time dimension

Can be obtained.

[Equation 16]

는 수학식 17을 통해 계산할 수 있다.Finally, the Grad-CAM of diagnostic feature C _d in class c

Can be calculated through Equation 17.

[Equation 17]

은 행렬을 취하여 대각선 벡터를 생성하는 함수를 나타내고,

은 외적 연산자를 나타내고,

는 k차원 벡터를 나타낸다.here,

Represents a function that takes a matrix and produces a diagonal vector,

Represents the cross product operator,

Represents a k-dimensional vector.

The Grad-CAM for the prescription feature C _m can be derived through the same procedure as the diagnostic feature C _d .

-Visualization using Grad-CAM (visualization module 430)

The end-to-end approach has learning efficiency as well as the ability to discover new features. In general, it is difficult to intuitively interpret a learned feature because the feature is represented by the weight of the deep network model. In order to solve this problem, in the present invention, information related to visit time important for disease classification can be obtained using Grad-CAM.

For example, referring to FIG. 7, the visualization module 430 may visualize the Grad-CAM of the diagnostic feature C _d for the positive class (abnormal) through a heat map.

In FIG. 7A, the x-axis shows the visit time for diagnosis, and the y-axis shows an example of positive classification of 10 randomly selected people. In the graph of FIG. 7 (A), one horizontal line represents the Grad-CAM of the diagnostic feature C _d for one patient. The heat map constituting the graph of FIG. 7 (A) shows which viewpoint of each feature vector has a relative influence on the classification of the corresponding positive class.

In the graph (B) of FIG. 7, the horizontal scale is a time domain for the visit time, and the darker the heat map of Grad-CAM may mean that the diagnosis at the time of the visit has a great influence on the result.

(B) of Figure 7 shows the disease code annotation for Grad-CAM of a patient whose ID number is 20065752. For example, it is possible to provide an onset prediction result that the patient has visited 11 times and the probability of disease occurrence within 6 months after the last visit is predicted as 0.7989. The highest intensity appeared at Visit 5, indicating that at Visit 5, the disease codes'E785','R42', and'E780' had a relatively more influence on the positive classification.

As described above, according to embodiments of the present invention, a new disease prediction model based on an RNN capable of predicting a high-risk disease may be provided as a prediction model using EHR data of a patient and an attention network. In particular, it is possible to provide more detailed and useful information by visualizing the relative influence of EHR data at a specific point in time to be classified as a specific type of disease in the form of a linear agreement on the time axis.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable gate array (PLU). It may be implemented using one or more general purpose computers or special purpose computers, such as a logic unit), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be embodyed in any type of machine, component, physical device, computer storage medium or device to be interpreted by the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And a ROM, RAM, flash memory, and the like, and may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (12)

In the computer-implemented method,
Generating sequence data by expressing the patient's medical data in the form of a sequence that is time-series data;
Predicting a disease of the patient from the sequence data through a disease prediction model using a recurrent neural network (RNN) and an attention network; And
Visualizing the attention weight of the sequence data for disease prediction of the patient using a gradient-weighted class activation map (CAM)
Including,
The generating step,
Generating sequence data for each information type of information included in diagnosis records and information included in prescription records as the medical data, and then embedding them to be input into the disease prediction model; And
Encoding the medical data into an embedding vector including the time interval by expressing an additional feature representing a time interval between successive medical events
Including,
The visualizing step,
Visualizing the time point of medical data that affects the patient's disease prediction result through a time domain for time information included in the medical data
The method characterized by. delete The method of claim 1,
The generating step,
Integrating the medical data into an embedding vector by taking the medical data as an input of bidirectional gated recurrent units (GRU)
How to include. delete The method of claim 1,
The disease prediction model,
Using an attention mechanism to learn the importance of the temporal information contained in the medical data for disease prediction.
The method characterized by. The method of claim 1,
The disease prediction model,
Calculating an attention weight for the sequence data by a time-distributed softmax layer of the attention network and reflecting it in the output of the RNN
The method characterized by. The method of claim 1,
The predicting step,
Merging the attention weights of the sequence data and performing class labeling according to disease prediction through a fully connected layer
How to include. The method of claim 1,
The visualizing step,
Calculating a gradient of a softmax output of a class with respect to the attention weight;
Obtaining a time average gradient by performing one-dimensional GAP (Global Average Pooling) on the time dimension; And
Computing a Grad-CAM of the attention weight using the time average gradient
How to include. The method of claim 1,
The visualizing step,
Visualizing the Grad-CAM of the attention weight for the class according to disease prediction through a heat map
The method characterized by. The method of claim 1,
The visualizing step,
Providing a heat map of the Grad-CAM showing the relative influence on the class according to disease prediction through the time domain for the visit time of the patient
The method characterized by. A computer-readable recording medium, characterized in that a program for executing the method of any one of claims 1, 3, and 5 to 10 is recorded on a computer. In a computer system,
At least one processor implemented to execute computer-readable instructions
Including,
The at least one processor,
A data processing unit for generating sequence data by expressing the patient's medical data in the form of a sequence that is time-series data; And
The patient's disease is predicted from the sequence data through a disease prediction model using an RNN and an attention network, and the attention weight of the sequence data for predicting the disease of the patient is calculated using a gradient-weighted class activation map (CAM). Visualization unit to visualize
Including,
The data processing unit,
As the medical data, the information included in the diagnosis record and the information included in the prescription record are generated for each information type, and then embedded to be input into the disease prediction model,
Encode the medical data into an embedding vector including the time interval by expressing an additional feature representing a time interval between successive medical events,
The visualization unit,
Visualizing the time point of medical data that affects the patient's disease prediction result through a time domain for time information included in the medical data
Computer system, characterized in that.

Images