RU2758550C1 - Method for diagnosing signs of bronchopulmonary diseases associated with covid-19 virus disease - Google Patents

Abstract

FIELD: medicine.

SUBSTANCE: invention relates to medicine and can be used in practical medicine for non-invasive diagnosis of diseases of the bronchopulmonary system. Three types of audio recordings from the patient are recorded: cough, breathing, speech, discrete integral transformation of audio recordings is performed, the result of which is a set of spectrograms of these audio recordings, additional segmentation of spectrograms into separate fragments with intersections in time is carried out, methods for preprocessing a signal with using ultra-precise linear layers and obtaining a set of feature vectors that are fed to the input of a convolutional neural network for classification with obtaining the generated feature vector at the output, the obtained vectors of signs are combined from the three original audio recordings, the combinations of the obtained vectors are transformed using a linear layer, and a conclusion about the patient's health is formed based on the results obtained.

EFFECT: invention provides automation and simplification of the technological process for the diagnosis of COVID using deep machine learning methods.

5 cl, 10 dwg

Description

The invention relates to medicine and can be used in practical medicine for non-invasive diagnosis of diseases of the bronchopulmonary system.

The developed technical solution characterizes the way of diagnosing acoustic signs caused by changes in the respiratory tract associated with Covid disease. Deep learning methods have been used to solve the problem of regression, determining the likelihood of a person being infected with viral diseases that affect the human respiratory tract, in particular those caused by the COVID-19 virus, based on records of coughing, breathing and speech.

Known (RU, patent 2304928, publ. 27.08.2007) a method of acoustic diagnosis of focal changes in the lungs of a person, including the registration and calculation of the spectrum of acoustic signals of the conducted sound of the voice on the surface of the chest in the points of examination symmetrically located on the right and left, measurement and comparative assessment of them amplitudes. The spectrum of the recorded signal is calculated in the frequency band from 80 to 2000 Hz on a logarithmic scale in amplitude at each point of the survey, the amplitudes and frequencies of the first (A1, f1), second (A2, f2), third (A3, f3) spectral maxima located at harmonically related frequencies and having a level of at least 60 dB from the level of the first maximum, calculate the ratio A12 / f21, equal to the ratio (A1-A2) to (f2-f1), A23 / f32, equal to (A2-A3) to (f3- f2), the difference ΔΑ12 values A1 and A2 over symmetric points on the right (D) and on the left (S), while the comparison of the obtained values is carried out with the corresponding threshold values for this type of disease, and the pathological decrease in pneumatization at the point of examination is recorded, if performed, according to at least one of the conditions: A12 / f21 is less than the first threshold value of this parameter (A12 / f21) _n1 , A23 / f32 is less than the first threshold value of this parameter (A23 / f32) _n1 , f1 is greater than the threshold value of this parameter (f1) _n , ΔΑ12 less than the first threshold value of this parameter (ΔΑ12) _n1 for examination points of the right lung, ΔΑ12 is greater than the second threshold value of this parameter (ΔΑ12) _n2 for examination points of the left lung, and a pathological increase in pneumatization at the examination point is recorded if A12 / f21 is greater than the second threshold value of this parameter (A12 / f21) _n2 and / or A23 / f32 is greater than the second threshold value of this parameter (A23 / f32) _n2 , and the values of the first and second threshold values are calculated as 5% and 95% of the percentile of the distribution of these parameters in the healthy group.

It is also known (RU, patent 2354285, publ. 10.05.2009) a method for diagnosing obstructive respiratory dysfunctions by conducting bronchophonography and registering the respiratory cycle, and the following parameters are assessed: the acoustic equivalent of the work of the respiratory muscles (ARD) in various frequency ranges: ARD0 - 200 -1200 Hz, ARD1 - 1200-12600 Hz, ARD2 - 5000-12600 Hz, ARD3 - 1200-5000 Hz; calculate the coefficients Κ1, K2, K3: K1 = DGS1 / DGS0 × 100, K2 = DGS2 / DGS0 × 100, K3 = DGS3 / DGS0 × 100; ΔK, corresponding to the increase in the indices of the coefficients K, namely, ΔK = K forced expiration - K calm breathing / K calm breathing × 100; coefficient growth index (IPC) = ΔK2 / ΔK3 and at values in the quiet breathing mode: ARD1 and ARD3 more than 100 mJ; K1 and K3 more than 15; ΔΚ1 and ΔΚ3 less than 200% and IPC 2 and more diagnose obstructive disturbances in the functions of external respiration.

Also known (RU, patent 2598051, publ. 09/20/2016) A method for determining changes in the voice function of a person with COPD, including measuring the parameters of changes in voice-forming function based on acoustic analysis using the Specta PLUS computer program, characterized in that the characteristics of the pitch frequency are determined , maximum phonation time and sections of voice noise, sequentially in dynamics and with an increase on the 10th day of treatment, the frequency of the fundamental tone to 142.6 ± 15.2, maximum phonation time to 20.5 ± 2.9, sections of voice noise to " + "determine the improvement of a person's vocal function.

The disadvantage of all the above technical solutions should be recognized as their inapplicability to the diagnosis of diseases caused by the COVID-19 virus.

The technical problem solved by using the developed method is to expand the arsenal of diagnostic tools for diseases caused by the COVID-19 virus.

The technical result achieved by implementing the developed method consists in automating and simplifying the technological process for diagnosing COVID using deep machine learning methods.

To achieve this technical result, it is proposed to use the developed method for diagnosing signs of bronchopulmonary diseases associated with the disease with the COVID-19 virus. When implementing the developed method, three types of audio recordings from a patient are registered: cough, breathing, speech, discrete integral transformation of audio recordings is carried out, the result of which is a set of spectrograms of these audio recordings, additional segmentation of spectrograms into separate fragments with intersections in time is carried out, applied to the obtained fragments of spectrograms methods of signal preprocessing using ultra-precise linear layers and obtaining a set of feature vectors that are fed to the input of a convolutional neural network for classification with obtaining a generated feature vector at the output, combining the obtained feature vectors from three original audio recordings, transforming the combination of the obtained vectors using a linear layer and based on the results obtained, a conclusion about the patient's health is formed.

In some embodiments of the developed method, after registering three types of audio recordings from a patient: cough, breathing, speech, the spectral characteristics of the audio recording are extracted and transferred to the input of classical machine learning algorithms.

To obtain spectrograms, you can use the windowed Fourier transform or wavelet transform.

In some variants of the developed method, after preprocessing the spectrogram fragments and obtaining the feature vectors, the vector is fed to the input of the recurrent neural network. Feature classification by the neural network is carried out using the attention mechanism.

The developed technical solution characterizes the way of diagnosing acoustic signs caused by changes in the respiratory tract associated with Covid disease. Deep learning methods are used to solve the problem of regression, to determine the likelihood, based on the records of coughing, breathing and speech, that a person is infected with viral diseases that affect the human respiratory tract. The method includes converting, preparing, preprocessing and analyzing data using deep learning methods. To classify diseases, it is proposed to use a recurrent network with a convolutional neural network as an encoder and an attention mechanism.

The presented technology is a server application for analyzing medical acoustic data of patients for the detection and classification of respiratory diseases, as well as complications and deviations caused by the presence of viruses, in particular, COVID-19.

Coronavirus infection has become a real challenge for the public. It is impossible not to appreciate the work of doctors who are faced with a huge number of patients. However, the outbreak of the coronavirus has exposed some health problems, in particular the lack of medical workers. In the age of high technology, it is worth thinking about supplying hospitals with special software that can help a doctor diagnose a disease. With the growing popularity of machine learning and deep learning methods, it becomes obvious that this area is being addressed to find a solution.

Today, there are several approaches to diagnosing respiratory and viral diseases. The main idea of most of them is based on the processing of audio signals from the human body: coughing, breathing, chest sounds. According to research by research groups, simple binary data classifiers based on logit model, gradient boosting, and support vector machines give a precision of up to 82%. The random forest approach obtained the accuracy (accuracy) of classification on test data reached 66.74%). Some researchers are following the path of developing a classifier, represented by three branches and a mediator, similar to the independent opinions of several doctors.

In the proposed implementation, a positive or negative result of Covid-19 is set only when the decisions of the three branches coincide, which reduces the probability of an error to 6.147⋅10 ^-4 . The classification used convolutional networks and support vector machines.

In addition to processing the sounds of the human body, X-rays and CT scans of the chest can also be used to diagnose Covid with deep learning methods.

The developed technical solution is a method for analyzing the acoustic data of a patient's cough, breathing and speech to identify and classify respiratory diseases or associated signs of a viral disease.

The model for the diagnosis of diseases is an ensemble of recurrent neural networks with an encoder, an attention mechanism and linear layers following it.

The invention is a method for processing records from users. The architecture of the method is represented by an ensemble of neural networks, which are represented by three independent branches with subsequent concretization of the results Fully connected by layers.

Figure 1 shows the general architecture used in the implementation of the method of the system for the diagnosis of COVID.

For analysis, three audio recordings are fed to the input of the system: speech, cough, breathing. Every audio recording has the same processing process. A diagram of the processing of each record is presented in parallel processed in a separate branch. The layout of each branch is the same and is shown in Fig. 1.

The sequence of processing audio recordings includes the following steps:

• verification and conversion of audio recording parameters;

• slicing and extraction of features for each separate audio recording window;

• obtaining a feature vector using RNN (recurrent neural network) for complete audio recording.

Then the audio recordings are checked and converted, while the audio recordings from users are sent to the processing unit. The block will check the audio file for compliance with the system requirements for data format, sampling frequency, bit rate, number of channels. If the parameters do not match, the data is converted to the required system parameters.

• Translation of an audio track into a numeric array

• Conversion from stereo to mono mode

• Resampling to 44.1 kHz sampling rate

If it is impossible to convert to the required parameters, the block generates an error indicating invalid parameters of the audio file.

Next comes the stage of cutting and extracting features. At the stage of feature extraction, the most significant features in audio files are selected for their subsequent submission to a recurrent neural network to extract patterns and patterns. Extraction of traits can be done in different ways, such as:

• integral transforms (windowed Fourier transform, wavelet transform, and others);

• extraction of i-vectors;

• hidden Markov models;

• other.

Then the stage of continuous integral transformations of the analysis of time signals passes. There are various families of integral transformations of non-stationary time signals. It is assumed that the time signal is transferred to the frequency range, where it is more convenient to analyze the behavior of the dynamics of the process and it is easier to extract numerical characteristics. At the same time, there are various types of time-frequency integral transformations that translate the signal into the frequency domain. In addition to Fourier Transform (FT), signal analysis applications also use Short-time Fourier Transform (STFT), Gabor Transform (GT), Wavelet Transform (WT), Wigner Distribution Function (WDF), etc.

STFT

By definition, the continuous windowed Fourier transform can be represented as an integral

where w (⋅) is a window function that allows you to select the time interval of interest and carry out additional processing inside it. When the Gaussian function is selected as the window function, the windowed Fourier transform (STFT) is called the Gabor transform (GT).

WT

A generalization of STFT is the wavelet transform. In the general case, the integral wavelet transform (2) is written in the form

where the kernel of the transformation is the wavelet function ψ (⋅), and the transformation itself uses its complex conjugation ψ (⋅). While the window function in STFT depends on one parameter τ, which determines the time shift, the wavelet in CWT depends on two parameters a, b, which are responsible for the scale (compression or expansion of the transformation kernel) and shift (translation), respectively. For example, a Morlet wavelet or Gabor wavelet is used as the ψ (⋅) kernel in medical applications, Fig. 2. which shows the Morlet wavelet and its first derivative, that is, a function of the form

In addition, the wavelet function must satisfy the following properties

1. The finiteness of energy

2. Condition of admissibility

3. For complex wavelet functions, the Fourier transform must be real and vanish for negative frequencies.

Note that there are various ways of constructing wavelet systems, both orthogonal and non-orthogonal. So, as an approximation basis for constructing various systems of wavelet functions, infinitely differentiable splines or atomic functions can be used. Examples of calculating the quantitative characteristics of temporal signals using similar synthesized systems of wavelet functions are also presented in. An example of the simplest atomic function that coincides with the Fabius function on the segment [0; 2] is shown in Fig. 3, which shows the form of the Fabius function and its first derivative

Note that there is a library for visualizing wavelet systems in Python and a library for wavelet transforms in Python.

Then the stage of discrete integral transformations of the analysis of time signals begins. Due to the discreteness of the input data, it becomes necessary to take into account the finiteness of the number of samples, and, as a consequence, discrete analogs of the continuous integral transformations indicated above appear.

DWFT

The discrete version of the continuous window transform takes the form

where X (k) is the discrete frequency of the time sequence x (n), n is the time index, k is the frequency index, N is the number of samples, w (n) are the samples of the window function. In this case, the window function can be selected in various ways. So, in practical applications, the Hann windows are used, which is defined as follows

и

and

extracting i-vectors.

The i-vectors or identity-vectors method is a method for extracting and using auxiliary features. At the moment, the class of methods i-vectors is a relatively new way of solving problems of recognizing objects of various nature. Initially, the i-vectors method arose to solve the speech recognition problem. The idea of the method is based on the representation of models of expressions of a Gaussian mixture

In this case, the image of this expression is also used as a feature vector in the language classifier.

Applying Windowed Fourier Transform

For example, consider a feature extraction scheme using the discrete windowed Fourier transform. The standard scheme for using DWFT is as follows. An area of interest is selected from the complete data signal for analysis, FIG. 4.

The part of the signal that fell into the area of interest is scalar multiplied by a certain window function, i.e. "weighting" occurs (Fig. 5 - Fig. 6).

In this case, the sum of the shifts of the Hann window function provides the decomposition of unity, (Fig. 7 - Fig. 8). However, wavelets and atomic functions can also be used as window functions, the sum of the shifts of which also satisfies the partition of unity.

b)

In the present invention, this approach allows obtaining a spectrogram, after which it is divided into fragments of 1 second duration with a step of 0.5 seconds, which are fed to the input of the CNN encoders.

In this case, CNN encoders are used to extract (extract) representative (significant) features and reduce the dimension of the input data into LSTM layers. The encoder consists of four blocks, which include a convolution operation with a 3 × 3 kernel, an activation layer with the Leaky ReLu function, a decimation method with a probability of excluding a neuron of 0.7 to prevent overfitting, and batch normalization. The encoders process the spectrogram windows received as input data and the received features, are fed to the input of the LSTM layers, Fig. 9.

A recurrent neural network with LSTM is designed according to the many-to-many principle. Each separate fragment of the audio recording, after passing through the feature extractors, falls on a separate LSTM layer with the dimension of 512 internal gates.

The output from each layer of the recurrent network is passed on to the attention block.

Attention

The output of each LSTM layer, which is a 512 vector, is passed through a linear layer with the hyperbolic tangent as its activation function.

The obtained vectors after the linear layer are scalar multiplied with the vector of weights, which, in the process of training the model, corrects the weights by the method of gradient descent .. and the generated features are transferred to softmax for normalization.

The normalized values are multiplied with the original features obtained on the LSTM layers and the obtained values are weightedly summed up with the outputs of all other layers. The architecture of the proposed deep learning algorithm is shown in Fig. 10. Obtaining a vector of weighted sums of all three audio recordings is fed to the input of the concatenation unit and subsequent linear transformation, and at the output we obtain the probability of a patient becoming infected with COVID-19.

When training the model, the adam optimization algorithm is used and the learning rate of the algorithm is reduced by 10 times every 100 steps.

The described method can be applied using any device that has a microphone and is able to use it for recording (including, but not limited to: a voice recorder, push-button mobile phone, smartphone, smart watch, terminal, smart speaker, etc.). Specialized software adapted to the specified device helps the user to complete the necessary sequence of steps to prepare and record audio files. The data recorded in the files is transmitted to the server with the file processing system deployed on it through any data transfer channels. The system on the server processes sound files in accordance with the method described above and transmits the result to the user (or to another addressee (either a person or another system determined by the system settings) using adaptable formats and any available communication channels.

Claims (5)

1. A method for diagnosing signs of bronchopulmonary diseases in COVID-19, characterized in that they register three types of audio recordings from a patient: cough, breathing, speech, perform discrete integral transformation of audio recordings, which results in obtaining a set of spectrograms of these audio recordings, carry out additional segmentation of spectrograms into separate fragments with intersection in time, apply to the obtained fragments of spectrograms the methods of signal preprocessing using ultra-precise linear layers and obtaining a set of feature vectors that are fed to the convolutional neural network for classification with obtaining the generated feature vector at the output, combining the obtained feature vectors from three original audio recordings , transform the union of the obtained vectors using a linear layer and, based on the results obtained, form a conclusion about the patient's health. 2. The method according to claim 1, characterized in that after registering three types of audio recordings from a patient: cough, breathing, speech, the spectral characteristics of the audio recording are extracted and transferred to the input of classical machine learning algorithms. 3. A method according to claim 1, characterized in that a windowed Fourier transform or a wavelet transform is used to obtain the spectrograms. 4. The method according to claim 1, characterized in that after preprocessing the spectrogram fragments and obtaining the feature vectors, the vector is fed to the input of the recurrent neural network. 5. The method according to claim 1, characterized in that the classification of the features by the neural network is carried out using the attention mechanism.

Images