CN116889388A - An intelligent detection system and method based on rPPG technology - Google Patents

Abstract

The invention relates to the field of rPPG technology, specifically an intelligent detection system and method based on rPPG technology. The system includes an information data preprocessing module, an rPPG signal segmentation and waveform selection module, a dual-channel feature fusion data prediction module and rPPG signal characteristics. Extraction and data prediction module, the information data preprocessing module is used to collect the user's palm information in real time through the camera, combine the collected user's palm information to divide the ROI area and extract the image G channel signal in the divided area, and average the G channel signal The pixel values are preprocessed as original rPPG signals. The present invention provides a non-contact data measurement method based on rPPG, which has the advantages of non-invasiveness, portability and universality.

Description

An intelligent detection system and method based on rPPG technology

Technical field

The invention relates to the field of rPPG technology, specifically an intelligent detection system and method based on rPPG technology.

Background technique

Remote photoplethysmography is a non-contact optical detection technology that can extract the human body's pulse wave signal through video, through which the blood volume in blood vessels changes periodically due to the contraction and relaxation of the heart, and the hemoglobin in different blood volumes changes Different light absorption abilities lead to regular changes in the reflected light on the skin surface. By shooting with video imaging equipment and using corresponding signal processing methods, the original pulse wave can be obtained from the video, and the changes in blood volume are related to the blood response. The lateral pressure generated by the blood vessel wall is directly related. As the blood volume increases, the lateral pressure of the blood on the blood vessel wall also increases. In the existing technology, trained professionals are used to measure relevant data, which does not allow for home monitoring and daily use. conditions, therefore an intelligent detection system and method based on rPPG technology is needed to achieve non-contact measurement.

Contents of the invention

The purpose of the present invention is to provide an intelligent detection system and method based on rPPG technology to solve the problems raised in the above background technology. The present invention provides the following technical solutions:

An intelligent detection method based on rPPG technology, the method includes the following steps:

S1. Collect the user's palm part information in real time through the camera, combine the collected user palm part information to divide the ROI area and extract the image G channel signal in the divided area, and preprocess the average pixel value in the G channel signal as the original rPPG signal;

S2. Perform targeted single-cycle segmentation on the preprocessed rPPG signal, conduct initial screening of the segmentation results based on the human standard pulse rate, and perform Tukey detection processing based on the screening results;

S3. Based on the analysis results of S2, extract the feature values in the rPPG signal after Tu-based detection processing, and conduct network parameter training through the Adma optimizer to build a dual-channel feature fusion data prediction model;

S4. Collect the current intensive care unit patient information in real time, and perform data training on the information through a dual-channel feature fusion data prediction model. The trained dual-channel feature fusion data prediction model will be obtained, and the preprocessed signal will be input to the The trained dual-channel feature fusion data prediction model is used for prediction, and then the feedback results are obtained.

Further, the method in S1 includes the following steps:

Step 1001: Collect the video of the user's palm movement in real time through the camera, and store each frame of the video as set A.

A=(A ₁ , A ₂ , A ₃ ,..., A _n ),

Where A _n represents the nth frame image, and n represents the total number of video frames collected;

Step 1002: arbitrarily extract one of the frame images to divide the ROI area, and use image recognition technology to mark 21 key points in the hand area in the n-th frame image.

Take the center of the junction of the palm and the wrist as the first key point, recorded as key point 0, take key point 0 as the origin, use the origin as the reference point, construct the first plane rectangular coordinate system with unit length as the interval, and divide the nth frame The corresponding key points in the hand area in the image are marked in the first plane rectangular coordinate system, and digitally marked in order;

Step 1003. Calculate the slope value of the line segment formed by each key point and the origin in the first plane Cartesian coordinate system and the distance value between the corresponding key point and the origin in the first plane Cartesian coordinate system, and analyze the corresponding key points and the origin. The result, as a combination, generates set B,

B={[(X ^A(n) _0,1 , D ^A(n) _0,1 ), (X ^A(n) _0,2 , D ^A(n) _0,2 ),..., (X ^A(n) _0,20 ，D ^A(n) _0,20 )]},

^Among _them ^, _{_} The distance value of the key point 0 in the first plane Cartesian coordinate system,

_Among ^them _{, _ _ _} ^_ _{_ _ _} ^_ _{_ 0} ) ² ] ^1/2 ;

Step 1004: Repeat steps 1002 to 1003 to obtain the slope value of the line segment formed by the corresponding key point and the origin of each frame of the video in the collected video, as well as the distance value between the corresponding key point and the origin in the first plane rectangular coordinate system, and calculate each step in turn. The difference between the key point positions in the frame image relative to the standard position is recorded as set C,

C=(C ₁ , C ₂ , C ₃ ,..., C _n ),

where _Cn represents the difference between the key point position in the nth frame image relative to the standard position,

Where C _n =α·∑ ²⁰ _a=1 |X ^A(n) _0,a -X ^a _standard |/20+β·∑ ²⁰ _a=1 |D ^A(n) _0,a -D ^a _standard |/ 20,

^Both α and β represent proportional coefficients, _which are preset values in the database. ^{Slope value} _{, _} The distance value between the key point marked a and the key point marked 0 in the first plane rectangular coordinate system. D ^a _standard means that the key point marked a and the key point marked 0 are at right angles to the first plane. The distance standard value in the coordinate system, the distance standard value is the database preset value;

Step 1005: Use the image corresponding to the minimum value of the difference in set C as the best image in the currently collected video, match the best image with the image in the Hand Landmarker model, and use the matching result to match the hand area in the model 21 key points are positioned in the best image. The Hand Landmarker model is trained by Google based on 30K real-world hand images. It realizes the positioning of 21 key points in the hand area. For each frame collected The image is used to judge hand landmarks. Because the palm area is rich in blood vessels, the palm ROI area is artificially divided according to the coordinates of key points. Even if there is hand movement, accurate ROI positioning can be achieved, minimizing external inspection interference;

Step 1006. Since blood tissue can absorb more light than other tissues, and the color of opaque objects is determined by the color of reflected light, blood will reflect red light and absorb green light, so the color of the green channel in the video collected by the camera The change is the most obvious. Since rPPG performs signal processing through changes in reflected light, the G channel is retained, the mean G channel pixels in the best image are read, and the read mean G channel pixels are used as the original rPPG signal through canny edge monitoring. Algorithm, separate the outline of the hand area from the background to obtain the background ROI, and correct the rPPG signal of the hand ROI through the change in background ROI brightness. The calculation formula for the change in background ROI brightness is I(t)=[∑ ^w _c=1 ∑ ^h _d=1 G _v (c,d,t)]/s,

Among them, I(t) represents the change of mild illumination changes with time t, G _v (c, d, t) represents the pixel point in the background ROI with an abscissa of c pixel intervals and a coordinate of d pixel intervals at time: The corresponding green channel value at time t, w represents the pixel width of the background ROI, h represents the pixel height of the background ROI, s represents the total number of pixels in the background ROI, and the discrete points whose brightness changes with time are fitted by a ninth-order polynomial. The brightness change curve is obtained, and the brightness change curve is subtracted from the original rPPG signal to eliminate illumination artifacts. On the other hand, in order to reduce the noise reflected when collecting the original rPPG, ensemble empirical mode decomposition is used to remove the original rPPG signal. At the same time, in order to provide a positioning starting point and end value point for single-cycle segmentation and reduce the error caused by insufficient sampling accuracy, a cubic sample interpolation function is performed on the rPPG signal to interpolate the original waveform from the 30Hz sampling frequency to 300Hz.

This invention collects the user's palm activity video in real time, analyzes the key point positions in the video, extracts each frame of image and matches it with the image in the Hand Landmarker model to obtain the accurate positioning of 21 key points in the hand area, and monitors the canny edge through canny edge monitoring. The algorithm separates the outline of the hand area from the background to obtain the background ROI. Through the brightness change of the background ROI, the rPPG signal of the hand ROI is corrected to provide data reference for subsequent data prediction.

Further, the method in S2 includes the following steps:

Step 2001: Obtain the preprocessed rPPG signal, use point o1 as the origin, time as the abscissa, and amplitude as the ordinate, construct a second plane rectangular coordinate system, and map the preprocessed rPPG signal to the second In the plane Cartesian coordinate system;

Step 2002: Mark all zero-crossing points in the decline period of the preprocessed rPPG signal in the second plane Cartesian coordinate system, combine two adjacent marking points in sequence, and combine two adjacent marking points in any combination. As the endpoint of the interval, extract the minimum value of the rPPG signal in the corresponding interval as the segmentation point to segment the rPPG signal;

Step 2003 and loop step 2002 divide the preprocessed rPPG signal into multiple single-cycle signal waveforms, conduct preliminary screening of the single-cycle signal waveforms based on the human body standard pulse rate range, and sequentially conduct waveform analysis of the single-cycle signal waveforms based on the preliminary screening results. Calibration, because the rPPG signal is easily affected by noise during the shooting process, it is necessary to perform relevant screening for rPPG exploitation. On the one hand, considering that the normal pulse rate range of the human body is 60 to 100 times per minute and the sampling frequency is 300Hz, the retention period A single-cycle signal waveform with a minimum of 120 sampling points and a maximum of 300 sampling points. After the preliminary screening of single-cycle signals is completed, on the other hand, considering the uniqueness of the human body's pulse rate, each person's pulse rate range is different. are all the same, so the graph-based detection of the preliminary filtered data is based on the number of sampling points of each single-cycle waveform. The retention interval of the single-cycle waveform is as follows:

P _calibration ={[Q ₁ -k(Q ₃ -Q ₁ )-r] ⊕ [Q ₃ +k(Q ₃ -Q ₁ )-r]}·ξ·G _anomaly ,

Q ₁ means that all single-cycle waveforms use the lower quartile of point data, Q ₃ means that all single-cycle waveforms use the upper quartile of point data, k means the anomaly coefficient, when k=3, it means extreme anomalies, when When k=1.5, it represents moderate anomaly, r represents the number of sampling points, ξ represents the proportion coefficient, and the proportion coefficient is the default value of the database. G _anomaly represents the total number of abnormal sampling points in the single-cycle signal waveform. If [Q ₁ -k(Q ₃ -Q ₁ )-r]＜0 and [Q ₃ +k(Q ₃ -Q ₁ )-r]＞0, then [Q ₁ -k(Q ₃ -Q ₁ )-r] ⊕ [Q ₃ +k(Q ₃ -Q ₁ )-r]=1, otherwise all are 0. The total number of abnormal sampling points in the single-cycle signal waveform is calculated by comparing the sampling points of the single-cycle signal waveform with the human body standard pulse rate. The difference operation is performed on the range, and the statistical difference operation is the number of negative values;

The present invention divides a single-cycle signal and uses the number of sampling points of each single-cycle waveform as a basis for waveform screening, and provides data reference for subsequent model training and data detection based on the screening results.

Further, the method in S3 includes the following steps:

Step 3001. Train the network parameters through the Adam optimizer, use the linear rectifier unit as the activation function, the root mean square error as the loss function, the average absolute error as the evaluation index, set the learning rate δ, and preprocess the data. The order of the rPPG signals obtained is scrambled and divided into a training set and a test set;

Step 3002: Send the signal of the input model to the FNN multi-layer perceptron branch for feature extraction. The FNN is completely composed of fully connected layers. The input features are combined through a multi-layer structure to mine the relationship between the input features and the data. strong correlation;

This invention sets up an input layer for receiving data, nine hidden layers for mining and extracting deep-level features in the input signal, the last output layer contains 160 nodes, and finally the vector containing the signal features is sent to the feature fusion module;

Step 3003: Send the input model signals to the CNN convolutional neural network module at the same time for multi-view feature extraction; this module uses the AlexNet network as the backbone, and the network structure contains a total of 9 layers. The first 8 layers are composed of convolutional layers and pooling layers. Composed, the convolution layer is used to extract features from the signal, the pooling layer uses maximum pooling to reduce the size of the feature map and reduce the computational complexity. The last layer is the flatten layer, which flattens the feature vector into one-dimensional data. Send it to the feature fusion module;

Step 3004: Fuse the output features of the branches of step 3002 and step 3003, and predict the data through two fully connected layers.

Further, the method in S4 collects current intensive care unit patient information in real time, and performs data training on the information through a dual-channel feature fusion data prediction model. The trained dual-channel feature fusion data prediction model will be obtained, and the preprocessed The final signal is input to the trained dual-channel feature fusion data prediction model for prediction, and then the feedback result is obtained.

An intelligent detection system based on rPPG technology, the system includes the following modules:

Information data preprocessing module: The information data preprocessing module is used to collect the user's palm information in real time through the camera, combine the collected user's palm information to divide the ROI area and extract the image G channel signal in the divided area, and average the G channel signal Pixel values are preprocessed as raw rPPG signals;

rPPG signal segmentation and waveform selection module: The rPPG signal segmentation and waveform selection module is used to perform targeted single-cycle segmentation of the preprocessed rPPG signal, divide the single-cycle signal according to the segmentation result, and perform rPPG waveform screening on the segmentation result. Perform Tukey detection processing based on the screening results;

Dual-channel feature fusion data prediction module: The dual-channel feature fusion data prediction module is used to construct a dual-channel feature fusion data prediction model by combining the analysis results of the rPPG signal segmentation and waveform selection modules;

rPPG signal feature extraction and data prediction module: The rPPG signal feature extraction and data prediction module is used to input the dual-channel feature fusion data prediction model through training data for training. The trained dual-channel feature fusion data prediction model will be obtained and the pre-trained dual-channel feature fusion data prediction model will be obtained. The processed signal is input to the trained dual-channel feature fusion data prediction model for prediction, and then the feedback result is obtained.

Further, the information data preprocessing module includes an image acquisition unit, an ROI area division unit, a channel data extraction unit and an rPPG signal preprocessing unit:

The image acquisition unit is used to collect real-time video of the user's palm movement through the camera, and extract each frame of the image in the collected video;

The ROI area division unit is used to determine the hand landmarks in each frame of image collected in combination with the analysis results of the image acquisition unit, and locate 21 key points in the hand area based on the determination results;

The channel data extraction unit is used to acquire RGB three-color channels in combination with the analysis results of the ROI area division unit, retain the G channel, and generate the original rPPG signal by calculating the average pixel value of the G channel;

The rPPG signal preprocessing unit is used to separate the outline of the hand area from the background to obtain the background ROI through the canny edge monitoring algorithm, and correct the rPPG signal of the hand ROI through changes in the brightness of the background ROI.

Further, the rPPG signal segmentation and waveform selection module includes a single-cycle segmentation unit, an rPPG waveform screening unit and a base detection processing unit:

The single-cycle segmentation unit is used to combine the analysis results of the information data preprocessing module to extract all zero-crossing points in the declining period of the original rPPG signal, and take the minimum value of the interval between two adjacent zero-crossing points as the starting point of a cycle. The original rPPG signal is segmented;

The rPPG waveform screening unit is used to obtain sampling point data based on the normal pulse rate range of the human body, and perform preliminary screening of the analysis results of the single-cycle segmentation unit based on the obtained results;

The base detection unit is used to perform base detection in combination with the analysis results of the rPPG waveform screening unit, and uses the number of sampling points of each single-cycle waveform as a basis for division to determine whether to retain the initially screened single-cycle waveform.

Further, the dual-channel feature fusion data prediction module includes a dual-channel feature fusion unit and a training dual-channel feature fusion unit:

The dual-channel feature fusion unit is used to combine the analysis results of the rPPG signal segmentation and waveform selection modules, send the rPPG signal to the FNN multi-layer perceptron branch for feature extraction, and combine the extracted features through the multi-layer structure;

The training dual-channel feature fusion unit is used to combine the analysis results of the rPPG signal segmentation and waveform selection modules, and send the rPPG signal to the CNN convolutional neural network module for multi-view feature extraction.

Further, the rPPG signal feature extraction and data prediction module includes a feature fusion unit and a data prediction unit:

The feature fusion unit is used to fuse the analysis results of the dual-channel feature fusion unit and the training dual-channel feature fusion unit;

The data prediction unit is used to perform data prediction in combination with the analysis results of the feature fusion unit.

The present invention provides a non-contact measurement method based on rPPG, which has the advantages of non-invasiveness, portability and universality. That is, it captures minute color changes on the skin surface through a camera. The measurement method based on rPPG is convenient and can be used through ordinary smartphones. The rear camera collects video of the palm area for data prediction, which allows users to monitor detection data in daily life without the need for special equipment or support from medical institutions. Moreover, the present invention does not require body surface measurements during measurement. Contact, more suitable for special use environments, thereby reducing interference to patients, making it more applicable and convenient.

Description of the drawings

Figure 1 is a schematic flow chart of an intelligent detection method based on rPPG technology in the present invention;

Figure 2 is a module schematic diagram of an intelligent detection system based on rPPG technology of the present invention;

Figure 3 is a schematic diagram of the position information of key points in the hand area of an intelligent detection method based on rPPG technology of the present invention;

Figure 4 is a schematic diagram of a dual-channel feature fusion neural network of an intelligent detection method based on rPPG technology of the present invention;

Figure 5 is a schematic diagram of data prediction results in an intelligent detection method based on rPPG technology of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

Embodiment 1: Please refer to Figure 1. In this embodiment:

An intelligent detection method based on rPPG technology, the method includes the following steps:

S1. Collect the user's palm part information in real time through the camera, combine the collected user palm part information to divide the ROI area and extract the image G channel signal in the divided area, and preprocess the average pixel value in the G channel signal as the original rPPG signal;

The method in S1 includes the following steps:

Step 1001: Collect the video of the user's palm movement in real time through the camera, and store each frame of the video as set A.

A=(A ₁ , A ₂ , A ₃ ,..., A _n ),

Where A _n represents the nth frame image, and n represents the total number of video frames collected;

Step 1002: arbitrarily extract one of the frame images to divide the ROI area, and use image recognition technology to mark 21 key points in the hand area in the n-th frame image (as shown in Figure 3).

Take the center of the junction of the palm and the wrist as the first key point, recorded as key point 0, take key point 0 as the origin, use the origin as the reference point, construct the first plane rectangular coordinate system with unit length as the interval, and divide the nth frame The corresponding key points in the hand area in the image are marked in the first plane rectangular coordinate system, and digitally marked in order;

Step 1003. Calculate the slope value of the line segment formed by each key point and the origin in the first plane Cartesian coordinate system and the distance value between the corresponding key point and the origin in the first plane Cartesian coordinate system, and analyze the corresponding key points and the origin. The result, as a combination, generates set B,

B={[(X ^A(n) _0,1 , D ^A(n) _0,1 ), (X ^A(n) _0,2 , D ^A(n) _0,2 ),..., (X ^A(n) _0,20 ，D ^A(n) _0,20 )]},

^Among _them ^, _{_} The distance value of the key point 0 in the first plane Cartesian coordinate system,

_Among ^them _{, _ _ _} ^_ _{_ _ _} ^_ _{_ 0} ) ² ] ^1/2 ;

Step 1004: Repeat steps 1002 to 1003 to obtain the slope value of the line segment formed by the corresponding key point and the origin of each frame of the video in the collected video, as well as the distance value between the corresponding key point and the origin in the first plane rectangular coordinate system, and calculate each step in turn. The difference between the key point positions in the frame image relative to the standard position is recorded as set C,

C=(C ₁ , C ₂ , C ₃ ,..., C _n ),

where _Cn represents the difference between the key point position in the nth frame image relative to the standard position,

Where C _n =α·∑ ²⁰ _a=1 |X ^A(n) _0,a -X ^a _standard |/20+β·∑ ²⁰ _a=1 |D ^A(n) _0,a -D ^a _standard |/ 20,

^Both α and β represent proportional coefficients, _which are preset values in the database. ^{Slope value} _{, _} The distance value between the key point marked a and the key point marked 0 in the first plane rectangular coordinate system. D ^a _standard means that the key point marked a and the key point marked 0 are at right angles to the first plane. The distance standard value in the coordinate system, the distance standard value is the database preset value;

Step 1005: Use the image corresponding to the minimum value of the difference in set C as the best image in the currently collected video, match the best image with the image in the Hand Landmarker model, and use the matching result to match the hand area in the model 21 key points are positioned in the best image;

Step 1006: Read the mean G channel pixels in the best image, and use the read mean G channel pixels as the original rPPG signal. Use the canny edge monitoring algorithm to separate the outline of the hand area from the background to obtain the background ROI. Use the background ROI brightness changes, correct the rPPG signal of the hand ROI, where the calculation formula for the background ROI brightness change is I(t)=[∑ ^w _c=1 ∑ ^h _d=1 G _v (c,d,t)]/s,

Among them, I(t) represents the change of light intensity with time t, and G _v (c, d, t) represents the pixel point in the background ROI with an abscissa of c pixel intervals and a coordinate of d pixel intervals at time t. The corresponding green channel value, w represents the pixel width of the background ROI, h represents the pixel height of the background ROI, and s represents the total number of pixels in the background ROI.

S2. Perform targeted single-cycle segmentation on the preprocessed rPPG signal, conduct initial screening of the segmentation results based on the human standard pulse rate, and perform Tukey detection processing based on the screening results;

The method in S2 includes the following steps:

Step 2001: Obtain the preprocessed rPPG signal, use point o1 as the origin, time as the abscissa, and amplitude as the ordinate, construct a second plane rectangular coordinate system, and map the preprocessed rPPG signal to the second In the plane Cartesian coordinate system;

Step 2002: Mark all zero-crossing points in the decline period of the preprocessed rPPG signal in the second plane Cartesian coordinate system, combine two adjacent marking points in sequence, and combine two adjacent marking points in any combination. As the endpoint of the interval, extract the minimum value of the rPPG signal in the corresponding interval as the segmentation point to segment the rPPG signal;

Step 2003 and loop step 2002 divide the preprocessed rPPG signal into multiple single-cycle signal waveforms, conduct preliminary screening of the single-cycle signal waveforms based on the human body standard pulse rate range, and combine the preliminary screening results to sequentially filter the single-cycle signal waveforms through calculations. To perform waveform calibration, the expression is:

P _calibration ={[Q ₁ -k(Q ₃ -Q ₁ )-r] ⊕ [Q ₃ +k(Q ₃ -Q ₁ )-r]}·ξ·G _anomaly ,

Q ₁ represents the lower quartile of point data for all single-cycle waveforms, Q ₃ represents the upper quartile of point data for all single-cycle waveforms, k represents the anomaly coefficient, and the anomaly coefficient is the database preset value, ξ represents a proportional coefficient, which is a database preset value. G _anomaly represents the total number of abnormal sampling points in a single-cycle signal waveform. If [Q ₁ -k (Q ₃ -Q ₁ ) -r] < 0 and [Q ₃ +k(Q ₃ -Q ₁ )-r]>0, then [Q ₁ -k(Q ₃ -Q ₁ )-r] ⊕ [Q ₃ +k(Q ₃ -Q ₁ )-r]=1 , otherwise all are 0.

S3. Based on the analysis results of S2, extract the feature values in the rPPG signal after Tu-based detection processing, and conduct network parameter training through the Adma optimizer to build a dual-channel feature fusion data prediction model;

The method in S3 includes the following steps:

Step 3001. Train the network parameters through the Adam optimizer, use the linear rectifier unit as the activation function, the root mean square error as the loss function, the average absolute error as the evaluation index, set the learning rate to 0.001, and preprocess the data. The order of the rPPG signals obtained is scrambled and divided into a training set and a test set;

Step 3002: Send the signal of the input model to the FNN multi-layer perceptron branch for feature extraction;

Step 3003: Send the input model signals to the CNN convolutional neural network module at the same time for multi-view feature extraction;

Step 3004: Fuse the output features of the branches of step 3002 and step 3003, and predict the data through two fully connected layers.

S4. Collect the current intensive care unit patient information in real time, and perform data training on the information through a dual-channel feature fusion data prediction model. The trained dual-channel feature fusion data prediction model will be obtained, and the preprocessed signal will be input to the The trained dual-channel feature fusion data prediction model is used for prediction, and then the feedback results are obtained.

The method in S4 collects the current intensive ward patient information in real time, and performs data training on the information through a dual-channel feature fusion data prediction model. The trained dual-channel feature fusion data prediction model will be obtained, and the preprocessed signal Input to the trained dual-channel feature fusion data prediction model for prediction, and then obtain feedback results.

In this embodiment: an intelligent detection system based on rPPG technology is disclosed (as shown in Figure 2), and the system is used to implement the specific solution content of the method.

Example 2: When constructing and training a dual-channel feature fusion neural network, use the Adam optimizer to train network parameters, use the linear rectifier unit as the activation function, use the root mean square error as the loss function, use the average absolute value as the evaluation index, and set learning The rate is 0.001, and the order of rPPG signals obtained after data preprocessing is scrambled and divided into training and test sets for subsequent training and prediction of the model (as shown in Figure 4). The dual-channel feature fusion neural network is specifically constructed. The process is as follows:

Step S4.1: Set up an input layer to receive data, and send the input model signal to the FNN multi-layer perceptron branch for feature extraction. FNN is completely composed of fully connected layers, and the input features are combined through a multi-layer structure. It is used to mine the strong correlation between input features and blood pressure. Nine hidden layers are used to mine and extract deep features in the input signal. The last output layer contains 160 nodes. Finally, the vector containing the signal features is fed into Feature fusion module;

Step S4.2: The signals input to the model are simultaneously sent to the CNN convolutional neural network module for multi-view feature extraction. This module uses the AlexNet network as the backbone. The network structure contains a total of 9 layers. The first 8 layers are composed of convolutional layers and pooling. Layer composition, the convolutional layer is used to extract features from the signal, the pooling layer uses maximum pooling to reduce the size of the feature map and reduce the computational complexity. The last layer is the flatten layer, which flattens the feature vector into one-dimensional data. , sent to the feature fusion module;

Step S4.3: Fusion the output features of the two branches, and finally predict blood pressure through the two fully connected layers. Use the training data to input the dual-channel feature fusion blood pressure prediction model for training, and obtain the dual-channel feature fusion blood pressure prediction model. , extract the data from the MIMIC intensive care database and send it to the model for training. The predicted diastolic blood pressure and systolic blood pressure have a good fit with the real values (as shown in Figure 5). The database records patient information in the intensive care unit. Including blood pressure and pulse wave signals.

It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as illustrative and non-restrictive from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is therefore intended that all claims falling within the claims All changes within the meaning and scope of equivalent elements are included in the present invention. Any reference signs in the claims shall not be construed as limiting the claim in question.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are mutually exclusive. any such actual relationship or sequence exists between them. Furthermore, the terms "comprises," "comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment.

Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it is still The technical solutions described in the foregoing embodiments may be modified, or some of the technical features may be equivalently replaced. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. An intelligent detection method based on rPPG technology, characterized in that the method includes the following steps: S1. Collect the user's palm part information in real time through the camera, combine the collected user palm part information to divide the ROI area and extract the image G channel signal in the divided area, and preprocess the average pixel value in the G channel signal as the original rPPG signal; S2. Perform targeted single-cycle segmentation on the preprocessed rPPG signal, conduct initial screening of the segmentation results based on the human standard pulse rate, and perform Tukey detection processing based on the screening results; S3. Based on the analysis results of S2, extract the feature values in the rPPG signal after Tu-based detection processing, and conduct network parameter training through the Adma optimizer to build a dual-channel feature fusion data prediction model; S4. Collect the current intensive care unit patient information in real time, and perform data training on the information through a dual-channel feature fusion data prediction model. The trained dual-channel feature fusion data prediction model will be obtained, and the preprocessed signal will be input to the The trained dual-channel feature fusion data prediction model is used for prediction, and then the feedback results are obtained. 2. An intelligent detection method based on rPPG technology according to claim 1, characterized in that the method in S1 includes the following steps: Step 1001: Collect the video of the user's palm movement in real time through the camera, and store each frame of the video as set A. A=(A ₁ , A ₂ , A ₃ ,..., A _n ), Where A _n represents the nth frame image, and n represents the total number of video frames collected; Step 1002: arbitrarily extract one of the frame images to divide the ROI area, and use image recognition technology to mark 21 key points in the hand area in the n-th frame image. Take the center of the junction of the palm and the wrist as the first key point, recorded as key point 0, take key point 0 as the origin, use the origin as the reference point, construct the first plane rectangular coordinate system with unit length as the interval, and divide the nth frame The corresponding key points in the hand area in the image are marked in the first plane rectangular coordinate system, and digitally marked in order; Step 1003. Calculate the slope value of the line segment formed by each key point and the origin in the first plane Cartesian coordinate system and the distance value between the corresponding key point and the origin in the first plane Cartesian coordinate system, and analyze the corresponding key points and the origin. The result, as a combination, generates set B, B={[(X ^A(n) _0,1 , D ^A(n) _0,1 ), (X ^A(n) _0,2 , D ^A(n) _0,2 ),..., (X ^A(n) _0,20 ，D ^A(n) _0,20 )]}, ^Among _them ^, _{_} The distance value of the key point 0 in the first plane Cartesian coordinate system, _Among ^them _{, _ _ _} ^_ _{_ _ _} ^_ _{_ 0} ) ² ] ^1/2 , Step 1004: Repeat steps 1002 to 1003 to obtain the slope value of the line segment formed by the corresponding key point and the origin of each frame of the video in the collected video, as well as the distance value between the corresponding key point and the origin in the first plane rectangular coordinate system, and calculate each step in turn. The difference between the key point positions in the frame image relative to the standard position is recorded as set C, C=(C ₁ , C ₂ , C ₃ ,..., C _n ), where _Cn represents the difference between the key point position in the nth frame image relative to the standard position, Where C _n =α·∑ ²⁰ _a=1 |X ^A(n) _0,a -X ^a _standard |/20+β·∑ ²⁰ _a=1 |D ^A(n) _0,a -D ^a _standard |/ 20, ^Both α and β represent proportional coefficients, _which are preset values in the database. ^{Slope value} _, ^_ _{_} The distance value between the key point marked a and the key point marked 0 in the first plane rectangular coordinate system. D ^a _standard means that the key point marked a and the key point marked 0 are at right angles to the first plane. The distance standard value in the coordinate system, the distance standard value is the database preset value; Step 1005: Use the image corresponding to the minimum value of the difference in set C as the best image in the currently collected video, match the best image with the image in the Hand Landmarker model, and use the matching result to match the hand area in the model 21 key points are positioned in the best image; Step 1006: Read the mean G channel pixels in the best image, and use the read mean G channel pixels as the original rPPG signal. Use the canny edge monitoring algorithm to separate the outline of the hand area from the background to obtain the background ROI. Use the background ROI brightness changes, correct the rPPG signal of the hand ROI, where the calculation formula for the background ROI brightness change is I(t)=[∑ ^w _c=1 ∑ ^h _d=1 G _v (c,d,t)]/s, Among them, I(t) represents the change of light intensity with time t, and G _v (c, d, t) represents the pixel point in the background ROI with an abscissa of c pixel intervals and a coordinate of d pixel intervals at time t. The corresponding green channel value, w represents the pixel width of the background ROI, h represents the pixel height of the background ROI, and s represents the total number of pixels in the background ROI. 3. An intelligent detection method based on rPPG technology according to claim 2, characterized in that the method in S2 includes the following steps: Step 2001: Obtain the preprocessed rPPG signal, use point o1 as the origin, time as the abscissa, and amplitude as the ordinate, construct a second plane rectangular coordinate system, and map the preprocessed rPPG signal to the second In the plane Cartesian coordinate system; Step 2002: Mark all zero-crossing points in the decline period of the preprocessed rPPG signal in the second plane Cartesian coordinate system, combine two adjacent marking points in sequence, and combine two adjacent marking points in any combination. As the endpoint of the interval, extract the minimum value of the rPPG signal in the corresponding interval as the segmentation point to segment the rPPG signal; Step 2003 and loop step 2002 divide the preprocessed rPPG signal into multiple single-cycle signal waveforms, conduct preliminary screening of the single-cycle signal waveforms based on the human body standard pulse rate range, and combine the preliminary screening results to sequentially filter the single-cycle signal waveforms through calculations. To perform waveform calibration, the expression is: P _calibration ={[Q ₁ -k(Q ₃ -Q ₁ )-r] ⊕ [Q ₃ +k(Q ₃ -Q ₁ )-r]}·ξ·G _anomaly , Q ₁ represents the lower quartile of point data for all single-cycle waveforms, Q ₃ represents the upper quartile of point data for all single-cycle waveforms, k represents the anomaly coefficient, and the anomaly coefficient is the database preset value, r represents the number of sampling points, ξ represents the proportion coefficient, which is the preset value of the database, G _anomaly represents the total number of abnormal sampling points in the single-cycle signal waveform, if [Q ₁ -k(Q ₃ -Q ₁ )-r ]<0 and [Q ₃ +k(Q ₃ -Q ₁ )-r]>0, then [Q ₁ -k(Q ₃ -Q ₁ )-r] ⊕ [Q ₃ +k(Q ₃ -Q ₁ )-r]=1, otherwise it is 0. 4. An intelligent detection method based on rPPG technology according to claim 3, characterized in that the method in S3 includes the following steps: Step 3001. Train the network parameters through the Adam optimizer, use the linear rectifier unit as the activation function, the root mean square error as the loss function, the average absolute error as the evaluation index, set the learning rate δ, and preprocess the data. The order of the rPPG signals obtained is scrambled and divided into a training set and a test set; Step 3002: Send the signal of the input model to the FNN multi-layer perceptron branch for feature extraction; Step 3003: Send the input model signals to the CNN convolutional neural network module at the same time for multi-view feature extraction; Step 3004: Fuse the output features of the branches of step 3002 and step 3003, and predict the data through two fully connected layers. 5. An intelligent detection method based on rPPG technology according to claim 4, characterized in that the method in S4 collects current intensive care unit patient information in real time, and passes the information through a dual-channel feature fusion data prediction model Perform data training to obtain a trained dual-channel feature fusion data prediction model, input the preprocessed signal into the trained dual-channel feature fusion data prediction model for prediction, and then obtain feedback results. 6. An intelligent detection system based on rPPG technology, characterized in that the system includes the following modules: Information data preprocessing module: The information data preprocessing module is used to collect the user's palm information in real time through the camera, combine the collected user's palm information to divide the ROI area and extract the image G channel signal in the divided area, and average the G channel signal Pixel values are preprocessed as raw rPPG signals; rPPG signal segmentation and waveform selection module: The rPPG signal segmentation and waveform selection module is used to perform targeted single-cycle segmentation of the preprocessed rPPG signal, divide the single-cycle signal according to the segmentation result, and perform rPPG waveform screening on the segmentation result. Perform Tukey detection processing based on the screening results; Dual-channel feature fusion data prediction module: The dual-channel feature fusion data prediction module is used to construct a dual-channel feature fusion data prediction model by combining the analysis results of the rPPG signal segmentation and waveform selection modules; rPPG signal feature extraction and data prediction module: The rPPG signal feature extraction and data prediction module is used to input the dual-channel feature fusion data prediction model through training data for training. The trained dual-channel feature fusion data prediction model will be obtained and the pre-trained dual-channel feature fusion data prediction model will be obtained. The processed signal is input to the trained dual-channel feature fusion data prediction model for prediction, and then the feedback result is obtained. 7. An intelligent detection system based on rPPG technology according to claim 6, characterized in that the information data preprocessing module includes an image acquisition unit, an ROI area division unit, a channel data extraction unit and an rPPG signal preprocessing unit. : The image acquisition unit is used to collect real-time video of the user's palm movement through the camera, and extract each frame of the image in the collected video; The ROI area division unit is used to determine the hand landmarks in each frame of image collected in combination with the analysis results of the image acquisition unit, and locate 21 key points in the hand area based on the determination results; The channel data extraction unit is used to acquire RGB three-color channels in combination with the analysis results of the ROI area division unit, retain the G channel, and generate the original rPPG signal by calculating the average pixel value of the G channel; The rPPG signal preprocessing unit is used to separate the outline of the hand area from the background to obtain the background ROI through the canny edge monitoring algorithm, and correct the rPPG signal of the hand ROI through changes in the brightness of the background ROI. 8. A kind of intelligent detection system based on rPPG technology according to claim 7, characterized in that the rPPG signal segmentation and waveform selection module includes a single-cycle segmentation unit, an rPPG waveform screening unit and a base detection processing unit: The single-cycle segmentation unit is used to combine the analysis results of the information data preprocessing module to extract all zero-crossing points in the declining period of the original rPPG signal, and take the minimum value of the interval between two adjacent zero-crossing points as the starting point of a cycle. The original rPPG signal is segmented; The rPPG waveform screening unit is used to obtain sampling point data based on the normal pulse rate range of the human body, and perform preliminary screening of the analysis results of the single-cycle segmentation unit based on the obtained results; The base detection unit is used to perform base detection in combination with the analysis results of the rPPG waveform screening unit, and uses the number of sampling points of each single-cycle waveform as a basis for division to determine whether to retain the initially screened single-cycle waveform. 9. A kind of intelligent detection system based on rPPG technology according to claim 8, characterized in that the dual-channel feature fusion data prediction module includes a dual-channel feature fusion unit and a training dual-channel feature fusion unit: The dual-channel feature fusion unit is used to combine the analysis results of the rPPG signal segmentation and waveform selection modules, send the rPPG signal to the FNN multi-layer perceptron branch for feature extraction, and combine the extracted features through the multi-layer structure; The training dual-channel feature fusion unit is used to combine the analysis results of the rPPG signal segmentation and waveform selection modules, and send the rPPG signal to the CNN convolutional neural network module for multi-view feature extraction. 10. An intelligent detection system based on rPPG technology according to claim 9, characterized in that the rPPG signal feature extraction and data prediction module includes a feature fusion unit and a data prediction unit: The feature fusion unit is used to fuse the analysis results of the dual-channel feature fusion unit and the training dual-channel feature fusion unit; The data prediction unit is used to perform data prediction in combination with the analysis results of the feature fusion unit.