CN116327155A - Remote heart rate measurement method, device and equipment based on adaptive ROI selection - Google Patents

Abstract

The invention discloses a remote heart rate measurement method, device and equipment based on self-adaptive ROI selection. The method obtains multiple regions by segmenting each frame of facial images; calculates the reflection angle, G channel value and signal Noise ratio, build a signal-to-noise ratio estimation model, according to the signal-to-noise ratio estimation model, you can find out the region (ROI) that is most suitable for iPPG heart rate extraction in a frame of image, and then synthesize the signal to estimate the heart rate. This method mainly solves the situation that the tracking of the target area is lost due to facial movement, and this method provides a new idea for the dynamic recognition of the ROI area of the face with iPPG technology. Experimental results show that, in consideration of facial movement and light and dark changes, using the method of the present invention before and after processing can increase the signal-to-noise ratio by 0.3152db, and PET6 (the absolute value of error heart rate is less than or equal to 6) by 0.5651%.

Description

Remote heart rate measurement method, device and equipment based on adaptive ROI selection

technical field

The invention relates to the technical field of heart rate measurement, in particular to a remote heart rate measurement method, device and equipment based on adaptive ROI selection.

Background technique

With the epidemic of infectious diseases and the increase in the death rate of heart disease, people's attention to heart health has gradually increased. iPPG technology is a contactless and non-invasive technology that measures and monitors our heart rate without the sensor touching the skin. Through long-term, accurate and timely detection of the human body, combined with doctors' diagnosis, the fatality rate of heart disease can be greatly reduced. The principle of iPPGs is that they are derived from the faint intensity changes in the skin area each cardiac cycle, caused by different blood volumes over time. There are many remote measurement technologies radar, optical depth measurement, which have been used to measure heart rate (HR), blood pressure, respiratory rate (RR) and heart rate variability (HRV). Especially because of its non-skin contact characteristics, it can be well applied to drivers, newborns and patients with skin diseases, so it has attracted the interest of many researchers.

One of the important steps in iPPG heart rate measurement is the selection of ROI. The iPPG signal is generated by extracting image averages over a period of time for selected ROIs; this is the method most commonly used by researchers. Image averaging of one ROI frame will give the average G channel value of all G channel values in the ROI. Therefore, the inherently weak variation of the G channel value in the ROI is also averaged; this makes the signal of iPPG in the facial video not strong enough. Over the past decade, a large number of methods for selecting ROIs have been proposed.

Since Poh et. Cheeks, forehead, and facial regions below the eyes are successively proposed as ROIs. Then a more accurate ROI selection method via face segmentation is proposed. A ROI selection criterion using facial landmarks is also proposed by Lam et al., which uses random patches of landmarks in the facial region below the eyes and inside the lips. These methods select these ROIs at the first frame and do not change as the video changes. Therefore, these methods are not suitable for head rotation and unstable illumination.

In order to overcome the unstable illumination, a method of selecting ROI by weight is proposed. Kumar et al. proposed to select ROIs from the whole face with 20×20 pixel patches and use weights to estimate the best combination of ROIs. Ewa M. Nowara et al. The face is segmented into many regions using facial landmarks, the signal quality of each region is evaluated to select an ROI, and five background regions are selected to remove the distraction of illumination changes. This method has good results and is often used to select ROIs.

However, the method of selecting ROIs by weight usually needs to select a time window to calculate the selection result, and measure the best ROI in this time period. On the other hand, the length of this time window varies from 5 seconds to tens of seconds, and this method will fail because the subject's facial region is not always captured by the camera with the head rotation, when the subject When the face moves, the tracking of the target area will be lost, so the best ROI cannot be selected, resulting in heart rate measurement failure.

Therefore, when the subject's face moves, improving the robustness of the ROI in the process of remote heart rate measurement is a technical problem to be solved urgently in this field.

Contents of the invention

The main technical problem to be solved by the present invention is to improve the robustness of the ROI in the remote heart rate measurement process when the subject’s face moves, thereby improving the accuracy of the remote heart rate measurement. In order to solve this technical problem, the present invention provides a The remote heart rate measurement method, device and equipment based on adaptive ROI selection are combined with a new ROI selection method to measure the heart rate under the condition of head rotation.

According to a first aspect of the present invention, the present invention provides a remote heart rate measurement method based on adaptive ROI selection, comprising the following steps:

Continuously acquire multiple frames of facial images with the subject's head still;

Divide each frame of facial image into multiple regions, and calculate the reflection angle of each region;

Calculate the signal-to-noise ratio of each region of each frame of facial image;

Construct a signal-to-noise ratio estimation model through the reflection angle, G channel value and signal-to-noise ratio of each area of each frame of facial image;

Select ROI for each frame of facial image according to the signal-to-noise ratio estimation model;

Combining the ROIs of multiple frames of facial images to obtain iPPG signals;

The subject's heart rate was calculated from the iPPG signal.

Further, before the step of dividing each frame of facial image into multiple regions and calculating the reflection angle of each region, it also includes:

Apply MediaPipeFaceMesh to each frame of facial image acquired to detect 468 3D landmarks on the subject's face;

Taking the 3D landmark of the face as the reference point to calculate the surface normal vector and reflection angle of the face pixel, the set of possible surface normal vectors at the 3D landmark j is defined as:

表示从地标j到邻近地标i的矢量，/>

表示从地标j到另一邻近地标k的矢量，m是一个正整数，代表用于估计地标j可能的表面法线向量集合P_j的相邻地标的数量；In the formula,

represents a vector from landmark j to neighboring landmark i, />

Represents the vector from landmark j to another neighboring landmark k, m is a positive integer representing the number of neighboring landmarks used to estimate the possible surface normal vector set Pj of landmark _j ;

The most representative surface normal at landmark _j is selected from the set of surface normal vectors Pj using analysis of covariance

Compute the reflection angle θ _j of landmark j from the surface normal:

是地标j的表面法线，/>

是相机轴的单位矢量。in,

is the surface normal of landmark j, />

is the unit vector of the camera axis.

Further, the step of dividing each frame of facial image into multiple regions and calculating the reflection angle of each region includes,

Segment the subject's facial image in three steps, and divide it into N quadrilateral ROI parts;

The reflection angle _θm of the segmented region is given by:

Among them, m is the sequence number of the N segmented regions, θ _{m, n} are the normal angles of the four vertices of the nth region, and the reflection angle of each region is represented by gray scale.

Further, the step of segmenting the subject's facial image in three steps includes:

Remove eye, eyebrow and mouth landmarks;

Remove landmarks caused by facial expression movements, including around the corners of the eyes, lips and nose;

Connect the remaining landmarks with straight lines.

Further, the step of constructing a signal-to-noise ratio estimation model through reflection angles, G channel values and signal-to-noise ratios of each region of each frame facial image includes:

For each frame of facial image _collected , record the average G channel value G _N and the average reflection angle value θ N of N regions, and then record the frame number t∈{1,...,T*FPS} of the current facial image , store the recorded data information in an N×t×2 matrix, where T is the number of seconds of video capture, and FPS is the frame rate of camera capture;

Record the average G channel value, average reflection angle value and signal-to-noise ratio within the preset time window as a set of data;

The average G channel value G _N and the average reflection angle value of the two influencing factors are separately tested for central tendency to remove outliers in the data;

Non-linear curve fitting is performed on the data after removing outliers, and the signal-to-noise ratio estimation model whose input is the reflection angle and G channel value and the output is the signal-to-noise ratio is obtained.

Further, the step of selecting the ROI for each frame of facial images according to the signal-to-noise ratio estimation model includes:

Calculate the minimum number of frames that can calculate the human heart rate according to the camera frequency and the human heart rate;

When the number of consecutive occurrences of the highest SNR area estimated by the SNR model is greater than or equal to the minimum number of frames, ROI selection is performed.

Preferably, the number of divided regions is N=91.

According to a second aspect of the present invention, the present invention provides a remote heart rate measurement device based on adaptive ROI selection, comprising the following modules:

An acquisition module, configured to continuously acquire multiple frames of facial images of the subject's head at rest;

Segmentation module, is used for dividing each frame face image into a plurality of regions;

Calculation module, for calculating the reflection angle of each area;

The calculation module is also used to calculate the signal-to-noise ratio of each region of each frame of facial image;

A building block for constructing a signal-to-noise ratio estimation model through reflection angles, G channel values and signal-to-noise ratios of each region of each frame of facial images;

A selection module is used to select the ROI for each frame of facial images according to the signal-to-noise ratio estimation model;

Synthesis module, for carrying out signal synthesis to the ROI of multi-frame face image, obtains iPPG signal;

The calculation module is also used to calculate the subject's heart rate according to the iPPG signal.

According to a third aspect of the present invention, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor implements the described program when executing the program. The steps of the remote heart rate measurement method.

According to still another aspect of the present invention, the present invention also provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the remote heart rate measurement method are implemented.

The technical solution provided by the invention has the following beneficial effects:

The present invention mainly proposes a remote heart rate measurement method based on adaptive ROI selection. The method obtains multiple regions by segmenting each frame of facial images, and calculates the reflection angle, G channel value and signal-to-noise ratio of each region. A signal-to-noise ratio estimation model is constructed. According to the signal-to-noise ratio estimation model, the most suitable region (ROI) for iPPG heart rate extraction in a frame of image can be found, so as to synthesize signals for heart rate estimation. This method mainly solves the situation that the tracking of the target area is lost due to facial movement, and this method provides a new idea for the dynamic recognition of the ROI area of the face with iPPG technology. Experimental results show that, in the case of considering facial movement and light and dark changes, using the method of the present invention before and after processing, SNR (signal-to-noise ratio) is improved by 0.3152db, and PET6 (absolute value of error heart rate is less than or equal to 6) improves 0.5651%.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing and embodiment, in the accompanying drawing:

Fig. 1 is the overall flowchart of a kind of remote heart rate measurement method based on self-adaptive ROI selection of the present invention;

Fig. 2 is a schematic diagram of the process of adaptive ROI selection in the present invention;

Figure 3 is the overall pipeline of the present invention to generate an angle map from an image, where (a) is the calculation of the surface reflection angle, (b) is the SNR estimation method, (c) is the establishment of the SNR estimation model, and (d) is the ROI robustness choose;

Fig. 4 is the data schematic diagram of the present invention;

Fig. 5 is a schematic diagram of window sliding in the present invention;

Fig. 6 is the frequency spectrum (b) of U _t (f) measurement signal template (a) and IPPG signal of the present invention;

Fig. 7 is a box plot of the G channel value and the signal-to-noise ratio of the present invention, and the solid line is the median value, which is used to observe the overall trend of the results;

Figure 8 shows that the ROI changes frequently in several consecutive frames of images;

Fig. 9 is a schematic diagram of ROI robustness selection method;

10 is a schematic structural diagram of a remote heart rate measurement device based on adaptive ROI selection in the present invention;

Fig. 11 is a schematic structural diagram of an electronic device of the present invention;

Fig. 12 is a comparison chart of SNR and PET6 results before and after the ROI selection method of the present invention is used.

Detailed ways

In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation manners of the present invention will now be described in detail with reference to the accompanying drawings.

Please refer to Fig. 1, the present invention provides a kind of remote heart rate measurement method based on adaptive ROI selection, and this method comprises the following steps:

S1: Continuously acquire multiple frames of facial images with the subject's head still;

S2: Divide each frame of facial image into multiple regions, and calculate the reflection angle of each region;

S3: Calculate the signal-to-noise ratio of each area of each frame of facial image;

S4: Construct a signal-to-noise ratio estimation model through reflection angles, G channel values, and signal-to-noise ratios of each region of each frame of facial images;

S5: Select an ROI for each frame of facial images according to the SNR estimation model;

S6: performing signal synthesis on ROIs of multiple frames of facial images to obtain iPPG signals;

S7: Calculate the heart rate of the subject according to the iPPG signal.

Among them, the selection process of adaptive ROI is shown in Figure 2, which mainly includes four steps:

1) Segment the face into 91 regions and calculate the reflection angle of each region. 2) Calculate the SNR of each region; 3) Construct the SNR estimation model through the reflection angle and G channel value and the calculated SNR. 4) Robustly select the ROI according to the SNR estimation model.

Next, the key points in the implementation process of the present invention will be described in detail in conjunction with specific drawings:

(1) Face segmentation and reflection angle calculation:

As shown in Figure 3, the overall pipeline for generating angle maps from images is shown. We apply MediaPipeFaceMesh to each image frame to detect 468 3D landmarks on the subject's face, as shown in Figure 3(a). The 3D facial landmarks serve as reference points for the subsequent computation of surface normal vectors and reflection angles of face pixels (Fig. 2). Define the set of possible surface normal vectors at landmark j as P _j , namely

表示从地标j到邻近地标i的矢量，/>

表示从地标j到另一邻近地标k的矢量，m是一个正整数，代表用于估计地标j可能的表面法线向量集合P_j的相邻地标的数量；In the formula,

represents a vector from landmark j to neighboring landmark i, />

Represents the vector from landmark j to another neighboring landmark k, m is a positive integer representing the number of neighboring landmarks used to estimate the possible surface normal vector set Pj of landmark _j ;

Then, the most representative surface normal at landmark _j is selected from the set of surface normal vectors Pj using the analysis of covariance (ANCOVA) method. The reflection angle _θj of landmark j is estimated as:

是地标j的表面法线，/>

是相机轴的单位矢量。估计的坐标点的法向量如图3(b)所示。in,

is the surface normal of landmark j, />

is the unit vector of the camera axis. The normal vectors of the estimated coordinate points are shown in Fig. 3(b).

Next, the subject's face is segmented in three steps. 1) Remove eye, eyebrow and mouth landmarks, which are useless for BVP signal. 2) Remove landmarks caused by facial expression movements, including around eye corners, lips and nose. 3) Connect the remaining landmarks with straight lines. Therefore, the subject's face is segmented into 91 quadrilateral ROI parts, as shown in Fig. 3(c). Finally, the reflection angles of the segmented regions can be given by:

Where m is the serial number of the 91 segmented regions, θ _{m, n} is the normal angle of the four vertices of the nth region. The reflection angle of each region is expressed in gray scale, as shown in Fig. 3(d).

In the divided 91 areas, the signal-to-noise ratio needs to be used to measure the influence of different reflection angles and different light intensities on the signal quality. In the previous research, it has been proved that among the RGB three channels of the color camera, the G channel contains the heart rate information with the highest correlation with the BVP signal. Therefore, the signal strength of the G channel is used here to measure the signal-to-noise ratio.

For each frame of image collected, record the average G channel value G _N and the average reflection angle value θ _N of N=91 regions respectively, where N∈{1,...,91}, and then record the frame of the current image The number t∈{1,...,T*FPS}, so the data information can be stored in an N×t×2 matrix, where T is the number of seconds of video capture, and FPS is the frame rate of camera capture.

(2) SNR estimation:

A brief time-frequency analysis is performed on the acquired G channel signal. In order to ensure the effectiveness of the research method in a short period of time, as shown in Figure 5, 10 seconds is used as a window for sliding analysis, and the pulses acquired within 10 seconds The signal spectrum is normalized as the estimated heart rate signal, and then the average value of the measurement results within 10 seconds of contact with the pulse oximeter is used as the real heart rate signal. Since the face of the subject is required to remain still, the reflection angle within 10 seconds can be used The average value represents the reflection angle of the signal in this time period.

The signal-to-noise ratio (SNR) evaluation method of DeHaan et al. was used to assess the availability and quality of rPPG signals. Frequency-domain evaluation of the time-domain dataset within 10 seconds of filtering. This method calculates the energy around the harmonics and the remainder in the power spectrum; the ratio of the two is then expressed as the signal-to-noise ratio in dB. The formulas for calculating signal, noise and signal-to-noise ratio are as follows:

为iPPG信号的频谱，f为频率，单位为节拍/分钟(bpm)。所需的信号波段在一般人体脉搏频率范围内，即30bpm至240bpm。in

is the spectrum of the iPPG signal, f is the frequency, and the unit is beat/minute (bpm). The required signal band is within the general human pulse frequency range, ie 30bpm to 240bpm.

As shown in FIG. 6 , the dotted line part is the U _t (f) metric signal template, and the solid line part is the spectrum of the iPPG signal. The signal-to-noise ratio calculation uses a metric signal template, in the estimated heart rate signal spectrum, centered on the signal peak, with a range of 2.5 frequency units above and below, and centered on the pulse rate of the contact sensor, with a range of 5 frequencies above and below, to take into account Heart rate variability. The signal-to-noise ratio is measured by the energy ratio of the components inside and outside the template.

(3) SNR estimation model establishment

Record the data in the 10-second time window. Since the reflection angle and the average value of the G channel in the fixed area of the face will not change greatly when the face is still and the light source is constant, the average G channel within 10 seconds can be recorded. In order to establish the mathematical relationship between them, the box plots are made for these two influencing factors to test the central tendency separately, and the deviation in the data is removed. Group points to achieve the purpose of preliminary screening of data and reduce the interference of outliers on the data model. The data results of the G channel and SNR are shown in Figure 7. The "+" in the figure is an outlier point. By observing the trend of the median value, the processed data is fitted with a nonlinear curve. The reflection angle and SNR data are processed in the same way, and the final input is the reflection angle and G channel value, and the output is the signal-to-noise ratio estimation model.

(4) ROI Robust Selection

The obtained signal-to-noise ratio estimation model is directly applied to the collected simulated driving data set. In each frame of image, an area with the highest signal-to-noise ratio will be estimated, and the signal of the best area in each frame will be synthesized frame by frame to obtain the original iPPG signal, but when the signal is synthesized for the selected area, it will be The situation shown in Figure 8 occurs, that is, selecting different regions as ROIs in several consecutive frames of images, which will cause the failure of the iPPG signal synthesized by these frames of images, resulting in a decrease in the signal-to-noise ratio. The obtained value is the change of light intensity caused by non-physiological phenomena, and the expected physiological information cannot be obtained from it. Therefore, it is necessary to increase the robustness of the selection of the ROI area here.

Here, the highest frequency of the normal heart rate is 4Hz, and the period is 0.25 seconds. When a camera with a frequency of 30Hz is used to collect, at least 7.5 frames of images are needed to restore and calculate the heart rate of the human body. As shown in Figure 9, a threshold F is set here to limit the transformation of the ROI region. First, the best region ID within 1 second is stored. When the best region is continuously the same and the number is greater than the threshold F (F≥8), Make ROI shifts.

A remote heart rate measurement device based on adaptive ROI selection provided by the present invention is described below, and the remote heart rate measurement device described below and the remote heart rate measurement method described above can be referred to in correspondence.

As shown in Figure 10, a remote heart rate measurement device based on adaptive ROI selection includes the following modules:

The acquisition module 001 is used to continuously acquire multiple frames of facial images of the subject in a state where the subject's head is still;

Segmentation module 002, for dividing each frame of face image into a plurality of regions;

Calculation module 003, for calculating the reflection angle of each area;

The calculation module 003 is also used to calculate the signal-to-noise ratio of each region of each frame of facial image;

Building block 004, for constructing the signal-to-noise ratio estimation model by the reflection angle, G channel value and signal-to-noise ratio of each region of each frame facial image;

Selection module 005, for selecting ROI for each frame of face image according to the signal-to-noise ratio estimation model;

Synthesis module 006, for carrying out signal synthesis to the ROI of multi-frame face image, obtains iPPG signal;

The calculation module 003 is also used to calculate the subject's heart rate according to the iPPG signal.

Based on but not limited to the above-mentioned device, the calculation module 003 is also used for:

Apply MediaPipeFaceMesh to each frame of facial image acquired to detect 468 3D landmarks on the subject's face;

Taking the 3D landmark of the face as the reference point to calculate the surface normal vector and reflection angle of the face pixel, the set of possible surface normal vectors at the 3D landmark j is defined as:

表示从地标j到邻近地标i的矢量，/>

表示从地标j到另一邻近地标k的矢量，m是一个正整数，代表用于估计地标j可能的表面法线向量集合P_j的相邻地标的数量；In the formula,

represents a vector from landmark j to neighboring landmark i, />

Represents the vector from landmark j to another neighboring landmark k, m is a positive integer representing the number of neighboring landmarks used to estimate the possible surface normal vector set Pj of landmark _j ;

The most representative surface normal at landmark _j is selected from the set of surface normal vectors Pj using analysis of covariance

Compute the reflection angle θ _j of landmark j from the surface normal:

是地标j的表面法线，/>

是相机轴的单位矢量。in,

is the surface normal of landmark j, />

is the unit vector of the camera axis.

Based on but not limited to the above devices, the segmentation module 002 is specifically used for:

Segment the subject's facial image in three steps: remove the landmarks on the eyes, eyebrows and mouth; remove the landmarks caused by facial expression movements, including the corners of the eyes, lips and around the nose; connect the remaining landmarks with straight lines, and finally segment into N= 91 quadrilateral ROI sections;

The reflection angle _θm of the segmented region is given by:

Among them, m is the sequence number of the N segmented regions, θ _{m, n} are the normal angles of the four vertices of the nth region, and the reflection angle of each region is represented by gray scale.

Based on but not limited to the above-mentioned device, the building block 004 is specifically used for:

For each frame of facial image _collected , record the average G channel value G _N and the average reflection angle value θ N of N regions, and then record the frame number t∈{1,...,T*FPS} of the current facial image , store the recorded data information in an N×t×2 matrix, where T is the number of seconds of video capture, and FPS is the frame rate of camera capture;

Record the average G channel value, average reflection angle value and signal-to-noise ratio within the preset time window as a set of data;

The two influencing factors of the average G channel value and the average reflection angle value are separately tested for central tendency to remove outliers in the data;

Non-linear curve fitting is performed on the data after removing outliers, and the signal-to-noise ratio estimation model whose input is the reflection angle and G channel value and the output is the signal-to-noise ratio is obtained.

Based on but not limited to the above devices, the selection module 005 is specifically used for:

Calculate the minimum number of frames that can calculate the human heart rate according to the camera frequency and the human heart rate;

When the number of consecutive occurrences of the highest SNR area estimated by the SNR model is greater than or equal to the minimum number of frames, ROI selection is performed.

As shown in FIG. 11 , a schematic diagram of the physical structure of an electronic device is illustrated, and the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630, and a communication bus 640, wherein, The processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute the steps of the remote heart rate measurement method, which specifically includes: continuously acquiring multiple frames of facial images in a still state of the subject's head; dividing each frame of facial images into multiple region, and calculate the reflection angle of each region; calculate the signal-to-noise ratio of each region of each frame of facial image; construct the signal-to-noise ratio estimation model by the reflection angle, G channel value and signal-to-noise ratio of each region of each frame of facial image; The signal-to-noise ratio estimation model selects the ROI for each frame of facial images; combines the ROIs of multiple frames of facial images to obtain iPPG signals; calculates the subject's heart rate according to the iPPG signals.

In addition, the logic instructions in the above-mentioned memory 630 may be implemented in the form of software functional units and when sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. And aforementioned storage medium comprises: U disk, removable hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random15 Access Memory), magnetic disk or optical disk etc. various mediums that can store program codes .

In yet another aspect, the embodiment of the present invention also provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned remote heart rate measurement method are implemented, specifically including: continuously acquiring the subject's head Multi-frame facial images in a static state; divide each frame of facial images into multiple regions, and calculate the reflection angle of each region; calculate the signal-to-noise ratio of each region of each frame of facial images; pass each frame of facial images through each region The reflection angle, G channel value and signal-to-noise ratio construct the signal-to-noise ratio estimation model; select the ROI for each frame of facial image according to the signal-to-noise ratio estimation model; perform signal synthesis on the ROI of multiple frames of facial images to obtain the iPPG signal; according to the iPPG The signal calculates the subject's heart rate.

In the present invention, by implementing the above-mentioned remote heart rate measurement method, device and equipment based on adaptive ROI selection, and the corresponding storage medium, it is possible to find out the most suitable area for iPPG heart rate extraction in a frame of image as the ROI, and perform multi-frame images. The iPPG signal is obtained by signal synthesis, and the iPPG signal is analyzed in the frequency domain using Fourier transform, and the frequency corresponding to the point with the largest amplitude is selected as the heartbeat frequency. This method mainly solves the situation that the tracking of the target area is lost due to facial movement, and this method provides a new idea for the dynamic recognition of the ROI area of the face with iPPG technology. The experimental results are shown in Figure 12, and Figure 12 shows that, in consideration of facial movement and light and dark changes, using the method of the present invention before and after processing can improve the signal-to-noise ratio SNR (signal-to-noise ratio) by 0.3152db, PET6 (The absolute value of the heart rate error is less than or equal to 6) increased by 0.5651%.

It should be noted that, as used herein, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or system comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or system. Without further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system comprising that element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third etc. does not indicate any order and these words may be construed as designations.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process conversion made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. A remote heart rate measurement method based on adaptive ROI selection, comprising the steps of:

continuously acquiring multi-frame facial images of a subject in a head static state;

dividing each frame face image into a plurality of areas, and calculating reflection angles of the areas;

calculating the signal-to-noise ratio of each region of each frame face image;

constructing a signal-to-noise ratio estimation model through the reflection angle, the G channel value and the signal-to-noise ratio of each region of each frame surface image;

selecting the ROI for each frame of facial image according to the signal-to-noise ratio estimation model;

performing signal synthesis on the ROI of the multi-frame facial image to obtain an iPG signal;

the heart rate of the subject was calculated from the iPPG signal.

2. The method of claim 1, wherein before the step of dividing each frame image into a plurality of regions and calculating the reflection angle of each region, further comprising:

applying MediaPipe FaceMesh to each acquired frame of facial images to detect 468 three-dimensional landmarks of the subject's face;

taking a three-dimensional landmark of a face as a reference point for calculating a surface normal vector and a reflection angle of a face pixel, and defining a set of possible surface normal vectors at the three-dimensional landmark j as:

in the method, in the process of the invention,

represents a vector from landmark j to adjacent landmark i, ">

Representing a vector from landmark j to another adjacent landmark k, m being a positive integer representing a set of possible surface normal vectors P for estimating landmark j _j Is a number of adjacent landmarks;

from a set of surface normal vectors P using covariance analysis _j Is the most representative surface normal at the selected landmark j

Calculating the reflection angle θ of landmark j from the surface normal _j ：

Wherein,,

is the surface normal of landmark j, +.>

Is the unit vector of the camera axis.

3. The method of claim 2, wherein the step of dividing each frame image into a plurality of regions and calculating the reflection angle of each region comprises,

dividing the facial image of the subject into three steps of segmentation into N quadrilateral ROI parts;

reflection angle θ of the divided region _m The method is given by the following steps:

where m is the number of N divided regions, θ _m，n Is the normal angle of the four vertices of the nth region, and the reflection angle of each region is represented in gray scale.

4. A remote heart rate measurement method according to claim 3, wherein the step of dividing the facial image of the subject in three steps comprises:

removing landmarks on the eyes, eyebrows, and mouth;

removing landmarks caused by facial expression movements, including corners of the eyes, lips, and surrounding the nose;

the straight line connects the remaining landmarks.

5. The method according to claim 1, wherein the step of constructing a signal-to-noise ratio estimation model from the reflection angle, the G-channel value, and the signal-to-noise ratio of each region of each frame image comprises:

for each acquired frame image, recording average G channel values G of N areas _N And the average reflection angle value theta _N The number of frames T e {1,..once again, T x FPS } of the current face image is recorded, the recorded data information is stored in an N x T x 2 matrix, wherein T is the number of seconds of video acquisition, and FPS is the frame rate of camera acquisition;

recording an average G channel value, an average reflection angle value and a signal to noise ratio in a preset time window as a group of data;

carrying out concentration trend test on two influence factors of the average G channel value and the average reflection angle value independently, and removing outliers in the data;

and performing nonlinear curve fitting on the data with the outliers removed, and obtaining a signal-to-noise ratio estimation model with the input quantity being the reflection angle and the G channel value and the output quantity being the signal-to-noise ratio.

6. The method of claim 1, wherein the step of selecting an ROI for each frame image based on a signal-to-noise ratio estimation model comprises:

calculating the minimum frame number capable of calculating the heart rate of the human body according to the frequency of the camera and the heart rate of the human body;

and when the number of times of continuous occurrence of the signal-to-noise ratio highest region estimated by the signal-to-noise ratio model is larger than or equal to the minimum frame number, selecting the ROI.

7. A remote heart rate measurement method according to claim 3 or 5, characterized in that the number of zones N = 91.

8. A remote heart rate measurement device based on adaptive ROI selection, comprising the following modules:

the acquisition module is used for continuously acquiring a plurality of frames of facial images of the head of the subject in a static state;

a segmentation module for segmenting each frame face image into a plurality of areas;

the calculation module is used for calculating the reflection angle of each region;

the calculating module is also used for calculating the signal-to-noise ratio of each region of each frame face image;

the construction module is used for constructing a signal-to-noise ratio estimation model through the reflection angle, the G channel value and the signal-to-noise ratio of each region of each frame of the image;

the selection module is used for selecting the ROI for each frame of the facial image according to the signal-to-noise ratio estimation model;

the synthesis module is used for carrying out signal synthesis on the ROI of the multi-frame facial image to obtain an iPG signal;

the calculation module is also used for calculating the heart rate of the subject according to the iPG signal.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the remote heart rate measurement method of any one of claims 1-7 when the program is executed.

10. A storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the remote heart rate measurement method as claimed in any one of claims 1 to 7.

Images