Open AccessArticle

Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review

Steffen Leonhardt

^1,*,†

Lennart Leicht

^1,† and

Daniel Teichmann

Chair for Medical Information Technology, Helmholtz-Institute for Biomedical Engineering, RWTH Aachen University, D-52076 Aachen, Germany

Institute for Medical Engineering and Science (IMES), Massachusetts Institute of Technology (M.I.T.), Boston, MA 02139, USA

Author to whom correspondence should be addressed.

^†

These authors contributed equally to this work.

Sensors 2018, 18(9), 3080; https://doi.org/10.3390/s18093080

Submission received: 13 June 2018 / Revised: 22 August 2018 / Accepted: 30 August 2018 / Published: 13 September 2018

(This article belongs to the Special Issue Noncontact and Unobtrusive Biomedical Sensors 2018)

Download

Browse Figures

Versions Notes

Abstract

This review provides an overview of unobtrusive monitoring techniques that could be used to monitor some of the human vital signs (i.e., heart activity, breathing activity, temperature and potentially oxygen saturation) in a car seat. It will be shown that many techniques actually measure mechanical displacement, either on the body surface and/or inside the body. However, there are also techniques like capacitive electrocardiogram or bioimpedance that reflect electrical activity or passive electrical properties or thermal properties (infrared thermography). In addition, photopleythysmographic methods depend on optical properties (like scattering and absorption) of biological tissues and—mainly—blood. As all unobtrusive sensing modalities are always fragile and at risk of being contaminated by disturbances (like motion, rapidly changing environmental conditions, triboelectricity), the scope of the paper includes a survey on redundant sensor arrangements. Finally, this review also provides an overview of automotive demonstrators for vital sign monitoring.

Keywords:

unobtrusive monitoring techniques; car seat, driver state monitoring; vehicle; electrocardiogram; steering wheel; capacitive electrocardiogram; magnetic impedance; eddy currents; ballistocardiography; photoplethysmography; PPG imaging; infrared thermography; RADAR

1. Introduction

To assess the state of a patient, clinicians classically employ four primary vital signs [1] for monitoring at the bedside: pulse (heart rate, HR), breathing rate (respiratory rate, RR), blood pressure (BP), and body (core) temperature (BT). Usually, oxygen saturation (SpO₂) is added as a fifth vital sign, and some authors even propose to add another three vital signs, namely pain, level of consciousness and urine output [2], despite the fact that they are more difficult to measure continuously.

These classical methods to measure vital signs typically require close physical interaction with the patient (cables, electrodes, cuffs and pneumatic lines, etc.), making the measurement suitable for a clinical environment, but limiting its usability for applications outside the hospital. Therefore, technologies for unobtrusive and contact-free monitoring of vital signs have attracted a lot of attention in recent years [3,4,5]. In fact, such techniques would open up completely new applications, including home care and sports, but also supervision during operation of vehicles.

Due to its impact on road safety and casualty statistics, car driver state monitoring has been an active area of research for a long time, and physiological sensors like electrocardiogram (ECG) and electroencephalogram (EEG) have often been employed in trials. For many years, researchers looked at e.g., stress [6,7] and drowsiness [8]. Based on video cameras and classical image processing technologies instead of physiological sensors, however, especially drowsiness and fatigue detection have been targeted much longer and relied on, for example, eye blink analysis and percentage of eye closure (PERCLOS) assessment (see [9,10]).

From a practical perspective, cable-bound solutions are not an option for driver state monitoring outside experimental trials. Hence, unobtrusive and non-contact driver state monitoring using physiological information is currently gaining increasing attention [11]. This trend is pushed in part by the ongoing R&D efforts towards autonomous driving and also by the demographic changes producing more elderly and potentially diseased drivers [12]. However, while simple schemes like the analysis of steering patterns to assess driver state have already been commercialized by several car manufacturers, the analysis of true physiological information coming from ECG, EEG and photoplethysmography (PPG) sensors still is in the research stage [13].

Hence, to support the ongoing developments of the scientific and industrial community, this review paper summarizes the state of the art in vital sign monitoring for this challenging application and discusses maturity as well as pros and cons of the individual concepts. It also points at potential solutions to increase coverage and vital sign accuracy resulting from motion artifacts, namely artifact compensation and/or signal and sensor fusion.

2. Unobtrusive Sensing of Physiological Parameters and Features

As mentioned above, the term vital signs typically refers to five physiological parameters: HR, RR, BP, BT and SpO₂. Since no attempts have been conducted to equip cars with unobtrusive BP monitoring, this vital sign will not be further addressed in this review.

In 2012, Scalise et al. [14] presented a comprehensive overview of contact-based (ECG and PPG) and non-contact (CW and PW (UWB) radar, laser vibrocardiography [15], magnetic induction, capacitive ECG, ultrasound transmitter) heart monitoring systems.

In 2014, Kranjec et al. [16] presented a review on heart rate and heart rate variability measurements. Apart from conventional measurement approaches (ECG and PPG), experimental approaches were also presented, including the measurement by thermal imaging, imaging using visible light, optical vibrocardiography, doppler radar, capacitive ECG and heart rate extraction from speech. Advantages and disadvantages of the experimental approaches were discussed. In the same year, Kranjec et al. [17] presented a feasability study for several experimental heart rate measurement methods. The authors separated the methods into two categories, depicted as “direct methods” measuring the electrical heart activity (capacitive ECG) and “indirect methods” measuring mechanical displacement of the patients body (microwave and ultrasound distance measurement, signal from microphone).

In 2015, Brüser et al. [3] from RWTH Aachen University provided a methodological review on nine different non-contact sensing modalities targeting the monitoring of cardiorespiratory activity. As described in this paper, there are three main categories of physical effects that can be sensed unobtrusively: bioelectrical, mechanical and thermal effects. Figure 1 provides an overview of the different effect categories, and how they relate to organ activity as well as to the measured vital signs.

The activity of the heart can be observed by measuring body surface potentials which are generated by the electrical excitation of heart muscle cells. The measurement of these body surface potentials is called ECG or in the case of capacitively coupled measurement capacitive ECG (cECG), see Section 3, and provides extensive information about the cardiac cycle.

Any intrathoracic organ motion will have mechanical effects. Such mechanical effects include the displacement of organ boundaries, deformation of surfaces and the displacement of fluids and may modulate the displacement of the body surface and the superficial perfusion, but also the intrathoracic impedance distribution. Body surface displacement can be obtained by radar, ballistocardiography (BCG), seismocardiography (SCG) (see Section 4) or video (see Section 5) and has been used for HR as well as RR monitoring. The superficial perfusion and its effect on the optical properties of the skin is the basis for PPG and PPG imaging (PPGi). These techniques provide information about the cardiac cycle and SpO₂ and are described in detail in Section 5. While variations in intrathoracic impedance distribution can also be observed with bioimpedance spectroscopy (BIS) and impedance cardiography (ICG), up to today only magnetic induction (MI) measurements allow unobtrusive (i.e., electrode-free) impedance analysis. In fact, MI has already shown its capability to measure HR and RR (see Section 6).

The body surface temperature, which can be monitored in an unobtrusive way by radiation-based thermography as described in Section 5, is modulated by changes in the body’s core temperature by direct conduction of heat between the organs and blood to the skin. Next to this, the body surface temperature is also modulated by heart and lung activity via heat convection by blood and air and, therefore, allows RR monitoring. In principle, body surface-based HR measurement could also be possible, but today’s thermography devices do not provide the necessary sensitivity.

Figure 1 covers unobtrusive sensors and the underlying physiological effects in general. Correspondingly, in the following chapters, we will cover the historical developments and the state-of-the art of these technologies in more detail, but specifically illuminate the current state of unobtrusive sensing inside a car.

3. ECG Monitoring in a Car

The electrocardiogram (ECG) is a biopotential, with the body floating with respect to the ground. Hence, its measurement requires at least two electrodes and a differential amplifier. The ECG reflects electrical and—to some extent—mechanical function of the heart muscle. It is fair to say that it is the most valuable physiological signal when it comes to heart diagnostics [18,19].

Classical ECG monitoring employs well-conducting adhesive electrodes made from silver with a thin silverchloride surface coating being in direct contact to the skin. Such electrodes are coupled to the body surface by well-conducting gels and are typically cable-bound, which prohibits regular use in non-clinical scenarios. Electrochemically, such electrodes are termed non-polarizable. The combination of a difficult to dissolve metal and its metal chloride make these electrodes less sensitive to ion concentration changes and hence their polarization is very stable. The electrode mainly behaves like a resistor.

Plain metal electrodes (like stainless steel) are different in that they represent polarizable dry electrodes. While cheap and simple, their polarization depends on the salt concentrations of the surroundings (e.g., skin) and hence their polarization is not stable. Electrically, such electrodes mainly behave like capacitors.

Low conduction or capacitive electrodes have no direct galvanic contact to the body. Typically, capacitive electrodes are separated from the body by cloth or other textile layers. As the measured ECG signals are at least one order of magnitude smaller than with galvanic electrodes, capacitive electrodes typically have an operational amplifier directly at the electrode. The compensation of common mode by an active driven ground is much more important than in the galvanic case.

For the car, electrodes can be placed at various locations, including the steering wheel (conductive), the backrest (capacitive, both thoracic or lumbar) and the seat (capacitive). For reduction of common mode, driven ground electrodes can be added. While thoracic placement of low-contact electrodes has the advantage of a larger signal strength, this location may also be the one which is subject to the biggest motion artifacts. Figure 2 gives an overview of published electrode locations for heart rate monitoring in the car.

3.1. Conductive ECG Monitoring

Placing electrodes on the steering wheel has been one of the early research targets. The available literature dates back at least a decade. In an attempt to quantify driver stress levels, in 2007, Jeong and colleagues [20] from Yonsei University in Korea investigated driver task load and stress response looking at heart rate variability (HRV) obtained with a steering wheel-based pair of dry ECG electrodes.

Similarly, in the same year, Lee and coworkers from Seoul National University in Korea reported about their study using dry electrodes integrated into the steering wheel [21]. These tape-type dry electrodes were made of copper tape (3M). Three male students were recruited as test drivers and asked to drive a route of 16 km in about 40–50 min. Heart rate variability was assessed using a modified Tompkins algorithm for R-peak detection and computation of R-peak intervals, heart rate, SDNN, RMSSD and pNN50. While just an abstract without providing any details, Hu and colleagues also reported on the integration of ECG into the steering wheel in 2008 [22].

In 2010, Shin et al. [23] and later Jung et al. [24] from Pukyong National University in Busan, Korea, presented steering wheel electrodes based on conductive fabric. In parallel, a multisensor system allowing to measure SpO₂, temperature and skin temperature at the steering wheel of a Mercedes S-Class vehicle (DAIMLER AG, Mercedesstraße 137, D-70327 Stuttgart, Germany) was published by Heuer and colleagues [25] from Karlsruhe Institute of Technology, Germany. Around the same time, D’Angelo and colleagues from the Technical University of Munich demonstrated a multisensor system integrated into the steering wheel of a BMW 730d test vehicle (BMW AG, Petuelring 130, D-80788 München, Germany) [26,27] featuring SpO₂ and conductive ECG sensing as well. The authors used a design similar to [20], integrating the electrodes into the steering wheel, the shift lever and the left arm rest. However, the precise electrode position was not revealed in this work.

In 2012, Gomez-Clapers and Casanella [28] from the Universitat Politecnica de Catalunya in Barcelona, Spain, presented a steering wheel-based ECG-measurement demonstrator which was based on dry ECG electrodes and wireless communication. Similarily, in that year, Silva and colleagues [29] from IT/IST-UTL in Lisbon, Portugal, demonstrated that dry Ag/AgCl electrodes on a steering wheel can produce ECG measurements of similar quality as traditional electrodes.

Another Korean project from Pukyong National University was presented in 2014. Jung and colleagues [24] proposed a design featuring a steering wheel with two conductive fabric electrodes, similiar to [29]. This electronic circuit design was based on an MSP 430 microcontroller and a Chipcon CC2420 RF Transceiver (both manufactured by Texas Instruments Inc., P.O. Box 660199, Dallas, TX, USA) for wireless data transmission at 2.4 GHz.

In 2015, Eßers et al. of Takata corporation (Takata corporation, Tokio, Japan) presented a steering wheel with sensors measuring the heart frequency. The actual integration is not well documented in the publication; it can only be assumed that it is similar to Lee [21] and Silva [29].

Figure 3 gives an overview of different electrode placements.

We note that ECG monitoring using steering wheel-based approaches is a feasible option for heart rate tracking but requires both hands to touch two different conductive parts of the wheel. Hence, significant steering movements with changes of the location of gripping and the quite common habit of steering with one hand pose a problem for this technique.

Hybrid ECG Monitoring

A somewhat different research approach was followed by Matsuda and Makikawa from Ritsumeikan University in Shiga, Japan. In 2008, these authors proposed using a conductive steering wheel and combine it with a capacitive electrode located in the driver seat [32] (see Figure 3c). This way, the habit of steering with one hand can be better accomodated. This proposal also combines the concepts of conductive measurements with capacitive sensing, which may justify the name hybrid sensing.

One year later, the idea of hybrid ECG sensing using the steering wheel as one electrode and the seat as the second (capacitive) electrode, was also proposed by Baek and coworkers, see [34], next to redundant sensing and sensor fusion.

In 2013, the idea of a steering wheel electrode measuring against a capacitve electrode in the seating area was again presented by Xu and Ta [33] from the Central South University, Hunan, China, possibly without knowledge of Matsuda’s work.

3.2. Capacitive ECG Monitoring

Unlike the classical conductive electrodes, low-contact electrodes do not rely on a galvanic skin contact, but achieve the signal coupling by means of a high-impedance or even capacitive coupling. In the case of capacitive electrodes, the surface is electrically insulated and remains stable in long-term applications. This concept is of course only of theoretical nature, since a purely capacitive electrode would require an infinitively high ohmic resistance—which no real electrode can provide. Thus, all existing capacitive electrodes are in fact low-contact electrodes. Consequently, they should also be named as such. However, in most of the recent literature, they are entitled as capacitive electrodes. For the sake of consistance, we will also use the term capacitive electrode in that synonymous sense, describing an electrode with a high-impedance coupling to the patient.

A timeline of the development of capacitive ECG systems is given in Table 1. Capacitive (insulated) electrodes date back more than 50 years and were first described by Richardson [35]. Early work regarding the integration of capacitive ECG measuring systems into objects of daily life (beds) was performed by Ishijima [36]. In his work, the author used conductive dry textile electrodes acting as an underlay or pillow. Proper ECG detection during sleep was reported. However, due to potential direct skin contact, the patient’s coupling may not have been exclusively capacitive.

After the millenium, Park and coworkers from Seoul National University in Korea began to explore integration into other objects of daily life. For example, in 2004 Lim et al. demonstrated the integration of capacitive electrodes into a bathtub [37] and also in 2004 a toilet seat was presented [38]. In 2006, this group published the integration of capacitive ECG electrodes into a chair [39]. Later, Lim et al. [40] and also Wu and Zhang [41] reported about integration of dry and capacitive ECG into beds.

At RWTH Aachen University in Aachen, Germany, the first applications of capacitive ECG electrodes were published in 2006 [56] and 2007 [57]. The so-called Aachen Smart chair is an office chair equipped with two solid copper plate electrodes integrated into the backrest. Beneath the seating area, hidden by the commercial textile cover, a large driven ground electrode is located. It acts as an active reference electrode. The backrest electrodes are coated by a protective black acrylic paint to achieve a purely capacitive coupling. Note that a similar design has been proposed in 2006 by Lim et al. [39].

In June 2008, capactive ECG monitoring reached the automotive domain, when the first paper proposing capacitive ECG sensing while driving a car was published by Leonhardt and Aleksandrowicz [42]. In this particular design, solid copper plate electrodes coated with a protective acrylic paint were placed on the backrest of a car seat at a 45° angle, with the aim of improving the signal-to-noise ratio (SNR) by simulating a typical heart axis. A driven right leg electrode (referred to as driven ground plane) was also placed in the backrest around the electrodes, as in this design the seat was already carrying an EMFi^TM foil (Emfit Ltd., Konttisentie 8, FI-40800 Vaajakoski, Finland) for simultaneous BCG measurements.

This design can be seen as one of the ancestors from which, over the years, a large variety of seats have evolved. Apart from differences in the the design of the electronics, they differ by the number of electrodes, their size, shape, location and material. Most, but not all, designs employ a driven reference electrode. An overview of several designs and their release date is presented in Figure 4.

Later in 2008, Chamadiya and colleagues from the Karlsruhe Institute of Technology (KIT) presented a concept integrating textile capacitive electrodes into a W221 Mercedes S-Class seat [43]. In this specific design, the electrodes were placed horizontally and rather lateral as the chosen seat already carried several other technical devices (ventilator, massage unit). However, this lateral electrode placement seems suboptimal as it may result in reduced body contact areas and subsequent lower coverage.

In 2011, Eilebrecht and colleagues from RWTH Aachen University published a multi-electrode design with six rectangular metal plate electrodes integrated into the backrest of a Ford S-Max driver seat [44]. This arrangement has the advantage that, when looking at different electrode pairs, selection of the strongest signal is possible. Implicitly, this allows for adapting to different heart axes and also to the torso size of different drivers.

With n = 7 test drivers, the practical usability of this design was tested on the Ford proving ground in Lommel (Belgium). Overall, ECG coverage during driving was 85% with three layers of clothing and moved up to 93% when limiting the allowed clothing to two layers [44]. Urban driving was found to be more challenging than smooth driving on a German highway. In specific drivers, the nervous reaction to fear (increased heart rate) and dedicated pathologies like extrasystoles were clearly visible.

Using this seat, Wartzek et al. compared the ECG measurement performance and coverage during Highway and city traffic in n = 5 test drivers [45]. Later in that year, the same group [46] described in detail that motion artifacts and especially their effective combination with local as well as global triboelectricity play an important role in capacitive monitoring and to date limit the robustness of this technique.

In a joint effort by the Karlsruhe Institute of Technology (KIT) and Daimler (DAIMLER AG, Mercedesstraße 137, D-70327 Stuttgart, Germany), textile integration into the car seat was further demonstrated by Chamadiya et al. in the same year [58]. A two electrode design featuring two textile electrodes placed horizontally and rather lumbar has been presented which provided decent ECG measurements in pilot trials.

Around that time, Schumm and colleagues from ETH Zurich in Switzerland obtained cECG test equipment from the Philips Chair at RWTH Aachen University. Based on the European SEAT project (EU 6th Framework Program, contract number: AST5-CT-2006-030958), the authors investigated if capacitive ECG measurements would be feasible for passenger monitoring in an airplane seat [47,62]. In their 2012 publication, Schumm et al. pointed out that the signal quality of unobtrusive sensing during various passenger activities might not always be sufficient for continuous monitoring [47]. The authors hence introduced a signal class label that reflects the signal quality and called it the quality label (QL).

In addition in 2012, Schneider et al. from the FZI Research Center for Information Technology in Karlsruhe, Germany, developed a framework for vital sign measurements inside a vehicle including a textile seat cover [48]. The design features two textile electrodes located horizontally in the lumbar region of the back and a driven seat electrode above the two active sensing electrodes. The design was demonstrated in an Audi Q5.

In the same year, Jung and coworkers from Pukyong National University presented another capacitive ECG monitoring system with two horizontal electrodes integrated into a car [49]. In this design, the active measurement electrodes were again planar copper plates located horizontally in the backrest, while the driven right leg electrode was made of conductive fabrics and located on the seat. In the presented trial, the authors investigated HRV analysis on measurements obtained from n = 5 subjects without any heart disease.

At the 2012 EXPO in Chicago, IL, USA, Plessey Ltd. (Plessey Semiconductors Ltd., Plymouth PL6 7BQ, UK) announced the availability of their capacitive ECG sensor system EPIC™. About two years later, the EPIC™ system was tested in an office chair [63]. In addition, a three-electrode system integrated into the car seat was announced in the media [59].

In 2014, Leicht et al. demonstrated that a local release of water vapor can dramatically increase the SNR of capacitive ECG [60]. As an underlying principle, it was shown that the increase of local air humidity can improve the ohmic coupling (hence the coupling will not be exclusively capacitive anymore). At the same time, this allows static charge difference induced by triboelectric effects to flow to ground. It hence reduces the impact of triboelectricity on capacitive monitoring. Leicht et al. proposed using conductive textile electrodes that are permeable for water vapor and to place chambers with silica gel as a source or sink for humidity behind the textile electrodes, but also other concepts for humidification are conceivable [64]. Figure 2h presents the arrangement of the electrodes used for prooving the concept.

In 2015, the RWTH Aachen group presented a multi-electrode design integrated into a Ford S-Max commercial test vehicle (Ford Motor Company, Dearborn, MI, USA). While in this design the electrodes remained solid metal, their shape was changed to round and the contract area was reduced as compared to previous designs. Note that the textile driven ground has now been completely integrated into the seat. In the same year, some of these authors started clinical trials with survivors of heart attacks or heart surgery in the Rosenquelle rehabilitation hospital in Aachen. Results from this study have been published in [55,61]. In this trial, patients (n = 10) were asked to drive in a driving simulator equipped with capacitive ECG in the seat and classical ECG monitoring for reference.

In addition in 2015, the approach of electrode humidification was evaluated by other groups. Fong and Chung [50] described special electrode designs allowing to release some humidity directly at the spot. Weder et al. [51] presented a textile breast belt incorporating a water reservoir used to moisten the electrodes. However, while highly integrated, these concepts are open-loop and provide no definition of the acting local humidity.

Hence, in 2017, Leicht et al. from RWTH Aachen University disclosed a closed-loop concept for feedback control of local humidity [53]. As storage and release of water vapor in silica gel is dependent on the local water vapour pressure, but is also a function of temperature, we proposed to establish closed-loop temperature control as a method to release water vapor from the gel and hence to enhance SNR. The corresponding car seat demonstrator is shown in Figure 4h.

In 2016, Plessey introduced WARDEN^TM, a six-electrode seat cover including a driven ground electrode built into the main electronic unit located at the opposite site of the backrest [52]. It can be used to easily retrofit car seats for contactless ECG recording and connects to in-vehicle data systems via Bluetooth. In late 2017, a six-electrode design featuring round capacitive electrodes in the backrest was presented to the public by the Belgian research organisation IMEC [54,65]. The overall design has a similiar look as the design presented in [60]. However, in this specific seat, radar sensors were integrated as well, which allows sensor fusion (see Section 9).

To simplify any comparison of designs, Table 2 compares specific features of capacitive ECG electrode placements for car seats as reported in the literature to date.

4. Ballistocardiography

Ballistocardiography (BCG) is an old medical sensing technique, with early BCG recordings dating back more than 100 years [66,67]. BCG allows monitoring of cardiac (and usually respiratory) activity unobtrusively. Typically relying on external pressure or strain-gauge sensors, a ballistocardiograph records the vibrations which are caused by the mechanical activity of heart and lung, as well as the momentum of the blood pulse traveling down the aorta. Hence, the diagnostic information is completely different from a classical electrocardiogram (ECG) which records electrical activity of the heart muscle.

In the late 1950s, Scarborough et al. introduced different categories of BCG measurements [68] and defined three axes of measurement for translational BCG recordings: longitudinal (head-foot, also called craniocaudal), transversal (side to side), and dorsoventral (back to chest). We note that typically the craniocaudal momentum is much bigger than the transversal and the dorsovental component. We also note that, to measure a BCG signal, contact (mechanical coupling) is required.

The early BCG recording systems were typically of the longitudinal kind [66,67]. For a standing person, this corresponds to the z-axis (see Figure 5a). Weighing scales which recently received increased attention [69] also measure along this axis. However, bed-based BCG systems [70] measure along a combination of the transversal and dorsoventral axes (see Figure 5b). If the coordinate system is not rotated with the person, but considered a fixed base coordinate system, then the dorsoventral axis corresponds to the z-axis as well.

For the bed, sensors can be placed above or between mattresses (for example, quasi-piezoelectric ferroelectret foils), inside the mattress (optical sensors), on the slatted frame (employing e.g., strain-gauge sensors) or in the bedposts (using, for example, pressure sensors).

If pressure or accelaration sensors are attached to the chest to record the heart motion (“thoracic sensors” in Figure 5), the resulting special form of BCG signal is referred to either as

Phonocardiogram, or
Apexcardiogram, or
Kinetocardiogram, or
Seismocardiogram.

BCG has gained renewed interest in recent years [71], mainly because of the simple and cheap instrumentation and the potential of this technique being used outside classical medical scenarios, for example in sleep analysis. In 2015, a review on recent advances was provided by Inan et al. [72].

However, the BCG signal is much more variable than the ECG signal and also dependent on the position of a (sleeping) body. Due to this high inter- and intravariability, proper heart rate detection is more difficult than ECG analysis. Despite these problems, a very powerful algorithm for individual heart rate detection was proposed by Brüser et al. [73].

The application of BCG sensing for driver monitoring has a lot of similarity with BCG monitoring in a chair. Generally, in this position, the craniocaudal BCG signal component is measured, while the dorsoventral signal component is much more difficult to measure as there may not always a proper mechanical coupling between the torso and the backrest, especially for thoracic sensor locations. Figure 6 sketches the options.

Early attempts to integrate BCG sensors into car seats started about 10 years ago. In 2008 and again in 2011, the placement of BCG sensors in the seating area of a SMART™ co-driver seat (DAIMLER AG, Mercedesstraße 137, D-70327 Stuttgart, Germany) was successfully demonstrated [42,74]. In this demonstrator, a quasi-piezoelectric ferroelectret EMFi^TM mat (Emfit Ltd., Konttisentie 8, FI-40800 Vaajakoski, Finland) was employed as a BCG sensor. However, data supplied in [74] indicates that motor vibrations may make BCG monitoring in a running car difficult.

In order to fight driver fatigue (which is said to account for 20–35% of serious accidents), the HARKEN (HARKEN: Heart and respiration in-car embedded non intrusive sensors, funded by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no.: 286265) project proposed to monitor both cardiac and respiratory activity by means of smart materials embedded in the seat cover and the safety belt of the car [75]. However, no further results on these BCG principles were published.

In 2015, the Faurecia Group (Faurecia S.A., Nanterre, France), a large automotive supplier, presented a car seat called Active Wellness^TM [76], which claims to detect the traveler’s drowsiness or stress level and then takes countermeasures to relieve the driver from these conditions [77]. The underlying sensing concept employs strain-gauge BCG sensors in the seating to detect cardiac and respiratory activity of drivers and/or occupants.

Very recently, in 2018, Wusk and Gabler presented heart and respiratory rate estimation obtained with a BCG sensor in a Ford Mustang passenger seat [78]. As a sensor, the authors proposed using a “fluid filled bladder connected to a solid-state pressure transducer”. In their study, data from eleven volunteers were collected. Under these controlled laboratory conditions, the authors were able to extract both vital signs with decent accuracy.

5. Optical Methods

Radiation has the advantage of unnoticeable transport of information from a driver or passenger without contact. Hence, as long as the optical pathways are free, optical modalities potentially offer unobtrusive sensing from a distance that is attractive for automotive applications, especially since, in modern cars, cameras are already used for external obstacle detection, etc.

In the recent past, there has been a boom in camera-based-technologies and applications for vital-sign monitoring. We note that, according to the utilized frequency ranges, the individual camera technologies differ with respect to materials, prices and size. Figure 7 highlights the frequency spectra suitable for cameras.

From Figure 7, we can conclude that there are three frequency ranges suitable for driver status monitoring: visible light (VIS, wavelength 350–740 nm), near-infrared light (NIR, wavelength 740–1000 nm) and far-infrared light (FIR, wavelength > 1 μm). Table 3 compares specific features of the different camera concepts and designs reported in the literature to date.

In the VIS and the NIR range, light can interact with living tissues. Due to conservation of energy, there are three principal modes of interaction: absorption, reflection and transmission. Transmitted light may travel through the tissue on a straight pathway (as in transmissive PPG), but may also be scattered. In fact, in skin tissue, the dominating effect of tissue–photon interaction is scattering (about 50 times stronger than absorption).

As for any optical method, a camera-based driver or passenger monitoring inside a car requires free line of sight. As the face offers uncovered skin, looking at the face of drivers and passengers easily comes to mind. Since current cockpit design requires free line of sight for the driver, in addition, the following camera locations have been explored (see Figure 8).

If the angle of attack

ρ

> 0, the nostrils may become visible, which is interesting for FIR respiratory monitoring. We note that FIR imaging is often referred to as infrared thermography (IRT).

5.1. Photoplethysmography

Photoplethysmography (PPG) is a well-known technique allowing measurement of blood volume changes in the skin or in a small part of the body [79]. In classical (local) PPG, the light source typically is a light-emitting diode (LED, either in the VIS and/or NIR frequency range) while the sensor is a photodetector (PD). In transmissive PPG (tPPG), the light transmitted through the tissue is detected by a photodetector opposite the light source, while in reflective PPG (rPPG), the photodetector detects light that is scattered and/or reflected from the skin (see Figure 9).

Note that, for tPPG, transmission length is limited to a few mm up to cm (allowing typical locations like earlobe or finger), while, in rPPG, penetration depth typically is up to 3 or 4 mm. Reflective PPG relies on sufficient ambient or actively injected light which is then modulated by blood flow in the upper and lower plexus of the skin capillaries. As a result, on their (typically banana-shaped) way through the skin, about 1 out of 20,000 photons reaching the skin from the light source will finally follow a path that allows them to exit from the skin and fly back to the detector collecting this flow of photons.

PPG Monitoring in the Car

In 2010, the integration of reflective PPG monitoring into the steering wheel of a BMW 650i test car was demonstrated by d’Angelo and colleagues [26]. The authors used a reflective pulse oximetry sensor connected to an OEM III module manufactured by Nonin Medical (Nonin Medical Inc., 13700 1st Avenue North, Plymouth, MI, 55441-5443 USA). In the same year, Heuer et al. demonstrated SpO₂ sensing in the steering wheel of a Mercedes S-Class demonstrator cockpit (DAIMLER AG, Mercedesstraße 137, D-70327 Stuttgart, Germany) [25], next to ECG monitoring in the steering wheel and capacitive ECG monitoring in the seat. These authors employed a Nonin 8000R reflective probe. Last but not least, also in 2010, Shin and colleagues [23] reported on a steering wheel design featuring conductive ECG fabrics and reflective PPG using a single frequency (infrared light at 940 nm).

However, a general drawback of this approach is that 100% sensor coverage can not be guaranteed. When integrating reflective PPG sensors into the wheel, the driver has to voluntarily touch/cover the hidden PPG sensor with perfused skin. Hence, this technique may be especially useful for on-demand applications.

5.2. PPG Imaging

Using a camera and a proper light source (VIS and/or NIR, either ambient or artificial illumination), the reflective PPG principle can be expanded to remote camera-based monitoring from a distance. Figure 10 explains the principle.

Note that both the light source and the camera can now operate remotely from a distance but require a free line-of-sight.

Camera-based PPG methods have recently attracted increasing scientific attention. The principle of PPG imaging is a simple one: as the heart beats, the apparent colour of the skin changes due to blood pulsation. In principle, remote PPG imaging allows unobtrusive measurements of heart rate and derived parameters like heart rate variability (HRV) or pulse transit time (PTT) [80], for example when different locations are related to each other or when combined with ECG.

To our knowledge, the earliest PPGI applications were published by Blazek and colleagues from RWTH Aachen University starting in 2000 and centered around heart rate and perfusion measurements at the leg [81,82,83].

In 2005, Wieringa et al. [84] reported on first steps towards SpO₂ imaging using a multi-wavelength approach. Employing a time-lapse image acquired from a CCD camera, Takano and Ohta [85] developed a non-contact and non-invasive device in 2007, which was able to measure both the respiratory and pulse rate simultaneously. In the same year, Zheng and Hu reported [86] that by using LEDs at three wavelengths (VIS and NIR), they were able to extract a PPG signal from skin with a CCD USB camera.

In 2008, Verkruysse et al. [87] from the University of California at Irvine and the Norwegian University of Science and Technology at Trondheim were among the first to demonstrate that simple low-cost digital video cameras and ambient daylight illumination might be sufficient to capture a remote reflectance photoplethysmography (rPPG) waveform in controlled scenarios. The authors reported that they were able to extract both heart rate and respiratory rate.

As mentioned in Wang’s Ph.D. thesis [88], Philips Research started camera-based work on what they call the Vital Signs Camera as early as 2004 [88]. Starting in 2012, Philips Research has been offering this software as an app for download to a smartphone for several years. While this applet was mainly targeting private customers for fun applications, Philips Intellectual Property and Standards department recently began licensing the Vital Signs Camera for commercial customers. Intended applications include wellness products and automotive applications (see https://www.ip.philips.com/licensing/program/115).

Significant work on camera-based monitoring has been performed by Tarassenko’s group from the University of Oxford, U.K. (see, e.g., [89]). As an example, in 2014, Villarroel et al. from that group reported on heart rate estimation in neonates using CCD cameras and visible light [90]. Similarily, Guazzi and coworkers published their results on SpO₂ imaging using RGB cameras in 2015 [91]. In order to estimate blood pressure from local pulse transit time signals, Daly presented the results of their clinical studies involving volunteers and dialysis patients in 2016 [80].

While not directly refering to the term “PPG” as the underlying measurement principle, in 2012, Wu and colleagues [92] have reported that, by employing Eulerian magnification, the subtle colour changes in the skin due to perfusion can nicely be visualized in relaxed subjects. Of course, the fascination of these visualizations results from the fact that the human eye can not see these sublte color changes directly. This explains why the corresponding videos (available on the author’s websites and on YouTube) have attracted signifcant attention.

Unfortunately, by now, a variety of names referring to remote PPG imaging exist in the literature (see Table 4).

While currently the terms remote PPG and sometimes remote PPG imaging/remote PPGI are becoming more and more popular, the term PPG imaging (PPGI) is actually much older and was originally coined at RWTH Aachen University in Germany (see [81,82,83]). From the authors perspective, this name is quite descriptive and clearly refers to the underlying methodology of using a camera for extracting physiological information. To end the Babylonian confusion on the name, it is suggested to have a naming debate in the scientific community. In any case, the abbreviation “rPPG” is an unlucky choice, as traditionally rPPG and tPPG refer to reflective and transmissive PPG, respectively.

All examples mentioned so far do not directly relate to monitoring in a car. Hence, the following subsection will look at literature with explicit automotive applications.

PPGI Vital Sign Monitoring in the Car

In 2014 and 2017, respectively, Zhang et al. from the Shenzhen Institutes of Advanced Technology and the Chinese University of Hong Kong, China [117,118] presented their results on PPGI monitoring of heart and respiratory rate by RGB image processing and independent component analysis (ICA) in the visible light frequency range. While the earlier publication focused on laboratory experiments, the later publication of this group included pilot measurements on one subject while driving in a real car.

In 2015, Kuo et al. [102] from Monash University in Melbourne, Victoria, Australia, reported on their results using imaging PPG. In their study, 10 participants completed an on-road driving task. A commercial heart rate monitor was used as a gold standard. As reported, IPPG produced a valid heart rate estimation for only 4 out of 10 participants.

In cooperation with Volvo (Volvo Car Corporation, Torslanda, Gothenburg, Sweden), in 2015 Rahman and colleagues from Mälardalen University in Sweden published a comprehensive review paper on the state-of-the-art in driver heart rate monitoring using cameras for the visible light range [119]. In the following year, this group suggested using a webcam for real-time heart rate monitoring [120]. While the paper focused on algorithms and lab experiments, the suggested applications included driver monitoring.

In 2016/2017, a similiar approach to heart rate detection in the car was presented by Wu et al. [107] from the National Chiao Tung University, Hsinchu, Taiwan. The authors also used RGB cameras and presented their results on lab experiments as well as test drives employing empirical mode decomposition (EMD) for filtering motion artifacts.

Similarly, in 2017, Blöcher et al. [97] from Karlsruhe Institute of Technology presented a camera-based online heart rate monitoring system for in-vehicle applications using such an off-the-shelf webcam as well. Their system reached a sensitivity of 93% under laboratory conditions (constant lightning and resting conditions). Their system was also tested under real-world driving conditions and showed encouraging results. However, due to a insufficient number of test persons (two), an additional evaluation on a larger test cohort is required to obtain statistically relevant results.

Employing cameras with active illumination in the NIR frequency has been proposed as well, but not too many reports exist thus far. In 2015, Gücüyener proposed to integrate a near infrared (NIR) camera with active LED illumination into the rear mirror and employ it for heart beat detection during driving [121]. In 2016, Tayibnapis and colleagues from the Daegu Gyeongbuk Institute of Science and Technology Daegu, South Korea, reported on using non-contact PPG in the NIR range for driver fatigue classification using support vector machine (SVM) algorithms.

5.3. Far Infrared Imaging (Thermography)

In contrast to VIS and NIR imaging, far infrared (FIR) imaging, also referred to as infrared thermography (IRT), is a passive method relying solely on the radiation emitted from the driver/passenger as a function of temperature. Hence, its biggest advantages are that no energy is transferred to the subject under test and that it can be used in complete darkness, which made this method a key technology for early adoption for military and border control applications.

While not targeting driver or passenger state monitoring at the time, early research applications of face thermography included quantifying stress and emotion [122,123] as well as vital sign monitoring (respiration) [124]. As originally demonstrated in 2007 by Pavlidis and coworkers [125], using an IRT camera to track the nostrils with a proper angle of attack allows for determining breathing rate and depth in an adult (see also [126]).

However, depending on the size of the subject, the angle of attack and the individual tidal volume, the temperature changes during inspiration and expiration may be small, usually ± 1 deg. C or less, as demonstrated in Figure 11.

Note that, in this example, the mean temperatures of the region of measurement (ROM) reached 31.17 deg. C and 31.44 deg. C during inspiration and expiration, respectively. Hence, any FIR camera used for this application must be sufficiently sensitive. Recently, the RWTH Aachen group has been able to demonstrate that it is possible to follow the nostrils during movements as well [127].

While in general vital sign monitoring using IRT is currently advancing, according to our knowledge, automotive applications of IRT-based vital sign measurements have not yet been reported. However, there is at least one publication aiming at quantifying facial expressions using IRT during driving (see Section 5.4.3).

5.4. Other Camera-Based Automotive Monitoring

5.4.1. Automotive Monitoring Using Visible Light (VIS)

With respect to early camera applications in the automotive arena, driver drowsiness detection as well as driver attention monitoring were among the first topics of interest.

As early as 2003, Smith et al. introduced an algorithm to monitor driver face orientation based on a single camera [128]. Shortly after, in 2004, Rongben and colleagues [129] reported on mouth tracking and shape analysis using CCD cameras to quantify drowsiness employing neural network (Perceptron-type) classifiers.

In 2007, Trivedi and coworkers investigated camera-based systems aiming at correlating the visual contextual information of vehicle interior and vehicle exterior [130]. Topics like occupant position and posture, driver distraction and estimation of face orientation were examined using a multi-camera system during daylight to look in and look out of the vehicle (LiLo concept). Shortly after, in 2008, Bergasa et al. reported on driver inattention analysis using a dual-camera system in the visible light range [131]. Their face detector was based on the known Viola and Jones face detection algorithm [132].

Capturing and quantifying emotions from facial images is a task that has applications far beyond the automotive scenario, but is of course quite important to access driver state. In fact, stress and emotions might be quantifiable based on vital signs (like heart and breathing rate and derivatives), but also by image processing. Correspondingly, in 2016, Manoharan and colleagues published an article on detecting stress and pain of drivers by analyzing facial expressions based on tracking dedicated fiducial points [133]. For classification of emotions, the authors employed support vector machines.

5.4.2. Automotive Monitoring Using Near Infrared Light (NIR)

Near infrared camera investigations aiming at driver drowsiness detection during darkness date back at least 20 years. As early as 1994, Ueno and colleagues from the Nissan Research Center reported on their experiences in face analysis and eye tracking using a CCD camera and NIR illumination [10].

For truck driving, Grace et al. [134] reported on using the well-known PERCLOS (percentage eye closure) index, a measure of drowsiness associated with slow eye closure, in 1998. Covering the state-of-the-art a decade ago, a survey on drowsiness detection and driver state monitoring using NIR cameras was published by Wang et al. in 2006 [135]. Interestingly, these authors state that the best techniques for fatigue measurement are “based on physiological phenomena like brain waves, heart rate, pulse rate and respiration. These techniques are intrusive, since they need to attach some electrodes on the drivers, causing annoyance to them” [135]. Hence, at the time, NIR-based computer vision was proposed as an (inferior?) alternative instead. That way, using a CCD camera with NIR illumination as a physiological sensor fuses the advantages of both approach.

For many years, the group of Jean-Philippe Thiran at EPFL in Lausanne, Switzerland has worked on signal processing on facial images in the car. A special focus has been the quantification of emotion, see, e.g., https://transport.epfl.ch/en/research-overview/vehicles-infrastructures/intelligent-vehicles/face_analysis/emotion-detector/. Note that, in 2014, Gao et al. published on using near-infrared (NIR) cameras on the dashboard of a car for that purpose [136]. Based on the tracking of 49 face landmarks, the authors report a correct classification rate of up to 90% in their experimental setting.

In 2016, Tayibnapis and coworkers [115] presented their results on the fatigue analysis of eyes, mouth, and head-pose using a NIR camera. For face analysis, the Viola–Jones algorithm [132] was employed. In addition, heart rate and heart rate variability were extracted using no-contact PPG imaging. Hence, their proposed system features two major contributions, namely “(1) integrating various facial feature methods using an NIR camera, and (2) measuring the driver’s physiological state with a non-contactable method” [115].

5.4.3. Automotive Monitoring Using Far Infrared Light (FIR)

As already mentioned in Section 5.3, both facial expression as well as vital sign monitoring (mainly respiratory activity) attracted the interest of several researchers quite early. Starting around the millenium, there has been continuing research activity in the area of quantifying emotions using far infrared imaging [123,137,138,139,140,141,142,143,144]. Among others, one rationale has been the observation that the perfusion of the forehead supplied by branches of the A. facialis changes during some emotional arousal affecting the thermal signatures of the face.

One of the first publications on using FIR imaging in the car was published by the University of Iowa in 2009 [145]. In fact, Reyes and coworkers used data obtained from n = 16 middle-aged test drivers during 2 h experimental drives in a driving simulator to quantify stress and emotions. Here again, perfusion changes in the eye corners and the root of the nose are key in the analysis.

5.5. Image Fusion

In automotive applications, the concept of image and sensor fusion dates back more than 20 years. In the beginning, imaging systems were often fused with other modalities (like radar) and were targeting mainly object recognition outside the car.

For example, in 1997, Zomotor et al. [146] described the fusion of infrared, laser and radar-based range information using a Kalman filter to improve the range detection at longer look-ahead distances required for “autonomous intelligent cruise control” (AICC) applications. A few years later, Fang et al. [147] used the fusion of radar and binocular stereo videos for a depth fusion algorithm. The improved depth information was then employed in a target segmention algorithm (like identification of other cars, pedestrians, etc.), which is required for autonomous driving. Furthermore, they demonstrated that the fusion of two camera image streams can be used for range detection, making an additional radar system obsolete.

She et al. [148] presented a system that fuses color and shape information for improved vehicle detection rates in 2004. In 2007, an interesting approach using a multi-wavelength and multi-camera approach for pedestrian detection was described by Krotosky et al. [149]. Their system employs two color and two infrared cameras in the front of the vehicle. Note that pedestrians are producing very different features in the two modalities, allowing for obtaining a unique fingerprint for pedestrian classification.

Apart from these systems directed outwards of the vehicle, systems monitoring the vehicle occupants have also been presented.

In 1999 and 2000, Pavlidis et al. [150,151] introduced a system using near-infrared imaging for vehicle occupant counting. Two near-infrared camera streams, below and above 1400 nm, were fused to segment vehicle occupants. The system was able to successfully distinguish between humans and mannequins. Possible applications include the surveillance of freeway lanes reserved for car pools.

Camera systems directed inside the vehicle can also be used to improve the safety of occupants. In 2002, Owechko et al. [152] described a vision system aiming at improving the safety of airbag deployment. Ignited airbags inflate very rapidly; they can cause severe head trauma if the vehicle occupant is too close to the device. In addition, airbags can actually kill small children in infant seats if they are placed in the front passenger seat. In such situations, the airbags must not be deployed. Owechko’s system uses two cameras and extracts four features from the images (range, edges, motion, shapes). Classifiers are used to distinguish the vehicle occupant, e.g., driver in normal position or infant in infant seat. A fusion algorithm weights the classifier outputs according to the confidence and decides about the airbag deployment.

Two years later, Trivedi et al. [153] extended the work of Owechko et al. by combining stereo and thermal video to determine the vehicle occupant position. As discussed before, thermal imaging has the benefit that it does not require illumination.

In 2004, Gyaourova et al. [154] combined visible and infrared images to improve face detection in people wearing eyeglasses. Infrared imaging has the benefit of being independent of illumination, but cannot pass through spectacles. On the other hand, visible imaging operates through eyeglass, but requires illumination. Therefore, a combination of both modalities can improve coverage rates. This, of course, is important to determine the PERCLOS ratio (see above) or the direction of view.

A yawning detection as a sign of driver fatigue was presented by Li [155] in 2009. It uses a real-time adaptation of the region of interest (ROI) of a camera. A low-resolution camera captures a larger region of the driving compartment. A face detection is performed on that image stream and a second camera is directed onto the mouth of the driver according to the detected face. It is argued that, by this approach, the driver’s mouth can be recorded at a higher resolution, providing a more accurate measure of fatigue.

Capturing and quantifying emotions from facial images is a task that has applications far beyond the automotive scenario, but is of course quite important to access driver state. For generic facial expression recognition, in 2016, Corneanu and coworkers [156] published a comprehensive survey on available databases containing faces in the RGB, NIR and far infrared range. Interestingly, there is far more data publically available in the RGB band than for IRT. A few databases (like NVIE and KTFE) offer labeled images in both frequency bands [156]. Recently, Wang et al. presented a deep learning approach for fusing visible and infrared images for facial expression recognition [157].

5.6. Video Motion

The general principle of ballistocardiography (BCG) has already been introduced in Section 4 of this review paper. While some groups including Verkruysse et al. [87] and Blanik et al. [158] have reported on BCG artifacts during PPG imaging before, Moco and colleagues [95] were the first to systematically quantify BCG artifacts for PPGI of the face.

However, treating the BCG component as an artifact in PPGI is not the only option in making use of this information. When heart rate estimation and not local PPG monitoring is of interest, the BCG artifact in image streams may actually be utilized as a separate signal source.

With respect to using cameras as a BCG sensor, in 2013, Balakrishnan and colleagues from the M.I.T. demonstrated that BCG can also be performed by video stream analysis [159]. In fact, these computer scientists demonstrated that proper face detection (OpenCV 2.4 Viola Jones algorithm [160]), subsequent OpenCV Lucas Kanade tracking of feature points and PCA decomposition allows for finding and magnifying the normally unnoticeable motions of the head in temporal image sequences, in that way similar to the color magnification performed by Wu et al. [92].

In 2015, Hoog Antink and colleagues have shown that robustness and coverage of heart rate estimation can be increased by fusing PPG and BCG information extracted from visible video streams [161].

Despite motion artifacts that can be expected during driving, an automotive application of camera-based BCG seems possible, but—according to our knowledge—has not been demonstrated thus far.

6. Magnetic Induction

In general, human tissue is an inhomogeneous and anisotropic electrical medium. Within the thorax of a human, the distribution of electrical impedance is modulated by cardiorespiratory activity due to organ motion and deformation as well as fluid displacement. Magnetic Induction (MI) monitoring is a no-contact method to measure this intrathoracic impedance distribution.

Basically, the principle is based on electromagnetic coupling between a coil and the thorax in its vicinity as illustrated in Figure 12.

When an alternating current (typically 10–20 MHz) flows through the air coils, this current induces a primary field affecting the tissue underneath, which results in eddy currents flowing on closed pathways perpendicular to the field. The eddy currents are modulated in amplitude, direction and orientation by air and blood volume changes, displacement of organ boundaries, and microscopic processes. As with any current flow, the eddy currents create a secondary field, which carries the information of the modulation. The penetration depth of this active method is estimated to be approximately the diameter of the coil. A brief introduction to the physics of the signal generation can be found in [162].

Single coil systems typically operate as part of a resonant oscillator circuit. The coil serves as a transmitting as well as a receiving coil and is a frequency-determining part of the oscillatory circuitry. When the coil’s reflective impedance changes due to the secondary field affecting the primary one, this will cause a change in the oscillatory frequency. This frequency change then correlates with physiological activity and is utilized as the sensor output signal.

By contrast, multi-coil approaches feature several coils, typically one transmitter and one or two receiver coils. In multi-coil arrangements, it is necessary to position the receiver coils such that the primary field induction is as small as possible while the secondary field induction should be as large as possible. When only one receiver coil is used, it has to be brought into an orthogonal orientation with the transmitter coil in order to ensure that the primary field does not induce any voltages into the receiver coil. When two receiver coils are used, they are typically connected as gradiometers such that the primary field is (almost) completely compensated. In [163], planar gradiometer arrangements have been proposed, but they have not been integrated into vehicles yet.

To date, three articles have been published which report on the use of MI monitoring within a car. Figure 13 gives an overview about the different locations where these MI sensors were located.

In 2011, Walter et al. presented the integration of a single coil MI sensor into the backrest of a vehicle seat [74] and demonstrated its usability for respiration monitoring. While these results were preliminary, this was the first time that the MI technique has been employed inside a vehicle. In 2017, Leicht et al. presented the PhysioBelt device [164], which comprised an MI device with a flexible coil sewn into the seat belt. In the same year, Vetter et al. [165] reported on progress with gradiometer technology for automotive integration. This new design features a gradiometer approach with one sending and two receiving coils forming a vertical gradiometer (see Figure 14).

While the former two devices only measured respiration, the latter MI device was also able to record a cardiac signal. While these results are preliminary and have not yet been thoroughly tested in larger studies, they at least demonstrate both ventilation and cardiac contraction can be observed in a MI signal measured in the backrest, with ventilation dominating by one order of magnitude in signal strength.

7. Radar-Based Methods

The term radar actually is an abbreviation refering to radio detecting and ranging. This technique uses high-frequent electromagnetic waves that are emitted from a transmitter (Tx) and reflected by the chest surface thus carrying information on human chest displacement and, to some extent and as a function of penetration depths, on inner organ movements. The reflected waves are sensed by a receiver (Rx) and analysed, e.g., in shift of phase, etc. Radar methods are mainly carrying information on movement, which can of course be caused by both respiratory and cardiac activity. As mentioned in [166], the heart can typically cause chest displacements up to ± 0.1 mm in relaxed humans (see also [167]), and respiration displacements of between 0.1 mm and several millimeters. Due to this, radar methods provide similiar information as ballistocardiography (BCG) methods.

There are two general radar techniques: the pulsed ultra wide-band (UWB) radar, often with a separate transmitter and receiver antenna, which is based on measurements of time-of-flight and derived quantities, and the continuous wave (CW) radar which basically relies on the Doppler effect.

In the U.S., the frequency range for UWB radar is regulated by the Federal Communications Commission (FCC) spans 3.1 GHz to 10.6 GHz. By contrast, in Europe, the allowed frequencies range from 3.4 GHz to 4.8 GHz and from 6 GHz to 8.5 GHz [168].

Vital sign monitoring in the frequency range of a few GHz is possible and has been demonstrated for UWB radar mode in many applications [169], including rescue missions (to locate human subjects burried under earthquake rubble or being covered under snow after avalanche accidents [170]). However, in this frequency range, antennas have been quite bulky and also the allowed transmission power is rather limited. Higher frequencies (like microwave radar in the near 60 GHz ISM band) offer some advantages such as higher sensitivity and smaller antennas [171]. However, at such high frequencies, the penetration depth of electromagnetic radiation into biological tissue is almost negligible and the radar system is mainly a BCG sensor. While this may be acceptable for respiration tracking, heart-related motions are very small and hard to detect.

Early attempts to use radar for measuring vital signs date back to the 1970s. In fact, in 1971, Caro and Bloice presented a 10 GHz radar system mounted on top of an infant incubator aiming at apnea detection [172].

In 2009, Ichapurapu and colleagues reported an accuracy of ± 1 heartbeat per minute for heart rate detection using a 2.4 GHz CW radar system [173]. Similiarly, in 2009, Massagram and colleagues demonstrated in 12 healthy subjects that heart-rate variability (HRV) and respiratory sinus arryhtmia (RSA) can be monitored using continuous Doppler radar in the 2.4 GHz range [174].

In 2010, Lazaro and colleagues shared their experiences using an impulse radio UWB radar system prototype manufactured by Geozondas, Lithuania, for vital sign monitoring [166]. In this work, the authors emphasized at the mathematics used for heart and respiration signal extraction. They also reported that the measurements were in compliance with the FCC mask (allowed dissipation power) almost everywhere in the 3.1 to 10.6 GHz frequency range.

In 2011, Scalise et al. presented a measurement method for respiration using a 6 GHz CW-radar system [175]. Despite successfully detecting respiration activity up to 2.5 m, the use in automotive environments might be difficult since the system uses a large double ridge horn antenna.

Bechet et al. [176] reported in 2013 on their results improving the signal quality of CW Doppler radar vital sign tracking at 2.4 GHz using the MUSIC (Multiple Signal Classification) algorithm.

For automotive applications, various sensor locations are possible, including integration into the cockpit, the safety belt, the steering wheel and into the backrest (see Figure 15).

In 2015, Vinci et al. proposed using continuous wave (CW) radar sensors for monitoring vital signs while driving [177]. The authors of this paper investigated two measurement scenarios: in the first scenario, a 24 GHz six-port configuration of the radar system is placed approximately half a meter in front of the driver, with the antenna pointing directly on its thorax. This distance between sensor and driver would allow future integration of the antenna, for example into the steering wheel. In the second measurement scenario, the same radar sensor has been installed on the backrest of the car seat (which is more difficult for such high frequencies, as mentioned above). Here, the authors intend to create a “biometric driver seat” while at the same time eliminating the necessity of a sensor integrated in the steering wheel. Therefore, the radar antenna is not aiming on the thorax of the person under test, but on his or her back.

A similiar approach was proposed by Lee and colleagues in 2016 [178]. Similiar to Vinci’s second scenario, the authors proposed to place a 24 GHz CW sensor in the backrest of the car seat. These authors also point out that, using CW radar, motion artefacts may significantly interfere with the frequency characteristics of the displacement signal and may hence be difficult to extract by signal analysis in the frequency domain (FFT).

In 2016, Izumi et al. presented a 24 GHz microwave Doppler sensor circuit placed on the chest of a volunteer sitting in a car seat. As demonstrated, the location is proper for seat belt integration [179]. Very recently, Schires and colleagues [180] demonstrated feasibility of heart and respiratory rate tracking using a new UWB radar design placed in the backrest of a car seat. A Ventricorder Impulse Radar module manufactured by the Norwegian company Novelda (Novelda AS, Gjerdrums vei 8, 0484 Oslo, Norwegen) was employed. In order to increase the penetration depth of the electromagnetic impulses, the authors chose a carrier frequency of 3.9 GHz using pulses of approximatively 2 GHz bandwidth.

8. Challenges of In-Vehicle Measurements

8.1. Motion Artifacts

When measuring biosignals unobtrusively, coverage and accuracy can be relatively low compared to clinical modalities. By using methods for sensor fusion, both quantities can be improved by exploiting redundancies in multiple measurement channels. When considering contact-free monitoring of vital signals, motion artifacts always pose a problem [181]. Compared to classical, cable-bound sensing, they are always more severe.

In particular, radar-based measurement systems are susceptible to motion artifacts, since they derive a vital sign by means of a distance measurement. To cope with this problem, Tang et al. developed a radar system using a transmit and re-transmit approach. The system uses two antennas, one being located in front and one being located behind the patient [182]. Vital signs, in particular respiration and heart rate, will move the patients towards or away from both antennas at the same time (in phase), while artifact movements will cause a out-of-phase movements (towards one antenna and away from the other antenna, and vice versa). The system makes use of these circumstances by first transmitting and receiving a signal with one antenna and then feeding the received signal into the other antenna. When received by this antenna, the signal contains in-phase signal parts associated with the vital signs (constructive interference) and out-of-phase signal parts from the artifact movement (destructive interference). Thus, despite artifact movement, the system can successfully detect heart and respiration activity.

Furthermore, most other modalities also have specific problems. Camera-based methods, for example, only work if the subject under test is in the field of view. Capacitive biopotentials may suffer from common mode problems resulting from maladjusted clothing combinations (e.g., polyester–cotton, silk–cotton, etc.).

Ottenbacher and Heuer [183] have analysed the contribution of lateral and transversal changes in distance between electrode and skin. While global triboelectricity equally affects all electrodes and acts as a common-mode interference, local transversal changes in distance directly affect the coupling capacity of the electrode–skin combination. This can be modelled as a time-dependent coupling capacity.

Wartzek et al. demonstrated in 2011 that additional factors contributing to motion artifacts in capacitive ECG are local triboelectric effects on the electrode–body interface. These seem to depend on the relative distance in the triboelectric series [46]. If combined with lateral movements, this relative distance determines how sensitive different fabric combinations are to common mode signals due to local triboelectricity.

Several groups have proposed algorithmic and technical methods to fight motion artifacts. In 2011, Eilebrecht and coworkers proposed the use of adaptive filters for artifact cancellation [184]. Afterwards, Eilebrecht et al. [185] and later Serteyn and coworkers [181] proposed to analyse the electrode–skin interface by injecting high frequent test currents. This means that the interface is characterized by some sort of impedance measurement. In 2016, Lee et al. demonstrated that motion artifacts in radar systems can be greatly reduced using multiple signal classification (MUSIC) [178]. The authors investigated three different driving conditions: halted vehicle, driving at 60 kmh and driving at 80 kmh. Using a classical, FFT-based approach, the average error rate regarding the heart rate was below 3 bpm when the vehicle was halted, but excessive (approx. 25 bmp) at 60 and 80 kmh. Using the MUSIC approach, the error rate was reduced in all three driving conditions and stayed below 3 bpm.

8.2. Variable Light Conditions

Camera-based vital sign monitoring requires, with the expection of cameras operating in the far infrared spectrum, light that can be detected by the camera. For in-vehicle applications, sufficient light cannot be guaranteed, for example while driving at night. Artificial illumination is therefore required, which can blind the driver in case light in the visible range is used and is thus a traffic risk.

A solution may be the use of near infrared light which cannot be detected by the human eye, but can be detected by camera systems. However, it is not yet decided whether NIR light is harmful to the human eye. NIR light can damage both the cornea by thermal injury and the lens by cataractogenesis. Devereux et al. [186] argue that, under daylight conditions, NIR light is always accompanied by visible light. The eye will regulate light intake by closing the iris and protect itself. However, under night conditions, the iris will be widely open, allowing artifical NIR illumination to enter the eye unattenutated. Damage to the eye after long-term exposion is therefore conceivable.

Apart from the absence of light and hence the requirement for artifical illumination, varying light conditions can also pose a challenge, e.g., driving through an avenue with large trees or shadows from other cars can rapidly change the signal recorded by the camera. Hence, a correction algorithm is required. For example, Jeanne et al. [187] applied spatio-temporal filtering and achieved a correlation with the reference of 0.99. Lee et al. [188] used knowledge about the external light source to correct the camera video.

9. Sensor Fusion

As described in Section 8, the measurement of physiological signals in a car can be a difficult challenge due to motion artifacts not only induced by the vehicle but also the driver himself. This is even more pronounced, when using unobtrusive sensor techniques. By applying sensor fusion, some parts of this challenge might be mitigated. The term sensor fusion here stands for the use of multiple sensors at different locations or the use of different sensor modalities at the same time or a combination of both.

Applying sensor fusion would allow

artifact compensation, e.g., by utilizing adaptive filtering with one sensor serving as the noise signal.
source separation, for example by applying algorithms based on statistical dependencies between the signals (like independent component analysis).
coverage rate enhancement, e.g., by utilizing multiple sensors measuring the same vital sign and, therefore, increasing the probability that this vital sign is obtainable at a point in time.

As mentioned before, in 2015, Hoog Antink and colleages from RWTH Aachen University have demonstrated that fusing unobtrusive measurements indeed improves coverage and robustness of vital sign estimation [161]. In their experimental setup, the authors actually fused camera-based (web cam for BCG and PPG) information and one-dimensional signal information (BCG in a seat).

Two years later, these authors further demonstrated [189] that the fusion of multiple signals (PPG, BCG, cECG, MI) can further improve vital sign estimation of seated subjects, in particular when motion artifacts are present.

Since the assessment and evaluation of several driver states like stress level, sleepiness or emotions require a variety of different physiological indicators, sensor fusion seems mandatory for driver state estimation. Several studies about driver state estimation have been conducted, using measurement setups with conventional, i.e., obtrusive, mainly electrode-based, sensors [7].

Already in 2009, Baek et al. from Seoul National University and some other research organisations in Korea proposed to combine several unobtrusive sensing modalities in the car [34]. In particular, the authors presented a demonstrator (a test vehicle called the U-car) in which they simultaneously measured cECG in the backrest of the seat, ECG, PPG and galvanic skin response (GSR) in the steering wheel and respiration using a piezoelectric sensor in the seat belt. Their concept is shown in Figure 16.

In a small study involving n = 4 young healthy volunteers, driver stress levels were assessed during night driving, cold environment, hot environment, noisy environment, narrow alleys and with simultaneous mental loads (arithmetic operations). For quantification, both heart rate variability (HRV) indices as well as pulse arrival time (PAT) were measured.

A similar system was presented by Vavrinsky et al. in 2010 [30] and 2012 [31]. The system comprises two electrodes for ECG measurement, two GSR electrodes and two newly designed IDAT microsensors. All devices are built into the steering wheel. The IDAT sensors integrate an inter-digital array (IDA) [190] and a temperature sensor (T) into one small device and can be used for the measurement of skin temperature, GSR and heart pulse. The system also includes a pressure sensor integrated into the seat measuring respiration, heart pulse and driver weight. In addition, a camera-based emotion recognition system was integrated, but no results were presented. While this system incorporates several sensors for the same modality, sensor fusion still needs to be demonstrated.

10. Conclusions

In this review, special attention was given to unobtrusive and no-contact sensors for vital sign monitoring that can be placed inside the car, i.e., in the seat, the safety belt, the steering wheel or the cockpit. Based on different physical principles, many technologies exist and were introduced. Table 5 compares different technical aspects of these sensors including the question of active energy injection, costs, allowable distance between subject and device.

According to the number of car-related publications per technology, the former and present efforts to incorporate these technologies into the car is not equally distributed among them. While the seat-integrated techniques ECG, cECG and BCG have been applied in the automotive environment quite extensively, camera-based techniques, although already at a high readiness level, have not been investigated to the same extent. This might be due to their comparably high costs and the potential conflict with drivers feeling observed and violated in their privacy.

In this review, pilot applications and demonstrators as well as commercial products (where existing) were presented and discussed. However, measurement scenarios using any of the upcoming wearables, alone or in combination with fixed sensors, were not covered. In the long run, sensor fusion with such sensors is considered beneficial and should be further investigated. Today, available no-contact sensors share the problem of motion artefacts. For some of the technologies, additional challenges exist (like triboelectricity for cECG and varying light conditions in the visible and NIR frequency range). To increase robustness for such delicate measurement scenarios, sensor/signal fusion is one of the options, next to motion artifact cancellation. This requires multi-sensor and multi-channel measurement setups with redundant as well as complementary sensor arrangements.

Author Contributions

Conceptualization: all authors participated in defining the overall structure of the review article, composing the tables and images and selecting the literature to be included. Writing: L.L. mainly wrote Section 3, Section 8 and Section 9, S.L. mainly wrote Section 4, Section 5 and Section 7, D.T. mainly wrote Section 2, Section 6 and Section 10. Review and Editing: All authors carefully reviewed and edited the complete text.

Funding

There are no funding sponsors for this manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

BCG	ballistocardiography
cbPPG	camera-based photoplethysmography
cECG	capacitive electrocardiogram
CW	continuous wave
ECG	electrocardiogram
EMD	empirical mode decomposition
EMFi	electromechanical film
EPIC	electric potential integrated circuit
FCC	Federal Communications Commission
FIR	far infrared light
HRV	heart rate variability
ICA	independent component analysis
iPPG	imaging photoplethysmography
IRT	infrared thermography
ISM	industrial, scientific and medical
KIT	Karlsruhe Institute of Technology
LED	light-emitting diode
LWIR	long wave infrared light
MI	magnetic impedance
MWIR	medium wave infrared light
ncPPG	non-contact photoplethysmography
NIR	near infrared light
PAT	pulse arrival time
PD	photodetector
PERCLOS	percentage eye closure
PPG	photoplethysmogram
PPGI	photoplethysmography imaging
PTT	pulse transit time
QL	quality label
RADAR	radio detecting and ranging
rPPG	reflective or remote photoplethysmography
RSA	respiratory sinus arrhythmia
Rx	receiver
SpO₂	oxygen saturation
SVM	support vector machine
tPPG	transmissive photoplethysmography
Tx	transmitter
UWB	ultra-wide band
VIS	visible light

References

Anonymous. Vital Signs (Wikipedia Article). Available online: https://en.wikipedia.org/wiki/Vital_signs (accessed on 14 May 2018).
Elliott, M.; Coventry, A. Critical care: The eight vital signs of patient monitoring. Br. J. Nurs. 2012, 21, 621–625. [Google Scholar] [CrossRef] [PubMed]
Brueser, C.; Hoog Antink, C.; Wartzek, T.; Walter, M.; Leonhardt, S. Ambient and Unobtrusive Cardiorespiratory Monitoring Techniques. IEEE Rev. Biomed. Eng. 2015, 8, 30–43. [Google Scholar] [CrossRef] [PubMed]
Naziyok, T.P.; Zeleke, A.A.; Röhrig, R. Contactless Patient Monitoring for General Wards: A Systematic Technology Review. Stud. Health Technol. Inform. 2016, 228, 707–711. [Google Scholar] [PubMed]
Zaunseder, S.; Henning, A.; Wedekind, D.; Trumpp, A.; Malberg, H. Unobtrusive acquisition of cardiorespiratory signals—Available techniques and perspectives for sleep medicine. Somnologie 2017, 21, 93–100. [Google Scholar] [CrossRef]
Healey, J.; Seger, J.; Picard, R. Quantifying driver stress: Developing a system for collecting and processing bio-metric signals in natural situations. Biomed. Sci. Instrum. 1999, 35, 193–198. [Google Scholar] [PubMed]
Healey, J.A.; Picard, R.W. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 2005, 6, 156–166. [Google Scholar] [CrossRef]
Liang, S.F.; Lin, C.T.; Wu, R.C.; Chen, Y.C.; Huang, T.Y.; Jung, T.P. Monitoring Driver’s Alertness Based on the Driving Performance Estimation and the EEG Power Spectrum Analysis. In Proceedings of the 27th Annual Conference of the IEEE Engineering in Medicine and Biology Society, Shanghai, China, 1–4 September 2005. [Google Scholar]
Knipling, R.R.; Wierwille, W.W. Vehicle-Based Drowsy Driver Detection: Current Status and Future Prospects. In Proceedings of the IVHS America Fourth Annual Meeting, Atlanta, GA, USA, 17–20 April 1994. [Google Scholar]
Ueno, H.; Kaneda, M.; Tsukino, M. Development of Drowsiness Detection System. In Proceedings of the Vehicle Navigation and Information Systems Conference, Yokohama, Japan, 31 August–2 September 1994. [Google Scholar]
Begum, S. Intelligent Driver Monitoring Systems based on Physiological Sensor Signals: A Review. In Proceedings of the 16th International IEEE Annual Conference on Intelligent Transportation Systems (ITSC 2013), The Hague, The Netherlands, 6–9 October 2013. [Google Scholar]
Rhiu, I.; Kwon, S.; Bahn, S.; Yun, M.H.; Yu, W. Research Issues in Smart Vehicles and Elderly Drivers: A Literature Review. Int. J. Hum. Comput. Interact. 2015, 31, 635–666. [Google Scholar] [CrossRef]
Choi, Y.J.; Han, S.I.; Kong, S.-H.; Ko, H. Driver Status Monitoring Systems for Smart Vehicles using Physiological Sensors: A safety enhancement system from automobile manufacturers. IEEE Signal Process. Mag. 2016, 33, 22–34. [Google Scholar] [CrossRef]
Scalise, L.; Millis, R.M. Non contact heart monitoring. Adv. Electrocardiogr. Methods Anal. 2012, 81–106. [Google Scholar] [CrossRef]
Morbiducci, U.; Scalise, L.; de Melis, M.; Grigioni, M. Optical Vibrocardiography: A Novel Tool for the Optical Monitoring of Cardiac Activity. Ann. Biomed. Eng. 2007, 35, 45–58. [Google Scholar] [CrossRef] [PubMed]
Kranjec, J.; Begus, S.; Gersak, G.; Drnovsek, J. Non-contact heart rate and heart rate variability measurements: A review. Biomed. Signal Process. Control 2014, 13, 102–112. [Google Scholar] [CrossRef]
Kranjec, J.; Beguš, S.; Drnovšek, J.; Geršak, G. Novel Methods for Noncontact Heart Rate Measurement: A Feasibility Study. IEEE Trans. Instrum. Meas. 2014, 63, 838–847. [Google Scholar] [CrossRef]
Wagner, G.S.; Strauss, D.G. Marriott’s Practical Electrocardiography, 12th ed.; LWW: Philadelphia, PA, USA, 2013. [Google Scholar]
Goldberger, A.L.; Goldberger, Z.D.; Shvilkin, A. Goldberger’s Clinical Electrocardiography: A Simplified Approach, 9th ed.; Elsevier: Philadelphia, PA, USA, 2017. [Google Scholar]
Jeong, I.C.; Lee, D.H.; Park, S.W.; Ko, J.I.; Yoon, H.R. Automobile driver’s stress index provision system that utilizes electrocardiogram. In Proceedings of the 2007 IEEE Intelligent Vehicles Symposium, Istanbul, Turkey, 13–15 June 2007. [Google Scholar]
Lee, H.B.; Choi, J.M.; Kim, J.S.; Kim, Y.S.; Baek, H.J.; Ryu, M.S.; Sohn, R.H.; Park, K.S. Nonintrusive biosignal measurement system in a vehicle. In Proceedings of the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, France, 23–26 August 2007; pp. 2303–2306. [Google Scholar]
Hu, S.; Bowlds, R.; Yu, X. Non-Intrusive ECG Measurement on Vehicle Steering Wheel and Driver Seat. J. Med. Devices 2008, 2, 027520. [Google Scholar] [CrossRef]
Shin, H.-S.; Jung, S.-J.; Kim, J.-J.; Chung, W.-Y. Real time car driver’s condition monitoring system. In Proceedings of the IEEE Sensors 2010 Conference, Waikoloa, HI, USA, 1–4 November 2010; pp. 951–954. [Google Scholar]
Jung, S.J.; Shin, H.S.; Chung, W.Y. Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intell. Transp. Syst. 2014, 8, 43–50. [Google Scholar] [CrossRef]
Heuer, S.; Chamadiyaz, B.; Gharbiy, A.; Kunze, C.; Wagner, M. Unobtrusive In-Vehicle Biosignal Instrumentation for Advanced Driver Assistance and Active Safety. In Proceedings of the IEEE EMBS Conference on Biomedical Engineering & Sciences (IECBES 2010), Kuala Lumpur, Malaysia, 30 November–2 December 2010. [Google Scholar]
D’Angelo, L.T.; Parlow, J.; Spiessl, W.; Hoch, S.; Lueth, T.C. A system for unobtrusive in-car vital parameter acquisition and processing. In Proceedings of the 4th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth 2010), Munich, Germany, 22–25 March 2010; pp. 1–7. [Google Scholar]
D’Angelo, L.T.; Lueth, T.C. Integrated Systems for Distraction—Free Vital Signs Measurement in Vehicles. Automobiltech. Z. ATZ-Worldw. 2011, 113, 52–56. [Google Scholar] [CrossRef]
Gomez-Clapers, J.; Casanella, R. A Fast and Easy-to-Use ECG Acquisition and Heart Rate Monitoring System using a Wireless Steering Wheel. IEEE Sens. J. 2012, 12, 610–616. [Google Scholar] [CrossRef]
Silva, H.; Lourenco, A.; Fred, A. In-Vehicle Driver Recognition based on Hands ECG Signals. In Proceedings of the 17th International Conference on Intelligent User Interfaces (IUI’12), Lisbon, Portugal, 14–17 February 2012. [Google Scholar]
Vavrinský, E.; Tvarožek, V.; Stopjaková, V.; Soláriková, P.; Brezina, I. Monitoring of car driver physiological parameters. In Proceedings of the Eighth International Conference on Advanced Semiconductor Devices and Microsystems, Smolenice, Slovakia, 25–27 October 2010; pp. 227–230. [Google Scholar]
Vavrinsky, E.; Tvarozek, V.; Gaspierik, P.; Stopjakova, V.; Donoval, M.; Teleka, P. Design of non-invasive setup for car driver biomonitoring. Procedia Chem. 2012, 6, 203–210. [Google Scholar] [CrossRef]
Matsuda, T.; Makikawa, M. ECG monitoring of a car driver using capacitively-coupled electrodes. In Proceedings of the 30th Annual International IEEE EMBS Conference, Vancouver, BC, Canada, 20–24 August 2008; pp. 1315–1318. [Google Scholar]
Xu, X.; Ta, L. A Novel Driver-friendly ECG Monitoring System Based on Capacitive-coupled Electrode. Inf. Technol. J. 2013, 12, 4730–4734. [Google Scholar] [CrossRef]
Baek, H.J.; Lee, H.B.; Kim, J.S.; Choi, J.M.; Kim, K.K.; Park, K.S. Nonintrusive biological signal monitoring in a car to evaluate a driver’s stress and health state. Telemed. J. E-Health 2009, 15, 182–189. [Google Scholar] [CrossRef] [PubMed]
Richardson, P.C. The Insulated Electrode. In Proceedings of the 20th Annual Conference on Engineering in Medicine and Biology, Boston, MA, USA, 5–6 September 1967; p. 157. [Google Scholar]
Ishijima, M. Monitoring of electrocardiograms in bed without utilizing body surface electrodes. IEEE Trans. Biomed. Eng. 1993, 40, 593–594. [Google Scholar] [CrossRef] [PubMed]
Lim, Y.K.; Kim, K.K.; Park, K.S. The ECG measurement in the bathtub using the insulated electrodes. In Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, CA, USA, 1–5 September 2004; Volume 4, pp. 2383–2385. [Google Scholar]
Kim, K K.; Lim, Y.K.; Park, K.S. The electrically non-contacting ECG measurement on the toilet seat using the capacitively-coupled insulated electrodes. In Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, CA, USA, 1–5 September 2004; Volume 4, pp. 2375–2378. [Google Scholar]
Lim, Y.G.; Kim, K.K.; Park, K.S. ECG measurement on a chair without conductive contact. IEEE Trans. Biomed. Eng. 2006, 53, 956–959. [Google Scholar] [PubMed]
Lim, Y.G.; Kim, K.K.; Park, K.S. ECG recording on a bed during sleep without direct skin-contact. IEEE Trans. Biomed. Eng. 2007, 54, 718–725. [Google Scholar] [CrossRef] [PubMed]
Wu, K.; Zhang, Y. Contactless and continuous monitoring of heart electric activities through clothes on a sleeping bed. In Proceedings of the 5th International Conference on Information Technology and Application in Biomedicine, in conjunction with the 2nd International Symposium & Summer School on Biomedical and Health Engineering, Shenzhen, China, 30–31 May 2008; pp. 282–285. [Google Scholar]
Leonhardt, S.; Aleksandrowicz, A. Non-contact ECG monitoring for automotive application. In Proceedings of the 5th International Workshop on Wearable and Implantable Body Sensor Networks (BSN 2008), Hong Kong, China, 1–3 June 2008. [Google Scholar]
Chamadiya, B.; Heuer, S.; Hofmann, U.G.; Wagner, M. Towards a capacitively coupled electrocardiography system for car seat integration. In Proceedings of the 4th European Conference of the International Federation for Medical and Biological Engineering, Antwerpen, Belgium, 23–27 November 2008; pp. 1217–1221. [Google Scholar]
Eilebrecht, B.; Wartzek, T.; Lem, J.; Vogt, R.; Leonhardt, S. Capacitive electrocardiogram measurement system in the driver seat. Automobiltech. Z. ATZ-Worldw. 2011, 113, 50–55. [Google Scholar] [CrossRef]
Wartzek, T.; Eilebrecht, B.; Lem, J.; Lindner, H.J.; Leonhardt, S.; Walter, M. ECG on the road: Robust and unobtrusive estimation of heart rate. IEEE Trans. Biomed. Eng. 2011, 58, 3112–3120. [Google Scholar] [CrossRef] [PubMed]
Wartzek, T.; Lammersen, T.; Eilebrecht, B.; Walter, M.; Leonhardt, S. Triboelectricity in capacitive biopotential measurements. IEEE Trans. Biomed. Eng. 2011, 58, 1268–1277. [Google Scholar] [CrossRef] [PubMed]
Schumm, J.; Arnrich, B.; Troester, G. ECG Monitoring in an Airplane Seat: Appraising the Signal Quality. IEEE Pervasive Comput. 2012, 11, 28–34. [Google Scholar] [CrossRef]
Schneider, J.; Koellner, C.; Heuer, S. An Approach to Automotive ECG Measurement Validation using a Car-Integrated Test Framework. In Proceedings of the 2012 Intelligent Vehicles Symposium, Alcala de Henares, Spain, 3–7 June 2012. [Google Scholar]
Jung, S.J.; Shin, H.S.; Yoo, J.H.; Chung, W.Y. Highly Sensitive Driver Condition Monitoring System using Nonintrusive Active Electrodes. In Proceedings of the 2012 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 13–16 January 2012; pp. 305–306. [Google Scholar]
Fong, E.M.; Chung, W.Y. A Hygroscopic Sensor Electrode for Fast Stabilized Non-Contact ECG Signal Acquisition. Sensors 2015, 15, 237–250. [Google Scholar] [CrossRef] [PubMed]
Weder, M.; Hegemann, D.; Amberg, M.; Hess, M.; Boesel, L.F.; Abächerli, R.; Meyer, V.R.; Rossi, R.M. Embroidered Electrode with Silver/Titanium Coating for Long-Term ECG Monitoring. Sensors 2015, 15, 1750–1759. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Plessey WARDEN System. Available online: http://www.plesseysemiconductors.com/products/warden/ (accessed on 3 June 2018).
Leicht, L.; Eilebrecht, B.; Weyer, S.; Leonhardt, S.; Teichmann, D. Closed-Loop Control of Humidification for Artifact Reduction in Capacitive ECG Measurements. IEEE Trans. Biomed. Circuits Syst. 2017, 11, 300–313. [Google Scholar] [CrossRef] [PubMed]
Van Hoof, C.; Torfs, T. Get Your Daily Health Check in the Car. IMEC Magazine. September 2017. Available online: https://www.imec-int.com/en/imec-magazine/imec-magazine-september-2017/get-your-daily-health-check-in-the-car (accessed on 23 April 2018).
Leicht, L.; Skobel, E.; Knackstedt, C.; Mathissen, M.; Sitter, A.; Wartzek, T.; Möhler, W.; Reith, S.; Leonhardt, S.; Teichmann, D. Capacitive ECG Monitoring in Cardiac Patients during Simulated Driving. IEEE Trans. Biomed. Eng. 2018. [Google Scholar] [CrossRef] [PubMed]
Leonhardt, S.; Aleksandrowicz, A.; Steffen, M. Magnetic and Capacitive Monitoring of Heart and Lung Activity as an Example for Personal Healthcare. In Proceedings of the 2006 3rd IEEE/EMBS International Summer School on Medical Devices and Biosensors, Cambridge, MA, USA, 4–6 September 2006; pp. 57–60. [Google Scholar]
Aleksandrowicz, A.; Leonhardt, S. Wireless and Non-contact ECG Measurement System—The “Aachen SmartChair”. Acta Polytech. 2007, 47, 68–71. [Google Scholar]
Chamadiya, B.; Heuer, S.; Wagner, M.; Hofmann, U.G. Textile Capacitive Electrocardiography for an automotive environment. In Proceedings of the International Conference on Biomedical Electronics and Devices (BIODEVICES 2011), Rome, Italy, 26–29 January 2011. [Google Scholar]
EPIC Seatback Sensor. Available online: http://www.plesseysemiconductors.com/wp-content/uploads/EPIC-Seatback-Sensor-Promotion-Sheet.pdf (accessed on 23 April 2018).
Leicht, L.; Eilebrecht, B.; Weyer, S.; Wartzek, T.; Leonhardt, S. Active humidification for capacitive-resistive ECG-systems. In Proceedings of the 48. DGBMT Jahrestagung (BMT 2014), Hannover, Germany, 8 October 2014; pp. 818–821. [Google Scholar]
Leicht, L.; Skobel, E.; Mathissen, M.; Leonhardt, S.; Weyer, S.; Wartzek, T.; Reith, S.; Moehler, W.; Teichmann, D. Capacitive ECG recording and beat-to-beat interval estimation after major cardiac event. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2015), Milano, Italy, 25–29 August 2015; pp. 7614–7617. [Google Scholar]
Arnrich, B.; Kappeler-Setz, C.; Schumm, J.; Troester, G. Design, Implementation and Evaluation of a Multimodal Sensor System Integrated into an Airplane Seat. Sensor Fusion—Foundation and Applications. Thomas, C., Ed.; Chapter 9. pp. 159–170. Available online: https://www.intechopen.com/books/sensor-fusion-foundation-and-applications (accessed on 13 June 2011).
Magno, M.; Spagnol, C.; Benini, L.; Popovici, E. A low power wireless node for contact and contactless heart monitoring. Microelectr. J. 2014, 45, 1656–1664. [Google Scholar] [CrossRef]
Leicht, L.; Eilebrecht, B.; Mathissen, M.; Lem, J.; Lindner, A.; Vogt, R.; Leonhardt, S.; Walter, M. Contactless Electrocardiographic Sensor with Moisture Generator. U.S. Patent US2015088317 (A1), 23 September 2013. [Google Scholar]
Van Hoof, C.; Torfs, T. How Your Car Can Check Your Daily Health. eeNews Automotive. 22 November 2017. Available online: http://www.eenewsautomotive.com/design-center/how-your-car-can-check-your-daily-health/page/0/2 (accessed on 6 June 2018).
Gordon, J.W. Certain molar movements of the human body produced by the circulation of the blood. J. Anatomy Physiol. 1877, 11, 533–536. [Google Scholar]
Starr, I.; Rawson, A.J.; Schroeder, H.A.; Joseph, N.R. Studies on the estimation of cardiac output in man, and of abnormalities in cardiac function, from the heart’s recoil and the blood’s impacts; the ballistocardiogram. Am. J. Physiol. 1939, 127, 1–28. [Google Scholar]
Scarborough, W.R.; Talbot, S.A.; Braunstein, J.R.; Rappaport, M.B.; Dock, W.; Hamilton, W.F.; Smith, J.E.; Nickerson, J.L.; Starr, I. Proposals for ballistocardiographic nomenclature and conventions: Revised and extended: Report of committee on ballistocardiographic terminology. Circulation 1956, 14, 435–450. [Google Scholar] [CrossRef] [PubMed]
Inan, O.T.; Etemadi, M.; Wiard, R.M.; Giovangrandi, L.; Kovacs, G.T.A. Robust ballistocardiogram acquisition for home monitoring. Physiol. Meas. 2009, 30, 169–185. [Google Scholar] [CrossRef] [PubMed]
Mack, D.C.; Patrie, J.T.; Suratt, P.M.; Felder, R.A.; Alwan, M.A. Development and preliminary validation of heart rate and breathing rate detection using a passive, ballistocardiography-based sleep monitoring system. IEEE Trans. Inf. Technol. Biomed. 2009, 13, 111–120. [Google Scholar] [CrossRef] [PubMed]
Jansen, B.H.; Larson, B.H.; Shankar, K. Monitoring of the ballistocardiogram with the static charge sensitive bed. IEEE Trans. Biomed. Eng. 1991, 38, 748–751. [Google Scholar] [CrossRef] [PubMed]
Inan, O.T.; Migeotte, P.F.; Park, K.S.; Etemadi, M.; Tavakolian, K.; Casanella, R.; Zanetti, J.; Tank, J.; Funtova, I.; Prisk, G.K.; et al. Ballistocardiography and Seismocardiography: A Review of Recent Advances. IEEE J. Biomed. Health Inform. 2015, 19, 1414–1427. [Google Scholar] [CrossRef] [PubMed]
Brueser, C.; Stadlthanner, K.; de Waele, S.; Leonhardt, S. Adaptive beat-to-beat heart rate estimation in ballistocardiograms. IEEE Trans. Inf. Technol. Biomed. 2011, 15, 778–786. [Google Scholar] [CrossRef] [PubMed]
Walter, M.; Eilebrecht, B.; Wartzek, T.; Leonhardt, S. The smart car seat: Personalized monitoring of vital signs in automotive applications. Pers. Ubiquitous Comput. 2011, 15, 707–715. [Google Scholar] [CrossRef]
HARKEN—Heart and Respiration in-Car Embedded Non Intrusive Sensors. Available online: http://cordis.europa.eu/result/rcn/156259_en.html (accessed on 3 September 2016).
Faurecia Press Release Kit. Available online: www.faurecia.com/files/corporate/publication/file/faurecia_active_wellness_140915_en.pdf (accessed on 5 September 2018).
Faurecia PRESS KIT AUTO SHANGHAI 2015. Available online: www.faurecia.com/files/corporate/publication/file/presskit_final_web.pdf (accessed on 5 September 2018).
Wusk, G.; Gabler, H. Non-Invasive Detection of Respiration and Heart Rate with a Vehicle Seat Sensor. Sensors 2018, 18, 1463. [Google Scholar] [CrossRef] [PubMed]
Sun, Y.; Thakor, N. Photoplethysmography Revisited: From Contact to Noncontact, From Point to Imaging. IEEE Trans. Biomed. Eng. 2016, 63, 463–477. [Google Scholar] [CrossRef] [PubMed]
Daly, J. Video Camera Monitoring to Detect Changes in Haemodynamics. Ph.D. Thesis, University of Oxford, Oxford, UK, 2016. [Google Scholar]
Blazek, V.; Wu, T.; Hoelscher, D. Near-infrared CCD imaging: Possibilities for noninvasive and contactless 2D mapping of dermal venous hemodynamics. In SPIE 3923, Optical Diagnostics of Biological Fluids V; The International Society for Optics and Photonics: San Diego, CA, USA, 2000. [Google Scholar]
Wu, T.; Blazek, V.; Schmitt, H.J. Photoplethysmography imaging: A new noninvasive and noncontact method for mapping of the dermal perfusion changes. In EBIOS 2000: European Conference on Biomedical Optics, Amsterdam, Proceedings of the SPIE 4163, Optical Techniques and Instrumentation for the Measurement of Blood Composition, Structure, and Dynamics; The International Society for Optics and Photonics: San Diego, CA, USA, 2000. [Google Scholar]
Huelsbusch, M.; Blazek, V. Contactless mapping of rhythmical phenomena in tissue perfusion using PPGI. In SPIE 4683, Medical Imaging 2002: Physiology and Function from Multidimensional Images; The International Society for Optics and Photonics: San Diego, CA, USA, 2002. [Google Scholar]
Wieringa, F.P.; Mastik, F.; van der Steen, A.F. Contactless multiple wavelength photoplethysmographic imaging: A first step toward “SpO₂ camera” technology. Ann. Biomed. Eng. 2005, 33, 1034–1041. [Google Scholar] [CrossRef] [PubMed]
Takano, C.; Ohta, Y. Heart rate measurement based on a time-lapse image. Med. Eng. Phys. 2007, 29, 853–857. [Google Scholar] [CrossRef] [PubMed]
Zheng, J.; Hu, S. The preliminary investigation of imaging photoplethysmographic system. In Proceedings of the Third International Conference on Optical and Laser Diagnostics, London, UK, 22–25 May 2007. Journal of Physics: Conference Series 85, 012031. [Google Scholar]
Verkruysse, W.; Svaasand, L.O.; Nelson, J.S. Remote plethysmographic imaging using ambient light. Opt. Express 2008, 16, 21434–21445. [Google Scholar] [CrossRef] [PubMed]
Wang, W. Robust and Automatic Remote Photoplethysmography. Ph.D. Thesis, Technische Universiteit Eindhoven, Eindhoven, The Netherlands, 2017. [Google Scholar]
Tarassenko, L.; Villarroel, M.; Guazzi, A.; Jorge, J.; Clifton, D.A.; Pugh, C. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol. Meas. 2014, 35, 807–831. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Villarroel, M.; Guazzi, A.; Jorge, J.; Davis, S.; Watkinson, P.; Green, G.; Shenvi, A.; McCormick, K.; Tarassenko, L. Continuous non-contact vital sign monitoring in neonatal intensive care unit. Healthc. Technol. Lett. 2014, 1, 87–91. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Guazzi, A.R.; Villarroel, M.; Jorge, J.; Daly, J.; Frise, M.C.; Robbins, P.A.; Tarassenko, L. Non-contact measurement of oxygen saturation with an RGB camera. Biomed. Opt. Express 2015, 6, 3320–3338. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Wu, H.Y.; Rubinstein, M.; Shih, E.; Guttag, J.; Durand, F.; Freeman, W. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 2012, 31, 1–8. [Google Scholar] [CrossRef] [Green Version]
Blazek, C.R.; Merk, H.F.; Schmid-Schoenbein, H.; Huelsbusch, M.; Blazek, V. Assessment of Allergic Skin Reactions and their Inhibition by Antihistamines using Photoplethysmography Imaging (PPGI). J. Allergy Clin. Immunol. 2006, 117, S226. [Google Scholar] [CrossRef]
Karlen, W.; Garde, A.; Myers, D.; Scheffer, C.; Ansermino, J.M.; Dumont, G.A. Estimation of Respiratory Rate from Photoplethysmographic Imaging Videos Compared to Pulse Oximetry. IEEE J. Biomed. Health Inform. 2015, 19, 1331–1338. [Google Scholar] [CrossRef] [PubMed]
Moco, A.V.; Stuijk, S.; de Haan, G. Ballistocardiographic Artifacts in PPG Imaging. IEEE Trans. Biomed. Eng. 2016, 63, 1804–1811. [Google Scholar] [CrossRef] [PubMed]
Blazek, V.; Blanik, N.; Blazek, C.R.; Paul, M.; Pereira, C.; Koeny, M.; Venema, B.; Leonhardt, S. Active and Passive Optical Imaging Modality for Unobtrusive Cardiorespiratory Monitoring and Facial Expression Assessment. Anesth. Analg. 2017, 124, 104–119. [Google Scholar] [CrossRef] [PubMed]
Bloecher, T.; Schneider, J.; Schinle, M.; Stork, W. An online PPGI approach for camera-based heart rate monitoring using beat-to-beat detection. In Proceedings of the 2017 IEEE Sensors Applications Symposium (SAS), Glassboro, NJ, USA, 13–15 March 2017. [Google Scholar]
Philips IP&S. Vital Signs Camera. Available online: www.vitalsignscamera.com (accessed on 10 June 2018).
Hu, S.; Zheng, J.; Chouliaras, V.; Summers, R. Feasibility of imaging photoplethysmography. In Proceedings of the 2008 International Conference on BioMedical Engineering and Informatics, Sanya, China, 28–30 May 2008. [Google Scholar]
Sun, Y.; Yu, X.B. An innovative nonintrusive driver assistance system for vital signal monitoring. IEEE J. Biomed. Health Inform. 2014, 18, 1932–1939. [Google Scholar] [CrossRef] [PubMed]
Kamshilin, A.A.; Nippolainen, E.; Sidorov, I.S.; Vasilev, P.V.; Erofeev, N.P.; Podolian, N.P.; Romashko, R.V. A new look at the essence of the imaging photoplethysmography. Sci. Rep. 2015, 5, 10494. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Kuo, J.; Koppel, S.; Charlton, J.L.; Rudin-Brown, C.M. Evaluation of a video-based measure of driver heart rate. J. Saf. Res. 2015, 54, 55–59. [Google Scholar] [CrossRef] [PubMed]
Blackford, E.B.; Piasecki, A.M.; Estepp, J.R. Measuring pulse rate variability using long-range, non-contact imaging photoplethysmography. In Proceedings of the 2016 IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 17–20 August 2016; pp. 3930–3936. [Google Scholar]
McDuff, D.J.; Estepp, J.R.; Piasecki, A.M.; Blackford, E.B. A survey of remote optical photoplethysmographic imaging methods. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2015), Milano, Italy, 25–29 August 2015; pp. 6398–6404. [Google Scholar]
Kwon, S.; Kim, J.; Lee, D.; Park, K. ROI analysis for remote photoplethysmography on facial video. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milano, Italy, 25–29 August 2015; pp. 4938–4941. [Google Scholar]
Van Gastel, M.; Stuijk, S.; de Haan, G. Motion Robust Remote-PPG in Infrared. IEEE Trans. Biomed. Eng. 2015, 62, 1425–1433. [Google Scholar] [CrossRef] [PubMed]
Wu, B.-F.; Chu, Y.W.; Huang, P.-W.; Chung, M.-L.; Lin, T.-M. A Motion Robust Remote-PPG Approach to Driver’s Health State Monitoring. In Proceedings of the ACCV 2016 International Workshops Computer Vision (ACCV 2016), Taipei, Taiwan, 20–24 November 2016; pp. 463–476. [Google Scholar]
Van Gastel, M.; Stuijk, S.; de Haan, G. Robust respiration detection from remote photoplethysmography. Biomed. Opt. Express 2016, 7, 4941–4957. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Kumar, M.; Veeraraghavan, A.; Sabharwal, A. DistancePPG: Robust non-contact vital signs monitoring using a camera. Biomed. Opt. Express 2015, 6, 1565–1588. [Google Scholar] [CrossRef] [PubMed]
Choe, J.; Chung, D.; Schwichtenberg, A.J.; Delp, E.J. Improving video-based resting heart rate estimation: A comparison of two methods. In Proceedings of the IEEE 58th International Midwest Symposium on Circuits and Systems (MWSCAS), Fort Collins, CO, USA, 2–5 August 2015; pp. 1–4. [Google Scholar]
Wedekind, D.; Malberg, H.; Zaunseder, S.; Gaetjen, F.; Matschke, K.; Rasche, S. Automated identification of cardiac signals after blind source separation for camera-based photoplethysmography. In Proceedings of the IEEE 35th International Conference on Electronics and Nanotechnology (ELNANO), KJiev, Ukraine, 24–26 April 2015; pp. 422–427. [Google Scholar]
Rasche, S.; Trumpp, A.; Waldow, T.; Gaetjen, F.; Plötze, K.; Wedekind, D.; Schmidt, M.; Malberg, H.; Matschke, K.; Zaunseder, S. Camera-based photoplethysmography in critical care patients. Clin. Hemorheol. Microcirc. 2016, 64, 77–90. [Google Scholar] [CrossRef] [PubMed]
Kevat, A.C.; Bullen, D.V.; Davis, P.G.; Kamlin, C.O.F. A systematic review of novel technology for monitoring infant and newborn heart rate. Acta Paediatr. 2017, 106, 710–720. [Google Scholar] [CrossRef] [PubMed]
Butler, M.J.; Crowe, J.A.; Hayes-Gill, B.R.; Rodmell, P.I. Motion limitations of non-contact photoplethysmography due to the optical and topological properties of skin. Physiol. Meas. 2016, 37, N27–N37. [Google Scholar] [CrossRef] [PubMed]
Tayibnapis, I.R.; Koo, D.Y.; Choi, M.K.; Kwon, S. A novel driver fatigue monitoring using optical imaging of face on safe driving system. In Proceedings of the International Conference on Control, Electronics, Renewable Energy and Communications (ICCEREC), Bandung, Indonesia, 13–15 September 2016; pp. 115–120. [Google Scholar]
Iozzia, L.; Cerina, L.; Mainardi, L. Relationships between heart-rate variability and pulse-rate variability obtained from video-PPG signal using ZCA. Physiol. Meas. 2016, 37, 1934–1944. [Google Scholar] [CrossRef] [PubMed]
Zhang, Q.; Xu, G.-Q.; Wang, M.; Zhou, Y.; Feng, W. Webcam based non-contact real-time monitoring for the physiological parameters of drivers. In Proceedings of the IEEE 4th Annual International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER 2014), Hong Kong, China, 4–7 June 2014. [Google Scholar]
Zhang, Q.; Xu, G.-Q.; Wang, M.; Zhou, Y.; Feng, W. Webcam-based, non-contact, real-time measurement for the physiological parameters of drivers. Measurement 2017, 100, 311–321. [Google Scholar] [CrossRef]
Rahman, H.; Barua, S.; Shahina, B. Intelligent driver monitoring based on physiological sensor signals: Application using camera. In Proceedings of the IEEE 18th International Conference on Intelligent Transportation Systems, Las Palmas, Spain, 15–18 September 2015; pp. 2637–2642. [Google Scholar]
Rahman, H.; Uddin Ahmed, M.; Begum, S.; Funk, P. Real Time Heart Rate Monitoring from Facial RGB Color Video using Webcam. In Proceedings of the 9th Annual Workshop of the Swedish Artificial Intelligence Society (SAIS) 2016, Malmö, Sweden, 2–3 June 2016. [Google Scholar]
Gücüyener, I. A novel design of heartbeat monitoring system for the motor vehicle. Int. J. Inj. Contr. Saf. Promot. 2015, 16, 1–5. [Google Scholar] [CrossRef] [PubMed]
Pavlidis, I.; Levine, J.; Baukol, P. Thermal imaging for anxiety detection. In Proceedings of the IEEE workshop on Computer Vision beyond the Visible Spectrum: Methods and Applications, Hilton Head, SC, USA, 16 June 2000. [Google Scholar]
Pavlidis, I.; Levine, J.; Baukol, P. Thermal image analysis for anxiety detection. In Proceedings of the 2001 International Conference on Image Processing, Thessaloniki, Greece, 7–10 October 2001. [Google Scholar]
Murthy, R.; Pavlidis, I.; Tsiamyrtzis, P. Touchless Monitoring of Breathing Function. In Proceedings of the 26th Annual International Conference of the IEEE EMBS, San Francisco, CA, USA, 1–5 September 2004. [Google Scholar]
Fei, J.; Pavlidis, I. Virtual Thermistor. In Proceedings of the 29th Annual International Conference of the IEEE EMBS, Cité Internationale, Lyon, France, 23–26 August 2007. [Google Scholar]
Fei, J.; Pavlidis, I. Thermistor at a distance: Unobtrusive measurement of breathing. IEEE Trans. Biomed. Eng. 2010, 57, 988–998. [Google Scholar] [PubMed]
Pereira, C.B.; Yu, X.; Czaplik, M.; Rossaint, R.; Blazek, V.; Leonhardt, S. Remote monitoring of breathing dynamics using infrared thermography. Biomed. Opt. Express 2015, 6, 4378–4394. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Smith, P.; Shah, M.; da Vitoria Lobo, N. Determining driver visual attention with one camera. IEEE Trans. Intell. Transp. Syst. 2003, 4, 205–218. [Google Scholar] [CrossRef] [Green Version]
Rongben, W.; Lie, G.; Bingliang, T.; Lisheng, J. Monitoring mouth movement for driver fatigue or distraction with one camera. In Proceedings of the 7th IEEE International Conference on Intelligent Transportation Systems, Washington, DC, USA, 3–6 October 2004; pp. 314–319. [Google Scholar]
Trivedi, M.M.; Gandhi, T.; McCall, J. Looking-In and Looking-Out of a Vehicle: Computer-Vision-Based Enhanced Vehicle Safety. IEEE Trans. Intell. Transp. Syst. 2007, 8, 108–120. [Google Scholar] [CrossRef] [Green Version]
Bergasa, L.M.; Buenaposada, J.M.; Nuevo, J.; Jimenez, P.; Baumela, L. Analysing driver’s attention level using computer vision. In Proceedings of the 11th International IEEE Conference on Intelligent Transportation Systems, Beijing, China, 12–15 October 2008; pp. 1149–1154. [Google Scholar]
Viola, P.; Jones, M.J. Robust real-time face detection. Int. J. Comput. Vis. 2004, 57, 137–154. [Google Scholar] [CrossRef]
Manoharan, R.; Chandrakala, S.; Khan, W. Drive Safe: An Intelligent System for Monitoring Stress and Pain from Drivers’ Facial Expressions. Int. J. Latest Technol. Eng. Manag. Appl. Sci. 2016, 46–51. [Google Scholar]
Grace, R.; Byrne, V.E.; Bierman, D.M.; Legrand, J.-M.; Gricourt, D.; Davis, B.K.; Staszewski, J.J.; Carnahan, B. A drowsy driver detection system for heavy vehicles. In Proceedings of the 17th AIAA/IEEE/SAE Digital Avionics Systems Conference (DASC), Bellevue, WA, USA, 31 October–7 November 1998. [Google Scholar]
Wang, Q.; Yang, J.; Ren, M.; Zheng, Y. Driver Fatigue Detection: A Survey. In Proceedings of the 6th World Congress on Intelligent Control and Automation, Dalian, China, 21–23 June 2006. [Google Scholar]
Gao, H.; Yüce, A.; Thiran, J.-P. Detecting emotional stress from facial expressions for driving safety. In Proceedings of the IEEE International Conference on Image Processing (ICIP 2014), Paris, France, 27–30 October 2014; pp. 5961–5965. [Google Scholar]
Levine, J.A.; Pavlidis, I.; Cooper, M. The face of fear. Lancet 2001, 357, 1757. [Google Scholar] [CrossRef]
Pavlidis, I.; Levine, J. Thermal image analysis for polygraph testing. IEEE Eng. Med. Biol. Mag. 2002, 21, 56–64. [Google Scholar] [CrossRef] [PubMed]
Puri, C.; Olson, L.; Pavlidis, I.; Levine, J.; Starren, J. Stresscam: Non-contact measurement of users’ emotional states through thermal imaging. In Proceedings of the Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, USA, 2–7 April 2005. [Google Scholar]
Khan, M.M.; Ward, R.D.; Ingleby, M. Infrared Thermal Sensing of Positive and Negative Affective States. In Proceedings of the IEEE Conference on Robotics, Automation and Mechatronics, Bangkok, Thailand, 1–3 June 2006. [Google Scholar]
Hernandez, B.; Olague, G.; Hammoud, R.; Trujillo, L.; Romero, E. Visual learning of texture descriptors for facial expression recognition in thermal imagery. Comput. Vis. Image Underst. 2007, 106, 258–269. [Google Scholar] [CrossRef]
Nhan, B.R.; Chau, T. Classifying Affective States Using Thermal Infrared Imaging of the Human Face. IEEE Trans. Biomed. Eng. 2010, 57, 979–987. [Google Scholar] [CrossRef] [PubMed]
Nguyen, H.; Kotani, K.; Chen, F.; Le, B. Estimation of Human Emotions using Thermal Facial Information. In Proceedings of the Fifth International Conference on Graphic and Image Processing (ICGIP 2013), Hong Kong, China, 26–27 October 2013. [Google Scholar]
Hong, K.; Liu, G. Facial Thermal Image Analysis for Stress Detection. Int. J. Eng. Res. Technol. 2017, 6, 94–98. [Google Scholar]
Reyes, M.L.; Lee, J.D.; Liang, Y.; Hoffman, J.D.; Huang, R.W. Capturing driver response to in-vehicle human-machine interface technologies using Facial Thermography. In Proceedings of the 5th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Big Sky, MT, USA, 22–25 June 2009. [Google Scholar]
Zomotor, Z.; Franke, U. Sensor fusion for improved vision based lane recognition and object tracking with range-finders. In Proceedings of the Conference on Intelligent Transportation Systems, Boston, MA, USA, 9–12 November 1997; pp. 595–600. [Google Scholar]
Fang, Y.; Masaki, I.; Horn, B. Depth-based target segmentation for intelligent vehicles: Fusion of radar and binocular stereo. IEEE Trans. Intell. Transp. Syst. 2002, 3, 196–202. [Google Scholar] [CrossRef]
She, K.; Bebis, G.; Gu, H.; Miller, R. Vehicle tracking using on-line fusion of color and shape features. In Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems, Washington, DC, USA, 3–6 October 2004; pp. 731–736. [Google Scholar] [Green Version]
Krotosky, S.J.; Trivedi, M.M. A Comparison of Color and Infrared Stereo Approaches to Pedestrian Detection. In Proceedings of the 2007 IEEE Intelligent Vehicles Symposium, Istanbul, Turkey, 13–15 June 2007; pp. 81–86. [Google Scholar]
Pavlidis, I.; Symosek, P.; Fritz, B.; Papanikopoulos, N.; Schwartz, K. Automatic detection of vehicle passengers through near-infrared fusion. In Proceedings of the IEEE Conference on Intelligent Transportation Systems (ITSC), Tokyo, Japan, 5–8 October 1999. [Google Scholar]
Pavlidis, I.; Morellas, V.; Papanikolopoulos, N. A vehicle occupant counting system based on near-infrared phenomenology and fuzzy neural classification. IEEE Trans. Intell. Transp. Syst. 2000, 1, 72–85. [Google Scholar] [CrossRef]
Owechko, Y.; Srinivasa, N.; Medasani, S.; Boscolo, R. Vision-based fusion system for smart airbag applications. IEEE Intell. Veh. Symp. 2002, 1, 245–250. [Google Scholar] [CrossRef]
Trivedi, M.M.; Cheng, S.Y.; Childers, E.M.C.; Krotosky, S.J. Occupant posture analysis with stereo and thermal infrared video: Algorithms and experimental evaluation. IEEE Trans. Veh. Technol. 2004, 53, 1698–1712. [Google Scholar] [CrossRef]
Gyaourova, A.; Bebis, G.; Pavlidis, I. Fusion of infrared and visible images for face recognition. In Proceedings of the 8th European Conference on Computer Vision (ECCV 2004), Prague, Czech Republic, 11–14 May 2004; pp. 456–468. [Google Scholar]
Li, L.; Chen, Y.; Li, Z. Yawning detection for monitoring driver fatigue based on two cameras. In Proceedings of the 12th International IEEE Conference on Intelligent Transportation Systems, St Louis, MO, USA, 4–7 October 2009; pp. 1–6. [Google Scholar]
Corneanu, C.A.; Simron, M.O.; Cohn, J.F.; Escalera Guerrero, S. Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 38, 1548–1568. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Wang, S.; Pan, B.; Chen, H.; Ji, Q. Thermal Augmented Expression Recognition. IEEE Trans. Cybern. 2018, 48, 2203–2214. [Google Scholar] [CrossRef] [PubMed]
Blanik, N.; Blazek, C.; Pereira, C.; Blazek, V.; Leonhardt, S. Frequency-selective quantification of skin perfusion behavior during allergic testing using photoplethysmography imaging. In Proceedings of the SPIE Medical Imaging 2014: Image Processing, San Diego, CA, USA, 15–20 February 2014. [Google Scholar]
Balakrishnan, G.; Durand, F.; Guttag, J. Detecting Pulse from Head Motions in Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013), Portland, OR, USA, 23–28 June 2013; pp. 3430–3437. [Google Scholar]
Viola, P.; Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 International Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai Island, HI, USA, 8–14 December 2001; pp. 511–518. [Google Scholar]
Hoog Antink, C.; Gao, H.; Brueser, C.; Leonhardt, S. Beat-to-beat heart rate estimation fusing multimodal video and sensor data. Biomed. Opt. Express 2015, 6, 2895–2907. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Teichmann, D.; Foussier, J.; Jia, J.; Leonhardt, S.; Walter, M. Noncontact monitoring of cardiorespiratory activity by electromagnetic coupling. IEEE Trans. Biomed. Eng. 2013, 60, 2142–2152. [Google Scholar] [CrossRef] [PubMed]
Cordes, A. Monitoring von Atmung und Herzaktivität Mittels Magnetischer Induktionsmessung. Ph.D. Thesis, RWTH Aachen University, Aachen, Germany, 2013. [Google Scholar]
Leicht, L.; Vetter, P.; Leonhardt, S.; Teichmann, D. The PhysioBelt: A Safety Belt Integrated Sensor System for Heart Activity and Respiration. In Proceedings of the IEEE International Conference on Vehicular Electronics and Safety (IEEE ICVES 2017), Vienna, Austria, 27–28 June 2017. [Google Scholar]
Vetter, P.; Leicht, L.; Leonhardt, S.; Teichmann, D. Integration of an Electromagnetic Coupled Sensor into a Driver Seat for Vital Sign Monitoring: Initial Insight. In Proceedings of the IEEE International Conference on Vehicular Electronics and Safety (IEEE ICVES 2017), Vienna, Austria, 27–28 June 2017. [Google Scholar]
Lazaro, A.; Girbau, D.; Villarino, R. Analysis of Vital Signs Monitoring using an IR-UWB Radar. Prog. Electromag. Res. 2010, 100, 265–284. [Google Scholar] [CrossRef]
Singh, M.; Ramachandran, G. Reconstruction of sequential cardiac in-plane displacement patterns on the chest wall by laser speckle interferometry. IEEE Trans. Biomed. Eng. 1991, 38, 483–489. [Google Scholar] [CrossRef] [PubMed]
Allen, B.; Brown, T.; Schwieger, K.; Zimmermann, E.; Malik, W.; Edwards, D.; Ouvry, L.; Oppermann, I. Ultra Wideband: Applications, Technology and Future perspectives. In Proceedings of the 1st International Workshop on Convergent Technologies (IWCT 2005), Oulu, Finland, 6–10 June 2005. [Google Scholar]
Immoreev, I.Y.; Samkov, S.V. Ultra-Wideband Radar for remote Detection and Measurement of Parameters of the moving Objects on a small Range. In Proceedings of the Conference on Ultrawideband and Ultrashort Impulse Signals, Sevastopol, Ukraine, 19–22 September 2004; pp. 214–216. [Google Scholar]
TES Electronic Solutions: Whitepaper on Vital Sign Detection Using 60 GHz Radar Technology. Available online: http://www.tes-dst.com/tes-dst/download/White_Paper_Vital_Sign_Detection_60GHz_ENG.pdf (accessed on 5 September 2018).
Kao, T.-Y.J.; Lin, J. Vital sign detection using 60-GHz Doppler radar system. In Proceedings of the 2013 IEEE International Wireless Symposium (IWS), Bejing, China, 14–18 April 2013. [Google Scholar]
Caro, C.G.; Bloice, J.A. Contactless apnoea detector based on radar. Lancet 1971, 298, 959–961. [Google Scholar] [CrossRef]
Ichapurapu, R.; Jain, S.; John, G.; Monday, T.; Lie, D.Y.C.; Banister, R.; Griswold, J. A 2.4 GHz non-contact biosensor system for continuous vital-signs monitoring on a single PCB. In Proceedings of the IEEE 8th International Conference on ASIC (ASICON ’09), Changsha, China, 20–23 October 2009. [Google Scholar]
Massagram, W.; Lubecke, V.M.; Host-Madsen, A.; Boric-Lubecke, O. Assessment of Heart Rate Variability and Respiratory Sinus Arrhythmia via Doppler Radar. IEEE Trans. Microw. Theory Tech. 2009, 57, 2542–2549. [Google Scholar] [CrossRef]
Scalise, L.; De Leo, A.; Mariani Primiani, V.; Russo, P.; Shahu, D.; Cerri, G. Non contact monitoring of the respiration activity by electromagnetic sensing. In Proceedings of the 2011 IEEE International Symposium on Medical Measurements and Applications, Bari, Italy, 30–31 May 2011; pp. 418–422. [Google Scholar]
Bechet, P.; Mitran, R.; Munteanu, M. A non-contact method based on multiple signal classification algorithm to reduce the measurement time for accurately heart rate detection. Rev. Sci. Instrum. 2013, 84, 084707. [Google Scholar] [CrossRef] [PubMed]
Vinci, G.; Lenhard, T.; Will, C.; Koelpin, A. Microwave Interferometer Radar-Based Vital Sign Detection for Driver Monitoring Systems. In Proceedings of the 2015 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM), Heidelberg, Germany, 27–29 April 2015. [Google Scholar]
Lee, K.J.; Park, C.; Lee, B. Tracking Driver’s Heart Rate by Continuous-Wave Doppler Radar. In Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’16), Orlando, FL, USA, 16–20 August 2016. [Google Scholar]
Izumi, S.; Matsunaga, D.; Nakamura, R.; Kawaguchi, H.; Yoshimoto, M. A Contact-Less Heart Rate Sensor System for Driver Health Monitoring. Available online: https://pdfs.semanticscholar.org/9059/6a41f8642c5854f88e02a3e121a151747434.pdf (accessed on 31 August 2018).
Schires, E.; Georgiou, P.; Lande, T.S. Vital Sign Monitoring through the Back using an UWB Impulse Radar with Body Coupled Antennas. IEEE Trans. Biomed. Circuits Syst. 2018, 12, 292–302. [Google Scholar] [CrossRef] [PubMed]
Serteyn, A.; Vullings, R.; Meftah, M.; Bergmans, J.W. Motion artifacts in capacitive ECG measurements: Reducing the combined effect of DC voltages and capacitance changes using an injection signal. IEEE Trans. Biomed. Eng. 2015, 62, 264–273. [Google Scholar] [CrossRef] [PubMed]
Tang, M.; Wang, F.; Horng, T. Single Self-Injection-Locked Radar With Two Antennas for Monitoring Vital Signs With Large Body Movement Cancellation. IEEE Trans. Microw. Theory Tech. 2017, 65, 5324–5333. [Google Scholar] [CrossRef]
Ottenbacher, J.; Heuer, S. Motion artefacts in capacitively coupled ECG electrodes. Proc. World Congr. Med. Phys. Biomed. Eng. 2010, 25, 1059–1062. [Google Scholar]
Eilebrecht, B.; Wartzek, T.; Willkomm, J.; Schommartz, A.; Walter, M.; Leonhardt, S. Motion Artifact Removal from Capacitive ECG Measurements by Means of Adaptive Filtering. In 5th European Conference of the International Federation for Medical and Biological Engineering; Springer: Heidelberg, Germany, 2011; Volume 37, pp. 902–905. [Google Scholar]
Eilebrecht, B.; Willkomm, J.; Pohl, A.; Wartzek, T.; Leonhardt, S. Impedance measurement system for determination of capacitive electrode coupling. IEEE Trans. Biomed. Circuits Syst. 2013, 7, 682–689. [Google Scholar] [CrossRef] [PubMed]
Devereux, H.; Smalley, M. Are infra red illuminators eye safe? In Proceedings of the IEEE 29th Annual 1995 International Carnahan Conference on Security Technology, Surrey, UK, 18–20 October 1995. [Google Scholar]
Jeanne, V.; Asselman, M.; den Brinker, B.; Bulut, M. Camera-based heart rate monitoring in highly dynamic light conditions. In Proceedings of the International Conference on Connected Vehicles and Expo (ICCVE 2013), Las Vegas, NV, USA, 2–6 December 2013. [Google Scholar]
Lee, D.; Kim, J.; Kwon, S.; Park, K. Heart rate estimation from facial photoplethysmography during dynamic illuminance changes. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milano, Italy, 25–29 August 2015. [Google Scholar]
Hoog Antink, C.; Schulz, F.; Leonhardt, S.; Walter, M. Motion Artifact Quantification and Sensor Fusion for Unobtrusive Health Monitoring. Sensors 2017, 18, 38. [Google Scholar] [CrossRef] [PubMed]
Ivanic, P.; Novotny, I.; Rehacek, V.; Tvarozek, V.; Weis, M. Thin film non-symmetric microelectrode array for impedance monitoring of human skin. Thin Solid Films 2003, 433, 332–336. [Google Scholar] [CrossRef]

Figure 1. Overview of physiological sources, effects, unobtrusive and non-contact sensors, and obtainable vital signals, modified and extended from [3] (see Figure 1).

Figure 2. Electrode locations for conductive and low-contact ECG monitoring around a car seat. Red blocks indicate tested locations for sensing electrodes, blue blocks indicate published locations for the driven ground electrode.

Figure 3. Overview of several designs for contact-based ECG monitoring (red: active electrodes, blue: driven ground electrode, green: electronics, precise electrode position on steering wheel not revealed): (a) Jeong [20]; (b) Lee [21], Silva [29], Vavrinsky [30,31]; (c) Matsuda [32], Xu [33]; (d) Heuer [25]; (e) D’Angelo [27]; (f) Gomez-Clapers [28]; (g) Jung [24].

Figure 4. Overview of several low-contact ECG electrode arrangements and their year of introduction to the public (red: active electrodes, blue: driven ground electrode). (a) SMART seat (Leonhardt, 2008) [42]; (b) Daimler S-Class (Chamadiya, 2008) [43]; (c) Ford S-Max (Eilebrecht, 2011) [44]; (d) Daimler car (Chamadiya, 2011) [58]; (e) Audi Q5 (Schneider, 2012) [48]; (f) Car seat (Jung, 2012) [49]; (g) EPIC System (Plessey, 2014) [59]; (h) Ford S-Max (Leicht, 2014) [60]; (i) Ford car (Leicht, 2015) [61]; (j) WARDEN (Plessey, 2017) [52].

Figure 5. Potential BCG sensor locations: (a) measuring cardiac acitivity on a weighing scale (craniocaudal component of the BCG momentum); (b) measuring both cardiac and respiratory activiy in bed (dorsoventral component of the BCG momentum).

Figure 6. Potential BCG sensor locations in a car seat. Note that the lumbar and especially the thoracic sensor location in the backrest are likely to have contact problems. By contrast, sensors in the seating area as well as in the safety belts will face vibrations coupled from the vehicle body.

Figure 7. Frequency ranges (VIS, NIR, FIR) usable for optical monitoring techniques.

Figure 8. Proper locations for cameras. Locations can be differentiated by the angle of attack

ρ

Figure 8. Proper locations for cameras. Locations can be differentiated by the angle of attack

ρ

Figure 9. LED and PD placement for transmissive and reflective photoplethysmography. In reflective mode, on average only 50 out of

10^{6}

photons leave the tissue to reach the PD. (a) transmissive PPG (tPPG); (b) reflective PPG (rPPG); (c) typical banana-shaped pathways of scattered photons in rPPG.

Figure 9. LED and PD placement for transmissive and reflective photoplethysmography. In reflective mode, on average only 50 out of

10^{6}

photons leave the tissue to reach the PD. (a) transmissive PPG (tPPG); (b) reflective PPG (rPPG); (c) typical banana-shaped pathways of scattered photons in rPPG.

Figure 10. Principle of PPG imaging (PPGI).

Figure 11. Temperature variation around the nostrils during inspiration and expiration. Thermograms of the nose during (a) inhalation and (b) exhalation from [127] (Figure 1).

Figure 12. Principle of magnetic induction monitoring. (a) single coil approach with frequency modulation and (b) multi-coil approach based on a gradiometer.

Figure 13. Coil locations of published in-car MI monitoring systems.

Figure 14. Gradiometer sensor inside a resin block integrated into the driver seat (a), modified and extended from [165]. Middle and right figure are magnified views of the sensor from top (b) and from the side (c) showing the spatial structure of the gradiometer.

Figure 15. Potential locations of radar sensors inside a car.

Figure 16. Early concept for sensor fusion inside the U-car [34].

Table 1. Timeline of the development of low-contact ECG systems.

1967	First description of insulated electrodes (Richardson, [35])
1993	cECG in objects of daily life (Ishijima, [36])
2004	cECG in a bathtub and toilet seat (Lim, [37], Kim [38])
2006	Integration of cECG into a chair (Lim, [39])
2007	cECG integrated into beds (Lim, [40])
2008	First integration of cECG into an automotive seat (Leonhardt, [42])
2008	Textile cECG electrodes in an automotive seat (Chamadiya, [43])
2011	Evaluation of cECG during test drives (Eilebrecht, [44])
2011	Measurement performance and coverage rate tested during highway and city traffic (Wartzek, [45])
2011	Triboelectricity and motion artifacts identified as factor for robustness (Wartzek, [46])
2011	cECG evaluated as a technology for passenger monitoring in airplanes (Schumm, [47])
2012	Introduction of a quality label to describe cECG signal quality (Schumm, [47])
2012	Textile seat cover for retrofitting seats with cECG systems (Schneider, [48])
2012	cECG used for heart rate variability analysis (HRV) (Jung, [49])
2012	First cECG sensor system commercially available: EPIC system by Plessey Inc.
2015	Active humidification further evaluated (Fong [50], Weder [51])
2016	WARDEN seat cover based system for retrofitting (Plessey, [52])
2017	Closed-loop humidification for artifact suppression (Leicht, [53])
2017	IMEC sensor system fusing cECG and radar (IMEC, [54])
2018	Clinical trial to evalute cECG as a diagnostic tool for heart attack survivors (Leicht, [55])

Table 2. Comparison of different cECG systems integrated into car seats.

System	Year	References	Electrode Properties			Special Properties
			#	Shape	Ground
SMART Test Vehicle	2008	[42]	2	rectangular	in backrest	isolated (coated) metal electrodes
Daimler S-class seat	2008	[43]	2	round	in seat plane
Ford S-Max seat	2011	[44]	6	rectangular	in seat plane	multi-electrode system
Daimler Test Vehicle	2011	[58]	2	rectangular		textile electrodes
Audi Q5	2012	[48]	2	rectangular	in backrest	textile electrodes, removable seat cover
Car seat	2012	[49]	2	rectangular	in seat plane
Ford S-Max seat	2014/2017	[53,60]	2	rectangular	in seat plane	textile electrodes, release of humidity
Plessey EPIC^TM seat	2014	[59]	3	round
Ford S-Max Test Vehicle	2015	[61]	6	round	in seat plane	deep-drawn electrodes
Plessey WARDEN^TM	2016	[52]	6	rectangular	backrest, built-into main unit	removable seat cover
IMEC car seat	2017	[54,65]	6	round		up to 64 electrodes possible

Table 3. Comparison of different camera-based monitoring systems.

Name	Freq. Range	Abb.	Price	Energy	Special Properties
Visible Light (Vis)	350 nm–740 nm	VIS	cheap	ambient light (passive) or	ambient light may fluctuate,
Visible Light (Vis)	350 nm–740 nm	VIS	cheap	active illumination	passive mode does not work at night
Near Infrared (NIR)	740 nm–1 μm	NIR	cheap	active illumination	works at night, long exposure may exhaust the retina
Far Infrared (FIR)	3 μm–5 μm	MWIR	expensive	passive	works at night, but glasses are not transparent
Far Infrared (FIR)	8 μm–14 μm	LWIR	expensive	passive	works at night, but glasses are not transparent

Table 4. Comparison of different names for PPG imaging.

Name	Abbreviation	First Mentioned	References (First Author, Year)
PPG imaging	PPGi/PPGI	2000	Blazek 2000 [81], Wu 2000 [82], Hulsbusch 2002 [83], Blazek 2006 [93],
PPG imaging	PPGi/PPGI	2000	Karlen 2015 [94], Moco 2016 [95], Blazek 2017 [96], Blöcher 2017 [97]
Vital Signs Camera	VSC	2004	Philips Research 2004 [98]
imaging PPG	iPPG/IPPG	2007	Zheng 2007 [86], Hu 2008 [99], Sun 2014 [100], Kamshilin 2015 [101],
imaging PPG	iPPG/IPPG	2007	Karlen 2015 [94], Kuo 2015 [102], Blackford 2016 [103], Sun 2016 [79]
remote PPG (imaging)	n.a.	2008	Verkruysse 2008 [87], McDuff 2015 [104]
remote PPG	rPPG	2015	Kwon 2015 [105], Gastel 2015 [106], Wu 2016 [107], Gastel 2016 [108]
DistancePPG	n.a.	2015	Kumar 2015 [109]
video-based HR	n.a.	2015	Choe 2015 [110]
camera-based PPG	cbPPG	2015	Wedekind 2015 [111], Rasche 2016 [112], Kevat 2017 [113]
non-contact PPG	ncPPG	2016	Butler 2016 [114], Tayibnapis 2016 [115]
video-PPG	vPPG	2016	Iozzia 2016 [116]

Table 5. Comparison of different ambient unobtrusive cardiorespiratory monitoring techniques by their technical aspects. Part of the information in columns 5 to 7 have been taken and adapted from [3]. RL stands for technology readyness level estimated by the authors.

Sensor Technique	Type of Contact	Measured Quantity	Energy Injection	Distance	Sensitivity to Positioning	Costs	RL	In-Car Use
ECG (steering wheel)	galvanic	electric biopotential	no	0	+	o	+	[20,21,24,25,27,28,29,31,32,33]
cECG	capacitive	electric biopotential	no	mm	+	o	o	[42,43,44,48,49,52,58,59,60,61]
BCG	mechanical	displacement, force	no	0	–	o	o	[42,74,75,76,78]
Video motion	optical	displacement	no	m	–	+	+	not yet
PPG (steering wheel)	optical	photon absorption	yes	mm	o	–	+	[23,25,26]
PPGi	optical	photon absorption	yes	m	–	+	o	[115,121]
Thermography	optical	radiation, temperature	no	m	–	++	+	[124,126]
MI	electromagnetic	electric bioimpedance	yes	cm	o	o	–	[74,164,165]
Radar	electromagnetic	displacement, velocity	yes	m	–	o	o	[177,178,179,180]

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Leonhardt, S.; Leicht, L.; Teichmann, D. Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review. Sensors 2018, 18, 3080. https://doi.org/10.3390/s18093080

AMA Style

Leonhardt S, Leicht L, Teichmann D. Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review. Sensors. 2018; 18(9):3080. https://doi.org/10.3390/s18093080

Chicago/Turabian Style

Leonhardt, Steffen, Lennart Leicht, and Daniel Teichmann. 2018. "Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review" Sensors 18, no. 9: 3080. https://doi.org/10.3390/s18093080

APA Style

Leonhardt, S., Leicht, L., & Teichmann, D. (2018). Unobtrusive Vital Sign Monitoring in Automotive Environments—A Review. Sensors, 18(9), 3080. https://doi.org/10.3390/s18093080

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu