- Edward Sazonov (IEEE M’02, SM’11) received the Diploma of Systems Engineer from Khabarovsk State University of Techno... moreEdward Sazonov (IEEE M’02, SM’11) received the Diploma of Systems Engineer from Khabarovsk State University of Technology, Russia, in 1993 and the Ph.D. degree in Computer Engineering from West Virginia University, Morgantown, WV, in 2002. Currently he is a Professor in the Department of Electrical and Computer Engineering at the University of Alabama, Tuscaloosa, AL and the head of the Computer Laboratory of Ambient and Wearable Systems (http://claws.eng.ua.edu). His research interests span wearable devices, sensor-based behavioral informatics and methods of biomedical signal processing and pattern recognition. Devices developed in his laboratory include a wearable sensor for objective detection and characterization of food intake (AIM – Automatic Ingestion Monitor)edit
Sensor-based food intake monitoring has become one of the fastest-growing fields in dietary assessment. Researchers are exploring imaging-sensor-based food detection, food recognition, and food portion size estimation. A major problem... more
Sensor-based food intake monitoring has become one of the fastest-growing fields in dietary assessment. Researchers are exploring imaging-sensor-based food detection, food recognition, and food portion size estimation. A major problem that is still being tackled in this field is the segmentation of regions of food when multiple food items are present, mainly when similar-looking foods (similar in color and/or texture) are present. Food image segmentation is a relatively under-explored area compared with other fields. This paper proposes a novel approach to food imaging consisting of two imaging sensors: color (Red–Green–Blue) and thermal. Furthermore, we propose a multi-modal four-Dimensional (RGB-T) image segmentation using a k-means clustering algorithm to segment regions of similar-looking food items in multiple combinations of hot, cold, and warm (at room temperature) foods. Six food combinations of two food items each were used to capture RGB and thermal image data. RGB and the...
Research Interests:
Research Interests:
Sensor networks and IoT systems have been widely deployed in monitoring and controlling system. With its increasing utilization, the functionality and performance of sensor networks and their applications are not the only design aims;... more
Sensor networks and IoT systems have been widely deployed in monitoring and controlling system. With its increasing utilization, the functionality and performance of sensor networks and their applications are not the only design aims; security issues in sensor networks attract more and more attentions. Security threats in sensor and its networks could be originated from various sectors: users in cyber space, security-weak protocols, obsolete network infrastructure, low-end physical devices, and global supply chain. In this work, we take one of the emerging applications, advanced manufacturing, as an example to analyze the security challenges in the sensor network. Presentable attacks—hardware Trojan attack, man-in-the-middle attack, jamming attack and replay attack—are examined in the context of sensing nodes deployed in a long-range wide-area network (LoRaWAN) for advanced manufacturing. Moreover, we analyze the challenges of detecting those attacks.
Research Interests:
Walking in real-world environments involves constant decision-making, e.g., when approaching a staircase, an individual decides whether to engage (climbing the stairs) or avoid. For the control of assistive robots (e.g., robotic... more
Walking in real-world environments involves constant decision-making, e.g., when approaching a staircase, an individual decides whether to engage (climbing the stairs) or avoid. For the control of assistive robots (e.g., robotic lower-limb prostheses), recognizing such motion intent is an important but challenging task, primarily due to the lack of available information. This paper presents a novel vision-based method to recognize an individual’s motion intent when approaching a staircase before the potential transition of motion mode (walking to stair climbing) occurs. Leveraging the egocentric images from a head-mounted camera, the authors trained a YOLOv5 object detection model to detect staircases. Subsequently, an AdaBoost and gradient boost (GB) classifier was developed to recognize the individual’s intention of engaging or avoiding the upcoming stairway. This novel method has been demonstrated to provide reliable (97.69%) recognition at least 2 steps before the potential mode...
Research Interests:
Extreme angles in lower body joints may adversely increase the risk of injury to joints. These injuries are common in the workplace and cause persistent pain and significant financial losses to people and companies. The purpose of this... more
Extreme angles in lower body joints may adversely increase the risk of injury to joints. These injuries are common in the workplace and cause persistent pain and significant financial losses to people and companies. The purpose of this study was to predict lower body joint angles from the ankle to the lumbosacral joint (L5S1) by measuring plantar pressures in shoes. Joint angle prediction was aided by a designed footwear sensor consisting of six force-sensing resistors (FSR) and a microcontroller fitted with Bluetooth LE sensors. An Xsens motion capture system was utilized as a ground truth validation measuring 3D joint angles. Thirty-seven human subjects were tested squatting in an IRB-approved study. The Gaussian Process Regression (GPR) linear regression algorithm was used to create a progressive model that predicted the angles of ankle, knee, hip, and L5S1. The footwear sensor showed a promising root mean square error (RMSE) for each joint. The L5S1 angle was predicted to be RMS...
Research Interests:
Research Interests:
Research Interests:
Research Interests:
BackgroundA fast rate of eating is associated with a higher risk for obesity but existing studies are limited by reliance on self-report and the consistency of eating rate has not been examined across all meals in a day. The goal of the... more
BackgroundA fast rate of eating is associated with a higher risk for obesity but existing studies are limited by reliance on self-report and the consistency of eating rate has not been examined across all meals in a day. The goal of the current analysis was to examine associations between meal duration, rate of eating, and body mass index (BMI) and to assess the variance of meal duration and eating rate across different meals during the day.MethodsUsing an observational cross-sectional study design, non-smoking participants aged 18–45 years (N = 29) consumed all meals (breakfast, lunch, and dinner) on a single day in a pseudo free-living environment. Participants were allowed to choose any food and beverages from a University food court and consume their desired amount with no time restrictions. Weighed food records and a log of meal start and end times, to calculate duration, were obtained by a trained research assistant. Spearman's correlations and multiple linear regressions ...
Research Interests:
Measuring humans' functional balance is important for clinical estimation of fall risk. Although many clinical assessments, such as Berg Balance Scale and Mini Balance Evaluation System Test, are available to test the functional... more
Measuring humans' functional balance is important for clinical estimation of fall risk. Although many clinical assessments, such as Berg Balance Scale and Mini Balance Evaluation System Test, are available to test the functional balance, the results are affected by the skills of different operators. This paper proposes an objective approach to access the functional balance by a wearable sensor system embedded in the shoe and a hip accelerometer. Support Vector Machine regression models are built with numerical features selected by mRMR algorithm to estimate the scores of the clinical assessments. Leave one out cross validation is employed to evaluate the regression models. The approach is validated on a group of 30 seniors ($76\pm 10.5$ years old), containing fallers and non-fallers. The results show that the wearable sensor system has a capability to estimate the Berg Balance Scale and Mini Balance Evaluation System Test scores with absolute mean errors and standard deviations $6.07\pm 3.76$ and $5.45\pm 3.65$, respectively, and demonstrates high agreement with falls history based risk assessment.
Research Interests:
ObjectiveTo describe best practices for manual nutritional analyses of data from passive capture wearable devices in free-living conditions.Method18 participants (10 female) with a mean age of 45 ± 10 years and mean BMI of 34.2 ± 4.6... more
ObjectiveTo describe best practices for manual nutritional analyses of data from passive capture wearable devices in free-living conditions.Method18 participants (10 female) with a mean age of 45 ± 10 years and mean BMI of 34.2 ± 4.6 kg/m2 consumed usual diet for 3 days in a free-living environment while wearing an automated passive capture device. This wearable device facilitates capture of images without manual input from the user. Data from the first nine participants were used by two trained nutritionists to identify sources contributing to inter-nutritionist variance in nutritional analyses. The nutritionists implemented best practices to mitigate these sources of variance in the next nine participants. The three best practices to reduce variance in analysis of energy intake (EI) estimation were: (1) a priori standardized food selection, (2) standardized nutrient database selection, and (3) increased number of images captured around eating episodes.ResultsInter-rater repeatabil...
Research Interests:
Objectives We examined the effect of gradually reduced guidance and feedback on portion size and daily reported energy intake (EI) estimation accuracy. We hypothesized that training would result in relatively accurate portion size... more
Objectives We examined the effect of gradually reduced guidance and feedback on portion size and daily reported energy intake (EI) estimation accuracy. We hypothesized that training would result in relatively accurate portion size estimation at a group level and that portion sizes would be most accurately estimated for solid foods compared with amorphous and liquid foods. We also expected that those who performed better in portion size testing to more accurately report EI. Methods In a single session, healthy, nonsmoking participants (n = 29; 38% F; aged 23.0 ± 4.8 y; BMI 22.7 ± 3.2 kg/m2 (mean ± SD)) were given training and gradually reduced written and oral guidance for portion size estimation over three training meals (meals 1–3) followed by one test meal. No instruction was provided for the test meal (meal 4), but oral feedback was provided afterwards. The meals used typical American foods and consisted of food replicas supplemented with real condiments and beverages. The next d...
Research Interests:
Objective:Passive, wearable sensors can be used to obtain objective information in infant feeding, but their use has not been tested. Our objective was to compare assessment of infant feeding (frequency, duration and cues) by self-report... more
Objective:Passive, wearable sensors can be used to obtain objective information in infant feeding, but their use has not been tested. Our objective was to compare assessment of infant feeding (frequency, duration and cues) by self-report and that of the Automatic Ingestion Monitor-2 (AIM-2).Design:A cross-sectional pilot study was conducted in Ghana. Mothers wore the AIM-2 on eyeglasses for 1 d during waking hours to assess infant feeding using images automatically captured by the device every 15 s. Feasibility was assessed using compliance with wearing the device. Infant feeding practices collected by the AIM-2 images were annotated by a trained evaluator and compared with maternal self-report via interviewer-administered questionnaire.Setting:Rural and urban communities in Ghana.Participants:Participants were thirty eight (eighteen rural and twenty urban) breast-feeding mothers of infants (child age ≤7 months).Results:Twenty-five mothers reported exclusive breast-feeding, which wa...
Research Interests:
Rapid weight gain during infancy expands the threat of obesity. Currently, there is a lack of tools to monitor the feeding behavior of infants. This paper presents a sensor system and a method for inspecting satiety cues exhibited by... more
Rapid weight gain during infancy expands the threat of obesity. Currently, there is a lack of tools to monitor the feeding behavior of infants. This paper presents a sensor system and a method for inspecting satiety cues exhibited by infants during feeding in the form of opening/closing of the eyes. Open eyes and vigorous sucking indicate an alert, actively feeding infant, while eye closing in combination with a slower frequency or lower strength of sucking indicate that an infant is getting full and falling asleep. The monitoring is performed by a sensor system consisting of a feeding bottle instrumented with a camera to monitor facial expression and a pressure sensor to monitor sucking. A method for open eye detection is proposed based on the histograms of oriented gradients features and multi-stage cascade classifier. In a pilot study of 4 infants, a recall value of 82.40% and F1-score of 80.79% was achieved while testing on 914 infant images.
Research Interests:
This paper presents a wireless charging system that can power up footwear-based sensors. Loosely coupled magnetic resonant technique was applied to achieve a 20 mm-long charging distance relative to the conventional Qi charging, which... more
This paper presents a wireless charging system that can power up footwear-based sensors. Loosely coupled magnetic resonant technique was applied to achieve a 20 mm-long charging distance relative to the conventional Qi charging, which requires close proximity and precise alignment. Transmitting and receiving resonators are modeled and simulated with a full-wave solver, ANSYS HFSS. Simulation results match measured results with respect to coupling coefficient (k). The resonator coupling efficiency (RCE) for the loosely coupled wireless charging system over a 20 mm gap is 65%, which seems feasible for a small battery-operated sensor system.
Research Interests:
With technological advancement, wearable egocentric camera systems have extensively been studied to develop food intake monitoring devices for the assessment of eating behavior. This paper provides a detailed description of the... more
With technological advancement, wearable egocentric camera systems have extensively been studied to develop food intake monitoring devices for the assessment of eating behavior. This paper provides a detailed description of the implementation of CNN based image classifier in the Cortex-M7 microcontroller. The proposed network classifies the captured images by the wearable egocentric camera as food and no food images in real-time. This real-time food image detection can potentially lead the monitoring devices to consume less power, less storage, and more user-friendly in terms of privacy by saving only images that are detected as food images. A derivative of pre-trained MobileNet is trained to detect food images from camera captured images. The proposed network needs 761.99KB of flash and 501.76KB of RAM to implement which is built for an optimal trade-off between accuracy, computational cost, and memory footprint considering implementation on a Cortex-M7 microcontroller. The image classifier achieved an average precision of 82%±3% and an average F-score of 74%±2% while testing on 15343 (2127 food images and 13216 no food images) images of five full days collected from five participants.
Research Interests:
Purpose The purpose of this study was to examine the dynamic association between lifestyle factors and both positive and negative effect in middle-aged African Americans. Methods Study participants (N = 69, Mean age=51 years, 80% female)... more
Purpose The purpose of this study was to examine the dynamic association between lifestyle factors and both positive and negative effect in middle-aged African Americans. Methods Study participants (N = 69, Mean age=51 years, 80% female) were recruited from two African American churches in the Deep South. Participants completed daily surveys on positive and negative effect, physical activity, sedentary behavior, diet quality, and sleep quality daily for up to 10-days. Mixed-effect models were used to examine associations between the variables of interest. Results On days that participants were more active, they experienced higher mean positive effect (P = .015) and lower mean negative effect (P = .028) scores. Conversely, more time spent sitting in lagged models (i.e., T-1) was associated with higher mean negative effect (P = .001) and lower mean positive effect (P = .040) scores. In lagged models, better sleep quality was associated with higher positive effects (P = .007) scores bu...
Research Interests: Demography and Medicine
Existing methods to assess infant nutritive sucking patterns are often limited to modified bottle apparatuses used solely in the clinic, and time-consuming analyses of videotaped meals by trained observers who count the number of sucks... more
Existing methods to assess infant nutritive sucking patterns are often limited to modified bottle apparatuses used solely in the clinic, and time-consuming analyses of videotaped meals by trained observers who count the number of sucks during a meal. To address the need for a method to efficiently assess meal and sucking patterns of infants across different settings, we developed an unobtrusive Instrumented Bottle (IB) based on a commercially available bottle. Sensors are located in a detachable bottom and include a digital 14-bit pressure sensor that measures pressure inside of the bottle $(range \ \pm 10\mathrm{cm} H20)$, a 3-axis accelerometer that measures orientation of the bottle, and a 5 megapixel camera that takes images of the infant face at preset intervals. In-vitro testing of the IB included bench testing where sucking patterns were generated by a commercially available breast pump, over a range of sucking frequencies and strengths offered by the pump. The mean relative error in suck count estimation was $2\%\pm 1.5{\%}$, assessed against observer annotation of the video of the experiments. Pilot in-vivo testing was performed in small cohort of 4 female infants, age 6–22 weeks, all consuming breast milk from the IB. The mean absolute error in suck count estimation by IB in reference to video-annotated counts was $14\%\pm 51{\%}$, These pilot results suggest that that IB may be an accurate instrument to assess infant feeding, however, larger human studies are needed to assess factors affecting accuracy in invivo.
Research Interests:
Diet is an important aspect of our health. Good dietary habits can contribute to the prevention of many diseases and improve overall quality of life. To better understand the relationship between diet and health, image-based dietary... more
Diet is an important aspect of our health. Good dietary habits can contribute to the prevention of many diseases and improve overall quality of life. To better understand the relationship between diet and health, image-based dietary assessment systems have been developed to collect dietary information. We introduce the Automatic Ingestion Monitor (AIM), a device that can be attached to one’s eye glasses. It provides an automated hands-free approach to capture eating scene images. While AIM has several advantages, images captured by the AIM are sometimes blurry. Blurry images can significantly degrade the performance of food image analysis such as food detection. In this paper, we propose an approach to pre-process images collected by the AIM imaging sensor by rejecting extremely blurry images to improve the performance of food detection.
Research Interests:
The walking speed is an important piece of information in many areas of scientific research. In recent studies, multi-camera based optical motion capture systems have largely been used to obtain accurate information on body motion and... more
The walking speed is an important piece of information in many areas of scientific research. In recent studies, multi-camera based optical motion capture systems have largely been used to obtain accurate information on body motion and walking speed. However, optical motion capture systems are expensive, complex, immobile, limited to laboratory usage as they require the installation of multiple (610) cameras in an enclosed place. The inertial sensor-based other motion capture systems also suffer from drift and have limited accuracy to linear motions. Lack of this motion ground truth has limited further development of the state of art methods for motion estimation. Motivated by that, this paper proposes to employ a single LIDAR based portable system to capture body motion and generate ground truth for walking speed both indoors and outdoors. In a laboratory test involving three participants walking 10 times toward the LIDAR sensor, the body motion obtained from the LIDAR matched with an 8camera based motion capture system with a mean cross-correlation coefficient of 0.997. A novel application area of the LIDAR based motion estimation is also provided in this paper where the LIDAR was employed to obtain a speed profile of the people approaching a staircase. LIDAR derived this speed profile might help develop assistive technologies (such as prosthesis or orthosis modules) for amputees or people in need.
Research Interests:
Objectives Traditional dietary assessment methods in low-middle income countries (LMICs) have considerable limitations. the objective of this study was to test the feasibility of using the Automatic Ingestion Monitor (AIM), an... more
Objectives Traditional dietary assessment methods in low-middle income countries (LMICs) have considerable limitations. the objective of this study was to test the feasibility of using the Automatic Ingestion Monitor (AIM), an eyeglasses-mounted wearable chewing sensor and micro-camera, to monitor food acquisition, preparation and consumption of households in a LMIC setting. Methods Data from households in Mampong-Akuapem (n = 5), a semi-rural community, and Kweiman (n = 7), a peri-urban community, in the Eastern and Greater Accra Regions of Ghana, respectively, were evaluated. The AIM was used to monitor household activities for one day. Within each household, the primary caregiver (mother) wore the AIM during waking hours on the chosen day as she went about her daily activities, except any activities where she wanted to preserve privacy. Mothers also responded to a socio-demographic questionnaire and evaluated their perceived burden of wearing the AIM. Images captured by the AIM w...
Research Interests:
Objectives Traditional dietary assessment methods in low-middle income countries (LMICs) have significant limitations. The objective of this study was to test the feasibility of using the Automatic Ingestion Monitor (AIM), an... more
Objectives Traditional dietary assessment methods in low-middle income countries (LMICs) have significant limitations. The objective of this study was to test the feasibility of using the Automatic Ingestion Monitor (AIM), an eyeglasses-mounted wearable chewing sensor and micro-camera, to monitor food acquisition, preparation and consumption of a household in a LMIC setting. Methods This is a case of an 8-member household from Mampong-Akuapem, a semi-rural community in the Eastern Region of Ghana. The household was made up of mother (35 years), father (37 years), and six children (ages 17 years, 13 years [twins], 8 years, 5 years, and 18 months). Mother has no formal education and works as a cook, whereas the father has elementary education and is a farmer/construction worker. All members of the household consume the same prepared meals. The AIM was used to monitor household activities for a day. The primary food preparer (mother) wore the AIM during all waking hours on the chosen d...
Research Interests:
The field of sensor-based dietary assessment and behavioral monitoring is rapidly expanding. New devices and methods for detection for food intake and characterization of ingestive behavior, energy intake and nutrition have been... more
The field of sensor-based dietary assessment and behavioral monitoring is rapidly expanding. New devices and methods for detection for food intake and characterization of ingestive behavior, energy intake and nutrition have been introduced. Quite often the testing of new devices is limited to restricted meals in laboratory setting, which has the advantage of being controlled, but may not be representative of real life conditions. To illustrate the importance of field testing, we performed a statistical comparison of meal microstructure metrics acquired in laboratory versus a field-like study. In the laboratory study, individual participants ate a self-selected meal in isolation. In the field-like study, participants consumed selfselected meals in a social setting. In both studies, the participants were monitored by both video observation and wearable food intake sensors. Statistically significant differences were observed in the duration of the meals, duration of ingestion, number o...
Research Interests:
Cerebral palsy (CP) is a group of nonprogressive neuro-developmental conditions occurring in early childhood that causes movement disorders and physical disability. Measuring activity levels and gait patterns is an important aspect of CP... more
Cerebral palsy (CP) is a group of nonprogressive neuro-developmental conditions occurring in early childhood that causes movement disorders and physical disability. Measuring activity levels and gait patterns is an important aspect of CP rehabilitation programs. Traditionally, such programs utilize commercially available laboratory systems, which cannot to be utilized in community living. In this study, a novel, shoe-based, wearable sensor system (pediatric SmartShoe) was tested on 11 healthy children and 10 children with CP to validate its use for monitoring of physical activity and gait. Novel data processing techniques were developed to remove the effect of orthotics on the sensor signals. Machine learning models were developed to automatically classify the activities of daily living. The temporal gait parameters estimated from the SmartShoe data were compared against reference measurements on a GAITRite mat. A leave-one-out cross-validation method indicated a 95.3% average accur...
Research Interests:
Insole based wearable sensors are becoming popular in applications such as gait and physical activity monitoring, energy expenditure estimation, in providing biofeedback, fall risk and others. In the existing systems, the insole needs to... more
Insole based wearable sensors are becoming popular in applications such as gait and physical activity monitoring, energy expenditure estimation, in providing biofeedback, fall risk and others. In the existing systems, the insole needs to be taken out of the shoe to recharge the battery, which is not a convenient task. The existing systems are application specific and can be used for a limited purpose. However, the desired system functionality for the shoe based wearable systems depends on factors such as the targeted age group of the individuals and laboratory set up vs free living conditions. We attempt to fill these gaps in this work, as we present the recent advancements in SmartStep monitors - SmartStep 2.0, making it a completely wireless, versatile gait data acquisition device. We discuss the implementation of the seamless wireless charging feature to resolve the battery charging problems, along with the implementation of the base station for charging purposes. In making the SmartStep 2.0 a versatile insole monitor, we discuss the implementation of the Wireless Firmware Upgrade (WFD) feature, which allows the same insole system to be used in different application scenarios. Two example application scenarios for wireless gait data acquisition are discussed, along with the power consumption figures in different modes. Results suggest that the SmartStep 2.0 can potentially be used in the ambulatory monitoring of physical activity and gait in laboratory as well as in free living conditions.
Research Interests:
Research Interests:
Objectives Herein we describe a new system we have developed for assessment of dietary intake, meal timing, and food-related activities, adapted for use in low- and middle-income countries. Methods System components include one or more... more
Objectives Herein we describe a new system we have developed for assessment of dietary intake, meal timing, and food-related activities, adapted for use in low- and middle-income countries. Methods System components include one or more wearable cameras (the Automatic Ingestion Monitor-2 (AIM), an eyeglasses-mounted wearable chewing sensor and micro-camera; ear-worn camera; the eButton, a camera attached to clothes; and eHat, a camera attached to a visor worn by the mother when feeding infants and toddlers), and custom software for evaluation of dietary intake from food-based images and sensor-detected food intake. General protocol: The primary caregiver of the family uses one or more wearable cameras during all waking hours. The cameras aim directly in front of the participant and capture images every few seconds, thereby providing multiple images of all food-related activities throughout the day. The camera may be temporarily removed for short periods to preserve privacy, such as d...
Research Interests:
Research Interests:
This paper presents a real-time classification method of ground-level walking and stair climbing, which is a crucial information of natural human locomotion in robotic prosthesis control. Two Inertial Measurement Units (IMU) were mounted... more
This paper presents a real-time classification method of ground-level walking and stair climbing, which is a crucial information of natural human locomotion in robotic prosthesis control. Two Inertial Measurement Units (IMU) were mounted on an earlier developed measurement exoskeleton system (one IMU in the shank and the other IMU on the thigh) to monitor the locomotion states. A pair of force-sensing resistors were also incorporated into the shoe insole for plantar pressure measurement. The sensors were interfaced with an STM32L476RG microcontroller powered by a rechargeable battery. The data collection was performed on two healthy subjects. Three features (Thigh IMU x-axis accelerometer minimum value, Shank IMU z-axis gyroscope maximum value, and x-axis gyroscope variance) were computed from the sensors signal. Classification of ground-level walking vs. stair climbing events was performed using Linear Discriminant Analysis (LDA). The accuracy, sensitivity, and specificity were obtained on the training set as 96.50%, 96.32%, and 96.66%, respectively. After implementing the classifier in the embedded system, the sensor system was tested in real-time for 26 minutes with an accuracy of 87.21%, the sensitivity of 90.48%, and the specificity of 86.75%. The results indicate that the system can detect the locomotion states with reasonable accuracy, which could be further implemented in determining the control strategy of a powered intelligent prosthesis in the real-time.
Research Interests:
Research Interests:
Objective: No data currently exist on the reproducibility of photographic food records compared to diet diaries, two commonly used methods to measure habitual dietary intake. Our aim was to examine the reproducibility of diet diaries,... more
Objective: No data currently exist on the reproducibility of photographic food records compared to diet diaries, two commonly used methods to measure habitual dietary intake. Our aim was to examine the reproducibility of diet diaries, photographic food records, and a novel electronic sensor, consisting of counts of chews and swallows using wearable sensors and video analysis, for estimating energy intake. Method: This was a retrospective analysis of data from a previous study, in which 30 participants (15 female), aged 29 ± 12 y and having a BMI of 27.9 ± 5.5, consumed three identical meals on different days. Four different methods were used to estimate total mass and energy intake on each day: 1) weighed food record; 2) photographic food record; 3) diet diary; and 4) novel mathematical model based on counts of chews and swallows (CCS models) obtained via the use of electronic sensors and video monitoring system. The Study staff conducted weighed food records for all meals, took pre...
Malnutrition, including both undernutrition and obesity, is a significant problem in low- and middle-income countries (LMICs). In order to study malnutrition and develop effective intervention strategies, it is crucial to evaluate... more
Malnutrition, including both undernutrition and obesity, is a significant problem in low- and middle-income countries (LMICs). In order to study malnutrition and develop effective intervention strategies, it is crucial to evaluate nutritional status in LMICs at the individual, household, and community levels. In a multinational research project supported by the Bill & Melinda Gates Foundation, we have been using a wearable technology to conduct objective dietary assessment in sub-Saharan Africa. Our assessment includes multiple diet-related activities in urban and rural families, including food sources (e.g., shopping, harvesting, and gathering), preservation/storage, preparation, cooking, and consumption (e.g., portion size and nutrition analysis). Our wearable device (“eButton” worn on the chest) acquires real-life images automatically during wake hours at preset time intervals. The recorded images, in amounts of tens of thousands per day, are post-processed to obtain the informat...
Research Interests:
Research Interests:
Research Interests:
Introduction Wearable sensors may be used for the assessment of behavioral manifestations of cigarette smoking under natural conditions. This paper introduces a new camera-based sensor system to monitor smoking behavior. The goals of this... more
Introduction Wearable sensors may be used for the assessment of behavioral manifestations of cigarette smoking under natural conditions. This paper introduces a new camera-based sensor system to monitor smoking behavior. The goals of this study were (1) identification of the best position of sensor placement on the body and (2) feasibility evaluation of the sensor as a free-living smoking-monitoring tool. Methods A sensor system was developed with a 5MP camera that captured images every second for continuously up to 26 hours. Five on-body locations were tested for the selection of sensor placement. A feasibility study was then performed on 10 smokers to monitor full-day smoking under free-living conditions. Captured images were manually annotated to obtain behavioral metrics of smoking including smoking frequency, smoking environment, and puffs per cigarette. The smoking environment and puff counts captured by the camera were compared with self-reported smoking. Results A camera loc...
Research Interests:
Research Interests:
Research Interests:
Imaging-based methods of food portion size estimation (FPSE) promise higher accuracies compared to traditional methods. Many FPSE methods require dimensional cues (fiducial markers, finger-references, object-references) in the scene of... more
Imaging-based methods of food portion size estimation (FPSE) promise higher accuracies compared to traditional methods. Many FPSE methods require dimensional cues (fiducial markers, finger-references, object-references) in the scene of interest and/or manual human input (wireframes, virtual models). This paper proposes a novel passive, standalone, multispectral, motion-activated, structured light-supplemented, stereo camera for food intake monitoring (FOODCAM) and an associated methodology for FPSE that does not need a dimensional reference given a fixed setup. The proposed device integrated a switchable band (visible/infrared) stereo camera with a structured light emitter. The volume estimation methodology focused on the 3-D reconstruction of food items based on the stereo image pairs captured by the device. The FOODCAM device and the methodology were validated using five food models with complex shapes (banana, brownie, chickpeas, French fries, and popcorn). Results showed that th...
Research Interests:
Research Interests:
Any body-mounted wearable sensor that takes periodic pictures is susceptible to motion blur in the captured images. Motion blur may render images useless, and capture of these images consumes battery power. In this paper, an attempt is... more
Any body-mounted wearable sensor that takes periodic pictures is susceptible to motion blur in the captured images. Motion blur may render images useless, and capture of these images consumes battery power. In this paper, an attempt is made to decrease the number of blurred images captured by an eyeglass-mounted camera and increase battery life of the device. The camera motion was detected using an onboard accelerometer and motion derived metrics were used to control image capture. A total of 825 images with corresponding accelerometer data were collected during 2.3 hours in different lighting conditions to train several capture models. Further 1564 images (4.3 hours) were used to test and compare the capture models. The performance of the best model was assessed in an independent experiment, where two devices, one taking pictures at a fixed frequency and one using the motion-adaptive capture were used to collect 650 images (1.8 hours). The motion-adaptive algorithm captured the same number of blur-free images, but reduced power consumption by 12%. The algorithm was found to perform better in the conditions with higher chances of motion blur e.g. in low lighting conditions.
Research Interests:
Research Interests:
Textile-based sensors are being integrated into garments for the monitoring of physiological signals from the human body. Commonly, textile sensors are implemented through knitting methods, while the response of these sensors from other... more
Textile-based sensors are being integrated into garments for the monitoring of physiological signals from the human body. Commonly, textile sensors are implemented through knitting methods, while the response of these sensors from other structures has been less studied. This work analyzed the feasibility of using a textile-based stretch sensor with a coverstitch formation integrated into a commercial shirt for monitoring breathing patterns. For comparison, a Respiratory Inductive Plethysmograph (RIP) based breathing system was used. Data from three subjects performing eight different activities were collected in a controlled environment. The performance of the textile sensor was evaluated based on the mean absolute error breathing rate captured using different segment sizes and as the degree of correlation to the RIP sensor. Results showed an average breathing rate error of 0.97+0.42 breaths/epoch for an epoch size of 10s. The average correlation with the RIP sensor signals was p=0.41+0.2. Results suggest that this garment-integrated sensor could be potentially used in the monitoring of breathing rate.
Research Interests:
Human activity recognition through wearable sensors is becoming integral to health monitoring and other applications. Typically, human activity is captured through signals from inertial sensors, while signals from other sensors have been... more
Human activity recognition through wearable sensors is becoming integral to health monitoring and other applications. Typically, human activity is captured through signals from inertial sensors, while signals from other sensors have been utilized less frequently. In this study, we explored the feasibility of classifying human activities by analyzing the temporal information of respiratory signals through hidden Markov models (HMMs). Left-to-right HMMs were trained for five activities: sedentary, walking, eating, talking, and cigarette smoking. The temporal information from every breathing segment was captured by fragmenting the tidal volume and airflow signals into smaller frames and computing features for each frame. These frames were used as observations to model the states of the HMMs through mixture of Gaussians. Using leave-one-out cross-validation, the classification performance showed an average precision, recall, and F-score of 60.37%, 67.01%, and 62.78%, respectively. Results suggest that respiratory signals can potentially be used as a primary or secondary source in the recognition of some human activities.
Research Interests:
Research Interests:
This paper presents a plantar pressure sensor system (P2S2) integrated in the insoles of shoes to detect thirteen commonly used human movements including walking, stooping left and right, pulling a cart backward, squatting, descending,... more
This paper presents a plantar pressure sensor system (P2S2) integrated in the insoles of shoes to detect thirteen commonly used human movements including walking, stooping left and right, pulling a cart backward, squatting, descending, ascending stairs, running, and falling (front, back, right, left). Six force sensitive resistors (FSR) sensors were positioned on critical pressure points on the insoles to capture the electrical signature of pressure change in the various movements. A total of 34 adult participants were tested with the P2S2. The pressure data were collected and processed using a Principal Component Analysis (PCA) for input to the multiple machine learning (ML) algorithms, including k-NN, neural network and Support-Vector Machine (SVM) algorithms. The ML models were trained using four-fold cross-validation. Each fold kept subject data independent from other folds. The model proved effective with an accuracy of 86%, showing a promising result in predicting human moveme...
Research Interests:
Knowing the amounts of energy and nutrients in an individual’s diet is important for maintaining health and preventing chronic diseases. As electronic and AI technologies advance rapidly, dietary assessment can now be performed using food... more
Knowing the amounts of energy and nutrients in an individual’s diet is important for maintaining health and preventing chronic diseases. As electronic and AI technologies advance rapidly, dietary assessment can now be performed using food images obtained from a smartphone or a wearable device. One of the challenges in this approach is to computationally measure the volume of food in a bowl from an image. This problem has not been studied systematically despite the bowl being the most utilized food container in many parts of the world, especially in Asia and Africa. In this paper, we present a new method to measure the size and shape of a bowl by adhering a paper ruler centrally across the bottom and sides of the bowl and then taking an image. When observed from the image, the distortions in the width of the paper ruler and the spacings between ruler markers completely encode the size and shape of the bowl. A computational algorithm is developed to reconstruct the three-dimensional b...
Research Interests:
In this study, information from surface electromyogram (sEMG) signals was used to recognize cigarette smoking. The sEMG signals collected from lower arm were used in two different ways: (1) as an individual predictor of smoking activity... more
In this study, information from surface electromyogram (sEMG) signals was used to recognize cigarette smoking. The sEMG signals collected from lower arm were used in two different ways: (1) as an individual predictor of smoking activity and (2) as an additional sensor/modality along with the inertial measurement unit (IMU) to augment recognition performance. A convolutional and a recurrent neural network were utilized to recognize smoking-related hand gestures. The model was developed and evaluated with leave-one-subject-out (LOSO) cross-validation on a dataset from 16 subjects who performed ten activities of daily living including smoking. The results show that smoking detection using only sEMG signal achieved an F1-score of 75% in person-independent cross-validation. The combination of sEMG and IMU improved reached the F1-score of 84%, while IMU alone sensor modality was 81%. The study showed that using only sEMG signals would not provide superior cigarette smoking detection perfo...
Research Interests:
Automatic Ingestion Monitor v2 (AIM-2) is an egocentric camera and sensor that aids monitoring of individual diet and eating behavior by capturing still images throughout the day and using sensor data to detect eating. The images may be... more
Automatic Ingestion Monitor v2 (AIM-2) is an egocentric camera and sensor that aids monitoring of individual diet and eating behavior by capturing still images throughout the day and using sensor data to detect eating. The images may be used to recognize foods being eaten, eating environment, and other behaviors and daily activities. At the same time, captured images may carry privacy concerning content such as (1) people in social eating and/or bystanders (i.e., bystander privacy); (2) sensitive documents that may appear on a computer screen in the view of AIM-2 (i.e., context privacy). In this paper, we propose a novel approach based on automatic, image redaction for privacy protection by selective content removal by semantic segmentation using a deep learning neural network. The proposed method reported a bystander privacy removal with precision of 0.87 and recall of 0.94 and reported context privacy removal by precision and recall of 0.97 and 0.98. The results of the study showe...