Abstract
Food sharing and communal eating are very common in some countries. To assess individual dietary intake in food sharing scenarios, this work proposes a vision-based approach to first capturing the food sharing scenario with a 360-degree camera, and then using a neural network to infer different eating states of each individual based on their body pose and relative positions to the dishes. The number of bites each individual has taken of each dish is then deduced by analyzing the inferred eating states. A new dataset with 14 panoramic food sharing videos was constructed to validate our approach. The results show that our approach is able to reliably predict different eating states as well as individual’s bite count with respect to each dish in food sharing scenarios.
Supported by the Innovative Passive Dietary Monitoring Project funded by the Bill & Melinda Gates Foundation (Opportunity ID: OPP1171395).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – Mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 446–461. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_29
Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., Sheikh, Y.A.: Openpose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. (2019)
Chen, J., Ngo, C.W.: Deep-based ingredient recognition for cooking recipe retrieval. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 32–41 (2016)
Doulah, A., Ghosh, T., Hossain, D., Imtiaz, M.H., Sazonov, E.: Automatic ingestion monitor version 2–a novel wearable device for automatic food intake detection and passive capture of food images. IEEE J. Biomed. Health Informatics (2020)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Liu, J., et al.: An intelligent food-intake monitoring system using wearable sensors. In: 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks, pp. 154–160. IEEE (2012)
Lo, F.P.W., Sun, Y., Qiu, J., Lo, B.: Food volume estimation based on deep learning view synthesis from a single depth map. Nutrients 10(12), 2005 (2018)
Lo, F.P.W., Sun, Y., Qiu, J., Lo, B.P.: Point2volume: a vision-based dietary assessment approach using view synthesis. IEEE Trans. Ind. Informatics 16(1), 577–586 (2019)
Martinel, N., Foresti, G.L., Micheloni, C.: Wide-slice residual networks for food recognition. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 567–576. IEEE (2018)
Meyers, A., et al.: Im2calories: towards an automated mobile vision food diary. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1233–1241 (2015)
Min, W., Liu, L., Luo, Z., Jiang, S.: Ingredient-guided cascaded multi-attention network for food recognition. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 1331–1339 (2019)
Qiu, J., Lo, F.P.W., Lo, B.: Assessing individual dietary intake in food sharing scenarios with a 360 camera and deep learning. In: 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN), pp. 1–4. IEEE (2019)
Qiu, J., Lo, F.P.W., Jiang, S., Tsai, C., Sun, Y., Lo, B.: Counting bites and recognizing consumed food from videos for passive dietary monitoring. IEEE J. Biomed. Health Informatics (2020)
Qiu, J., Lo, F.P.W., Sun, Y., Wang, S., Lo, B.: Mining discriminative food regions for accurate food recognition. In: British Machine Vision Conference (2019)
Rouast, P.V., Adam, M.T.: Learning deep representations for video-based intake gesture detection. IEEE J. Biomed. Health Informatics 24(6), 1727–1737 (2019)
Sun, M., et al.: An exploratory study on a chest-worn computer for evaluation of diet, physical activity and lifestyle. J. Healthcare Eng. 6(1), 1–22 (2015)
Yanai, K., Kawano, Y.: Food image recognition using deep convolutional network with pre-training and fine-tuning. In: 2015 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1–6. IEEE (2015)
Zhu, F., Bosch, M., Khanna, N., Boushey, C.J., Delp, E.J.: Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE J. Biomed. Health Informatics 19(1), 377–388 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lei, J., Qiu, J., Lo, F.PW., Lo, B. (2021). Assessing Individual Dietary Intake in Food Sharing Scenarios with Food and Human Pose Detection. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12665. Springer, Cham. https://doi.org/10.1007/978-3-030-68821-9_45
Download citation
DOI: https://doi.org/10.1007/978-3-030-68821-9_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68820-2
Online ISBN: 978-3-030-68821-9
eBook Packages: Computer ScienceComputer Science (R0)
