An Efficient Motion Adjustment Method for a Dual-Arm Transfer Robot Based on a Two-Level Neural Network and a Greedy Algorithm
<p>Operational steps of a patient transfer robot.</p> "> Figure 2
<p>Structure of the developed network.</p> "> Figure 3
<p>Function of the encoder.</p> "> Figure 4
<p>Structure of the first level subnetwork. <b><span class="html-italic">S</span></b> is the heatmap 1, <b>L</b> is the vector field, <b><span class="html-italic">S</span></b>′ is the heatmap 2, C: X-Y represents a convolutional layer, which includes X convolutional kernels with the size of Y × Y, and P:N is a maxpooling layer, where N is the stride of filter.</p> "> Figure 5
<p>Structure of the second level subnetwork. <b>F</b>, <b>F</b>′, <b>Z</b>, and <b>Z</b>′ are feature maps, <b>N</b> is the number of iterations.</p> "> Figure 6
<p>Structure of the multi-head attention module.</p> "> Figure 7
<p>Flowchart for lifting state adjustment. A represents a virtual action set; a is an action in the action set A; s is the current lifting state, and s_ is the next lifting state, which is generated from s and a through a real human-machine system; S_ is the next virtual lifting state set. The virtual human-machine system functions to generate the next virtual lifting state set based on s and an A. While the real human-machine system functions to generate the real next lifting state based on s and a, DNN is the proposed neural network, and <span class="html-italic">Min</span>() is a minimized function, which is capable of choosing a minimum value from an array. <span class="html-italic">F</span> is the virtual contact force set, while <span class="html-italic">f</span> is the real contact force. Arrow in dash functions as updating.</p> "> Figure 8
<p>Sequences of generating the virtual lifting state.</p> "> Figure 9
<p>Platform for data collection.</p> "> Figure 10
<p>Markers for data collection.</p> "> Figure 11
<p>Lifting state adjustment for data collection.</p> "> Figure 12
<p>Typical examples of the CFPR dataset. The first column shows the position of human joints and the lifting points, the second column indicates the weight of the subjects, and the last column depicts the contact force on thigh and back of the experimenters.</p> "> Figure 13
<p>Examples of lifting state adjustment. (<b>a</b>) the lifting state before adjustment; (<b>b</b>) the lifting state after adjustment.</p> ">
Abstract
:1. Introduction
2. Two-Level DNN for Contact Force Estimation
2.1. Encoder Design
2.2. First-Level Subnetwork
2.3. Second-Level Subnetwork
3. Experimental Results
3.1. Validation of the Developed DNN in Nursing Environment
3.2. Practical Application to Dual-Arm Patient Transfer Robot
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Mukai, T.; Hirano, S.; Nakashima, H.; Sakaida, Y.; Guo, S. Realization and safety measures of patient transfer by nursing-care assistant robot riba with tactile sensors. J. Robot. Mechatron. 2011, 23, 360–369. [Google Scholar] [CrossRef]
- Ding, M.; Ikeura, R.; Mori, Y.; Mukai, T.; Hosoe, S. Measurement of human body stiffness for lifting-up motion generation using nursing-care assistant robot—RIBA. In Proceedings of the SENSORS, 2013 IEEE: Piscataway, NJ, USA, Baltimore, MD, USA, 3–6 November 2013; pp. 1–4. [Google Scholar] [CrossRef]
- Mukai, T.; Hirano, S.; Nakashima, H.; Yoshida, M.; Guo, S.; Hayakawa, Y. Manipulation using tactile information for a nursing-care assistant robot in whole-body contact with the object. Trans. Jpn. Soc. Mech. Eng. Ser. C 2011, 77, 3794–3807. [Google Scholar] [CrossRef]
- Mukai, T.; Onishi, M.; Odashima, T.; Hirano, S.; Luo, Z. Development of the tactile sensor system of a human-interactive robot “RI-MAN”. IEEE Trans. Robot. 2008, 24, 505–512. [Google Scholar] [CrossRef]
- Liu, Y.; Guo, S.; Yin, Y.; Jiang, Z.; Liu, T. Design and compliant control of a piggyback transfer robot. J. Mech. Robot. 2022, 14, 031009. [Google Scholar] [CrossRef]
- Liu, Y.; Chen, G.; Liu, J.; Guo, S.; Mukai, T. Biomimetic design of a chest carrying nursing-care robot for transfer task. In Proceedings of the 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia, 12–15 December 2018. [Google Scholar]
- Li, Y.; Guo, S.; Zhu, L.; Mukai, T.; Gan, Z. Enhanced probabilistic inference algorithm using probabilistic neural networks for learning control. IEEE Access 2019, 7, 184457–184467. [Google Scholar] [CrossRef]
- Li, Y.; Guo, S.; Zhu, L.; Mukai, T.; Gan, Z. A recurrent reinforcement learning approach applicable to highly uncertain environments. Int. J. Adv. Robot. Syst. 2020, 17, 1729881420916258. [Google Scholar] [CrossRef]
- Mukai, T.; Hirano, S.; Nakashima, H.; Kato, Y.; Sakaida, Y.; Guo, S.; Hosoe, S. Development of a nursing-care assistant robot RIBA that can lift a human in its arms. In Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, 18–22 October 2010. [Google Scholar]
- Mukai, T.; Hirano, S.; Yoshida, M.; Nakashima, H.; Guo, S.; Hayakawa, Y. Whole-body contact manipulation using tactile information for the nursing-care assistant robot riba. In Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA, 25–30 September 2011. [Google Scholar]
- Mukai, T.; Hirano, S.; Yoshida, M.; Nakashima, H.; Guo, S.; Hayakawa, Y. Tactile-based motion adjustment for the nursing-care assistant robot RIBA. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011. [Google Scholar]
- Zyada, Z.; Hayakawa, Y.; Hosose, S. Kinematic analysis of a two-link object for whole arm manipulation. In Proceedings of the Proceedings of the 9th WSEAS International Conference on Signal Processing, Robotics and Automation, Stevens Point, WI, USA, 20–22 February 2010. [Google Scholar]
- Ding, M.; Ikeura, R.; Mukai, T.; Nagashima, H.; Hirano, S.; Matsuo, K.; Hosoe, S. Comfort estimation during lift-up using nursing-care robot—RIBA. In Proceedings of the 2012 First International Conference on Innovative Engineering Systems, Alexandria, Egypt, 7–9 December 2012. [Google Scholar]
- Ji, Z.; Wang, H.; Jiang, G.; Li, L. Analysis of muscle activity utilizing bench presses in the AnyBody simulation modelling system. Model. Simul. Eng. 2016, 1, 3649478. [Google Scholar] [CrossRef]
- Ueda, J.; Ming, D.; Krishnamoorthy, V.; Shinohara, M.; Ogasawara, T. Individual muscle control using an exoskeleton robot for muscle function testing. IEEE Trans. Neural Syst. Rehabil. Eng. 2010, 18, 339–350. [Google Scholar] [CrossRef]
- Ding, M.; Matsubara, T.; Funaki, Y.; Ikeura, R.; Mukai, T.; Ogasawara, T. Generation of comfortable lifting motion for a human transfer assistant robot. Int. J. Intell. Robot. Appl. 2017, 1, 74–85. [Google Scholar] [CrossRef]
- Frazier, R.M.; Carter-Templeton, H.; Wyatt, T.H.; Wu, L. Current trends in robotics in nursing patents—A glimpse into emerging innovations. CIN Comput. Inform. Nurs. 2019, 37, 290–297. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Kim, H.; Phan, T.Q.; Hong, W.; Chun, S.Y. Physiology-based augmented deep neural network frameworks for ECG biometrics with short ECG pulses considering varying heart rates. Pattern Recognit. Lett. 2022, 156, 1–6. [Google Scholar] [CrossRef]
- Yan, P.; Duan, S.; Luo, X.; Zhang, N.; Deng, Y. Development and validation of a deep neural network–based model to predict acute kidney injury following intravenous administration of iodinated contrast media in hospitalized patients with chronic kidney disease: A multicohort analysis. Nephrol. Dial. Transplant. 2022, 38, 352–361. [Google Scholar] [CrossRef] [PubMed]
- Olamat, A.; Ozel, P.; Atasever, S. Deep learning methods for multi-channel EEG-based emotion recognition. Int. J. Neural Syst. 2022, 32, 2250021. [Google Scholar] [CrossRef] [PubMed]
- Yun, M.; Kim, J.; Do, K. Estimation of wave-breaking index by learning nonlinear relation using multilayer neural network. J. Mar. Sci. Eng. 2022, 10, 50–66. [Google Scholar] [CrossRef]
- Gao, Q.; Liu, J.; Ju, Z.; Zhang, X. Dual-hand detection for human–robot interaction by a parallel network based on hand detection and body pose estimation. IEEE Trans. Ind. Electron. 2019, 66, 9663–9672. [Google Scholar] [CrossRef]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar]
- Wang, X.; Girshick, R.; Gupta, A.; He, K. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Cao, Z.; Hidalgo, G.; Simon, T.; Wei, S.E.; Sheikh, Y. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 172–186. [Google Scholar] [CrossRef] [PubMed]
- Chen, M.; Wu, J.; Li, S.; Liu, J.; Yokota, H.; Guo, S. Accurate and real-time human-joint-position estimation for a patient-transfer robot using a two-level convolutional neutral network. Robot. Auton. Syst. 2021, 139, 103735. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Jiang, S.; Min, W.; Mei, S. Hierarchy-dependent cross-platform multi-view feature learning for venue category prediction. IEEE Trans. Multimed. 2018, 21, 1609–1619. [Google Scholar] [CrossRef]
- Min, W.; Mei, S.; Liu, L.; Wang, Y.; Jiang, S. Multi-task deep relative attribute learning for visual urban perception. IEEE Trans. Image Process. 2019, 29, 657–669. [Google Scholar] [CrossRef] [PubMed]
- Abdeljawad, A.; Grohs, P. Approximations with deep neural networks in sobolev time-space. Anal. Appl. 2022, 20, 499–541. [Google Scholar] [CrossRef]
- Lin, Z.; Akin, H.; Rao, R.; Hie, B.; Zhu, Z.; Lu, W.; Rives, A. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 2023, 379, 1123–1130. [Google Scholar] [CrossRef] [PubMed]
- Vokoun, C.R.; Huang, X.; Jackson, M.B.; Basso, M.A. Response normalization in the superficial layers of the superior colliculus as a possible mechanism for saccadic averaging. J. Neurosci. 2014, 34, 7976–7987. [Google Scholar] [CrossRef] [PubMed]
- Tang, J.; Tang, X.; Lim, A.; Han, K.; Li, C.; Yuan, J. Revisiting modified greedy algorithm for monotone submodular maximization with a knapsack constraint. Proc. ACM Meas. Anal. Comput. Syst. 2021, 5, 1–22. [Google Scholar]
- Moon, G.; Chang, J.Y.; Suh, Y.; Lee, K.M. Holistic planimetric prediction to local volumetric prediction for 3d human pose estimation. arXiv 2017, arXiv:1706.0475. [Google Scholar]
Action | a1 | a2 | a4 | a5 | a6 | a7 | a8 | a9 |
---|---|---|---|---|---|---|---|---|
σ1, σ2 | 0, 0 | 0, 5 | 5, 0 | 5, 5 | 5, −5 | −5, 0 | −5, 5 | −5, −5 |
Method | Back | Thigh | Average | Speed (ms) |
---|---|---|---|---|
Method in [12] | 5 | 20.1 | 12.6 | 1 |
Method in [16] | 21 | 62.5 | 41.8 | 2 |
Our method | 80.7 | 87.3 | 84.0 | 30 |
Subjects | Gender | Age (Y/O) | Weight (kg) |
---|---|---|---|
Sub1 | Male | 29 | 81 |
Sub2 | Female | 41 | 66 |
Sub3 | Male | 29 | 78 |
Sub4 | Female | 35 | 52 |
Sub5 | Male | 44 | 65 |
Sub6 | Male | 56 | 70 |
Sub7 | Male | 30 | 76 |
Sub8 | Female | 30 | 65 |
Sub9 | Male | 21 | 76 |
Sub10 | Male | 46 | 60 |
Subjects | Thigh | Back | ||||
---|---|---|---|---|---|---|
Before | After | Change | Before | After | Change | |
Sub1 | 512.75 | 436.15 | −2.96 | 161.47 | 79.61 | −79.20 |
Sub2 | 447.07 | 444.11 | −65.23 | 209.10 | 129.90 | −72.32 |
Sub3 | 477.16 | 411.93 | −75.53 | 189.67 | 117.35 | −36.26 |
Sub4 | 322.27 | 246.74 | −16.22 | 129.85 | 93.59 | −34.94 |
Sub5 | 375.87 | 359.65 | −6.22 | 111.03 | 76.09 | −39.63 |
Sub6 | 423.66 | 417.44 | −102.05 | 183.40 | 143.77 | −36.92 |
Sub7 | 387.91 | 285.86 | −1.89 | 158.71 | 121.79 | +41.00 |
Sub8 | 359.55 | 357.66 | −56.00 | 45.75 | 86.75 | −78.23 |
Sub9 | 423.25 | 367.25 | +43.81 | 130.71 | 52.48 | +60.38 |
Sub10 | 278.46 | 322.27 | −76.60 | 69.46 | 129.85 | −81.86 |
Subject | Before | After | Change |
---|---|---|---|
Sub1 | 4 | 5 | +1 |
Sub2 | 2 | 5 | +3 |
Sub3 | 5 | 4 | +1 |
Sub4 | 2 | 3 | +1 |
Sub5 | 2 | 3 | +1 |
Sub6 | 3 | 4 | +1 |
Sub7 | 1 | 4 | +3 |
Sub8 | 1 | 4 | +3 |
Sub9 | 4 | 3 | −1 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, M.; Liu, Q.; Wang, K.; Yang, Z.; Guo, S. An Efficient Motion Adjustment Method for a Dual-Arm Transfer Robot Based on a Two-Level Neural Network and a Greedy Algorithm. Electronics 2024, 13, 3090. https://doi.org/10.3390/electronics13153090
Chen M, Liu Q, Wang K, Yang Z, Guo S. An Efficient Motion Adjustment Method for a Dual-Arm Transfer Robot Based on a Two-Level Neural Network and a Greedy Algorithm. Electronics. 2024; 13(15):3090. https://doi.org/10.3390/electronics13153090
Chicago/Turabian StyleChen, Mengqian, Qiming Liu, Kai Wang, Zhiqiang Yang, and Shijie Guo. 2024. "An Efficient Motion Adjustment Method for a Dual-Arm Transfer Robot Based on a Two-Level Neural Network and a Greedy Algorithm" Electronics 13, no. 15: 3090. https://doi.org/10.3390/electronics13153090