Abstract
Developing robots with animal-like flexibility, adaptability, and robustness remains challenging. However, the neuromuscular system of animals can provide bioinspiration for robotic controller design. In this work, we have developed a bio-inspired simulation environment, GymSlug, for reinforcement learning of motor control sequences based on our prior models of feeding behavior in the marine mollusk Aplysia californica. Using a range of model-free deep reinforcement learning algorithms, we train agents capable of producing motor neural control sequences, muscle activities, and feeding apparatus behavior that are qualitatively similar to behaviors observed in the animal during swallowing of unbreakable seaweed. The robustness of the trained agent is demonstrated by its ability to adapt to a previously unseen environment with breakable seaweed of varying strength. In addition, the environment can be easily reconfigured to train agents for additional tasks, including effective egestion of inedible objects. Our extensible simulation environment provides a platform for developing novel controllers to test biological hypotheses, learn control policies for neurorobotic models, and develop new approaches for soft robotic grasping control inspired by Aplysia.
This work was supported by NSF DBI 2015317 as part of the NSF/CIHR/DFG/FRQ/UKRI-MRC Next Generation Networks for Neuroscience Program.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Gill, J.P., Chiel, H.J.: Rapid adaptation to changing mechanical load by ordered recruitment of identified motor neurons. eNeuro. 7, 1–18 (2020)
Lyttle, D.N., Gill, J.P., Shaw, K.M., Thomas, P.J., Chiel, H.J.: Robustness, flexibility, and sensitivity in a multifunctional motor control model. Biol. Cybern. 111(1), 25–47 (2017)
Shaw, K.M., et al.: The significance of dynamical architecture for adaptive responses to mechanical loads during rhythmic behavior. J. Comput. NeuroSci. 38, 25–51 (2015)
Szczecinski, N.S., et al.: Introducing MantisBot: hexapod robot controlled by a high-fidelity, real-time neural simulation. IEEE Int. Conf. Intell. Robots Syst. 2015, 3875–3881 (2015)
Szczecinski, N.S., Quinn, R.D.: Leg-local neural mechanisms for searching and learning enhance robotic locomotion. Biol. Cybern. 112(1–2), 99–112 (2018)
Sutton, G.P., Mangan, E.V., Neustadter, D.M., Beer, R.D., Crago, P.E., Chiel, H.J.: Neural control exploits changing mechanical advantage and context dependence to generate different feeding responses in Aplysia. Biol. Cybern. 91(5), 333–345 (2004)
Hunt, A., Szczecinski, N., Quinn, R.: Development and training of a neural controller for hind leg walking in a dog robot. Front. Neurorobot. 11, 1–16 (2017)
Webster-Wood, V.A., Gill, J.P., Thomas, P.J., Chiel, H.J.: Control for multifunctionality: bioinspired control based on feeding in Aplysia californica. Biol. Cybern. 114(6), 557–588 (2020)
Mitchell, B.A., Petzold, L.R.: Control of neural systems at multiple scales using model-free, deep reinforcement learning. Sci. Rep. 8, 1–12 (2018)
Li, Z., et al.: Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots (2021)
Song, S., et al.: Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. J. NeuroEng. Rehabil. 18, 126 (2020)
Brockman, G., et al.: OpenAI Gym. arXiv:1606.01540, pp. 1–4 (2016)
Mnih, V., et al.: Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602v1, pp. 1–9 (2013)
Schulman, J., Levine, S., Moritz, P., Jordan, M., Abbeel, P.: Trust region policy optimization. In: 32nd International Conference on Machine Learning, ICML 2015, vol. 3, pp. 1889–1897 (2015)
Hill, A., et al.: Stable Baselines (2018). https://github.com/hill-a/stable-baselines
Yu, H., Xu, W., Zhang, H.: Towards Safe Reinforcement Learning with a Safety Editor Policy (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, W., Xu, M., Gill, J.P., Thomas, P.J., Chiel, H.J., Webster-Wood, V.A. (2022). GymSlug: Deep Reinforcement Learning Toward Bio-inspired Control Based on Aplysia californica Feeding. In: Hunt, A., et al. Biomimetic and Biohybrid Systems. Living Machines 2022. Lecture Notes in Computer Science(), vol 13548. Springer, Cham. https://doi.org/10.1007/978-3-031-20470-8_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-20470-8_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20469-2
Online ISBN: 978-3-031-20470-8
eBook Packages: Computer ScienceComputer Science (R0)