at the Chair of Automatic Control (Faculty of Engineering - CAU Kiel)
(Eine Anwendung des Reinforcement Learning zur Regelung dynamischer Systeme)
This Git-Repository holds all the Code written for my Bachelor Thesis at the Chair of Automatic Control at TF Uni Kiel. The Programming Language is Python. NOW IN PYTHON 3!
This bachelor thesis implements a way to achieve a reliable and stable control of a dynamic system through approaches of reinforcement learning. For a neuronal network, the "Touch Withdrawal Circuit" of the worm C. Elegans is examined in great detail and the structures are transformed into a simulator. As a simulation environment, the inverted pendulum is being used with one degree of freedom (1 DOF). To simulate the neural network and gurantee reliable control for the inverted pendulum, a simulator is being developed and implemented using the programming languarge Python. Using the well known Leaky Integrate and Fire model, simulation of internal neural dynamics and processing information within the network is made possible. Furthermore, Pparameters of the network are found using reinforcement learning algorithms and applied to the environment CartPole v0 from OpenAI Gym. The result of this work shows, that it is possible to implement a functional simulator for biological neural networks and to link it with methods of reinforcement learning. After computing multiple simulations, suitable Parameters for the network, which ensure stable control of the inverse pendulum, are found. An application to other simulation environments or with similar neural networks is also possible due to the structure of the simulator.
- Simulation for my neural Circuit was initally to use Roboschool's inverted Pendulum. Because of the high degree of non-linerarity, a new approach is being developed: A new Simulation Environment containing a Wagon which starts some distance out of its reference point. Control Theory would suggest a PT2, the neural Network should solve this on it's own using no control Theory at all. - Depricated due to the good integration of CartPole_v0 from OpenAI Gym
- Got the OpenAI Gym CartPole_v0 Environment up and running - now we have to develop a reliable Reward-System and some good reinforcement Learning algorithms
- Genetic Algorithms still under development.. Need to implement some tensor multiplications
Some Libraries that I used:
Computing is getting more and more complex and takes a huge amount of resources. I'll be using the Google Cloud Plattform from now on to train parameters and weights of my neural Network. More Information in my Bachelor Thesis
- Genetic Algorithm is finally done and works fine - searches for a local maximum instead of a global maximum - as predicted
- BA got some more description of the GA.
- Bachelor Thesis is finally done and being reviewed by several people.
- Genetic Algorithm gets more and more complex
- Cleaning up this Repo - Everything under the master branch now
- Implemented a GUI into main.py for Desktop use and created main_cmd.py for headless simulation use
- More writing for the BA Thesis (see Commit History)
- Wrote the last chapters in my Bachelor Thesis and made the first commit - more to come
- More writing for the BA Thesis (see Commit History)
- New Simulation Results from GCP came in - Parameter Run hit the 200 Reward - Mark and performed very well even without weight optimization
- Did a Screencast of a good run - see /footage
- More writing for the BA Thesis (see Commit History)
- New approach concerning Parameter sweeps - been simulating only half of the desired parameters and then duplicating them for symmetrical use - did improve performance very much
- More writing for the BA Thesis (see Commit History)
- Some little Performance tweaks as well as cleaning up the Repo..
- More writing for the BA Thesis (see Commit History)
- Have been writing a lot this week and tewaking a little bit on my Observe-Method.
- Did some Commentarys on my code as well as some very important little changes
- Added LATEX Files containing my Bachelor Thesis. 4 Weeks until deadline.
- Made my Code ready for Google Cloud Plattform to get some serious simulation done
- Try merging to master-branch
- Merged NeuralCircuit_v2 into master-Branch and did some Debugging
- Finally switched to Python3 in order to use tensorflow for my next projects
- More Error-Debugging and logically thinking about the odd behaviour. Maybe I figure it out in the next days.
- New FileSaver "Hickle" for improved Performance and cross-version Compatibility.
- Got the Google Cloud Plattform Up and Running for training my Network - faster, efficient and free.
- Opened my first GitHub Project Kanban Board to keep track of my tasks
- Created a new Feature-Branch "NeuralCircuit_v2" - rather a Version 2 of my original neural Circuit with many improvements - still trying to figure out this odd "LEFT, RIGHT, LEFT, ..." - Behaviour
- Added Runtimes to calculate how long the trainings will take - need to get some kind of cloud support :)
- Worked very hard on gathering information and writing my Thesis over the last week
- Made some serious changes in the Ecosystem - main.py as the caller-file for modules
- Created random_search_v2 with serious performance improvements and lightweight architecture
- Implemented a new weight-logic and increased my average score from 20/200 to 150/200 - still working on visiualizing the good simulations
- Still somme odd characteristics within the circuit. Maybe letting the RL-Algrithms take control over synapses...
- Halftime-Talk with Prof. Meurer and Alexander Schaum, showing my current Work and presenting the Slide (see docs)
- Got some more Simulation-Runs and I am pretty sure now, why the inverted Pendulum behaves incorrectly sometimes - implementation to be done
- Made a new Inspection Module to print out the parameter Matrices
- First preperations for a genetic algorithm to learn new Parameters
- Did a Status Slide to present my current work on Wed., 18.07.2018 to Prof. Meurer - see docs
- Completely new structures within my Repo - Splitted the Code from main.py to some modules to ensure modular building structures and better work on features.
- Still need to increase my OpenAI Gym Score - it is still not higher than 30/200...
- Random Search for parameter matrices is finally working now. The OpenAI Gym Environment 'Cartpole_v0' performs well and leaves me with a reward of around 50/200 after 10.000 Simulations. It is important to know, that we are randomly simulating 6 [4x4] Matrices and 4 [4x1] vectors.
- New Ideas for further tuning: Genetic Algorithms and Function minimizing through Cost-Function
- Yet another feature folder added: TW-Circuit/new_circuit_symm now with integration on OpenAI Gym CartPole_v0. Still working on tuning the Parameters and weights, still haven't got a good reward system.. but we are getting there!
- Did some Clean-Up to this Repository - only the important code is hosted, other things will remain on my private NextCloud Server. Also tried Branching - need some more info.
- TW-Circuit/new_circuit_SIMCE released based on SIM-CE Simulink Simulation - New neual Network with potential!
- TW-Circuit/new_circuit_TWCIRCUIT released based on Worm Level Control Paper with advanced Matrices and more computational power - Working on this for the next days and maybe implement some kind of simple Simulation (see ToDo's)
- TW-Circuit now takes Gap-Junction-Currents - BUT still outputs weird Plots and Results - PLEASE HELP!
- Some breakthroughs at modelling the formulas from the Paper "TW Neuronal Circuit" - got a very promising LIF Simulation out of variating the Constants in Project "TW-Circuit" - will continue this Idea and maybe implement a Network from there.
- Added a new Project "WaterLevel_RL" which was a first try to implement a Water Level regulator with a nerual network - still under development and not quite sure, if I continue this Idea
- Implementation of a very substantial neural network in lif_model/single_neuron_net.py
- New Library "DyNetwork" for my own neural Network and some other purposes initialized - Still no serious content.