8000 Allow Subprocesses from Filesystem : ESP32? : · Issue #4480 · micropython/micropython · GitHub
[go: up one dir, main page]
Skip to content

Allow Subprocesses from Filesystem : ESP32? : #4480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
oclyke opened this issue Feb 8, 2019 · 6 comments
Closed

Allow Subprocesses from Filesystem : ESP32? : #4480

oclyke opened this issue Feb 8, 2019 · 6 comments

Comments

@oclyke
Copy link
Contributor
oclyke commented Feb 8, 2019

Hello everyone.

My goal is to be able to dynamically select Python files from the filesystem and run them 'concurrently' - also choosing when to stop one and begin another. In an ideal world you could provide the equivalent to a 'yield()' call in those python files that would allow others to execute, but running them end to start could work as well. I am raising this issue to a) seek advice from those who are more familiar with the complexities here and b) see if anyone is interested in such an ability / would like to collaborate to make it happen.

I have considered a few possible ways to make this happen, but I don't know which fits best in the ideology of MicroPython or which is the easiest to implement.

(My sights are focused on the ESP32 port but if this could be a MicroPython wide addition then I'm game)

  1. It seems like the Python 'subprocess' module could at least run one file and then another from a Python script, and I could imagine setting up a method for users to start/stop execution of certain files with some kind of user interface. But from my very limited knowledge of the module it seems like there would be no way to yield execution part-way though.
    • If this is the right way to go I'd need to read up on how to port a module. With a little time, guidance, or combination I could make it happen.
    • This might be a nice way to avoid having to worry about memory and storing states of new interpreters?
  2. Since the interpreter is run underneath a FreeRTOS task, and the 'pyexec_file()' function exists I'd be very happy to be able to create another task, and allow users to call the FreeRTOS 'yield()' function from a python module. I think this is my preferred solution but it goes against 'anything that can be written in Python should be.'
    • If going this way my main concern would be making sure there are no conflicts between states of the main and spawned interpreters.
    • I've begun looking through the code to try to identify global state variables that would need to be isolated, but I feel a little overwhelmed. Advice from someone who is familiar with the state requirements of the interpreter would be so greatly appreciated.
    • One last concern here is how to decide how much memory each process should be allotted.
  3. I briefly considered the '_thread' module but I don't think it will work to start a whole file... Am I wrong? Maybe users could write module files with a common 'run' function and then import / del the modules as needed - and somehow call the run function for the right module as a thread in the main process?
    • The way I see it this is the least favorable solution, though if it could work it would require little to no new development.

I'd really appreciate advice / thoughts on this topic. If it can be a worthy addition to MicroPython then I'll be eager to help out! If it's not right then I'd still appreciate advice on how to proceed for my own extension that I can offer.

I guess that about wraps it up for now. Looking forward to talking about this. Thanks!
@oclyke
Copy link
Contributor Author
oclyke commented Feb 18, 2019

In the meantime I've begun poking around trying to make this happen - mostly doing a lot of reading about cPython and some digging around in MicroPython source. One thing I noticed of interest is this comment from py/mpstate.h:

// This file contains structures defining the state of the MicroPython memory system, runtime and virtual machine. The state is a global variable, but in the future it is hoped that the state can become local.

So it looks like at one point there were other people interested in this improvement.

@dpgeorge
Copy link
Member

My goal is to be able to dynamically select Python files from the filesystem and run them 'concurrently' - also choosing when to stop one and begin another.

There are 3 main ways to do things concurrently in MicroPython:

  1. cooperatively within a single execution environment, most notably with uasyncio
  2. preemptively within a single execution environment, with _thread
  3. within separate execution environments

1 and 2 already exist and work well, and are characterised by the fact that all Python code has access to the same heap (so can share objects). But 3 is probably what you are after here: the ability to instantiate separate VM contexts to run completely isolated Python code. In such a case you'd need a separate heap for each instance, so the separately executing code could only share data via OS primitives. This would be pretty much equivalent to subprocesses in CPython.

One thing I noticed of interest is this comment from py/mpstate.h:

// This file contains structures defining the state of the MicroPython memory system, runtime and virtual machine

Yes, this is how it would be done, to provide a separate mp_state_ctx for each "isolated process", which could independently load and execute a .py file. I've had success with multiple mp_state_ctx in a few projects. The main point is that, because mp_state_ctx is accessed as a global variable, it needs to be converted to a pointer (via the MP_STATE_xxx macros) and this pointer swapped whenever there is a switch from one "process" to the next, so that the currently running "process" points to its corresponding state.

@oclyke
Copy link
Contributor Author
oclyke commented Feb 19, 2019

@dpgeorge A big thanks for sharing your time with me.

You're right, option #3 is my aim. May have jumped the gun on naming this issue after the subprocess module - upon re-inspection it seems to handle general command line arguments to the OS which is more functionality than I need.

I'm encouraged to hear that you've used multiple mp_state_ctx to run isolated processes - were there any other 'gotchyas' that I should be aware of? So far I've been hunting around in the source to find global variables but that whole process is getting to be a little too 'Salem' for me.

A rough outline of my upcoming strategy is this:

  1. Make a fork of MicroPython dedicated to this feature addition
  2. Identify the majority of places where conflicts between states may occur, either with the advice of experts or by trying to run clang's 'scan-build' (or ideally both)
  3. Encapsulate these conflicts in a logical manner so as to reduce their scope
  4. Come up with a port-independent API to instantiate new processes - ideally to create a standard MicroPython external module
  5. Implement that API in the ESP32 port as an example

I feel very comfortable with steps 1 and 3-5, particularly if there is input from other people as to what they'd like to see from such a module. The part that worries me is still trying to be sure that I don't have any scope conflicts with other state variables - any advice about this would certainly be appreciated.

Soon I will make a fork for this purpose and post the link here so that curious, helpful, or any other kind of person can see what's happening.

@oclyke
Copy link
Contributor Author
oclyke commented Feb 21, 2019 •
edited
Loading

I've created a fork of micropython to work on localizing the interpreter. The goal is to ultimately create a standardized method of creating new interpreters to run alongside existing interpreters.

Check out the fork to see what's happening, contribute to the discussion, and try it out on your own hardware.

Branches

  • master : micropython master branch for reference (as of 2/21/19)
  • multi-master : solidified changes for this development
  • multi-localization : identifying and removing global state variables
  • multi-interface : defining a standard interface to be included as a module in micropython

Edit: I re-forked to make a future PR as simple as possible with only minimal changes. No more crazy branches.

@oclyke oclyke closed this as completed Feb 28, 2019
@Joseph-93
Copy link

I'm attempting to do a similar process soon, and I'd like to see whatever code you have, to better understand how you solved the problem. Is there a link I may see for it?

@oclyke
Copy link
Contributor Author
oclyke commented Jan 4, 2021

@Joseph-93 sure thing, here it is. Got it working at a basic level with a simple module to control individual processes, though I have since abandoned the effort.
https://github.com/oclyke/multipython-archive

tannewt pushed a commit to tannewt/circuitpython that referenced this issue Mar 25, 2021
@ladyada
Merge pull request micropython#4480 from tannewt/raspberrypi_link_lin…
7831be5 
…e_length

Use GCC @file to shorten linker command length
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dpgeorge @oclyke @Joseph-93
0