WIP adding a new tool to access a MicroPython target with host filesystem mounted on the board #3034
Conversation
|
Because this tool creates Python-level file objects it requires something like #2824 to get to a fully working state. Alternatively the RemoteFile object could be implemented in C but that's a pretty dirty solution. |
|
Sounds interesting. I'm going to have to play with this. |
|
Here is an example session: |
|
I've now pushed some more commits to this branch which add io.IOBase so that files can be implemented in pure Python, and improvements to extmod/vfs.c to allow importing of files to go via a Python hook. I've also updated the mprepl.py script to allow writing to files on the host PC, and also importing files from the host PC. It now enables the following workflow:
Any feedback on this new tool is welcome. |
|
Really nice! I have played around with it a little bit and have not found any issues. A future improvement could be to keep the command history in the tool, that way it would survive a soft reboot. |
|
So, I'm surprised to see such tool, I would think it as not a natural direction to want to move "PC" into a "pyboard", the normal direction, of making "pyboard" seamless accessible from "PC", i.e. elaborating pyboard.py tool as a more natural. It's also pretty cunning way to sneak in the stream protocol implementation for Python classes ;-). But my only real concern is the following: how about we don't propagate usage of unfinished VFS API, with known drawbacks? As mentioned in the previous reviews, listdir() should NOT be in the basis of a VFS class. Instead, ilistdir() should be, which allows to enumerate files in a fixed amount of RAM. listdir() then should be implemented in terms of ilistdir() in a generic way in VFS support code. Summing up, it would be nice to finalize VFS API first, which also includes #2950 (if something there depends on me, please push me, though I'm in general +1 to merge it if it provides needed external functionality, we can optimize implementation later). Thanks. |
|
+1 +1
|
|
Nice! A couple of oddities. If an SD card is fitted I'm seeing $ ./mprepl.py
Traceback (most recent call last):
File "./mprepl.py", line 384, in <module>
main()
File "./mprepl.py", line 379, in main
main_loop(console, dev)
File "./mprepl.py", line 321, in main_loop
pyb.exec_(fs_hook_code)
File "/mnt/qnap2/data/Projects/MicroPython/micropython/tools/pyboard.py", line 336, in exec_
raise PyboardError('exception', ret, ret_err)
pyboard.PyboardError: ('exception', b'', b'Traceback (most recent call last):\r\n File "<stdin>", line 1, in <module>\r\n File "io.py", line 1, in <module>\r\nImportError: no module named \'uasyncio\'\r\n')As the error suggests, there is no copy of uasyncio available on /flash, /sd, or as a frozen module. If I make uasyncio available on the SD card I get $ ./mprepl.py
Traceback (most recent call last):
File "./mprepl.py", line 384, in <module>
main()
File "./mprepl.py", line 379, in main
main_loop(console, dev)
File "./mprepl.py", line 321, in main_loop
pyb.exec_(fs_hook_code)
File "/mnt/qnap2/data/Projects/MicroPython/micropython/tools/pyboard.py", line 336, in exec_
raise PyboardError('exception', ret, ret_err)
pyboard.PyboardError: ('exception', b'About to yield\r\n', b'Traceback (most recent call last):\r\n File "<stdin>", line 1, in <module>\r\n File "io.py", line 36, in <module>\r\n File "uasyncio/core.py", line 129, in run_forever\r\n File "uasyncio/core.py", line 102, in run_forever\r\n File "uasyncio/__init__.py", line 21, in add_reader\r\nOSError: stream operation not supported\r\n')Secondly, if I import a file it sometimes appears to hang but a second enter runs it. |
|
@pfalcon I'm thinking it would be nice to merge the C-level patches from this PR (addition of uio.IOBase and VFS being able to import-stat a generic FS) for v1.9. Why? because it goes that "last mile" with the VFS and rounds it off nicely, allowing one to implement a full FS -- and files -- in pure Python. |
|
If it's configurable and off by default - sure, why not. |
|
Ok, I decided not to merge the C-level patches from this PR just yet. There're some things to discuss and get right before it can go it. So leaving it until after v1.9. |
|
Wow, that’s a very cool tool! Why have I never heard of this before? Now if it would work over WebREPL that would be even cooler! It doesn’t totally work for me yet on macOS with an ESP8266 board running MicroPython v1.10-8-g8b7039d7d. I can open(), read(), and write() files, but importing doesn’t work. If I have a file hello.py on the host containing Also, files can’t be used in a I haven’t tried debugging or fixing any of this yet. |
|
👍 That appears to have fixed the import case! Here’s how I added the context manager and --- mprepl.py
+++ mprepl.py
@@ -27,6 +27,7 @@
CMD_CLOSE = 5
CMD_READ = 6
CMD_WRITE = 7
+CMD_SEEK = 8
fs_hook_code = """\
import os, io, select, ustruct as struct, micropython
@@ -37,6 +38,7 @@
CMD_CLOSE = 5
CMD_READ = 6
CMD_WRITE = 7
+CMD_SEEK = 8
class RemoteCommand:
def __init__(self):
try:
@@ -136,6 +138,18 @@
n = self.cmd.rd_int32()
self.cmd.end()
return n
+ def seek(self, offset, whence=0):
+ self.cmd.begin(CMD_SEEK)
+ self.cmd.wr_int32(self.fd)
+ self.cmd.wr_int32(offset)
+ self.cmd.wr_uint32(whence)
+ pos = self.cmd.rd_int32()
+ self.cmd.end()
+ return pos
+ def __enter__(self):
+ return self
+ def __exit__(self, exc_type, exc_value, traceback):
+ self.close()
class RemoteFS:
def mount(self, readonly, mkfs):
@@ -221,7 +235,7 @@
self.fin = fin
self.fout = fout
def rd_uint32(self):
- return struct.unpack('<I', self.rd(4))[0]
+ return struct.unpack('<I', self.fin.read(4))[0]
def wr_uint32(self, i):
self.fout.write(struct.pack('<I', i))
def rd_int32(self):
@@ -315,6 +329,13 @@
n = data_files[fd][0].write(buf)
cmd.wr_int32(n)
+def do_seek(cmd):
+ fd = cmd.rd_int32()
+ offset = cmd.rd_int32()
+ whence = cmd.rd_uint32()
+ pos = data_files[fd][0].seek(offset, whence)
+ cmd.wr_int32(pos)
+
cmd_table = {
CMD_STAT: do_stat,
CMD_ILISTDIR_START: do_ilistdir_start,
@@ -323,6 +344,7 @@
CMD_CLOSE: do_close,
CMD_READ: do_read,
CMD_WRITE: do_write,
+ CMD_SEEK: do_seek,
}
def main_loop(console, dev): |
|
Heads-up: I almost have it running over WebREPL (with some hacks and open questions). The current roadblock is that WebREPL cannot transfer binary data that is not UTF-8, because it just packs it into WebSocket text messages without checking, which causes conformant recipients to close the connection. Will have to find a way to fix that first. |
To run before interactive console.
... from pc to micropython When the repl serial port is shared between commands and binary data, it can be mixed up if text is written on the console while commands are trying to run in the background. Using the second usb serial port (optionally) available in micropython prvents this from being possible, meaning large blocks of text can be pasted onto the console and it all runs correctly.
…read 6D40 . Flush input buffer when a command begins
On remote repl or in provided script, running exit() with or without exit code, eg exit(-1) will close the script on pc.
M/S Terminology cleanup
|
Superseded by #6375 |
This has been under development since April 2017. See micropython#3034 and micropython#6375. Signed-off-by: Damien George <damien@micropython.org>
This has been under development since April 2017. See micropython#3034 and micropython#6375. Signed-off-by: Damien George <damien@micropython.org>
This has been under development since April 2017. See micropython#3034 and micropython#6375. Signed-off-by: Damien George <damien@micropython.org>
The aim of this tool is to allow easy and rapid development of programs that run on pyboards (in the general sense, ie anything that runs MicroPython and has a REPL available via a serial port).
The main feature is that the host PC's filesystem is mounted on the target pyboard. This is kind of reverse to what @dhylands' rshell currently does (and also mpfshell and ampy). Essentially the PC becomes a (huge) filesystem on the pyboard (with data coming in over the serial connection) and all filesystem commands work transparently.
Using the tool is very easy: you just run it and optionally specify the serial port to connect to. Then you get a normal REPL running directly on the board, just like with picocom/miniterm/etc. But before you get access to the REPL a script is run on the pyboard which installs a hook in the pyboard's filesystem so that the pyboard has access to the PC's filesystem (the directory that you run the tool from).
So at the REPL if you do
os.listdir()you'll see the contents of the PC's current directory. You can also stat and open and read and close files.Ultimately this can be extented to allow you to import scripts from your PC without ever copying them to the pyboard. This would allow for very fast workflow to develop an entire program on your PC and run it on the pyboard without ever copying any scripts.
This tool is currently only a proof-of-concept and there are a few things that need to be implemented on the firmware side before everything works (eg importing doesn't yet work).