8000 Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM (WIP) by dpgeorge · Pull Request #8381 · micropython/micropython · GitHub
[go: up one dir, main page]

Skip to content

Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM (WIP) #8381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 33 commits into
base: master
Choose a base branch
from

Conversation

dpgeorge
Copy link
Member
@dpgeorge dpgeorge commented Mar 4, 2022

This is part 2 of #8191 (new mpy v6 file format). The PR adds the ability to place .mpy files in a special filesystem ROM and import them inplace so they don't take up much RAM. This is essentially dynamically frozen .mpy files.

Things included here are:

  • os.mount(path) can be used to query a mount point, to return the VFS object mounted there
  • new os.VfsMap filesystem which is a very simple linear read-only filesystem that can be memory mapped
  • when importing .mpy files from a mounted VfsMap filesystem, the data in the files there can be referenced directly (ie no need to load it into RAM)
  • new mpremote deploy-mapfs <directory> command that takes a directory on the host PC, packs it into a mapfs filesystem and deploys it to the MicroPython device
  • support on stm32, rp2 and esp32 for a small (~128k) mapfs filesystem which is mounted at /mapfs on boot (and also added to sys.path)

Workflow for the user:

  • install new firmware from this PR
  • create a <directory> on your PC containing all the code to go in the mapfs (must precompile .py to .mpy beforehand if you want to take advantage of RAM savings, but any types of files can go in this filesystem)
  • mpremote deploy-mapfs <directory>
  • /mapfs now contains all files from <directory> and they can be listed/read/imported as though it were a normal filesystem

@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Mar 4, 2022
@dpgeorge
Copy link
Member Author
dpgeorge commented Mar 4, 2022

Here is an example using aioble on PYBD-SF2.

First, importing aioble from .py files stored on the device:

MicroPython v1.18-183-g3dcff7935-dirty on 2022-03-04; PYBD-SF2W with STM32F722IEK
Type "help()" for more information.
>>> import aioble
>>> import gc, micropython
>>> gc.collect(); micropython.mem_info(1)
stack: 484 out of 15360
GC: total: 183040, used: 20048, free: 162992
 No. of 1-blocks: 196, 2-blocks: 53, max blk sz: 72, max free sz: 8731
GC memory layout; from 2000f4f0:
00000: MDh.hh=======================================h..hh=h==........h=
00400: ======h==========.hh.hhh..h....MD.Sh...............MDh.....D....
00800: ..M...D.S......................h==================Sh=Shh========
00c00: =======...h=======...h=T=MD....D..h=======h.h===.D..............
01000: .Dh=.Dh.h===hhh.B=BBB.Bh==h===DBBh=======.B=Bh.DBBBBBBBh===Bh===
01400: =BBBhhh=hDDBS.BBBh=======BBBBh===.B=.h=====h==============h====h
01800: .B...MDS....h..BB...B..D.BBhShS....BLhLh=h.hBBBh==============h=
01c00: ==h.....D..hBB...B.DBh===h==B.h========BBB..h===MDS...h=Dh===h==
02000: =BDB.h.h=h=.B=h.h=BB.h=.h=.h=h===....hhhh========Bh=======h=====
02400: ======...Bh=======..MDS............B...h===...Sh===.....h..h====
02800: =======..h..........h===========h.......h=h=BBh...h=h=..h===h=h=
02c00: =h=h==h====================================...................hh
03000: ====h.DBBhBBhh===========.h=h===h====.Dh..B=Bhhh..h===h===.h.h==
03400: =============........Dh============.hh=...hB.h===..h=..BB.DBB.DB
03800: B..BB=h=h===h===========BBBh.h=======ShShShh=h===h====.D......B=
03c00: =B.B=....h=...h==h===.DhB.h.B=h=...hh......h======...D........B.
04000: B=...h=h===h=h===................h===........h=h=..h===....hh...
04400: .............h=....h=...........................................
04800: ................................h==============.................
04c00: ..................................h===h===h======h===h===h===h==
05000: ===h====h==h=h======h====h==h==hh=====h==h==hh=h===============.
05400: .............................h=======h=........h=...............
05800: ......h=...h=...................................................
05c00: ..................h======h=======================h=======.......
06000: ..h=======h=====================================================
06400: ==================...................h..........................
       (3 lines all free)
07400: .............................h==================================
07800: ================................................................
07c00: ................................................................
08000: ......................................h=........................
08400: .....................................h=============h========....
08800: ....h==h========h=====h==h======h=====h====h===h==========h===h=
08c00: ===========.....................................................
       (2 lines all free)
09800: ............h........h=...h=....................................
09c00: ....................................h=====.........h========hh==
0a000: hh==Shh==h====hh======h==.......................................
0a400: ..........................h===========h==h========h====h==h==h==
0a800: ===h=h=h=============...........................................
       (135 lines all free)
2c800: ................................................

Second, importing aioble from .mpy files in /mapfs:

MicroPython v1.18-183-g3dcff7935-dirty on 2022-03-04; PYBD-SF2W with STM32F722IEK
Type "help()" for more information.
>>> import aioble
>>> import gc, micropython
>>> gc.collect(); micropython.mem_info(1)
stack: 484 out of 15360
GC: total: 183040, used: 11856, free: 171184
 No. of 1-blocks: 175, 2-blocks: 30, max blk sz: 72, max free sz: 10406
GC memory layout; from 2000f4f0:
00000: MDh.hh=======================================h..hh=h==........h=
00400: =====Bh==========.hh.hhh..h....MD.Sh.........h===SSS.h==========
00800: ========.D..MD........h===============h=========================
00c00: ===========SSS........................................M...D.Sh=.
01000: T=MD....Dh=.D.Dh.h===.Dh.h===hhh.B=BBB.Bh==h===.B=Bh.DBBDBBBBBBB
01400: h===Bh====BBBhhh=hDDB..BBBBBBBh===.B=h=====h==============h====h
01800: .BMD.......SSS.h==============h====..............h=======BB.D.BL
01c00: hLBh=h===BhBBBh....Dhh========BB.Bh===DBBBBh==h===MD....B.h=h===
02000: h===.DBDB.h.h=h=.B=h.h=BB.h=.h=.h=h===....Shhh===========h======
02400: ==..MD.....h=======......B...BB......h===h===.h===========......
02800: h============S..........................h...BBhh.D..BBhBBh.h=h==
02c00: =========h===h====.Dh..B=Bhhh..h===h===h=...hB.h===.h===========
03000: ==h=============================================================
03400: ==========SSSS......hh..........................hh..h=======....
03800: DBB.DB..DBB..BB=Bh=h===h===========BBBh=h===h====.D....B==B.B=.h
03c00: =h==h===.DhB.B=hh=h.h===.D...B=h=h===Bh==============h===.......
04000: ........h=......................................................
       (161 lines all free)
2c800: ................................................

Third, importing aioble from frozen code (in .frozen):

MicroPython v1.18-180-g19b74d4e3-dirty on 2022-03-04; PYBD-SF2W with STM32F722IEK
Type "help()" for more information.
>>> import aioble
>>> import gc,micropython
>>> gc.collect();micropython.mem_info(1)
stack: 484 out of 15360
GC: total: 183104, used: 8304, free: 174800
 No. of 1-blocks: 156, 2-blocks: 30, max blk sz: 40, max free sz: 10798
GC memory layout; from 2000f4b0:
00000: MDh.hh=======================================h..hh=h==........h=
00400: ======h==========.hh.hhh..h....MD.Sh..D...MD.M...D.Sh=.T=MD....D
00800: h=.D.Dh.h===.Dh.h===hhh.B=BBB.Bh==h===.B=Bh.DBBDBBBBBBBh===Bh===
00c00: =BBBhhh=hDDB.BBBBBBBBh===.B=h=====h==============h====h.BMD.BB.D
01000: .BLhLBh==============h=h===BhBBBh....Dhh========BB.Bh===DBBBBh==
01400: h===MD.B.BB.h=h===h===.DBDB.h.h=h=.B=h.h=BB.h=.h=.h=h===....Bhhh
01800: ===========h========..MD........h===.hh===========h===...BBhh.D.
01c00: .BBhBBh.h=h===========h===h====.Dh..B=Bhhh.Dh===h===h=...hB.....
02000: BB.DBB.DBB..BB=h=h===h===========BBBh=h===h====.D....B==B.B=.h=h
02400: ==h===.DhB.B=hh=h.h===.D...B=h=h===Bh==============h===..h===...
02800: ....h=..........................................................
       (167 lines all free)
2c800: ....................................................

Summary:

  • from .py, memory used: 20048
  • .mpy from /mapfs, memory used: 11856
  • from frozen, memory used: 8304

Note: importing .mpy from a traditional filesystem (eg FAT, littlefs) uses pretty much the same RAM as importing from .py.

@dpgeorge
Copy link
Member Author
dpgeorge commented Mar 4, 2022

In the aioble example above, note that aioble creates a lot of classes on import and so that's why the frozen memory use is 8304 bytes of heap. Baseline (fresh soft reset memory use) for PYBD-SF2 is about 1300 bytes, so aioble allocates about 7000 bytes due to the import.

Taking frozen memory use (8304) as the baseline, we have:

  • .py: 11744 bytes heap (bytecode, qstr data, qstr tables, function wrappers)
  • .mpy from /mapfs: 3552 bytes heap (qstr tables and function wrappers)

So importing .mpy from /mapfs uses about 30% of the memory compared to importing .py (or .mpy) from a normal filesystem.

@iabdalkader
Copy link
Contributor
iabdalkader commented Mar 4, 2022

Hi, I'd like to share some random thoughts on this feature:

  • Can we have a more convenient C API ? Actual use case: I want to add binary blobs to fs image and read them as C arrays from C code, files such as ML models and so on, so I need an address and a size. Basically I need to detect if mapfs is mounted and to call mp_vfs_map_search_filesystem from C.
  • Why isn't MICROPY_VFS_MAP used throughout the code to gate this feature's code when it's disabled ? For example in ports/stm32/storage.c and a few more places.
  • _boot.py is added to the manifest but wouldn't it fail if MICROPY_VFS_MAP is disabled ? I see there's a MICROPY_BOARD_FROZEN_BOOT_FILE defined but not used anywhere. Perhaps there should be a way to conditionally include files in the manifest or a way to detect if compile time features are enabled/disabled from Python ? Asking for this feature, and for my _boot_fat.py in MSC PR.
  • I think it should be called romfs 🙄

@dpgeorge
Copy link
Member Author
dpgeorge commented Mar 7, 2022

Can we have a more convenient C API ? Actual use case: I want to add binary blobs to fs image and read them as C arrays from C code, files such as ML models and so on, so I need an address and a size. Basically I need to detect if mapfs is mounted and to call mp_vfs_map_search_filesystem from C

I did want to add a way for user Python code to memory map a file and get its address in flash/ROM. That would allow Python code to load resources (eg images) from the mapfs very efficiently.

Yes, it would be good to have a convenient C API for this feature as well. Will look at adding that.

Why isn't MICROPY_VFS_MAP used throughout the code to gate this feature's code when it's disabled ? For example in ports/stm32/storage.c and a few more places

You're right, it should be used. But also my thoughts are that this feature is very fundamental and should probably be enabled everywhere. This feature is ultimately more fundamental than the Python compiler, because you can always compile offline and upload code to a mapfs.

_boot.py is added to the manifest but wouldn't it fail if MICROPY_VFS_MAP is disabled ?

Yes that needs to be fixed. Either selectively include different _boot.py files, or detect the feature at runtime (eg via try/except around the new code). 8000

I think it should be called romfs

Yeah, that's a decent name (and then VfsMap would be renamed VfsRom). It isn't strictly read-only though, because this filesystem could potentially support append as an operation (ie add a new file). Or even update an existing file by appending a new version which overrides the previous one. So that's why I went with "map", to describe that it can be mapped to address space.

The naming is definitely still open for discussion.

@dpgeorge dpgeorge changed the title Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM (WIP) Mar 7, 2022
@dpgeorge
Copy link
Member Author
dpgeorge commented Mar 7, 2022

I added WIP to the title to indicate there is still some work to do here, and some decisions to be made.

@hoihu
Copy link
Contributor
hoihu commented Mar 7, 2022

This is a terrific feature (this rom FS and #8191). I'm really looking forward to this!

@iabdalkader
Copy link
Contributor
iabdalkader commented Mar 7, 2022

I did want to add a way for user Python code to memory map a file and get its address in flash/ROM

Actually I just realized that if I just open a file from a mapfs partition, from Python, I'll get back a mp_obj_vfs_map_file_t and I can pass that to the C function as a normal file, check the type, cast to mp_obj_vfs_map_file_t and get the address and size in C. If that's right then a C API may not be needed.

It isn't strictly read-only though, because this filesystem could potentially support append as an operation

Do you mean in runtime ? If so, then romfs may not be entirely accurate.

@chrisovergaauw
Copy link
Contributor

I added WIP to the title to indicate there is still some work to do here, and some decisions to be made.

Hi George,

This seems like a valuable feature. Have you got any update on this?
What work still needs to be done?

regards,
Chris

@dpgeorge
Copy link
Member Author

This is a big feature and needs more time on the design.

laurensvalk added a commit to pybricks/micropython that referenced this pull request Jul 26, 2022
This is a partial implementation of MicroPython micropython#8381. We could use
it until that PR gets merged upstream, or use this as a smaller solution
since it doesn't require the various extmod modules. We can use
something like this in our minimal port-variant.

This allows us to run an .mpy blob from RAM without making a near-copy.
This way we don't have to use free_len to free up the original blob, so
the user can run it again without re-uploading.
laurensvalk added a commit to pybricks/micropython that referenced this pull request Jul 26, 2022
This is a partial implementation of MicroPython micropython#8381. We could use
it until that PR gets merged upstream, or use this as a smaller solution
since it doesn't require the various extmod modules. We can use
something like this in our minimal port-variant.

This allows us to run an .mpy blob from RAM without making a near-copy.
This way we don't have to use free_len to free up the original blob, so
the user can run it again without re-uploading.
@laurensvalk
Copy link
Contributor

This is great work!

I just tried a partial implementation of this to test a slightly different use case, and it seems to work well. I took only the parts relating to py/persistentcode for now because our targets are closer to the minimal port than to the stm32 ports. If we have enough space, the rest of this PR might come in handy too.

In Pybricks, we transfer the mpy blob over the air to the LEGO hub. Since most LEGO hubs don't have any form of user data storage, we have to run it from RAM. Previously, running a program this way created a near-copy of the mpy data, and we always discarded the original mpy blob so the user had some RAM left to use in their code. But this meant the program could not easily be run again.

With the tools from this PR, we get to keep the original mpy in memory without sacrificing scarce user RAM more than we need to.

laurensvalk added a commit to pybricks/micropython that referenced this pull request Aug 8, 2022
This is a partial implementation of MicroPython micropython#8381. We could use
it until that PR gets merged upstream, or use this as a smaller solution
since it doesn't require the various extmod modules. We can use
something like this in our minimal port-variant.

This allows us to run an .mpy blob from RAM without making a near-copy.
This way we don't have to use free_len to free up the original blob, so
the user can run it again without re-uploading.
@massimosala
Copy link
massimosala commented May 30, 2023
  • support on stm32, rp2 and esp32 for a small (~128k) mapfs filesystem which is mounted at /mapfs on boot (and also added to sys.path)

Workflow for the user:

  • install new firmware from this PR

Hi Damien

For a project I have to use esp8266.
I started developing on micropython about 20 days ago, full time.
I read a lot of docs and discussions about memory usage:

#2709
#4073
#4124
#8191
#8381
f2040bf

and also the good guidelines by @peterhinch
https://github.com/peterhinch/micropython-samples/blob/master/import/IMPORT.md

The project is made of about 20 .mpy files.
I spent many efforts to minimize RAM usage.
The program ASAP allocates a bytearray of 512 bytes and uses that single bytearray for all functions (apart from json ... there aren't json_load_using_buffer / json_dump_using_buffer).
I use gc.collect() extensively.

At runtime I have about 2-3KB of free heap.
I need to add two more features. Unfortunately, adding them causes the program to drop below 1 KB of heap and eventually crash.

I really need to load .mpy while minimizing heap usage.
I'm unable to compile a custom firmware and that's not a solution: the program also does OTA updates, fetching only the required modules (.mpy).

Are these new features ready for the esp8266 ?

Other thoughts... an alternative firmware, without some features... like web REPL.
Would this help to increase the available heap at startup?

BTW thanks for micropython!
Best regards, Massimo

iabdalkader and others added 22 commits February 19, 2025 23:20
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Thanks to @robert-hh.

Signed-off-by: Damien George <damien@micropython.org>
esp32/partitions-4MiB.csv: Reserve 128K for romfs.

Signed-off-by: Damien George <damien@micropython.org>
Available ROMFS size:
- FLASH_512K: 8KiBytes
- FLASH_1M: 16KiBytes
- FLASH_2M+: 256KiBytes
- OTA: 8KiBytes

Signed-off-by: Damien George <damien@micropython.org>
The default size is 256k. It can be changed in mpconfigboard.h by
defining MICROPY_HW_ROMFS_BYTES to a different value, but it must
not be smaller than a single sector.

Signed-off-by: robert-hh <robert@hammelrath.com>
The VfsRom file system has a size of 15k or 16k and is placed at the
upper end of the flash. Implication:

- Without SPIFLASH: Reduce the internal LFS file system from 64k to 42k,
  leaving about 7k headroom for future code changes.
  Use the object-based capabilities for VfsRom. The VfsRom size is 15k.
- With SPIFLASH: Remove the onewire, ds18x20 and dht drivers from
  frozen bytecode as they can be placed into the vfsRom files when
  needed. The VfsRom size is 16k.
  Use the rom_ioctl functions for VfsRom, since the flash devices for
  the regular file system and VfsRom are different.

Signed-off-by: robert-hh <robert@hammelrath.com>
The VfsRom file system has a size default of 64k for SAMD51x19 and
256K for SAMD51x20. It is placed at the upper end of the flash.
Implication:

- Without SPIFLASH: Reduce the internal LFS file system from 128k to
  64k.
- With SPIFLASH: Reduce the code size from 496K to 432K. If that is
  not sufficient for some boards or configurations, it can be
  changed for each board or board variant.

Signed-off-by: robert-hh <robert@hammelrath.com>
Using the object based capabilities, which avoids complication about
different flash block sizes. VfsROM fs sizes:

NRF51x22: 12K
NRF52632: 128K
NRF52840: 128K
NRF9160:  128K

Tested with Microbit and  Arduino Nano Connect. Problem with the test:
the MicroBit RAM is too small to run mpremote even if that is the board
where VfsROM would be most helpful. So a special small loader would be
needed.

Signed-off-by: robert-hh <robert@hammelrath.com>
Replacing a mix of C-Code and Python code. The mkfs part has been
simplified to save code.

Signed-off-by: robert-hh <robert@hammelrath.com>
VfsRom file system sizes:

EK_RA4M1: 12K
RA4M1_CLICKER: 12k
EK_RA4W1: 64k
EK_RA6M1: 64k
EK_RA6M2: 128k
EK_RA6M5: 256k
ARDUINO_PORTENTA_C33: 192k

Tested with Weact RA4M1 core board with EK_RA4M1 firmware and EK_RA6M2.
More boards have to be tested.

Signed-off-by: robert-hh <robert@hammelrath.com>
Define and use vfsrom_start and vfsrom_end for the VfsRom file system.
The way they are set in the MCU loader files, VfsRom will be placed
between the text segment and the existing vfs files system area.
The size of the VfsRom file system is set in Makefile or one of the
board .mk files by setting the symbol MICROPY_HW_ROMFS_BYTES.

Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
To get newer arm-none-eabi-gcc that can do better size optimisations.

Signed-off-by: Damien George <damien@micropython.org>
It is needed for VfsRom at devices without external flash.

Simplify the code for "case MP_VFS_ROM_IOCTL_WRITE_PREPARE:".

Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: Damien George <damien@micropython.org>
This reverts commit 3cd0c45c432c4d28bfa2d0499da3499a7e398c1f.
won't work when mpremote is pip installed, because micropython-lib is
not available!  so would need to fetch via URL

Signed-off-by: Damien George <damien@micropython.org>
@@ -129,8 +134,8 @@ void PORTENTA_board_osc_enable(int enable);
// QSPI flash #1 for storage
#define MICROPY_HW_QSPI_PRESCALER (2) // 100MHz
#define MICROPY_HW_QSPIFLASH_SIZE_BITS_LOG2 (27)
// Reserve 1MiB at the end for compatibility with alternate firmware that places WiFi blob here.
#define MICROPY_HW_SPIFLASH_SIZE_BITS (120 * 1024 * 1024)
// Reserve 4MiB for romfs and 1MiB at for WiFi/BT firmware.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a typo in all of the Arduino headers: "at for" -> "for"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
py-core Relates to py/ directory in source
Projects
None yet
Development

Successfully merging this pull request may close these issues.

0