-
-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM (WIP) #8381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Here is an example using First, importing
Second, importing
Third, importing
Summary:
Note: importing .mpy from a traditional filesystem (eg FAT, littlefs) uses pretty much the same RAM as importing from .py. |
In the Taking frozen memory use (8304) as the baseline, we have:
So importing .mpy from |
Hi, I'd like to share some random thoughts on this feature:
|
I did want to add a way for user Python code to memory map a file and get its address in flash/ROM. That would allow Python code to load resources (eg images) from the Yes, it would be good to have a convenient C API for this feature as well. Will look at adding that.
You're right, it should be used. But also my thoughts are that this feature is very fundamental and should probably be enabled everywhere. This feature is ultimately more fundamental than the Python compiler, because you can always compile offline and upload code to a
Yes that needs to be fixed. Either selectively include different
Yeah, that's a decent name (and then The naming is definitely still open for discussion. |
I added WIP to the title to indicate there is still some work to do here, and some decisions to be made. |
This is a terrific feature (this rom FS and #8191). I'm really looking forward to this! |
Actually I just realized that if I just open a file from a
Do you mean in runtime ? If so, then |
Hi George, This seems like a valuable feature. Have you got any update on this? regards, |
This is a big feature and needs more time on the design. |
This is a partial implementation of MicroPython micropython#8381. We could use it until that PR gets merged upstream, or use this as a smaller solution since it doesn't require the various extmod modules. We can use something like this in our minimal port-variant. This allows us to run an .mpy blob from RAM without making a near-copy. This way we don't have to use free_len to free up the original blob, so the user can run it again without re-uploading.
This is a partial implementation of MicroPython micropython#8381. We could use it until that PR gets merged upstream, or use this as a smaller solution since it doesn't require the various extmod modules. We can use something like this in our minimal port-variant. This allows us to run an .mpy blob from RAM without making a near-copy. This way we don't have to use free_len to free up the original blob, so the user can run it again without re-uploading.
This is great work! I just tried a partial implementation of this to test a slightly different use case, and it seems to work well. I took only the parts relating to In Pybricks, we transfer the mpy blob over the air to the LEGO hub. Since most LEGO hubs don't have any form of user data storage, we have to run it from RAM. Previously, running a program this way created a near-copy of the mpy data, and we always discarded the original mpy blob so the user had some RAM left to use in their code. But this meant the program could not easily be run again. With the tools from this PR, we get to keep the original mpy in memory without sacrificing scarce user RAM more than we need to. |
This is a partial implementation of MicroPython micropython#8381. We could use it until that PR gets merged upstream, or use this as a smaller solution since it doesn't require the various extmod modules. We can use something like this in our minimal port-variant. This allows us to run an .mpy blob from RAM without making a near-copy. This way we don't have to use free_len to free up the original blob, so the user can run it again without re-uploading.
19b74d4
to
0664d36
Compare
Hi Damien For a project I have to use esp8266. #2709 and also the good guidelines by @peterhinch The project is made of about 20 .mpy files. At runtime I have about 2-3KB of free heap. I really need to load .mpy while minimizing heap usage. Are these new features ready for the esp8266 ? Other thoughts... an alternative firmware, without some features... like web REPL. BTW thanks for micropython! |
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Thanks to @robert-hh. Signed-off-by: Damien George <damien@micropython.org>
esp32/partitions-4MiB.csv: Reserve 128K for romfs. Signed-off-by: Damien George <damien@micropython.org>
Available ROMFS size: - FLASH_512K: 8KiBytes - FLASH_1M: 16KiBytes - FLASH_2M+: 256KiBytes - OTA: 8KiBytes Signed-off-by: Damien George <damien@micropython.org>
The default size is 256k. It can be changed in mpconfigboard.h by defining MICROPY_HW_ROMFS_BYTES to a different value, but it must not be smaller than a single sector. Signed-off-by: robert-hh <robert@hammelrath.com>
The VfsRom file system has a size of 15k or 16k and is placed at the upper end of the flash. Implication: - Without SPIFLASH: Reduce the internal LFS file system from 64k to 42k, leaving about 7k headroom for future code changes. Use the object-based capabilities for VfsRom. The VfsRom size is 15k. - With SPIFLASH: Remove the onewire, ds18x20 and dht drivers from frozen bytecode as they can be placed into the vfsRom files when needed. The VfsRom size is 16k. Use the rom_ioctl functions for VfsRom, since the flash devices for the regular file system and VfsRom are different. Signed-off-by: robert-hh <robert@hammelrath.com>
The VfsRom file system has a size default of 64k for SAMD51x19 and 256K for SAMD51x20. It is placed at the upper end of the flash. Implication: - Without SPIFLASH: Reduce the internal LFS file system from 128k to 64k. - With SPIFLASH: Reduce the code size from 496K to 432K. If that is not sufficient for some boards or configurations, it can be changed for each board or board variant. Signed-off-by: robert-hh <robert@hammelrath.com>
Using the object based capabilities, which avoids complication about different flash block sizes. VfsROM fs sizes: NRF51x22: 12K NRF52632: 128K NRF52840: 128K NRF9160: 128K Tested with Microbit and Arduino Nano Connect. Problem with the test: the MicroBit RAM is too small to run mpremote even if that is the board where VfsROM would be most helpful. So a special small loader would be needed. Signed-off-by: robert-hh <robert@hammelrath.com>
Replacing a mix of C-Code and Python code. The mkfs part has been simplified to save code. Signed-off-by: robert-hh <robert@hammelrath.com>
VfsRom file system sizes: EK_RA4M1: 12K RA4M1_CLICKER: 12k EK_RA4W1: 64k EK_RA6M1: 64k EK_RA6M2: 128k EK_RA6M5: 256k ARDUINO_PORTENTA_C33: 192k Tested with Weact RA4M1 core board with EK_RA4M1 firmware and EK_RA6M2. More boards have to be tested. Signed-off-by: robert-hh <robert@hammelrath.com>
Define and use vfsrom_start and vfsrom_end for the VfsRom file system. The way they are set in the MCU loader files, VfsRom will be placed between the text segment and the existing vfs files system area. The size of the VfsRom file system is set in Makefile or one of the board .mk files by setting the symbol MICROPY_HW_ROMFS_BYTES. Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
To get newer arm-none-eabi-gcc that can do better size optimisations. Signed-off-by: Damien George <damien@micropython.org>
It is needed for VfsRom at devices without external flash. Simplify the code for "case MP_VFS_ROM_IOCTL_WRITE_PREPARE:". Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: Damien George <damien@micropython.org>
This reverts commit 3cd0c45c432c4d28bfa2d0499da3499a7e398c1f.
won't work when mpremote is pip installed, because micropython-lib is not available! so would need to fetch via URL Signed-off-by: Damien George <damien@micropython.org>
9f93264
to
2431419
Compare
@@ -129,8 +134,8 @@ void PORTENTA_board_osc_enable(int enable); | |||
// QSPI flash #1 for storage | |||
#define MICROPY_HW_QSPI_PRESCALER (2) // 100MHz | |||
#define MICROPY_HW_QSPIFLASH_SIZE_BITS_LOG2 (27) | |||
// Reserve 1MiB at the end for compatibility with alternate firmware that places WiFi blob here. | |||
#define MICROPY_HW_SPIFLASH_SIZE_BITS (120 * 1024 * 1024) | |||
// Reserve 4MiB for romfs and 1MiB at for WiFi/BT firmware. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a typo in all of the Arduino headers: "at for" -> "for"
This is part 2 of #8191 (new mpy v6 file format). The PR adds the ability to place .mpy files in a special filesystem ROM and import them inplace so they don't take up much RAM. This is essentially dynamically frozen .mpy files.
Things included here are:
os.mount(path)
can be used to query a mount point, to return the VFS object mounted thereos.VfsMap
filesystem which is a very simple linear read-only filesystem that can be memory mappedVfsMap
filesystem, the data in the files there can be referenced directly (ie no need to load it into RAM)mpremote deploy-mapfs <directory>
command that takes a directory on the host PC, packs it into a mapfs filesystem and deploys it to the MicroPython device/mapfs
on boot (and also added tosys.path
)Workflow for the user:
<directory>
on your PC containing all the code to go in the mapfs (must precompile .py to .mpy beforehand if you want to take advantage of RAM savings, but any types of files can go in this filesystem)mpremote deploy-mapfs <directory>
/mapfs
now contains all files from<directory>
and they can be listed/read/imported as though it were a normal filesystem