stm32/main: Allow freezing boot.py and main.py into firmware image #4348
Conversation
|
As it stands, all the ports implement boot files (templated or otherwise) separately in their respective At this stage I've only updated the stm32 I'll deal with the build failure shortly. |
|
Yeah, like that, except the order should be the opposite - first consolidation, then adding new features. And you're pretty right that it's a boring work which falls on the responsibility of maintainers. There're a lot of such things in MicroPython (well, definitely less than in many other projects), which block further progress. But regressing the consolidation work already done (a lot!) by adding port-specific features at the expense of all other ports (which also need the same feature) should not be a way to go. |
|
I think it would be a lot simple to just allow a given board to specify a custom factory filesystem. In the simplest case main.c could just But I agree that something needs to be done about this, there needs to be an efficient way to build and deploy custom applications. (Eg one approach is to build a FAT filesystem image on the PC and copy it to the board, which can be done over DFU. And to protect against corruption the filesystem would need to be mounted read-only.) |
|
You're right, I did over-complicate this. @pfalcon I do agree with your sentiments. My back story is I needed this functionality to ship a release of our firmware at work a couple of weeks ago and this is what I came up with quickly, so no it was not thought out as a platform wide improvement. Seeing as it was already done however, I pushed it here for this discussion. @dpgeorge Yeah a simple I do prefer the idea of providing a complete initial filesystem would be much more flexible. I do have a small set of other files I'd like to have included in the image myself, so I think this angle is definitely worth pursuing. A boardconfig or make define could point this to a new board or user specific fs folder. I can think of a couple of options for inclusion methods that would suit different targets better. For read only filesystems (teensy is always hardcoded / RO isn't is? and could be an option on other platforms/boards for deployment) the approach of pre-generating the fat fs is pretty clean, then it can be just linked into the appropriate location. That way there's no duplication of template/filesystem using extra flash. Actually for RO use it would probably make more sense to use the memzip (#147) or finish the ZipFile (#1797) support and use that, though I guess that's a cpu+ram vs flash tradeoff, probably best to support both zip and fat as the image format. In the case of RW filesystems, either a prebuilt FAT image or a zip image could still be included in the image and used as the template to install onto the formatted fs on initial install / erase. Using zip probably makes a lot more sense as it'll reduce flash usage, at the expense of a one-off cpu+ram when extracting it into a new fs. |
Yes that would be much simpler (but I'd want to keep away from symlinks in the repo).
FAT FS has a lot of overhead, its minimum size is about 20k (see eg #4305), excluding actual file data. So implementing a very simple linear RO filesystem structure, and using that instead of FAT for RO filesystems, would more than pay for that 20k. There are a few different classes of filesystems which would be good to support:
|
|
You can also use zip files which contain uncompressed files (which is what memzip did) |
|
Allowing non- .py/.mpy files to be frozen would be pretty close to using memzip with uncompressed files. |
…O_FS_TEMPLATE is defined and set. These files are often undesirable in a production setting and take up code space.
6700ccc to
94acf41
Compare
|
FWIW I've just pushed the much simpler version of this I've been running for some time now (still stm32 only), albeit just updated to suit latest master. Two commits; the first gives a define option The second updates utils/pyexec: This means you can put your main.py and boot.py in a folder make -C micropython/ports/stm32 FROZEN_DIR=bootand it'll always use your frozen boot/main files. As an interim solution for production use I'm finding it quite stable and effective, secure against users being able to inject their own boot/main files, and consistent in behavior even when the fat filesystem gets corrupted and you lose your runtime files. |
|
Thanks @andrewleech for the update. I like the
So I'd be happy to merge that commit. The other (disable initial files) would need separate consideration.
It also works with |
|
👍 + |
|
I merged support for frozen boot/main scripts in 859596c There's still the remaining commit here regarding not including the default files for stm32. |
|
See #4752 for an alternative proposal to make default files configurable. |
|
Default files were made configurable in 97753a1 |
…I_wrover Franzininho wifi ESP32-S2 Wrover
If boot.py and/or main.py are in frozen scripts, use them rather than the static copy in code for creating initial files on filesystem.
This was originally inspired by #4248, however taking a different direction.
Currently, the template for
main.pyandboot.pyis just contained in char string in code. When a new filesystem is being created the initial files are created from these templates.The PR allows a firmware board developer to use their own copies of boot.py or main.py instead of the code templates by putting those files into a separate folder for frozen scripts, eg:
Doing it this way allows the end user to still make modifications to those files during development, etc and the modified copies will be used. However if something goes wrong they can erase the filesystem and go back to the frozen templates.
This PR combined with the existing FROZEN_DIR / FROZEN_MPY_DIR system allows us to have our entire micropython application frozen into the rom dfu image for easy deployment without needing to manually copy any files onto the filesystem afterwards.