LittleFS Filesystem Support #5167
Conversation
|
Is this PR port-agnostic? Will it work on ESP32? I would also like to introduce second partition for logging in my application. |
Yes most of it is port-agnostic. It should work on esp32 with a block device written in Python, interfacing between littlefs and the esp32's flash storage. |
|
Thanks @andrewleech for publishing this. I do want to get littlefs support in master ASAP. There are two main discussion points around this:
The existing block device protocol is rather simple, with 3 methods defined: For littlefs the requirements are different:
(The existing block protocol could be used for littlefs with read=write=erase the same size, but it would be inefficient.) As I see it there are two options to define the new/extended block device protocol: I'd prefer (b), because it's less confusing, and allows an entity to implement both protocols at the same time (by implementing both sets of methods). An idea for the new protocol, extending the old one, is: class AbstractBlockDevice:
def readchunks(self, rd_chunk_num, buf):
# similar to readblocks, read starting at chunk_num into given buf
def overwritechunks(self, wr_chunk_num, buf):
# don't erase, just overwrite existing data from buf, starting at chunk_num
def erasechunks(self, er_chunk_num, bytes_to_erase):
# erase starting from chunk_num
def ioctl(self, op, arg):
# 1-5 are existing ops
# 6 = get read chunk size
# 7 = get write chunk size
# 8 = get erase chunk size(read, write and erase chunk sizes can be different) |
As a test/example, external SPI flash is mounted automatically on boot. The configuration for this is defined at the top of pyblittlefs.c.
Runs on unix coverage and stm32.
This needs to be enabled by a particular board when it wants to have a native littlefs filesystem.
|
I feel supporting both v1 and v2 via different interfaces could be somewhat difficult as the lfs function interfaces are pretty much exactly the same, only selected by different pins of the submodule. Perhaps if v2 has had quite a few releases since I first started looking at this, perhaps it's now stable enough to ignore v1 altogether. I like your suggestion for an extended version of the python block interface, that looks quite flexible without a lot of overhead. On an unrelated note, might need some adjustments to the default enabled status of the module: Seems the M0's will need it off by default as a minimum. |
|
If we want to extend the block interface for improved performance in lfs, we'll probably also want to expose some of the other lfs settings to the end user:
An interesting discussion on the sizing and usage of blocks/cache: littlefs-project/littlefs#277 |
IMO all the technical issues supporting v1 and v2 at the same time can be overcome: like with oofatfs we just include the littlefs code verbatim in this repo (not as a submodule, it's pretty minimal so no real need to have another submodule just for a few files), then we can have two copies at once. Littlefs has a script to rename the API functions to include a prefix, so we can have the following directory/file layout: Then the extmod driver So supporting two versions is possible. It's then up to a port/board to decide whether to enable them both or not. |
I don't think these settings need to be part of the block device interface. Instead they belong to the constructor of |
|
Sounds good. Yes I did mean those arguments as parameters for I'm having a go at implementing an initial chunks interface today, see how far I get! |
| return 0; | ||
| } | ||
| #if (LFS_VERSION >= 0x00020000) | ||
| static uint8_t __attribute__ ((aligned (64))) lookahead_buffer[128/8]; |
There was a problem hiding this comment.
@andrewleech I'm pretty sure this should be lookahead_buffer[128]...! LFS2 changed this, compared to LFS1. Having a small buffer will lead to memory corruption.
There was a problem hiding this comment.
Ah yes, here's the malloc for that buffer when lfs is used with dynamic memory.
https://github.com/ARMmbed/mbed-littlefs/blob/master/littlefs/lfs2.c#L3218
It doesn't have the divide by 8, you're right. That'll be why I had some corruption when I first tested LFSv2!
|
Closed in favor of newer work & PR's breaking down the feature. |
This is based predominantly on a rebased version of #3847 with some additions for library version support, minor bugs and mpconfigboard based configuration.
I don't expect it to be ready to merge in the current form, but wanted to share it as a working example.
I've been using this implementation for some months now with no issues of reliability, no filesystem corruption.
My custom hardware has an external spiflash chip which is divided in half, the first half being used as a FAT filesystem (written once during calibration), the second half formatted as littlefs which has runtime information, logging, etc written to it quite regularly.
Neither of these filesystems are exposed on usb mass storage in my application.
The littlefs bindings support the library at both version v1.7.2 (currently pinned) and the newer v2.0.x with no extra configuration needed, however in my previous testing of v2.0.3 I saw some (suspected) fs corruption so rolled back to the stable v1.7.2
I was battling some other FS corruption causing problems around the same time, so it might not have been a littlefs version/bug at fault.
mpconfigboard.h
bdev.c
Note our hardware has a 32MB spiflash chip which also requires #5166