ESP32 S2/S3 USB MSC support? #8426
Comments
|
MSC mode is a can of worms for this reason. |
|
Bump, this would be a neat feature. In my application, it'd be preferable to present a controlled list as opposed to automatically using the literal file listing. Maybe like how ble does it, an irq callback when a listing is requested, file read or write. I know very little about USB Msc support, maybe a naive idea on my part, but I think would be cool! |
|
test result shows filesystem corruption maybe happen |
|
@peterhinch - From experience I'll counter that USB MSC is not at all a can of worms but actually a significant development accelerator. It already works beautifully on STM32's. I'd never cripple my development system by turning it off (during development). Espressif also realized the value of USB and so went to great trouble to trade silicon real estate for USB support. As such, I'm have a serious look at ESP32-S3 as a challenging STM32's in some use-cases. IMHO, it would be a huge mistake for the micropython community to delay or limit TinyUSB integration. Many will simply drop mpy and move on to circuitpython over something like this. For good reasons, Adafruit appears to be making it a policy to focus on devices which support USB MSC. After doing some significant development on STM32's with USB MSC and having zero issues with flash corruption, I'll strongly favor devices with USB MSC. Note that ESP32-S3 with USB MSC storage enabled works right now on the circuitpython fork. I begrudgingly flashed cpy onto my hardware and it "just worked". Now looking forward to same on micropython (or micropython on zephyr). |
|
The only time this would be an actual issue is if someone wrote self writing code which edited I feel like this answer is another cop out for not wanting to fix the mess that is the USB stack on micropython. |
|
File corruption is a risk if a program modifies the filesystem in any way. The issue crops up repeatedly in the forum. I found myself answering the question so frequently that I wrote the FAQ referenced above. |
|
@Red-M @t35tB0t As Peter said, the problem with MSC origins from a PC and Pyboard assuming to have exclusive access to the filesystem and being able to modify it - writing files, directories and the FAT. In addition to the PC caching and deferring accesses this will a file system corruption sooner or later. Even if you understand the risks and try to avoid it. |
It was implemented, but it was never merged. I kept the commit that adds it here: 3589583 if anyone wants to reuse it somewhere feel free to do so. |
|
8000
@Red-M Yes, thank you for detailing the factors for others here. However, I really do understand these factors and have been using USB MSC on mounted SDcards with STM32 for 1000's hours with no corruption. The repeated assertion here that using USB MSC will sooner or later corrupt the file system is not accurate. What will corrupt your flash is power disruption, or bad code, or bad user behavior. But fundamentally, an embedded system that is writing to its system flash is fundamentally less robust regardless if you have USB or not. Note that I'm somewhat centric to SDcard usage because I'm developing applications that are too large for typical MCU internal flash. The latest ESP32's are interesting but have storage performance issues. SD cards have an embedded ARM controller and do their own wear leveling now. If they are not powered down properly and are crashed during a write, their wear leveling will corrupt the entire file system. So, the recommendation is to use Industrial Grade SDcards which have more support for avoiding such corruption. As far as making USB MSC safe to use, simply consider the following and you'll be fine...
I hope I'm not just in a minority who is writing larger/more complex MicroPython applications (500K to 1MB source code)? Unfortunately, I have seen no alternative development path that compares to the efficiency and cycle time of using USB MSC. It really works really well. If unavoidable flash corruption exists, with these large projects and heavy USB MSC use, then file system corruption would be a problem for me. It isn't. What is a problem is a lack of any good alternative to USB MSC. Thonny and rshell are incapable or supporting these projects effectively. If there is anything that compares to the performance and utility of USB MSC, please let us know here. |
mpremote is the official solution. |
|
@peterhinch - As far as link behavior goes, the official mpremote is not architecturally different than Thonny or rshell. I've found both Thonny and rshell to be unusable for larger projects. A faster interface is needed: USB MSC is an obvious choice that has proven to be fast and useful without corrupting SDcards. USB over Ethernet might be interesting as an extension but even having FTP in c-code below micropython would be a relief. I must discount these other tools which are using REPL over a slow default serial connection. The raw USB rate is nearly 100x faster. The default UART speed at 0.1152 Mbits/sec is actually 10x slower than the UART hardware can actually do. Even with a UART speed-up, the USB will still be 10x faster to we should be using it. Thus, we have no comparison to USB MSC. Two key facts here are that USB MSC has been extremely reliable on STM32s and that CircuitPython has USB MSC working on the ESP32-S3 already. Are we aware of any intention to implement USB MSC on Micropython for the ESP32-S3? |
|
I plan to but I'd prefer a regular dev to provide such a feature as this codebase seems to have very particular rules for contributions. I really want to get MSC working and then work on the actual thing I want being USB HID as honestly, MP should really be noticing the features that CP has since its a major driving force to the amount of people around it. If I can get some traction on #8689 as I have a bunch of expresif ESP-S3 devboards that need this compile time flag to work (I have N8R8 DevKits), I can start to work on some of the shortcomings of USB under MP. |
|
@Red-M - I have some N8R8's as well. I fiddled with TinyUSB in MP but only got it partly integrated in an evening. The USB event callback with ESP SDK is a bit convoluted and CP's approach is vastly different than the legacy MP USB so I stopped there. If you do make progress on USB MSC on the N8R8s, I'll gladly do some debugging/testing. Fortunately, I also have a stack of STM32's - apparently the global supply of STM32 ICs has all but vanished. |
|
Realistically I just want to make a keyboard with some custom firmware but this would be way easier with threads on an RTOS because I want low latency on the key loop. Somehow this is a hard ask as CP has a good USB stack but has decided that threads are "too hard" for most people and thus have removed threads entirely (not disabled). MP on the other hand will let you use such interfaces but the USB stack is a mess and doesn't have a concept of HALs for interfacing with hardware so the same library interfaces have to be maintained between ports instead of a unified interface into the port specific HAL, which makes maintenance a huge strain. I'm fine to try for USB MSC support but I already have some really dumb tooling that did what I needed enough to test out and play with MP but this constant dodging of features is making people move to CP instead because they have features but not much power in terms of multicore support (where as MP has much better support) |
|
@Red-M - Agreed. CP as a dumbed down MP fork was a waste of developer time and my time to try to make something useful with it. Perhaps Zephyr will mature and change the solution space here. Meanwhile, patching in USB MSC (along with other USB modes) to the ESP32-S3 would make these N8R8s pretty nice. It would be a leap forward in usefulness. Of course, some serious work addressing Octal Flash and PSRAM issue(s) is also needed. We could also try to find some high-speed UART rate combos on PC/target to work around the divider mis-matched bauds. Not as sweet as USB MSC but we might at least give folks more useful interface performance with e.g. Thonny. I've done this before so just need to dig up some notes, re-implement, and then test stability perhaps with a BER pattern. |
|
Hi, all Any interest in adding MSC support for ESP32 S3? #8699 I have added this feature by referring to the pull request of @iabdalkader #7402 The MSC feature is configurable with sdkconfig.usb_msc. MSC with sdcard will be later. |
|
@M-D-777 - Nice. Thank you for this. Will try your MSC implementation this weekend then see if I can add SDcard and maybe RAM disk blk dev support as well. But these are just stop-gap measures. Zephyr is moving to fill this hardware interface gap far better than MP or CP ever can. So we're eager to have that kind of full featured platform running MP. |
|
First of all, I am not familiar with the internals of MP at all, so maybe this is a bad idea that has been rejected already. Why not have two partitions in flash, one that is writable by PC but only readable by MP (so this partition will hold .py code, configuration etc.), and another partition that is only writable by MP (it would be nice, but not strictly necessary for it to be readable by PC) so that it can be used by MP code for logging data, saving states, etc.) This should cover most of the common usage that people want to use USB MSC for, i.e., be able to save modified Python code quickly to the ESP32, and for MP code to save data without worrying about data corruption. The only issue is how to get the saved data out from the ESP32 to PC, but this can probably done using some kind of file transfer over Wifi. If this a bad idea, or it is too much trouble to implement, I'd say make USB MSC an optional feature that can be turn on and off. It should be OFF by default, and one should be given a warning that this is an "adult feature" that could cause data corruption if used indiscriminately. Those who consider themselves adult can then turn it on and bear the responsibility for the consequences :-) Edit: Sorry for making this unnecessary suggestion. Lots of ideas, pros and cons, have already been discussed in rp2: Add MSC support. |
You can already set up this exact scenario on the stm32 and rp2 ports (and it would work on ESP32 if we had MSC support). It's unfortunate to have to fragment the flash into two parts though (in terms of wasted space). With a read-only-from-micropython partition, you still have to have a way to know exactly when the filesystem is in a valid state for reading. i.e. if the PC is in the middle of of a file operation (e.g. saving code.py), the device needs to know that all blocks are flushed (at every layer from the host PC to the host disk cache, to the device's flash cache) before being able to read. Obviously it's not nearly as bad as two concurrent writers, but even a concurrent writer and reader isn't safe. I think CircuitPython has some heuristics to detect completed write operations to make this work (e.g. idle timeout?), i.e. this is how they trigger the "code.py has change, do soft reset" thing. Honestly I think the way CircuitPython implement this is a good compromise in the complexity-vs-ease-of-use tradeoff. If only MTP was usable on the host PC... and we wouldn't need to worry about this.
Yes, see #7402 (comment) (Edit: sorry just saw your edit to link to the exact same comment :) ) |
CircuitPython waits 0.75 seconds (used to be 0.5 seconds) after the last write before triggering an auto-reload. We still see delayed writes on Linux and Windows. macOS is much better about it. Windows supposedly fixed the worst delayed writes (which were on FAT12 only, probably a legacy of floppy support) many years after we reported the problem, but I haven't checked again recently. |
|
@iabdalkader made a PR #8814 for rp2 and I did for the mimxrt port #8820, setting the filesystem to r/o by Micropython as long as the drive is mounted at the PC, and setting it to r/w after it had been unmounted. For my test configuration of mimxrt boards and debian Linux I found, that the Micro had to go through a warm reset to ensure, that all modifications done by the PC are noticed by the Micro. So it it a three state logic. That's not overly comfortable, but seems safe. For the rp2 set-up this did not seem to be necessary, which I'm skeptic about. |
|
The dual-mode USB solutions here are good ideas but, IMHO, they
unnecessarily complicated. The STM32's work perfectly well as-is. So,
unless there's something adversely different about the ESP32, I see no
good reason for putting much effort into that complexity. Try
developing with USB on STM32's and then come back here with real
problems instead of imagined collisions and contrived scenarios.
From hundreds of application development hours on the STM32's I am able
to assert that, as-is, the MPY USB functionality is by far the most
superior interface solution (for development). This includes working
from windows and linux host machines. The Micropython paradigm with the
STM32's has been there for a very long time and it "just works".
Why make the ESP32's different? What's sad is that the ESP32's hardware
with USB was a welcome arrival. Having it broken like this is quite a
letdown. Seeing that CircuitPython made it work (albeit in a gimped
way) is irritating. The alternate development interfaces perform poorly
to the point of being unusable for larger application development. I
challenge anyone to prove otherwise.
So, having USB work the same in MPY, across supported hardware, is a
reasonable expectation.
|
|
I have encountered numerous corruptions when using MSC with STM32, which is what motivated me to write rshell, so while they may be less frequent, they are definitely still there. With MSC, the host assumes it has exclusive access and it will happily overwrite changes made by the micropython board. This is why the volume needs to be RO when the host has it mounted. |
|
The fundamental issue here is that the flash access is way too slow for
any reasonably sized project when using any currently available
interface than USB. We welcome any similar solution but are dismayed
with the apparent lack of understanding surrounding USB and flash usage.
rshell is clever but useless for any large development project. USB on
STM32 has been the only useful solution to that. So, this is not a code
development or host issue - if we're trying to break a paradigm of
protecting code flash from corrupted execution, then we're already on a
path to compromised designs. Good luck when the power fails suddenly or
the code has an evil bug.
But firstly, hat's off to the awesome work MPY Devs have done to make
MPY as stable as it is. MPY has greatly matured and works very well for
real-world professional applications --- when used with solid
architecture. If you have encountered numerous corruptions when using
MSC with STM32 then, in all due respect, you were not using that flash
"properly". I'll explain...
A robust target application should not be actively writing to program
flash. Why on earth would one do that?! Its not safe for an
application to be actively writing to code flash during runtime.
Consider what corruption can happen when power is lost. This is one of
those things you can do but should not. We do not want to accidentally
create self-modifying code here.
My team has had zero flash corruption issues with MicroPython running in
a production environment running million dollar products in complex test
environments. The MPY applications are relatively large, complex, and
must reliably operate the networked production test equipment. In
addition to local stepper motors on RS485, we have many socket services
(had to increase socket resources significantly). And have many async
tasks running cooperatively. Yet we are very restrictive on any writing
to flash. Without USB development, the project would have been impractical.
We would not be using USB if it were going to put expensive products at
risk. And that's why we never write to flash during normal runtime
execution either. So, no, the volume does NOT need to be RO when a
development host is mounted - MPY needs to use the flash RO when the
host is mounted or, smarter, use it RO all the time. This is a
fundamental architecture that most professional products use. Add
separate data flash or support data partitions if you must log
continuously. But NEVER write to your own code space. That's like
dumping garbage in your own bed. Don't do it.
|
|
@jimmo I also agree that there will be wasted space, but the two partitions solution also offer the advantage that the "internal" partition writable by MP can then be set to Littlefs and have better wear leveling. Actually, as you pointed out, the current STM32 port already offered this exact solution, as explained by Hybrid (STM32) @t35tB0t I agree with your sentiment regarding the usefulness of USB MSC, and I'd like to see it supported by MP whenever the hardware supports it. But wouldn't the slow uploading problem be partially alleviated by breaking a large project into a number of smaller .py modules during development? |
|
@tsaost USB is operating at 12MHz and SerialCom defaults to 0.115MHz.
Above ~400Kbits, the SerialCom BAUD error can cause issues. With some
effort, we may be able to get some target serial consoles running at
~1MBAUD. However, this is just a serial port and in now way offers the
simplicity of a file system mount over USB. I edited the hard-coded
Thonny baud rate to try this out and the experience was underwhelming.
There was also an issue EPS32-S3 flash caching when running PSRAM along
with some very slow performing work-arounds. This may be related to the
reluctance from some here.
In any case, my example project has like a hundred py files and related
configuration files. We have large dictionaries so I created a caching
dictionary object to preserve RAM. Just the py files alone total 12,000
lines. So it isn't a tremendous amount of code.
So, it actually is mostly small files. And, no, rshell and Thonny would
fail to support development on a system like this. In looking for nearby
alternatives, I'm looking at MTP - just like our cell phones use. This
would support file browsing but not in-situ editing.
Thus, there are no alternatives on the able here with the same anywhere
near utility level as USB MSC both in terms of performance and
functionality. CircuitPython clear made their position on this. Zephyr
with MicroPython reduced to just being an application may be the future
answer. Especially if the MPY team has having so much difficult staying
on top of powerful hardware development.
The request is simple: add "adult" USB MSC support to all relevant
platforms without complicated nannys.
|
That is good advice for professional users and advanced amateurs who can easily add flash chips. Many of the users we support are hobbyists whose first project is a simple data logger - they are unhappy when their only filesystem is corrupted. |
|
@peterhinch Agreed that implementing a data logger using only internal
flash presents a problem. And that problem is that data logging to code
flash really isn't simple (because of the issues I've already
highlighted). If Devs really think that one tiny logger use-case is so
important to focus on, then I'll suggest creating a special
safe-data-logger function. One which does protected writes to reserved
data blocks so code flash can not be corrupted. That's just one
possible approach that adds value instead of reducing functionality for
the rest of us. Actually, I might even use such a function myself.
|
|
Y'all heard of a semaphore? |
|
I think ESP32 S2 S3 can completely follow the way of RP2(#7402 ) first, and it is up to the user to decide whether to use MSC or not. |
|
I received a lolin s2 mini yesterday and my mind was blown when I saw how easy it is to update code in CP compared to the painfully slow rshell. Seriously considering teaching CP instead of MP in my next workshop just for that reason alone. The choice to withhold this extremely time saving and user friendly feature because it’ll affect some users down the road leads to bad learning experience, cripples adoption, slows down community growth. if you care about a small subset of users who will write to flash and hit a bug it is logical to care even more about a much larger subset of users who will want to upload custom code to their board. MSC should be supported, enabled by default or at least mentioned on the first pages of the tutorial. If the problem is as serious as some argue (while others, seemingly just as experienced, are claiming it’s a non-issue in real life scenarios) I as a user would understand if enabling MSC disables writing to FS from Python. |
|
Uploading custom code to flash is easy, using mpremote or ftp for devices with network support.
For the other ports, someone has to implement it. I could do that for the SAMD port, once the other previous commits for that port are merged. An implementation for the NRF port should be possible based on the rp2 implementation. ESP32 is more difficult F438 since it uses a different USB stack. |
|
@paulftw - Awesome to hear that you're teaching embedded Python and
agreed that the utility of USB MSC is literally a 100x benefit. The
effect of using USB MSC as-is with STM32 is not serious and its not
really a problem. The issue is that it is an easy feature to access.
People expect it to "just work" and it does. Where they get into
trouble is when they run off and craft code to write to flash without
understand what they are really doing and why the choice to write to
code space is inherently dangerous irregardless of USB MSC. Turning off
USB MSC doesn't fix user ignorance of flash usage risks.
This can be ameliorated with a bunch of explanatory documentation and
good code examples. @Red-M might even provide some semaphore examples
to show how MP can protect itself from aggressive users on the USB.
Architecting for robust solutions and avoiding the pitfalls of flash
corruption could be an informative workshop in itself.
@robert-hh - actually, we found that developing custom code using
mpremote or ftp is not all that effective. We wrote a multi-instancing
ftp server in MP so can push data over e.g. 8 sockets in parallel. Its
cute and works well file tools like FileZilla - most of the time. The
ftp server is functional as a mass code update tool. However, it is not
effective for code development. If MP crashes or is halted, then you're
stuck. So, ftp and mpremote were found to not be truly useful for heavy
development work.
|
|
I too have had various issues with corruption using MSC on a pyboard, due to simultaneous FS access from the PC and microcontroller. However, it's pretty useful, as long as you're careful. The main way I ensure safety is to drop the microcontroller into a REPL before using MSC. Would it be feasible to only allow MSC access when the micro is in a REPL mode? |
|
@victorallume Reading the source code, it certainly is feasible to
block MSC access when MPy is not in REPL. But really we don't need the
added complexity and hassles from that. We didn't need it for the more
sophisticated STM32's so why add MSC nanny's for the ESP32s? Just
design well for what you have in hand and you will have zero problems
with MSC. I've seen enough development and deployment hours with MSC on
STM32s to assert that the much of the debate over MSC flash corruption
here is FUD. Tell us a specific corruption scenario and we can discuss
how to fix that system's design so MSC usage is not a flash corruption
concern.
|
|
@iabdalkader has implemented a lock mechanism for the RP2 board, and I did so in the MSC PR for the mimxrt port. It sets the file system to read-only for the board as long as it is mounter by the host. Similar to what Adafruit does, but kind of automatic. To get R/W access at the board, you just have to eject it at the host. For most users it is useful. If you have a usage pattern which does not use mixed write accesses, then this is not needed. |
|
@peterhinch - As a long time contributor to MicroPython, you and
others have definitely produced a reliable and powerful tool for
embedded systems. Many of us out here have benefited from a focus on
efficient memory usage and practical implementation in constrained
systems. Hence, your concern regarding file system corruption is
appreciated. Yet, respectfully, I feel the concern is misapplied in the
case of USB MSC. I'll also suggest that your referenced FAQ is misses
the point in calling USB MSC a "pitfall" when the true pitfall comes
whens designing target code that writes to flash.
Generally speaking, constantly or randomly writing to flash file system
with your code base is bad practice. Power disruption or other run-time
issues can result in storage corruption. So, this is not a just a USB
MSC risk but a general flash-use risk that novice user's should be made
aware of. Add separate SPI flash for logging if you must. Just stop
writing code that modifies the system flash or you will eventually
corrupt it.
The discussion here is regarding implementation of USB MSC on the
ESP32-S3. This feature is vastly more effective for complex systems
development than using Thonny or rshell. On the STM32 with USB MSC, we
can attach am 8GB SDcard, lay down a git repo on the sdcard, and develop
a complex project with many modules from a PC with a favorite editor.
Code, debug, check-in, push to server; fast cycle time; seamless
development; zero file system corruption.
I tried Thonny and rshell with the ESP32-S3 and it was such a sluggish
experience that I'm back to writing ESP32 code on the STM32 with USB MSC
and then doing final segments on the ESP32. I tried CircuitPython on
the ESP32-S3 and, while the USB-MSC is working beautifully, the overall
departures from MicroPython were beyond disappointing. So, the plea
here is for USB MSC on ESP32-S3.
|
|
@Red-M ... semaphores? cute and even a funny redneck jab - but it misses
the point. The premise here is that USB MSC access to flash can result
in flash corruption. The emphasis here is that ANY writes to code flash
are dangerous operations that require operational considerations or you
will get bit. In addition, @peterhinch is asserting that data logging
in a minimalist hardware configuration is a highly prized feature.
Semaphores and the like are all tiny mechanisms and not a complete
solutions. Let me disrupt the power while you're using your semaphore
to demonstrate...
Writing to flash pages containing code is fundamentally dangerous and
not a safe beginner move. If the intention is to make these things
unbrickable, then there's plenty of other MPY features that a curious
beginner can use to break their system. And thus a lot more thinking
and code would need to be written - and it only protects users from
learning.
I'm left here wondering if other challenges with the ESP32 itself or
affection for LittleFS may be reluctance factors here? After all, STM32
with USB MSC has been around for years and without a need for this
discussion thread. So why worry now?
Is there something fundamentally buggy in the current ESP32-S3
hardware? We know the PSRAM caching design is problematic. Is the ESP
USB hw interface also suffering some bug that makes it undesirable?
Seem to function well enough with CPY...
|
|
I have no desire to enter a debate on this, merely to reiterate an empirical observation from forum support: novice Pyboard users were regularly scribbling over their filesystems. Advice from experienced users was to turn off MSC support in boot.py. The automatic r/o mechanism described by @robert-hh sounds good. |
|
@peterhinch yes, there is no debate here. As a denizen of the MPy
community there is great respect for you. Yet we shall correct your
empirical observation because it may be misleading folks into thinking
that all experienced users are telling folks to turn MSC off. Some have
suggested otherwise.
Experienced users are making it clear that MSC must be available on
hardware that was powerful enough that the chip designers included the
USB core. Or they'll drop MPy and move on. Flash corruption is a trivial
thing to recover from but slow development tools are a permanent
impediment. There are plenty of MPy features that allow us to corrupt
our flash and few will actually brick a unit. Novices should play; they
be allowed to build fast and fail often. Its good for them.
Having the auto r/o MSC mode as default as @robert-hh suggests would be
fine. As long as we can flip a switch and leave it where we want it,
then its all good. But a MPy development policy of protecting novices at
the expense of access to basic hardware capabilities isn't sustainable.
|
Peter is just reporting what he's seen. There's nothing to "correct". I have the same empirical observations, and as an experienced MicroPython user I also provide the same advice -- for most users, using MSC to transfer files to/from the device is not the best option when you weigh up all the competing factors (assumed experience, convenience, risk of corruption, etc). But one of the key things about MicroPython is that we strive provide building blocks, and users who want to combine them in ways that make sense for them should definitely do so! #7402 (comment) is a good summary from Damien for the same issue on the rp2 port. We're aiming to get to the point that this can all be configured in boot.py however the user likes, and at a per-partition level. (But there is some work to get there!) For deployment of larger projects such as the sort of thing you're talking about, we're also working on the "mapfs" feature #8381 -- this has the advantage of saving significant amounts of RAM (because the .mpy files run from flash after being copied to the device) as well as making deployment a lot faster... see notes on this at the bottom of this post!
This may be true when accessing serial over a USB/UART converter, but in this case the point is moot -- MSC would have been unavailable anyway. When accessing USB serial over a CDC ACM device it does not run at 115200 -- the speed setting when opening an ACM device is just forwarded through to the other end and it is free to do whatever it chooses with this information (e.g. if you were implementing a USB/UART that is the speed you should open your UART at).
When you are teaching MicroPython, from a lot of personal experience I find that although MSC is a huge benefit in terms of simplicity, it is basically guaranteed filesystem corruption for new MicroPython users. And the only really efficient workflow that involves working with MSC is to edit directly from the device, and so filesystem corruption means the student loses their code. CircuitPython has fine tuned their workflow to make this as reliable as possible (and it's one of the key differentiators of CircuitPython and one of the many reasons why I'm glad that CircuitPython exists -- it's a great alternative to MicroPython in the education space). Another note -- one thing that trips up new users is that when they copy the files, they still need a serial console to do anything useful. So it's neat to be able to "just copy main.py" but anything above the most trivial program you need a serial terminal -- you might as well install mpremote or similar. And it's confusing when you copy the file over MSC, you still need to trigger a soft reset to remount the filesystem to be able to see the new files for importing). (CircuitPython again solve this by having a heuristic to trigger soft reset automatically on write, which works well for them). For small projects, "mpremote mount" is an amazing workflow that gives you the best of all worlds -- the files live on the PC, they run automatically from the device, and no risk of filesystem corruption.
I tried out a similar scenario on a pyboard v1.1 by creating 50 small files (random sizes between 200 and 1400 bytes).
I think we can possibly optimise mpremote-littlefs a bit here, and in particular a rsync-like partial update would be useful. Additionally, at least half of the mpremote-littlefs copy time is rewriting the same pages over and over again and this could be improved. Given the other benefits of both LittleFS and mpremote, the extra copy time is a worthwhile tradeoff for most users and I will continue to recommend disabling MSC. As maintainers though, as you can see the the mapfs feature is likely to take a higher priority because it is such a huge win in every way (reduced RAM, > 10x faster copy, etc). The copy is so fast because it does a single erase for the whole region and then it's just streaming writes. |
|
The "correction" is that experienced developers are also suggesting
folks leave MSC on and avoid that nastly habit of writing data to code
flash!
Projects many times larger than the tiny speed test referenced here will
simply not be practical without MSC. mpremote-littlefs with an rsync
like partial update is a workflow benedfit for projects as they get larger.
Also, the file system restart to sync process shouldn't be confusing -
its so fast it becomes muscle-memory yet still and with intent and
control. Teach why its necessary and move on. We need to be able to
tweak multiple files on a running system without having it automatically
restart with each edit. MSC paired with a terminal console has been by
far the most streamlined tool chain we've tried with our Mpy projects.
I'm pretty sure I can find ways to corrupt flash even with MSC missing.
As can we likely fix designs where MSC was deemed to be the corruption
villain. I have specific cases where MSC workflow works fine. Where's
a real example where we can recreate the flash corruption and study the
root causes? I'd like to see real cases where one must use a serial
interface to be safe.
|
|
Its been a while since this was updated, but to add my 2c. Regardless of if MSC is a 'Can or worms' or not. The ESP32/S2/S3 has the hardware to support it, so support should be developed. Unless MSC is marked as deprecated across the entire micropython project, we should not be deciding what or what does not go into the specific ESP32 port. Otherwise the ESP32 will become divergent from the micropython standard - this is the worse option, in my opinion, and will hurt micropython as a whole, with developers now uncertain which boards their code will run on, or not. A solution to the corruption problem, would simply to lock the filesystem to python writes, if that filesystem is opened on the host PC. If, MSC is never developed (because the ESP32 micropython developers have decided that MSC should not be there) then solutions like this would never come to light. |
|
MSC is atm supported for the stm32 port and one rp2040 board variant. So it is not very common across MP ports. There is an PR to support Python coded USB devices. That would be a portable way to have various USB device types and modes. Then users can decide whether the install it to their boards or not. |
|
MSC is not the problem with corruption - humans are. If a system on a chip includes a mainstream hardware feature - it is expected to be supported by firmware. Cutting out mainstream features in order to chase a goal of fully normalizing MicroPython across all platforms is not going to end well. An exaggerated example would be limiting precision to integers because not all hardware has floating point. The reason a particular piece of silicon invests all the transistors needed to implement hardware USB block is likely because that device has the horsepower to justify such an interface. It follows then that a more powerful development interface is going to be useful if not essential if one expects to use all that horsepower effectively. From experience, I'll assert that its really simple to use MSC safely and effortlessly even without extra locks. What is the "PR for Python-coded USB devices" trying to solve? The MSC Interface needs to be fast and not running in python environment. Plan A - beg for MicroPython Devs to retain and extend embedded USB support to all relevant hardware modules After so much time has past on this topic and ESP-S3's still don't have a solid MSC solution, I'm suspecting that only Plan B will yield useful results in the long run. |
|
I was going with Plan B anyway/aswell, its probably better that micropython as a whole moves to becoming a zepher app because its easier for board/hardware support. |
|
USBDevice support landed in recent MicroPython. And there is an MR for ESP32-S3 at #15108 |
|
So general idea of this issue is that it would be elegant solution of this is PTP, a light file-based I have micropython proof of concept here for PTP protocol needs some unique 32-bit id micropython should probably buffer at least part So here is WISHLIST if micropython would implement in function
|
|
@emard that's fantastic, thanks for sharing! I've long wanted MTP / PTP working on micropython and have been occasionally working on my own driver for it here: https://github.com/andrewleech/micropython-lib/tree/usb-mtp2/micropython/usb/usb-device-mtp Mine connects on PC, shows file listings, delete works etc but actually transferring files from device to PC is locking up, I'm super keen to look at your version and compare the two, maybe there's learnings we can both share here! Regarding inode's I don't think it's really possible to have them provided by micropython because the default underlying filesystem drivers (LFS2 or FAT) don't have these as far as I'm aware. I didn't think about the inode / handles that much, it just |
|
For me the write support PTP was long search,
sniffing correct USB traffic at android and comparing
the differences. After file update, PTP device must
send filechange event message to interrupt endpoint with
filehandle id, then it finally started to upload file without
error. If PC uploads a file but doesn't receive filechange
event it will throw some error .
I think that if PC opens filemanager window and
while it is open then esp32 changes a file of its own
and sends a "filechange event" - then PC could refresh
file list with for example new size, date, filename etc..
I havent made full micropython ptp vfs browser yet
but obviously PTP is a good and simple solution for
what was impossible people are trying to mount
FAT from 2 sides, changing from both sides while
mounted and hope for no corruption if they are lucky
If there's no inode available probably a handle_id+=1
should be good enough. In practice we can see should
we really buffer full tree or only current browsed directory
and maybe workaround sending filechange id to directory
which havent' really changed but just to
to force PC to refresh and get new set from handle_id+=1
…On Thu, May 1, 2025 at 5:17 AM Andrew Leech ***@***.***> wrote:
*andrewleech* left a comment (micropython/micropython#8426)
<#8426 (comment)>
@emard <https://github.com/emard> that's fantastic, thanks for sharing!
I've *long* wanted MTP / PTP working on micropython and have been
occasionally working on my own driver for it here:
https://github.com/andrewleech/micropython-lib/tree/usb-mtp2/micropython/usb/usb-device-mtp
Mine connects on PC, shows file listings, delete works etc but actually
transferring files from device to PC is locking up, I'm super keen to look
at your version and compare the two, maybe there's learnings we can both
share here!
Regarding inode's I don't think it's really possible to have them provided
by micropython because the default underlying filesystem drivers (LFS2 or
FAT) don't have these as far as I'm aware. I didn't think about the inode /
handles that much, it just handle += 1 in my code currently when it first
builds up the filesystem tree. Not particularly repeatable though, might be
one of the bugs affecting it. I would assume that
hash("/full/path/to/file") would be a better option, it's repeatable and
fast to generate, though not completly guaranteed to be unique.
—
Reply to this email directly, view it on GitHub
<#8426 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC4K2RYLVVY6U4ECPD3EAXD24GG3LAVCNFSM6AAAAAB4FTGPB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBTHE4TAMBXG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
most things work file browser of micropythong esp32s3
vfs tree, read write rename delete files (mkdir/rmdir not yet)
https://github.com/emard/esp32s3-ptp
your code represents to host as MTP device (icon looks like tablet)
and shows storage name first while mine represents as PTP device (icon
looks like camera) and shows root folder first
To try MTP mode and see storage names,
I tried to copy few of your descriptor things
but still always I get PTP. do you have some quick
hint what part is important to determine is it PTP or MTP
…On Thu, May 1, 2025 at 10:31 AM D EMARD ***@***.***> wrote:
For me the write support PTP was long search,
sniffing correct USB traffic at android and comparing
the differences. After file update, PTP device must
send filechange event message to interrupt endpoint with
filehandle id, then it finally started to upload file without
error.
I haven't tried but guess that for example
if both PC opens filemanager window and then esp32
changes a file on its own and then sends a
filechange event, PC should refresh this file with
for example new size, date, filename etc.. I havent tried
this myself but I guess PTP is solution what
was impossible if FAT is mounted and changing
from both sides.
On Thu, May 1, 2025 at 5:17 AM Andrew Leech ***@***.***>
wrote:
> *andrewleech* left a comment (micropython/micropython#8426)
> <#8426 (comment)>
>
> @emard <https://github.com/emard> that's fantastic, thanks for sharing!
> I've *long* wanted MTP / PTP working on micropython and have been
> occasionally working on my own driver for it here:
> https://github.com/andrewleech/micropython-lib/tree/usb-mtp2/micropython/usb/usb-device-mtp
>
> Mine connects on PC, shows file listings, delete works etc but actually
> transferring files from device to PC is locking up, I'm super keen to look
> at your version and compare the two, maybe there's learnings we can both
> share here!
>
> Regarding inode's I don't think it's really possible to have them
> provided by micropython because the default underlying filesystem drivers
> (LFS2 or FAT) don't have these as far as I'm aware. I didn't think about
> the inode / handles that much, it just handle += 1 in my code currently
> when it first builds up the filesystem tree. Not particularly repeatable
> though, might be one of the bugs affecting it. I would assume that
> hash("/full/path/to/file") would be a better option, it's repeatable and
> fast to generate, though not completly guaranteed to be unique.
>
> —
> Reply to this email directly, view it on GitHub
> <#8426 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AC4K2RYLVVY6U4ECPD3EAXD24GG3LAVCNFSM6AAAAAB4FTGPB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBTHE4TAMBXG4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
|
It would be fantastic if we could get a working PTP and/or MTP example code that we could include in micropython-lib together with the other USB device examples. |
|
This would be great, if included in
micropython-lib that would make
development so comfortable much
better than mpremote cp
my coding style is messy
I can try to clean my code,
shorten, fix bugs...
…On Sun, May 4, 2025 at 7:27 PM Jon Nordby ***@***.***> wrote:
*jonnor* left a comment (micropython/micropython#8426)
<#8426 (comment)>
It would be fantastic if we could get a working PTP and/or MTP example
code that we could include in micropython-lib together with the other USB
device examples.
—
Reply to this email directly, view it on GitHub
<#8426 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC4K2R3MLKX25JZHRDF3PEL24ZEZFAVCNFSM6AAAAAB4FTGPB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBZGMZDOOBTGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
If interface is named "MTP" then protocol will become MTP |
Are there any plans to increase MSC support here?
CDC+MSC may be great
Thanks
The text was updated successfully, but these errors were encountered: