doc,esp32: Improve and de-duplicate the flashing instructions. #16535

projectgus · 2025-01-07T04:03:14Z

Summary

This PR aims to clean up the flashing instructions for the esp32 port, and also make them easier to maintain in the future.

The big change for the download pages is using string formatting to expand the very few things (chip name, flash offset) which are different between most boards. This allows using the same two deploy.md and deploy_nativeusb.md for almost all ESP32 boards, which also made it easier to add some additional troubleshooting docs there.

The initial motivation comes from a few places:

@andrewleech pointed out that esptool port auto-detection works well these days: docs: Add reference for reset and boot sequence, document factory reset for esp32 & rp2. #15956 (comment)
I noticed that the instructions on the download page are also pretty bare-bones for most boards. They also include redundant options like -z (which is enabled by default).
@donburch888 pointed out that the setup instructions give the wrong flashing address for newer chips: https://github.com/orgs/micropython/discussions/16417 - that page is also missing a faster baud rate, etc.

Specific changes applied across here include:

Remove unnecessary command line arguments.
~~Harmonise the flashing steps between the download pages and the ESP32 getting started doc.~~ EDIT: Now the tutorial page refers to the download page for flashing steps - it's much simpler as these are the steps for this particular board, rather than "any" board.
Expanded our troubleshooting steps a bit, to include getting into download mode on most boards.
Also link to the esptool.py docs when possible, as these are quite good.

Testing

Used ./build-downloads.py to generate download pages from master and this branch, and diffed the directories to ensure no non-esp32-port files changed and that nothing important was missing from the updated esp32 download pages.
I've uploaded the generated files from this branch to a gist, here: https://gist.github.com/projectgus/f2d24fa2e4689c41ce8002b70f0926bf#file-esp32c6-md

Trade-offs and Alternatives

Having lots of copy-paste deploy.md is less "clever", but much more painful to maintain and update.
If we want even more flexibility in how we generate these in the future then we could migrate to something like Jinja2 templating. This would let us generate quite sophisticated sets of download instructions (for example, we could add DFU flashing instructions for all boards where the chip supports USB DFU-mode). Probably overkill for now.

This work was funded through GitHub Sponsors.

projectgus · 2025-01-07T04:06:29Z

@UnexpectedMaker I'd be interested to know what you think of this, as your boards have the best download instructions included so far 🤩 . I think I picked up all the per-board variants (flashing mode buttons, and the boards which need external flashing mode wiring).

You can see a copy of the suggested new generated deploy.md files for UM boards here: https://gist.github.com/projectgus/f2d24fa2e4689c41ce8002b70f0926bf#file-um_feathers2-md

and also these ones where the name has no UM_ prefix: nanos3 and omgs3

Please let me know if you think anything here is wrong, or might lead to a worse user experience.

codecov · 2025-01-07T04:07:10Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.58%. Comparing base (b6649b9) to head (84e0aca).
Report is 4 commits behind head on master.

Additional details and impacted files 8000

@@           Coverage Diff           @@
##           master   #16535   +/-   ##
=======================================
  Coverage   98.58%   98.58%           
=======================================
  Files         167      167           
  Lines       21596    21596           
=======================================
  Hits        21291    21291           
  Misses        305      305

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2025-01-07T04:38:39Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

UnexpectedMaker · 2025-01-07T04:54:30Z

@projectgus Hiya!

A few things...

I'm not sure what you mean by NANOS3 and OMGS3 not having UM_ ? All of my boards have an UM_ prefix
For the S2/S3 boards (those that use native USB) there should be no --baud setting as it's fixed speed and adding that makes no difference and is confusing for people. The same thing applies to the message about dropping it to slow transfer if there are transfer issues.
Seems to be inconsistencies with board naming within each doc - for instance in FeatherS2 - sometimes the board is called FeatherS2 and sometimes Feather S2

Otherwise, I think consistency across all of the boards is great, good job!

projectgus · 2025-01-07T06:07:00Z

Thanks for the quick review, @UnexpectedMaker!

I'm not sure what you mean by NANOS3 and OMGS3 not having UM_ ? All of my boards have an UM_ prefix

Oh, right. I put direct links to those two as they're not grouped with the other files in the gist. All the other UM_SOMETHING boards generate markdown files named UM_SOMETHING.md but these ones are nanos3.md and omgs3.md. This name comes from the board.json contents, not the board directory name. I don't think it changes anything apart from the download URLs, i.e. there is currently:

Happy to change omgs3 and nanos3 to be consistent with the others in this PR, but I don't particularly mind (and leaving them as-is will avoid breaking any existing links).

For the S2/S3 boards (those that use native USB) there should be no --baud setting as it's fixed speed and adding that makes no difference and is confusing for people. The same thing applies to the message about dropping it to slow transfer if there are transfer issues.

Good catch! I've added a separate deploy_nativeusb.md and pointed all the boards with native USB to this one. The gist is updated as well, please take a look.

Seems to be inconsistencies with board naming within each doc - for instance in FeatherS2 - sometimes the board is called FeatherS2 and sometimes Feather S2

Ah, this is where the board.md file uses one form and the board.json file uses the other. Which form do you prefer: with or without the space? I'll clean it up in this PR.

UnexpectedMaker · 2025-01-07T06:13:41Z

Thanks for the quick review, @UnexpectedMaker!

Oh, right. I put direct links to those two as they're not grouped with the other files in the gist. All the other UM_SOMETHING boards generate markdown files named UM_SOMETHING.md but these ones are nanos3.md and omgs3.md. This name comes from the board.json contents, not the board directory name. I don't think it changes anything apart from the download URLs, i.e. there is currently:

https://micropython.org/download/omgs3/

https://micropython.org/download/UM_TINYS2/

Happy to change omgs3 and nanos3 to be consistent with the others in this PR, but I don't particularly mind (and leaving them as-is will avoid breaking any existing links).

Ahh, hmm, my mistake then, sorry. They should all be um_ - I can fix that later if you want, or if you are already doing stuff, happy for you to do it ;)

Good catch! I've added a separate deploy_nativeusb.md and pointed all the boards with native USB to this one. The gist is updated as well, please take a look.

Excellent!

Seems to be inconsistencies with board naming within each doc - for instance in FeatherS2 - sometimes the board is called FeatherS2 and sometimes Feather S2

Ah, this is where the board.md file uses one form and the board.json file uses the other. Which form do you prefer: with or without the space? I'll clean it up in this PR.

Hmm, again, seems my mistake/inconsistency when adding the board data over time. There should be no spaces in my names other than when there is a Neo - so FeatherS3 and FeatherS3 Neo - All S2/S3 bits have no spaces before them.

Thanks for being so responsive Angus - Happy New Year btw! I hope you and the family are well :)

projectgus · 2025-01-07T06:35:11Z

@UnexpectedMaker Great! I've pushed a commit to change the spacings, as this PR otherwise will otherwise make the download pages inconsistent.

I've left the omgs3/tinys3 naming alone, thanks for offering to pick that one up. It probably makes more sense to do it in a separate PR as it's separate from these changes. Feel free to tag me for review if you do submit a PR for it.

And Happy New Year to you and your family, as well! 😁

Josverl

On >90% of Windows the command esptool.py will fail
below is for pipx installed esptool

I suggest to clarify this here , or in another place where the installation of the esptool package is described.

and HNY 🥂

Josverl · 2025-01-07T11:24:53Z

One other thing that I found when creating mpflash was that of the ESP32 port , the ARDUINO_NANO_ESP32 cannot be flashed with esptool , but needs dfutil

The documentation is in place , but its easy to miss as its an outlier from most of the ESPxxx boards
https://github.com/micropython/micropython/blob/bf35eefc625b35aa301aefbc743dcf1395829865/ports/esp32/boards/ARDUINO_NANO_ESP32/deploy.md

https://github.com/Josverl/micropython-stubber/blob/main/src/mpflash/mpflash/flash/esp.py

projectgus · 2025-01-08T05:14:29Z

Thanks @Josverl for the insights! I've added notes for Windows users, and updated the gist with the previews.

The documentation is in place , but its easy to miss as its an outlier from most of the ESPxxx boards

Yes, that one is kept unchanged in this PR (I did add a spurious flash_offset key to the boards.json file to be consistent with all the others, but the actual deploy.md file remains the same.

Do you suggest additional changes? We could mention this in the intro.rst file as well, I suppose, but I think if we go that far then we should probably remove the flashing instructions from there entirely and direct everyone to their board's download page instead.

Josverl · 2025-01-08T08:00:41Z

Do you suggest additional changes? We could mention this in the intro.rst file...

Not in detail, but perhaps a note like:

While many board in a port share the same flashing tools and common parameters, some boards will use different parameters or different flashing tools and methods.

dpgeorge

Thanks @projectgus for this clean up, it's definitely a good improvement and will make it easier to maintain these instructions.

Will you squash the two commits that modify docs/esp32/tutorial/intro.rst? It seems like the latter one fixes the first.

tools/autobuild/build-downloads.py

ports/esp32/boards/ARDUINO_NANO_ESP32/board.json

projectgus · 2025-01-15T00:47:10Z

Thanks @dpgeorge and @Josverl for the review comments. Have updated accordingly (and the preview gist is also refreshed).

I ended up removing the flashing steps from the tutorial entirely and directing users to the download page. In most cases they already have that page open, and it's much easier to follow because it's the specific steps for that board only.

I also made some more tweaks, particularly around the instructions for choosing a board to download as I realised these were also quite out of date. I moved the information about ESP32_GENERIC variants out of the tutorial as well, and expanded the descriptions in that board's board.md file.

mattytrentini

A nice improvement; queries about flashing ESP32's are very common. I think this will help!

docs/esp32/tutorial/intro.rst

Allows two source files (ports/esp32/boards/deploy.md and deploy_nativeusb.md for boards with only native USB) for all esp32 installation steps, with templated chip name and flash offset inserted via string formatting. The new files add more text to explain the esptool.py port auto-detection, remove the unnecessary -z feature (already enabled by default), and add a bit of troubleshooting and port detection info. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

Includes fixing the flashing address for newer SoCs, as reported in discussion https://github.com/orgs/micropython/discussions/16417 Also removes some redundant or out of date information, and adds links to the Espressif esptool docs which are quite comprehensive. Information about ESP32_GENERIC variants is moved to the board page, as it only applies to that board. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

The previous deploy.md refactors revealed that these boards had a different "product" entry in boards.json compared to the name given in the board.md file. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

The user already has it open, and its customised for their particular board. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

projectgus added docs port-esp32 labels Jan 7, 2025

projectgus requested review from mattytrentini, andrewleech and dpgeorge January 7, 2025 04:03

projectgus force-pushed the doc/esp32_flashing branch 3 times, most recently from 06e9206 to c86075d Compare January 7, 2025 04:28

Josverl suggested changes Jan 7, 2025

View reviewed changes

projectgus force-pushed the doc/esp32_flashing branch from bfc948c to 57b7896 Compare January 8, 2025 05:10

projectgus force-pushed the doc/esp32_flashing branch from 57b7896 to 47fb89c Compare January 8, 2025 05:40

dpgeorge mentioned this pull request Jan 14, 2025

esp32/boards: Remove remaining "id" entries from board.json. #16584

Merged

dpgeorge reviewed Jan 14, 2025

View reviewed changes

tools/autobuild/build-downloads.py Outdated Show resolved Hide resolved

ports/esp32/boards/ARDUINO_NANO_ESP32/board.json Outdated Show resolved Hide resolved

projectgus force-pushed the doc/esp32_flashing branch 2 times, most recently from f38eb5a to 8e991c0 Compare January 15, 2025 00:44

dpgeorge added this to the release-1.25.0 milestone Jan 15, 2025

mattytrentini approved these changes Jan 16, 2025

View reviewed changes

docs/esp32/tutorial/intro.rst Outdated Show resolved Hide resolved

projectgus force-pushed the doc/esp32_flashing branch from 9fa9227 to 6b0f7c6 Compare January 17, 2025 03:28

projectgus mentioned this pull request Jan 17, 2025

esp32c2 support (IDF v5.3) #15906

Open

projectgus added 4 commits January 17, 2025 16:03

docs/esp32: Defer to the download page for flashing steps.

84e0aca

The user already has it open, and its customised for their particular board. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

dpgeorge approved these changes Jan 17, 2025

View reviewed changes

dpgeorge force-pushed the doc/esp32_flashing branch from 6b0f7c6 to 84e0aca Compare January 17, 2025 05:54

dpgeorge merged commit 84e0aca into micropython:master Jan 17, 2025
59 of 64 checks passed

projectgus deleted the doc/esp32_flashing branch January 17, 2025 06:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

doc,esp32: Improve and de-duplicate the flashing instructions. #16535

doc,esp32: Improve and de-duplicate the flashing instructions. #16535

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doc,esp32: Improve and de-duplicate the flashing instructions. #16535

doc,esp32: Improve and de-duplicate the flashing instructions. #16535

Uh oh!

Conversation

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!