diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs new file mode 100644 index 0000000000000..195e1e06eea46 --- /dev/null +++ b/.git-blame-ignore-revs @@ -0,0 +1,68 @@ +# all: Prune trailing whitespace. +dda9b9c6da5d3c31fa8769e581a753e95a270803 + +# all: Remove the "STATIC" macro and just use "static" instead. +decf8e6a8bb940d5829ca3296790631fcece7b21 + +# renesas-ra: Fix spelling mistakes found by codespell. +b3f2f18f927fa2fad10daf63d8c391331f5edf58 + +# all: Update Python formatting to ruff-format. +bbd8760bd9a2302e5abee29db279102bb11d7732 + +# all: Fix various spelling mistakes found by codespell 2.2.6. +cf490a70917a1b2d38ba9b58e763e0837d0f7ca7 + +# all: Fix spelling mistakes based on codespell check. +b1229efbd1509654dec6053865ab828d769e29db + +# top: Update Python formatting to black "2023 stable style". +8b2748269244304854b3462cb8902952b4dcb892 + +# all: Reformat remaining C code that doesn't have a space after a comma. +5b700b0af90591d6b1a2c087bb8de6b7f1bfdd2d + +# ports: Reformat more C and Python source code. +5c32111fa0e31e451b0f1666bdf926be2fdfd82c + +# all: Update Python formatting to latest Black version 22.1.0. +ab2923dfa1174dc177f0a90cb00a7e4ff87958d2 + +# all: Update Python formatting to latest Black version 21.12b0. +3770fab33449a5dadf8eb06edfae0767e75320a6 + +# tools/gen-cpydiff.py: Fix formatting of doc strings for new Black. +0f78c36c5aa458a954eed39a46942209107a553e + +# tests/run-tests.py: Reformat with Black. +2a38d7103672580882fb621a5b76e8d26805d593 + +# all: Update Python code to conform to latest black formatting. +06659077a81b85882254cf0953c33b27614e018e + +# tools/uncrustify: Enable more opts to remove space between func and '('. +77ed6f69ac35c1663a5633a8ee1d8a2446542204 + +# tools/codeformat.py: Include extmod/{btstack,nimble} in code formatting. +026fda605e03113d6e753290d65fed774418bc53 + +# all: Format code to add space after C++-style comment start. +84fa3312cfa7d2237d4b56952f2cd6e3591210c4 + +# tests: Format all Python code with black, except tests in basics subdir. +3dc324d3f1312e40d3a8ed87e7244966bb756f26 + +# all: Remove spaces inside and around parenthesis. +1a3e386c67e03a79eb768cb6e9f6777e002d6660 + +# all: Remove spaces between nested paren and inside function arg paren. +feb25775851ba0c04b8d1013716f442258879d9c + +# all: Reformat C and Python source code with tools/codeformat.py. +69661f3343bedf86e514337cff63d96cc42f8859 + +# stm32/usbdev: Convert files to unix line endings. +abde0fa2267f9062b28c3c015d7662a550125cc6 + +# all: Remove trailing spaces, per coding conventions. +761e4c7ff62896c7d8f8c3dfc3cc98a4cc4f2f6f diff --git a/.gitattributes b/.gitattributes index 133e2625c1f38..2d8496db50488 100644 --- a/.gitattributes +++ b/.gitattributes @@ -7,17 +7,18 @@ *.bat text eol=crlf # These are binary so should never be modified by git. +*.a binary +*.ico binary *.png binary *.jpg binary *.dxf binary *.mpy binary +*.der binary # These should also not be modified by git. tests/basics/string_cr_conversion.py -text tests/basics/string_crlf_conversion.py -text ports/stm32/pybcdc.inf_template -text -ports/stm32/usbd_* -text -ports/stm32/usbdev/** -text ports/stm32/usbhost/** -text ports/cc3200/hal/aes.c -text ports/cc3200/hal/aes.h -text diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000000000..15186fd5c1916 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1 @@ +github: micropython diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000000000..1ec6c7067f9cd --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,109 @@ +name: Bug report +description: Report a bug or unexpected behaviour +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Please provide as much detail as you can, it really helps us find and fix bugs faster. + + #### Not a bug report? + + * If you have a question \"How Do I ...?\", please post it on [GitHub Discussions](https://github.com/orgs/micropython/discussions/) or [Discord](https://discord.gg/RB8HZSAExQ) instead of here. + * For missing or incorrect documentation, or feature requests, then please [choose a different issue type](https://github.com/micropython/micropython/issues/new/choose). + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) matching this bug before reporting. + - type: input + id: port-board-hw + attributes: + label: Port, board and/or hardware + description: | + Which MicroPython port(s) and board(s) are you using? + placeholder: | + esp32 port, ESP32-Fantastic board. + validations: + required: true + - type: textarea + id: version + attributes: + label: MicroPython version + description: | + To find the version: + + 1. Open a serial REPL. + 2. Type Ctrl-B to see the startup message. + 3. Copy-paste that output here. + + If the issue is about building MicroPython, please provide output of `git describe --dirty` and as much information as possible about the build environment. + + If the version or configuration is modified from the official MicroPython releases or the master branch, please tell us the details of this as well. + placeholder: | + MicroPython v6.28.3 on 2029-01-23; PyBoard 9 with STM32F9 + validations: + required: true + - type: textarea + id: steps-reproduce + attributes: + label: Reproduction + description: | + What steps will reproduce the problem? Please include all details that could be relevant about the environment, configuration, etc. + + If there is Python code to reproduce this issue then please either: + a. Type it into a code block below ([code block guide](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks)), or + b. Post longer code to a [GitHub gist](https://gist.github.com/), or + c. Create a sample project on GitHub. + + For build issues, please provide the exact build commands that you ran. + placeholder: | + 1. Copy paste the code provided below into a new file + 2. Use `mpremote run` to execute it on the board. + validations: + required: true + - type: textarea + id: expected + attributes: + label: Expected behaviour + description: | + What did you expect MicroPython to do? If comparing output with CPython or a different MicroPython port/version then please provide that output here. + placeholder: | + Expected to print "Hello World". + + Here is the correct output, seen with previous MicroPython version v3.14.159: + + > [...] + - type: textarea + id: what-happened + attributes: + label: Observed behaviour + description: | + What actually happened? Where possible please paste exact output, or the complete build log, etc. Very long output can be linked in a [GitHub gist](https://gist.github.com/). + placeholder: | + This unexpected exception appears: + + > [...] + validations: + required: true + - type: textarea + id: additional + attributes: + label: Additional Information + description: | + Is there anything else that might help to resolve this issue? + value: No, I've provided everything above. + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true + - type: markdown + attributes: + value: | + Thanks for taking the time to help improve MicroPython. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000000000..f3662464da337 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,11 @@ +blank_issues_enabled: false +contact_links: + - name: MicroPython GitHub Discussions + url: https://github.com/orgs/micropython/discussions + about: Community discussion about all things MicroPython. This is the best place to start if you have questions about using MicroPython or getting started with MicroPython development. + - name: MicroPython Documentation + url: https://docs.micropython.org/ + about: Documentation for using and working with MicroPython and libraries. + - name: MicroPython Downloads + url: https://micropython.org/download/ + about: Pre-built firmware and information for most supported boards. diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml new file mode 100644 index 0000000000000..93051e51c80c7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -0,0 +1,46 @@ +name: Documentation issue +description: Report areas of the documentation or examples that need improvement +title: "docs: " +labels: ["documentation"] +body: + - type: markdown + attributes: + value: | + This form is for reporting issues with the documentation or examples provided with MicroPython. + + If you have a general question \"How Do I ...?\", please post it on [GitHub Discussions](https://github.com/orgs/micropython/discussions/) or [Discord](https://discord.gg/RB8HZSAExQ) instead of here. + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) before reporting a new one. + - type: input + id: page + attributes: + label: Documentation URL + description: | + Does this issue relate to a particular page in the [online documentation](https://docs.micropython.org/en/latest/)? If yes, please paste the URL of the page: + placeholder: | + https://docs.micropython.org/en/latest/ + - type: textarea + id: version + attributes: + label: Description + description: | + Please describe what was missing from the documentation and/or what was incorrect/incomplete. + validations: + required: true + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true + - type: markdown + attributes: + value: | + Thanks for taking the time to help improve MicroPython. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000000000..7d5162a32a4af --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,74 @@ +name: Feature request +description: Request a feature or improvement +labels: ['enhancement'] +body: + - type: markdown + attributes: + value: | + This form is for requesting features or improvements in MicroPython. + + #### Get feedback first + + Before submitting a new feature idea here, suggest starting a discussion on [Discord](https://discord.gg/RB8HZSAExQ) or [GitHub Discussions](https://github.com/orgs/micropython/discussions/) to get early feedback from the community and maintainers. + + #### Not a MicroPython core feature? + + * If you have a question \"How Do I ...?\", please post it on GitHub Discussions or Discord instead of here. + * Could this feature be implemented as a pure Python library? If so, please open the request on the [micropython-lib repository](https://github.com/micropython/micropython-lib/issues) instead. + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) before opening a new one. + - type: textarea + id: feature + attributes: + label: Description + description: | + Describe the feature you'd like to see added to MicroPython. What does this feature enable and why is it useful? + + * For core Python features, where possible please include a link to the relevant PEP or CPython documentation. + * For new architectures / ports / boards, please provide links to relevant documentation, specifications, and toolchains. Any information about the popularity and unique features about this hardware would also be useful. + * For features for existing ports (e.g. new peripherals or microcontroller features), please describe which port(s) it applies to, and whether this is could be an extension to the machine API or a port-specific module? + * For drivers (e.g. for external hardware), please link to datasheets and/or existing drivers from other sources. + + If there is an existing discussion somewhere about this feature, please add a link to it as well. + validations: + required: true + - type: textarea + id: size + attributes: + label: Code Size + description: | + MicroPython aims to strike a balance between functionality and code size. Can this feature be optionally enabled? + + If you believe the usefulness of this feature would outweigh the additional code size, please explain. (It's OK to say you're unsure here, we're happy to discuss this with you.) + - type: dropdown + id: implementation + attributes: + label: Implementation + description: | + What is your suggestion for implementing this feature? + + (See also: [How to sponsor](https://github.com/sponsors/micropython#sponsors), [How to submit a Pull Request](https://github.com/micropython/micropython/wiki/ContributorGuidelines).) + options: + - I hope the MicroPython maintainers or community will implement this feature + - I intend to implement this feature and would submit a Pull Request if desirable + - I would like to sponsor development of this feature + multiple: true + validations: + required: true + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true + - type: markdown + attributes: + value: | + Thanks for taking the time to suggest improvements for MicroPython. diff --git a/.github/ISSUE_TEMPLATE/security.yml b/.github/ISSUE_TEMPLATE/security.yml new file mode 100644 index 0000000000000..57c2a5885ed85 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/security.yml @@ -0,0 +1,60 @@ +name: Security report +description: Report a security issue or vulnerability in MicroPython +labels: ["security"] +body: + - type: markdown + attributes: + value: | + This form is for reporting security issues in MicroPython that are not readily exploitable. + + 1. For issues that are readily exploitable or have high impact, please email contact@micropython.org instead. + 1. If this is a question about security, please ask it in [Discussions](https://github.com/orgs/micropython/discussions/) or [Discord](https://discord.gg/RB8HZSAExQ) instead. + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) before reporting a new one. + + - type: input + id: port-board-hw + attributes: + label: Port, board and/or hardware + description: | + Which MicroPython port(s) and board(s) are you using? + placeholder: | + esp32 port, ESP32-Duper board. + - type: textarea + id: version + attributes: + label: MicroPython version + description: | + To find the version: + + 1. Open a serial REPL. + 2. Type Ctrl-B to see the startup message. + 3. Copy-paste that output here. + + If the version or configuration is modified from the official MicroPython releases or the master branch, please tell us the details of this as well. + placeholder: | + MicroPython v6.28.3 on 2029-01-23; PyBoard 9 with STM32F9 + - type: textarea + id: report + attributes: + label: Issue Report + description: | + Please provide a clear and concise description of the security issue. + + * What does this issue allow an attacker to do? + * How does the attacker exploit this issue? + validations: + required: true + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000000000..2c7d1708395e2 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,7 @@ +version: 2 +updates: + # Maintain dependencies for GitHub Actions + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "daily" diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000000..e11cebddb3710 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,33 @@ + + +### Summary + + + + +### Testing + + + + +### Trade-offs and Alternatives + + + diff --git a/.github/workflows/biome.yml b/.github/workflows/biome.yml new file mode 100644 index 0000000000000..88744f16ca7d6 --- /dev/null +++ b/.github/workflows/biome.yml @@ -0,0 +1,16 @@ +name: JavaScript code lint and formatting with Biome + +on: [push, pull_request] + +jobs: + eslint: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Biome + uses: biomejs/setup-biome@v2 + with: + version: 1.5.3 + - name: Run Biome + run: biome ci --indent-style=space --indent-width=4 tests/ ports/webassembly diff --git a/.github/workflows/code_formatting.yml b/.github/workflows/code_formatting.yml new file mode 100644 index 0000000000000..9f30f048cfdbe --- /dev/null +++ b/.github/workflows/code_formatting.yml @@ -0,0 +1,20 @@ +name: Check code formatting + +on: [push, pull_request] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + code-formatting: + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + - name: Install packages + run: source tools/ci.sh && ci_c_code_formatting_setup + - name: Run code formatting + run: source tools/ci.sh && ci_c_code_formatting_run + - name: Check code formatting + run: git diff --exit-code diff --git a/.github/workflows/code_size.yml b/.github/workflows/code_size.yml new file mode 100644 index 0000000000000..67261933798cb --- /dev/null +++ b/.github/workflows/code_size.yml @@ -0,0 +1,50 @@ +name: Check code size + +on: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'ports/bare-arm/**' + - 'ports/mimxrt/**' + - 'ports/minimal/**' + - 'ports/rp2/**' + - 'ports/samd/**' + - 'ports/stm32/**' + - 'ports/unix/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 100 + - name: Install packages + run: source tools/ci.sh && ci_code_size_setup + - name: Build + run: source tools/ci.sh && ci_code_size_build + - name: Compute code size difference + run: tools/metrics.py diff ~/size0 ~/size1 | tee diff + - name: Save PR number + if: github.event_name == 'pull_request' + env: + PR_NUMBER: ${{ github.event.number }} + run: echo $PR_NUMBER > pr_number + - name: Upload diff + if: github.event_name == 'pull_request' + uses: actions/upload-artifact@v4 + with: + name: code-size-report + path: | + diff + pr_number + retention-days: 1 diff --git a/.github/workflows/code_size_comment.yml b/.github/workflows/code_size_comment.yml new file mode 100644 index 0000000000000..521afad709d17 --- /dev/null +++ b/.github/workflows/code_size_comment.yml @@ -0,0 +1,105 @@ +name: Code size comment + +on: + workflow_run: + workflows: [Check code size] + types: [completed] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + comment: + runs-on: ubuntu-22.04 + steps: + - name: 'Download artifact' + id: download-artifact + uses: actions/github-script@v7 + with: + result-encoding: string + script: | + const fs = require('fs'); + + const allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({ + owner: context.repo.owner, + repo: context.repo.repo, + run_id: context.payload.workflow_run.id, + }); + + const matchArtifact = allArtifacts.data.artifacts.filter((artifact) => { + return artifact.name == "code-size-report" + }); + + if (matchArtifact.length === 0) { + console.log('no matching artifact found'); + console.log('result: "skip"'); + + return 'skip'; + } + + const download = await github.rest.actions.downloadArtifact({ + owner: context.repo.owner, + repo: context.repo.repo, + artifact_id: matchArtifact[0].id, + archive_format: 'zip', + }); + + fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/code-size-report.zip`, Buffer.from(download.data)); + + console.log('artifact downloaded to `code-size-report.zip`'); + console.log('result: "ok"'); + + return 'ok'; + - name: 'Unzip artifact' + if: steps.download-artifact.outputs.result == 'ok' + run: unzip code-size-report.zip + - name: Post comment to pull request + if: steps.download-artifact.outputs.result == 'ok' + uses: actions/github-script@v7 + with: + github-token: ${{secrets.GITHUB_TOKEN}} + script: | + const fs = require('fs'); + + const prNumber = Number(fs.readFileSync('pr_number')); + const codeSizeReport = `Code size report: + + \`\`\` + ${fs.readFileSync('diff')} + \`\`\` + `; + + const comments = await github.paginate( + github.rest.issues.listComments, + { + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + } + ); + + comments.reverse(); + + const previousComment = comments.find(comment => + comment.user.login === 'github-actions[bot]' + ) + + // if github-actions[bot] already made a comment, update it, + // otherwise create a new comment. + + if (previousComment) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: previousComment.id, + body: codeSizeReport, + }); + } else { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + body: codeSizeReport, + }); + } diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml new file mode 100644 index 0000000000000..1d6b1dc9d2b27 --- /dev/null +++ b/.github/workflows/codespell.yml @@ -0,0 +1,13 @@ +name: Check spelling with codespell + +on: [push, pull_request] + +jobs: + codespell: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + # codespell version should be kept in sync with .pre-commit-config.yml + - run: pip install --user codespell==2.4.1 tomli + - run: codespell + diff --git a/.github/workflows/commit_formatting.yml b/.github/workflows/commit_formatting.yml new file mode 100644 index 0000000000000..fcbcaa7092ee2 --- /dev/null +++ b/.github/workflows/commit_formatting.yml @@ -0,0 +1,18 @@ +name: Check commit message formatting + +on: [pull_request] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 100 + - uses: actions/setup-python@v5 + - name: Check commit message formatting + run: source tools/ci.sh && ci_commit_formatting_run diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000000000..d01a4b50c9810 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,23 @@ +name: Build docs + +on: + push: + pull_request: + paths: + - docs/** + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + - name: Install Python packages + run: pip install -r docs/requirements.txt + - name: Build docs + run: make -C docs/ html diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml new file mode 100644 index 0000000000000..6613f106625a2 --- /dev/null +++ b/.github/workflows/examples.yml @@ -0,0 +1,25 @@ +name: Check examples + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'examples/**' + - 'ports/unix/**' + - 'py/**' + - 'shared/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + embedding: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: make -C examples/embedding -f micropython_embed.mk && make -C examples/embedding + - name: Run + run: ./examples/embedding/embed | grep "hello world" diff --git a/.github/workflows/mpremote.yml b/.github/workflows/mpremote.yml new file mode 100644 index 0000000000000..ee91b6360b9b4 --- /dev/null +++ b/.github/workflows/mpremote.yml @@ -0,0 +1,29 @@ +name: Package mpremote + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + # Setting this to zero means fetch all history and tags, + # which hatch-vcs can use to discover the version tag. + fetch-depth: 0 + - uses: actions/setup-python@v5 + - name: Install build tools + run: pip install build + - name: Build mpremote wheel + run: cd tools/mpremote && python -m build --wheel + - name: Archive mpremote wheel + uses: actions/upload-artifact@v4 + with: + name: mpremote + path: | + tools/mpremote/dist/mpremote*.whl diff --git a/.github/workflows/mpy_format.yml b/.github/workflows/mpy_format.yml new file mode 100644 index 0000000000000..b6768a46c3cdc --- /dev/null +++ b/.github/workflows/mpy_format.yml @@ -0,0 +1,24 @@ +name: .mpy file format and tools + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'examples/**' + - 'tests/**' + - 'tools/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-22.04 # use 22.04 to get python2 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_mpy_format_setup + - name: Test mpy-tool.py + run: source tools/ci.sh && ci_mpy_format_test diff --git a/.github/workflows/ports.yml b/.github/workflows/ports.yml new file mode 100644 index 0000000000000..1f262b0ba4bee --- /dev/null +++ b/.github/workflows/ports.yml @@ -0,0 +1,22 @@ +name: Build ports metadata + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - ports/** + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - name: Build ports download metadata + run: mkdir boards && ./tools/autobuild/build-downloads.py . ./boards diff --git a/.github/workflows/ports_alif.yml b/.github/workflows/ports_alif.yml new file mode 100644 index 0000000000000..0e96e7d816e50 --- /dev/null +++ b/.github/workflows/ports_alif.yml @@ -0,0 +1,33 @@ +name: alif port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/alif/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_alif: + strategy: + fail-fast: false + matrix: + ci_func: # names are functions in ci.sh + - alif_ae3_build + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_alif_setup + - name: Build ci_${{matrix.ci_func }} + run: source tools/ci.sh && ci_${{ matrix.ci_func }} diff --git a/.github/workflows/ports_cc3200.yml b/.github/workflows/ports_cc3200.yml new file mode 100644 index 0000000000000..f178a140587db --- /dev/null +++ b/.github/workflows/ports_cc3200.yml @@ -0,0 +1,28 @@ +name: cc3200 port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/cc3200/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_cc3200_setup + - name: Build + run: source tools/ci.sh && ci_cc3200_build diff --git a/.github/workflows/ports_esp32.yml b/.github/workflows/ports_esp32.yml new file mode 100644 index 0000000000000..4c07f89437757 --- /dev/null +++ b/.github/workflows/ports_esp32.yml @@ -0,0 +1,57 @@ +name: esp32 port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/esp32/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_idf: + strategy: + fail-fast: false + matrix: + ci_func: # names are functions in ci.sh + - esp32_build_cmod_spiram_s2 + - esp32_build_s3_c3 + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - id: idf_ver + name: Read the ESP-IDF version (including Python version) + run: source tools/ci.sh && echo "IDF_VER=${IDF_VER}-py${PYTHON_VER}" | tee "$GITHUB_OUTPUT" + + - name: Cached ESP-IDF install + id: cache_esp_idf + uses: actions/cache@v4 + with: + path: | + ./esp-idf/ + ~/.espressif/ + !~/.espressif/dist/ + ~/.cache/pip/ + key: esp-idf-${{ steps.idf_ver.outputs.IDF_VER }} + + - name: Install ESP-IDF packages + if: steps.cache_esp_idf.outputs.cache-hit != 'true' + run: source tools/ci.sh && ci_esp32_idf_setup + + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2 + with: + key: esp32-${{ matrix.ci_func }} + + - name: Build ci_${{matrix.ci_func }} + run: source tools/ci.sh && ci_${{ matrix.ci_func }} diff --git a/.github/workflows/ports_esp8266.yml b/.github/workflows/ports_esp8266.yml new file mode 100644 index 0000000000000..5236edf40b959 --- /dev/null +++ b/.github/workflows/ports_esp8266.yml @@ -0,0 +1,28 @@ +name: esp8266 port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/esp8266/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_esp8266_setup && ci_esp8266_path >> $GITHUB_PATH + - name: Build + run: source tools/ci.sh && ci_esp8266_build diff --git a/.github/workflows/ports_mimxrt.yml b/.github/workflows/ports_mimxrt.yml new file mode 100644 index 0000000000000..7743e036ab377 --- /dev/null +++ b/.github/workflows/ports_mimxrt.yml @@ -0,0 +1,33 @@ +name: mimxrt port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/mimxrt/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + defaults: + run: + working-directory: 'micropython repo' # test build with space in path + steps: + - uses: actions/checkout@v4 + with: + path: 'micropython repo' + - name: Install packages + run: source tools/ci.sh && ci_mimxrt_setup + - name: Build + run: source tools/ci.sh && ci_mimxrt_build diff --git a/.github/workflows/ports_nrf.yml b/.github/workflows/ports_nrf.yml new file mode 100644 index 0000000000000..76727c9d1f6bd --- /dev/null +++ b/.github/workflows/ports_nrf.yml @@ -0,0 +1,28 @@ +name: nrf port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/nrf/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_nrf_setup + - name: Build + run: source tools/ci.sh && ci_nrf_build diff --git a/.github/workflows/ports_powerpc.yml b/.github/workflows/ports_powerpc.yml new file mode 100644 index 0000000000000..c41b13e5ddffe --- /dev/null +++ b/.github/workflows/ports_powerpc.yml @@ -0,0 +1,28 @@ +name: powerpc port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/powerpc/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_powerpc_setup + - name: Build + run: source tools/ci.sh && ci_powerpc_build diff --git a/.github/workflows/ports_qemu.yml b/.github/workflows/ports_qemu.yml new file mode 100644 index 0000000000000..57192c43936e5 --- /dev/null +++ b/.github/workflows/ports_qemu.yml @@ -0,0 +1,44 @@ +name: qemu port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/qemu/**' + - 'tests/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_and_test_arm: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_qemu_setup_arm + - name: Build and run test suite + run: source tools/ci.sh && ci_qemu_build_arm + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + build_and_test_rv32: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_qemu_setup_rv32 + - name: Build and run test suite + run: source tools/ci.sh && ci_qemu_build_rv32 + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ports_renesas-ra.yml b/.github/workflows/ports_renesas-ra.yml new file mode 100644 index 0000000000000..b9fa74331dc0b --- /dev/null +++ b/.github/workflows/ports_renesas-ra.yml @@ -0,0 +1,29 @@ +name: renesas-ra port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/renesas-ra/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_renesas_ra_board: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_renesas_ra_setup + - name: Build + run: source tools/ci.sh && ci_renesas_ra_board_build + diff --git a/.github/workflows/ports_rp2.yml b/.github/workflows/ports_rp2.yml new file mode 100644 index 0000000000000..748f38e143893 --- /dev/null +++ b/.github/workflows/ports_rp2.yml @@ -0,0 +1,33 @@ +name: rp2 port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/rp2/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + defaults: + run: + working-directory: 'micropython repo' # test build with space in path + steps: + - uses: actions/checkout@v4 + with: + path: 'micropython repo' + - name: Install packages + run: source tools/ci.sh && ci_rp2_setup + - name: Build + run: source tools/ci.sh && ci_rp2_build diff --git a/.github/workflows/ports_samd.yml b/.github/workflows/ports_samd.yml new file mode 100644 index 0000000000000..5bf1826cd17bb --- /dev/null +++ b/.github/workflows/ports_samd.yml @@ -0,0 +1,28 @@ +name: samd port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/samd/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_samd_setup + - name: Build + run: source tools/ci.sh && ci_samd_build diff --git a/.github/workflows/ports_stm32.yml b/.github/workflows/ports_stm32.yml new file mode 100644 index 0000000000000..8800f145189b8 --- /dev/null +++ b/.github/workflows/ports_stm32.yml @@ -0,0 +1,36 @@ +name: stm32 port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/stm32/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_stm32: + strategy: + fail-fast: false + matrix: + ci_func: # names are functions in ci.sh + - stm32_pyb_build + - stm32_nucleo_build + - stm32_misc_build + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_stm32_setup + - name: Build ci_${{matrix.ci_func }} + run: source tools/ci.sh && ci_${{ matrix.ci_func }} + diff --git a/.github/workflows/ports_unix.yml b/.github/workflows/ports_unix.yml new file mode 100644 index 0000000000000..2547015038e4d --- /dev/null +++ b/.github/workflows/ports_unix.yml @@ -0,0 +1,264 @@ +name: unix port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'examples/**' + - 'mpy-cross/**' + - 'ports/unix/**' + - 'tests/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + minimal: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: source tools/ci.sh && ci_unix_minimal_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_minimal_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + reproducible: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build with reproducible date + run: source tools/ci.sh && ci_unix_minimal_build + env: + SOURCE_DATE_EPOCH: 1234567890 + - name: Check reproducible build date + run: echo | ports/unix/build-minimal/micropython -i | grep 'on 2009-02-13;' + + standard: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: source tools/ci.sh && ci_unix_standard_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_standard_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + standard_v2: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: source tools/ci.sh && ci_unix_standard_v2_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_standard_v2_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + coverage: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_coverage_setup + - name: Build + run: source tools/ci.sh && ci_unix_coverage_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_coverage_run_tests + - name: Test merging .mpy files + run: source tools/ci.sh && ci_unix_coverage_run_mpy_merge_tests + - name: Build native mpy modules + run: source tools/ci.sh && ci_native_mpy_modules_build + - name: Test importing .mpy generated by mpy_ld.py + run: source tools/ci.sh && ci_unix_coverage_run_native_mpy_tests + - name: Run gcov coverage analysis + run: | + (cd ports/unix && gcov -o build-coverage/py ../../py/*.c || true) + (cd ports/unix && gcov -o build-coverage/extmod ../../extmod/*.c || true) + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v5 + with: + fail_ci_if_error: true + verbose: true + token: ${{ secrets.CODECOV_TOKEN }} + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + coverage_32bit: + runs-on: ubuntu-22.04 # use 22.04 to get libffi-dev:i386 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_32bit_setup + - name: Build + run: source tools/ci.sh && ci_unix_coverage_32bit_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_coverage_32bit_run_tests + - name: Build native mpy modules + run: source tools/ci.sh && ci_native_mpy_modules_32bit_build + - name: Test importing .mpy generated by mpy_ld.py + run: source tools/ci.sh && ci_unix_coverage_32bit_run_native_mpy_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + nanbox: + runs-on: ubuntu-22.04 # use 22.04 to get python2, and libffi-dev:i386 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_32bit_setup + - name: Build + run: source tools/ci.sh && ci_unix_nanbox_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_nanbox_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + float: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: source tools/ci.sh && ci_unix_float_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_float_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + stackless_clang: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_clang_setup + - name: Build + run: source tools/ci.sh && ci_unix_stackless_clang_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_stackless_clang_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + float_clang: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_clang_setup + - name: Build + run: source tools/ci.sh && ci_unix_float_clang_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_float_clang_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + settrace: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + # Python 3.12 is the default for ubuntu-24.04, but that has compatibility issues with settrace tests. + # Can remove this step when ubuntu-latest uses a more recent Python 3.x as the default. + with: + python-version: '3.11' + - name: Build + run: source tools/ci.sh && ci_unix_settrace_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_settrace_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + settrace_stackless: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + # Python 3.12 is the default for ubuntu-24.04, but that has compatibility issues with settrace tests. + # Can remove this step when ubuntu-latest uses a more recent Python 3.x as the default. + with: + python-version: '3.11' + - name: Build + run: source tools/ci.sh && ci_unix_settrace_stackless_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_settrace_stackless_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + macos: + runs-on: macos-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.8' + - name: Build + run: source tools/ci.sh && ci_unix_macos_build + - name: Run tests + run: source tools/ci.sh && ci_unix_macos_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + qemu_mips: + # ubuntu-22.04 is needed for older libffi. + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_qemu_mips_setup + - name: Build + run: source tools/ci.sh && ci_unix_qemu_mips_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_qemu_mips_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + qemu_arm: + # ubuntu-22.04 is needed for older libffi. + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_qemu_arm_setup + - name: Build + run: source tools/ci.sh && ci_unix_qemu_arm_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_qemu_arm_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + qemu_riscv64: + # ubuntu-22.04 is needed for older libffi. + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_qemu_riscv64_setup + - name: Build + run: source tools/ci.sh && ci_unix_qemu_riscv64_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_qemu_riscv64_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ports_webassembly.yml b/.github/workflows/ports_webassembly.yml new file mode 100644 index 0000000000000..880f15ab34469 --- /dev/null +++ b/.github/workflows/ports_webassembly.yml @@ -0,0 +1,32 @@ +name: webassembly port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'ports/webassembly/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_webassembly_setup + - name: Build + run: source tools/ci.sh && ci_webassembly_build + - name: Run tests + run: source tools/ci.sh && ci_webassembly_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ports_windows.yml b/.github/workflows/ports_windows.yml new file mode 100644 index 0000000000000..84e018ba15d16 --- /dev/null +++ b/.github/workflows/ports_windows.yml @@ -0,0 +1,151 @@ +name: windows port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'ports/unix/**' + - 'ports/windows/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build-vs: + strategy: + fail-fast: false + matrix: + platform: [x86, x64] + configuration: [Debug, Release] + variant: [dev, standard] + visualstudio: ['2017', '2019', '2022'] + include: + - visualstudio: '2017' + runner: windows-latest + vs_version: '[15, 16)' + - visualstudio: '2019' + runner: windows-2019 + vs_version: '[16, 17)' + - visualstudio: '2022' + runner: windows-2022 + vs_version: '[17, 18)' + # trim down the number of jobs in the matrix + exclude: + - variant: standard + configuration: Debug + - visualstudio: '2019' + configuration: Debug + env: + CI_BUILD_CONFIGURATION: ${{ matrix.configuration }} + runs-on: ${{ matrix.runner }} + steps: + - name: Install Visual Studio 2017 + if: matrix.visualstudio == '2017' + run: | + choco install visualstudio2017buildtools + choco install visualstudio2017-workload-vctools + choco install windows-sdk-8.1 + - uses: microsoft/setup-msbuild@v2 + with: + vs-version: ${{ matrix.vs_version }} + - uses: actions/setup-python@v5 + if: matrix.runner == 'windows-2019' + with: + python-version: '3.9' + - uses: actions/checkout@v4 + - name: Build mpy-cross.exe + run: msbuild mpy-cross\mpy-cross.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} + - name: Update submodules + run: git submodule update --init lib/micropython-lib + - name: Build micropython.exe + run: msbuild ports\windows\micropython.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} -property:PyVariant=${{ matrix.variant }} + - name: Get micropython.exe path + id: get_path + run: | + $exePath="$(msbuild ports\windows\micropython.vcxproj -nologo -v:m -t:ShowTargetPath -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} -property:PyVariant=${{ matrix.variant }})" + echo ("micropython=" + $exePath.Trim()) >> $env:GITHUB_OUTPUT + - name: Run tests + id: test + env: + MICROPY_MICROPYTHON: ${{ steps.get_path.outputs.micropython }} + working-directory: tests + run: python run-tests.py + - name: Print failures + if: failure() && steps.test.conclusion == 'failure' + working-directory: tests + run: python run-tests.py --print-failures + - name: Run mpy tests + id: test_mpy + env: + MICROPY_MICROPYTHON: ${{ steps.get_path.outputs.micropython }} + working-directory: tests + run: python run-tests.py --via-mpy -d basics float micropython + - name: Print mpy failures + if: failure() && steps.test_mpy.conclusion == 'failure' + working-directory: tests + run: python run-tests.py --print-failures + + build-mingw: + strategy: + fail-fast: false + matrix: + variant: [dev, standard] + sys: [mingw32, mingw64] + include: + - sys: mingw32 + env: i686 + - sys: mingw64 + env: x86_64 + runs-on: windows-2022 + env: + CHERE_INVOKING: enabled_from_arguments + defaults: + run: + shell: msys2 {0} + steps: + - uses: actions/setup-python@v5 + # note: can go back to installing mingw-w64-${{ matrix.env }}-python after + # MSYS2 updates to Python >3.12 (due to settrace compatibility issue) + with: + python-version: '3.11' + - uses: msys2/setup-msys2@v2 + with: + msystem: ${{ matrix.sys }} + update: true + install: >- + make + mingw-w64-${{ matrix.env }}-gcc + pkg-config + git + diffutils + path-type: inherit # Remove when setup-python is removed + - uses: actions/checkout@v4 + - name: Build mpy-cross.exe + run: make -C mpy-cross -j2 + - name: Update submodules + run: make -C ports/windows VARIANT=${{ matrix.variant }} submodules + - name: Build micropython.exe + run: make -C ports/windows -j2 VARIANT=${{ matrix.variant }} + - name: Run tests + id: test + run: make -C ports/windows test_full VARIANT=${{ matrix.variant }} + - name: Print failures + if: failure() && steps.test.conclusion == 'failure' + working-directory: tests + run: python run-tests.py --print-failures + + cross-build-on-linux: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_windows_setup + - name: Build + run: source tools/ci.sh && ci_windows_build diff --git a/.github/workflows/ports_zephyr.yml b/.github/workflows/ports_zephyr.yml new file mode 100644 index 0000000000000..eb85af6a36154 --- /dev/null +++ b/.github/workflows/ports_zephyr.yml @@ -0,0 +1,60 @@ +name: zephyr port + +on: + push: + pull_request: + paths: + - '.github/workflows/ports_zephyr.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'ports/zephyr/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: jlumbroso/free-disk-space@main + with: + # Only free up a few things so this step runs quickly. + android: false + dotnet: true + haskell: true + large-packages: false + docker-images: false + swap-storage: false + - uses: actions/checkout@v4 + - id: versions + name: Read Zephyr version + run: source tools/ci.sh && echo "ZEPHYR=$ZEPHYR_VERSION" | tee "$GITHUB_OUTPUT" + - name: Cached Zephyr Workspace + id: cache_workspace + uses: actions/cache@v4 + with: + # note that the Zephyr CI docker image is 15GB. At time of writing + # GitHub caches are limited to 10GB total for a project. So we only + # cache the "workspace" + path: ./zephyrproject + key: zephyr-workspace-${{ steps.versions.outputs.ZEPHYR }} + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2 + with: + key: zephyr + - name: Install packages + run: source tools/ci.sh && ci_zephyr_setup + - name: Install Zephyr + if: steps.cache_workspace.outputs.cache-hit != 'true' + run: source tools/ci.sh && ci_zephyr_install + - name: Build + run: source tools/ci.sh && ci_zephyr_build + - name: Run main test suite + run: source tools/ci.sh && ci_zephyr_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ruff.yml b/.github/workflows/ruff.yml new file mode 100644 index 0000000000000..4c4a2a3162ed6 --- /dev/null +++ b/.github/workflows/ruff.yml @@ -0,0 +1,13 @@ +name: Python code lint and formatting with ruff + +on: [push, pull_request] + +jobs: + ruff: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + # ruff version should be kept in sync with .pre-commit-config.yaml & also micropython-lib + - run: pipx install ruff==0.11.6 + - run: ruff check --output-format=github . + - run: ruff format --diff . diff --git a/.gitignore b/.gitignore index 5e841a89c059d..b5010dfd14db6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,44 +1,26 @@ -# Compiled Sources -################### -*.o -*.a -*.elf -*.bin -*.map -*.hex -*.dis -*.exe - -# Packages -############ - -# Logs and Databases -###################### -*.log - -# VIM Swap Files -###################### -*.swp - -# Build directory -###################### +# This .gitignore file is intended to be minimal. +# +# If you find that you need additional rules, such as IDE temporary +# files, please do so either via a global .gitignore file (registered +# with core.excludesFile), or by adding private repository-specific +# rules to .git/info/exclude. See https://git-scm.com/docs/gitignore +# for more information. + +# Build directories build/ +build-*/ +docs/genrst/ -# Test failure outputs -###################### -tests/*.exp -tests/*.out +# Test failure outputs and intermediate artefacts +tests/results/* +tests/ports/unix/ffi_lib.so # Python cache files -###################### __pycache__/ -*.pyc # Customized Makefile/project overrides -###################### GNUmakefile user.props -# Generated rst files -###################### -genrst/ +# MacOS desktop metadata files +.DS_Store diff --git a/.gitmodules b/.gitmodules index c3f4c55aca2b6..02849ec9bdd11 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,17 +1,76 @@ [submodule "lib/axtls"] path = lib/axtls - url = https://github.com/pfalcon/axtls - branch = micropython + url = https://github.com/micropython/axtls.git [submodule "lib/libffi"] path = lib/libffi - url = https://github.com/atgreen/libffi + url = https://github.com/libffi/libffi [submodule "lib/lwip"] path = lib/lwip - url = https://git.savannah.gnu.org/r/lwip.git + url = https://github.com/lwip-tcpip/lwip.git [submodule "lib/berkeley-db-1.xx"] path = lib/berkeley-db-1.xx - url = https://github.com/pfalcon/berkeley-db-1.xx + url = https://github.com/micropython/berkeley-db-1.xx [submodule "lib/stm32lib"] path = lib/stm32lib url = https://github.com/micropython/stm32lib - branch = work-F4-1.13.1+F7-1.5.0+L4-1.3.0 + branch = work-F0-1.9.0+F4-1.16.0+F7-1.7.0+G0-1.5.1+G4-1.3.0+H7-1.6.0+L0-1.11.2+L1-1.10.3+L4-1.17.0+WB-1.10.0+WL-1.1.0 +[submodule "lib/nrfx"] + path = lib/nrfx + url = https://github.com/NordicSemiconductor/nrfx.git +[submodule "lib/mbedtls"] + path = lib/mbedtls + url = https://github.com/ARMmbed/mbedtls.git +[submodule "lib/asf4"] + path = lib/asf4 + url = https://github.com/adafruit/asf4 + branch = circuitpython +[submodule "lib/tinyusb"] + path = lib/tinyusb + url = https://github.com/hathach/tinyusb +[submodule "lib/mynewt-nimble"] + path = lib/mynewt-nimble + url = https://github.com/micropython/mynewt-nimble.git +[submodule "lib/btstack"] + path = lib/btstack + url = https://github.com/bluekitchen/btstack.git +[submodule "lib/nxp_driver"] + path = lib/nxp_driver + url = https://github.com/hathach/nxp_driver.git +[submodule "lib/libhydrogen"] + path = lib/libhydrogen + url = https://github.com/jedisct1/libhydrogen.git +[submodule "lib/pico-sdk"] + path = lib/pico-sdk + url = https://github.com/raspberrypi/pico-sdk.git +[submodule "lib/fsp"] + path = lib/fsp + url = https://github.com/renesas/fsp.git +[submodule "lib/wiznet"] + path = lib/wiznet5k + url = https://github.com/andrewleech/wiznet_ioLibrary_Driver.git + # Requires https://github.com/Wiznet/ioLibrary_Driver/pull/120 + # url = https://github.com/Wiznet/ioLibrary_Driver.git +[submodule "lib/cyw43-driver"] + path = lib/cyw43-driver + url = https://github.com/georgerobotics/cyw43-driver.git +[submodule "lib/micropython-lib"] + path = lib/micropython-lib + url = https://github.com/micropython/micropython-lib.git +[submodule "lib/protobuf-c"] + path = lib/protobuf-c + url = https://github.com/protobuf-c/protobuf-c.git +[submodule "lib/open-amp"] + path = lib/open-amp + url = https://github.com/OpenAMP/open-amp.git +[submodule "lib/libmetal"] + path = lib/libmetal + url = https://github.com/OpenAMP/libmetal.git +[submodule "lib/arduino-lib"] + path = lib/arduino-lib + url = https://github.com/arduino/arduino-lib-mpy.git +[submodule "lib/alif_ensemble-cmsis-dfp"] + path = lib/alif_ensemble-cmsis-dfp + url = https://github.com/alifsemi/alif_ensemble-cmsis-dfp.git +[submodule "lib/alif-security-toolkit"] + path = lib/alif-security-toolkit + url = https://github.com/micropython/alif-security-toolkit.git diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000000000..ac9785bb59232 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,27 @@ +repos: + - repo: local + hooks: + - id: codeformat + name: MicroPython codeformat.py for changed C files + entry: tools/codeformat.py -v -c -f + language: python + - id: verifygitlog + name: MicroPython git commit message format checker + entry: tools/verifygitlog.py --check-file --ignore-rebase + language: python + verbose: true + stages: [commit-msg] + - repo: https://github.com/charliermarsh/ruff-pre-commit + # Version should be kept in sync with .github/workflows/ruff.yml & also micropython-lib + rev: v0.11.6 + hooks: + - id: ruff + - id: ruff-format + - repo: https://github.com/codespell-project/codespell + # Version should be kept in sync with .github/workflows/codespell.yml + rev: v2.4.1 + hooks: + - id: codespell + name: Spellcheck for changed files (codespell) + additional_dependencies: + - tomli diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index d50a5d2615442..0000000000000 --- a/.travis.yml +++ /dev/null @@ -1,76 +0,0 @@ -sudo: required -dist: trusty -language: c -compiler: - - gcc -cache: - directories: - - "${HOME}/persist" -env: - - MAKEOPTS="-j4" - -before_script: -# Extra CPython versions -# - sudo add-apt-repository -y ppa:fkrull/deadsnakes -# Extra gcc versions -# - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test - - sudo add-apt-repository -y ppa:terry.guo/gcc-arm-embedded - - sudo dpkg --add-architecture i386 - - sudo apt-get update -qq || true - - sudo apt-get install -y python3 gcc-multilib pkg-config libffi-dev libffi-dev:i386 qemu-system gcc-mingw-w64 - - sudo apt-get install -y --force-yes gcc-arm-none-eabi - # For teensy build - - sudo apt-get install realpath - # For coverage testing (upgrade is used to get latest urllib3 version) - - sudo pip install --upgrade cpp-coveralls - - gcc --version - - arm-none-eabi-gcc --version - - python3 --version - -script: - - make ${MAKEOPTS} -C mpy-cross - - make ${MAKEOPTS} -C ports/minimal CROSS=1 build/firmware.bin - - ls -l ports/minimal/build/firmware.bin - - tools/check_code_size.sh - - mkdir -p ${HOME}/persist - # Save new firmware for reference, but only if building a main branch, not a pull request - - 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then cp ports/minimal/build/firmware.bin ${HOME}/persist/; fi' - - make ${MAKEOPTS} -C ports/unix deplibs - - make ${MAKEOPTS} -C ports/unix - - make ${MAKEOPTS} -C ports/unix nanbox - - make ${MAKEOPTS} -C ports/bare-arm - - make ${MAKEOPTS} -C ports/qemu-arm -f Makefile.test test - - make ${MAKEOPTS} -C ports/stm32 - - make ${MAKEOPTS} -C ports/stm32 BOARD=PYBV11 MICROPY_PY_WIZNET5K=5200 MICROPY_PY_CC3K=1 - - make ${MAKEOPTS} -C ports/stm32 BOARD=STM32F769DISC - - make ${MAKEOPTS} -C ports/stm32 BOARD=STM32L476DISC - - make ${MAKEOPTS} -C ports/teensy - - make ${MAKEOPTS} -C ports/cc3200 BTARGET=application BTYPE=release - - make ${MAKEOPTS} -C ports/cc3200 BTARGET=bootloader BTYPE=release - - make ${MAKEOPTS} -C ports/windows CROSS_COMPILE=i686-w64-mingw32- - - # run tests without coverage info - #- (cd tests && MICROPY_CPYTHON3=python3.4 ./run-tests) - #- (cd tests && MICROPY_CPYTHON3=python3.4 ./run-tests --emit native) - - # run tests with coverage info - - make ${MAKEOPTS} -C ports/unix coverage - - (cd tests && MICROPY_CPYTHON3=python3.4 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests) - - (cd tests && MICROPY_CPYTHON3=python3.4 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests -d thread) - - (cd tests && MICROPY_CPYTHON3=python3.4 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests --emit native) - - (cd tests && MICROPY_CPYTHON3=python3.4 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests --via-mpy -d basics float) - - # test when input script comes from stdin - - cat tests/basics/0prelim.py | ports/unix/micropython_coverage | grep -q 'abc' - - # run coveralls coverage analysis (try to, even if some builds/tests failed) - - (cd ports/unix && coveralls --root ../.. --build-root . --gcov $(which gcov) --gcov-options '\-o build-coverage/' --include py --include extmod) - - # run tests on stackless build - - rm -rf ports/unix/build-coverage - - make ${MAKEOPTS} -C ports/unix coverage CFLAGS_EXTRA="-DMICROPY_STACKLESS=1 -DMICROPY_STACKLESS_STRICT=1" - - (cd tests && MICROPY_CPYTHON3=python3.4 MICROPY_MICROPYTHON=../ports/unix/micropython_coverage ./run-tests) - -after_failure: - - (cd tests && for exp in *.exp; do testbase=$(basename $exp .exp); echo -e "\nFAILURE $testbase"; diff -u $testbase.exp $testbase.out; done) - - (grep "FAIL" ports/qemu-arm/build/console.out) diff --git a/ACKNOWLEDGEMENTS b/ACKNOWLEDGEMENTS index 65f731b1fffed..41ed6bf24b5a0 100644 --- a/ACKNOWLEDGEMENTS +++ b/ACKNOWLEDGEMENTS @@ -762,7 +762,6 @@ today. The names appear in order of pledging. 1642 Udine 1643 Simon Critchley 1644 Sven Haiges, Germany - 1645 Yi Qing Sim 1646 "silicium" ("silicium_one", if "silicium" is busy) 1648 Andy O'Malia, @andyomalia 1650 RedCamelApps.com diff --git a/CODECONVENTIONS.md b/CODECONVENTIONS.md index 982b958313166..d3f71cb083deb 100644 --- a/CODECONVENTIONS.md +++ b/CODECONVENTIONS.md @@ -11,7 +11,7 @@ It's also ok to drop file extensions. Besides prefix, first line of a commit message should describe a change clearly and to the point, and be a grammatical sentence with -final full stop. First line should fit within 78 characters. Examples +final full stop. First line must fit within 72 characters. Examples of good first line of commit messages: py/objstr: Add splitlines() method. @@ -19,16 +19,17 @@ of good first line of commit messages: docs/machine: Fix typo in reset() description. ports: Switch to use lib/foo instead of duplicated code. -After the first line, add an empty line and in following lines describe -a change in a detail, if needed. Any change beyond 5 lines would likely -require such detailed description. +After the first line add an empty line and in the following lines describe +the change in a detail, if needed, with lines fitting within 75 characters +(with an exception for long items like URLs which cannot be broken). Any +change beyond 5 lines would likely require such detailed description. To get good practical examples of good commits and their messages, browse the `git log` of the project. -MicroPython doesn't require explicit sign-off for patches ("Signed-off-by" -lines and similar). Instead, the commit message, and your name and email -address on it construes your sign-off of the following: +When committing you must sign-off your commit by adding "Signed-off-by:" +line(s) at the end of the commit message, e.g. using `git commit -s`. You +are then certifying and signing off against the following: * That you wrote the change yourself, or took it from a project with a compatible license (in the latter case the commit message, and possibly @@ -42,19 +43,153 @@ address on it construes your sign-off of the following: copyright for your changes (for smaller changes, the commit message conveys your copyright; if you make significant changes to a particular source module, you're welcome to add your name to the file header). -* Your signature for all of the above, which is the 'Author' line in - the commit message, and which should include your full real name and - a valid and active email address by which you can be contacted in the - foreseeable future. +* Your contribution including commit message will be publicly and + indefinitely available for anyone to access, including redistribution + under the terms of the project's license. +* Your signature for all of the above, which is the "Signed-off-by" line, + includes your full real name and a valid and active email address by + which you can be contacted in the foreseeable future. + +Code auto-formatting +==================== + +Both C and Python code formatting are controlled for consistency across the +MicroPython codebase. C code is formatted using the `tools/codeformat.py` +script which uses [uncrustify](https://github.com/uncrustify/uncrustify). +Python code is linted and formatted using +[ruff & ruff format](https://github.com/astral-sh/ruff). +After making changes, and before committing, run `tools/codeformat.py` to +reformat your C code and `ruff format` for any Python code. Without +arguments this tool will reformat all source code (and may take some time +to run). Otherwise pass as arguments to the tool the files that changed, +and it will only reformat those. + +uncrustify +========== + +Only [uncrustify](https://github.com/uncrustify/uncrustify) v0.71 or v0.72 can +be used for MicroPython. Different uncrustify versions produce slightly +different formatting, and the configuration file formats are often +incompatible. v0.73 or newer *will not work*. + +Depending on your operating system version, it may be possible to install a pre-compiled +uncrustify version: + +Ubuntu, Debian +-------------- + +Ubuntu versions 21.10 or 22.04LTS, and Debian versions bullseye or bookworm all +include v0.72 so can be installed directly: + +``` +$ apt install uncrustify +``` + +Arch Linux +---------- + +The current Arch uncrustify version is too new. There is an [old Arch package +for v0.72](https://archive.archlinux.org/packages/u/uncrustify/) that can be +installed from the Arch Linux archive ([more +information](https://wiki.archlinux.org/title/Downgrading_packages#Arch_Linux_Archive)). Use +the [IgnorePkg feature](https://wiki.archlinux.org/title/Pacman#Skip_package_from_being_upgraded) +to prevent it re-updating. + +Brew +---- + +This command may work, please raise a new Issue if it doesn't: + +``` +curl -L https://github.com/Homebrew/homebrew-core/raw/2b07d8192623365078a8b855a164ebcdf81494a6/Formula/uncrustify.rb > uncrustify.rb && brew install uncrustify.rb && rm uncrustify.rb +``` + +Code spell checking +=================== + +Code spell checking is done using [codespell](https://github.com/codespell-project/codespell#codespell) +and runs in a GitHub action in CI. Codespell is configured via `pyproject.toml` +to avoid false positives. It is recommended run codespell before submitting a +PR. To simplify this, codespell is configured as a pre-commit hook and will be +installed if you run `pre-commit install` (see below). + +If you want to install and run codespell manually, you can do so by running: + +``` +$ pip install codespell tomli +$ codespell +``` + +Automatic Pre-Commit Hooks +========================== + +To have code formatting and commit message conventions automatically checked, +a configuration file is provided for the [pre-commit](https://pre-commit.com/) +tool. + +First install `pre-commit`, either from your system package manager or via +`pip`. When installing `pre-commit` via pip, it is recommended to use a +virtual environment. Other sources, such as Brew are also available, see +[the docs](https://pre-commit.com/index.html#install) for details. + +``` +$ apt install pre-commit # Ubuntu, Debian +$ pacman -Sy python-precommit # Arch Linux +$ brew install pre-commit # Brew +$ pip install pre-commit # PyPI +``` + +Next, install [uncrustify (see above)](#uncrustify). Other dependencies are managed by +pre-commit automatically, but uncrustify needs to be installed and available on +the PATH. + +Then, inside the MicroPython repository, register the git hooks for pre-commit +by running: + +``` +$ pre-commit install --hook-type pre-commit --hook-type commit-msg +``` + +pre-commit will now automatically run during `git commit` for both code and +commit message formatting. + +The same formatting checks will be run by CI for any Pull Request submitted to +MicroPython. Pre-commit allows you to see any failure more quickly, and in many +cases will automatically correct it in your local working copy. + +To unregister `pre-commit` from your MicroPython repository, run: + +``` +$ pre-commit uninstall --hook-type pre-commit --hook-type commit-msg +``` + +Tips: + +* To skip pre-commit checks on a single commit, use `git commit -n` (for + `--no-verify`). +* To ignore the pre-commit message format check temporarily, start the commit + message subject line with "WIP" (for "Work In Progress"). + +Running pre-commit manually +=========================== + +Once pre-commit is installed as per the previous section it can be manually +run against the MicroPython python codebase to update file formatting on +demand, with either: +* `pre-commit run --all-files` to fix all files in the MicroPython codebase +* `pre-commit run --file ./path/to/my/file` to fix just one file +* `pre-commit run --file ./path/to/my/folder/*` to fix just one folder Python code conventions ======================= -Python code follows [PEP 8](http://legacy.python.org/dev/peps/pep-0008/). +Python code follows [PEP 8](https://legacy.python.org/dev/peps/pep-0008/) and +is auto-formatted using [ruff format](https://docs.astral.sh/ruff/formatter) +with a line-length of 99 characters. Naming conventions: - Module names are short and all lowercase; eg pyb, stm. -- Class names are CamelCase, with abreviations all uppercase; eg I2C, not +- Class names are CamelCase, with abbreviations all uppercase; eg I2C, not I2c. - Function and method names are all lowercase with words separated by a single underscore as necessary to improve readability; eg mem_read. @@ -64,16 +199,28 @@ Naming conventions: C code conventions ================== -When writing new C code, please adhere to the following conventions. +C code is auto-formatted using [uncrustify](https://github.com/uncrustify/uncrustify) +and the corresponding configuration file `tools/uncrustify.cfg`, with a few +minor fix-ups applied by `tools/codeformat.py`. When writing new C code please +adhere to the existing style and use `tools/codeformat.py` to check any changes. +The main conventions, and things not enforceable via the auto-formatter, are +described below. + +As the MicroPython code base is over ten years old, not every source file +conforms fully to these conventions. If making small changes to existing code, +then it's usually acceptable to follow the existing code's style. New code or +major changes should follow the conventions described here. + +## White space -White space: - Expand tabs to 4 spaces. - Don't leave trailing whitespace at the end of a line. - For control blocks (if, for, while), put 1 space between the keyword and the opening parenthesis. - Put 1 space after a comma, and 1 space around operators. -Braces: +## Braces + - Use braces for all blocks, even no-line and single-line pieces of code. - Put opening braces on the end of the line it belongs to, not on @@ -81,18 +228,43 @@ Braces: - For else-statements, put the else on the same line as the previous closing brace. -Header files: +## Header files + - Header files should be protected from multiple inclusion with #if directives. See an existing header for naming convention. -Names: +## Names + - Use underscore_case, not camelCase for all names. - Use CAPS_WITH_UNDERSCORE for enums and macros. - When defining a type use underscore_case and put '_t' after it. -Integer types: MicroPython runs on 16, 32, and 64 bit machines, so it's -important to use the correctly-sized (and signed) integer types. The -general guidelines are: +### Public names (declared in headers) + +- MicroPython-specific names (especially any declared in `py/` and `extmod/` + directories) should generally start with `mp_` or `MP_`. +- Functions and variables declared in a header should generally share a longer + common prefix. Usually the prefix matches the file name (i.e. items defined in + `py/obj.c` are declared in `py/obj.h` and should be prefixed `mp_obj_`). There + are exceptions, for example where one header file contains declarations + implemented in multiple source files for expediency. + +### Private names (specific to a single .c file) + +- For static functions and variables exposed to Python (i.e. a static C function + that is wrapped in `MP_DEFINE_CONST_FUN_...` and attached to a module), use + the file-level shared common prefix, i.e. name them as if the function or + variable was not static. +- Other static definitions in source files (i.e. functions or variables defined + in a .c file that are only used within that .c file) don't need any prefix + (specifically: no `s_` or `_` prefix, and generally avoid adding the + file-level common prefix). + +## Integer types + +MicroPython runs on 16, 32, and 64 bit machines, so it's important to use the +correctly-sized (and signed) integer types. The general guidelines are: + - For most cases use mp_int_t for signed and mp_uint_t for unsigned integer values. These are guaranteed to be machine-word sized and therefore big enough to hold the value from a MicroPython small-int @@ -101,11 +273,13 @@ general guidelines are: - You can use int/uint, but remember that they may be 16-bits wide. - If in doubt, use mp_int_t/mp_uint_t. -Comments: +## Comments + - Be concise and only write comments for things that are not obvious. - Use `// ` prefix, NOT `/* ... */`. No extra fluff. -Memory allocation: +## Memory allocation + - Use m_new, m_renew, m_del (and friends) to allocate and free heap memory. These macros are defined in py/misc.h. @@ -125,7 +299,7 @@ Braces, spaces, names and comments: foo(x + TO_ADD, some_value - 1); } - for (int my_counter = 0; my_counter < x; my_counter++) { + for (int my_counter = 0; my_counter < x; ++my_counter) { } } @@ -140,7 +314,7 @@ Documentation conventions ========================= MicroPython generally follows CPython in documentation process and -conventions. reStructuredText syntax is used for the documention. +conventions. reStructuredText syntax is used for the documentation. Specific conventions/suggestions: diff --git a/CODEOFCONDUCT.md b/CODEOFCONDUCT.md new file mode 100644 index 0000000000000..07cf87713bd86 --- /dev/null +++ b/CODEOFCONDUCT.md @@ -0,0 +1,53 @@ +MicroPython Code of Conduct +=========================== + +The MicroPython community is made up of members from around the globe with a +diverse set of skills, personalities, and experiences. It is through these +differences that our community experiences great successes and continued growth. +When you're working with members of the community, this Code of Conduct will +help steer your interactions and keep MicroPython a positive, successful, and +growing community. + +Members of the MicroPython community are open, considerate, and respectful. +Behaviours that reinforce these values contribute to a positive environment, and +include: acknowledging time and effort, being respectful of differing viewpoints +and experiences, gracefully accepting constructive criticism, and using +welcoming and inclusive language. + +Every member of our community has the right to have their identity respected. +The MicroPython community is dedicated to providing a positive experience for +everyone, regardless of age, gender identity and expression, sexual orientation, +disability, physical appearance, body size, ethnicity, nationality, race, or +religion (or lack thereof), education, or socio-economic status. + +Unacceptable behaviour includes: harassment, trolling, deliberate intimidation, +violent threats or language directed against another person; insults, put downs, +or jokes that are based upon stereotypes, that are exclusionary, or that hold +others up for ridicule; unwelcome sexual attention or advances; sustained +disruption of community discussions; publishing others' private information +without explicit permission; and other conduct that is inappropriate for a +professional audience including people of many different backgrounds. + +This code of conduct covers all online and offline presence related to the +MicroPython project, including GitHub and the forum. If a participant engages +in behaviour that violates this code of conduct, the MicroPython team may take +action as they deem appropriate, including warning the offender or expulsion +from the community. Community members asked to stop any inappropriate behaviour +are expected to comply immediately. + +Thank you for helping make this a welcoming, friendly community for everyone. + +If you believe that someone is violating the code of conduct, or have any other +concerns, please contact a member of the MicroPython team by emailing +contact@micropython.org. + +License +------- + +This Code of Conduct is licensed under the Creative Commons +Attribution-ShareAlike 3.0 Unported License. + +Attributions +------------ + +Based on the Python code of conduct found at https://www.python.org/psf/conduct/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 06f970607b9bb..73004ac5181ab 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,6 +3,6 @@ make sure that you are acquainted with Contributor Guidelines: https://github.com/micropython/micropython/wiki/ContributorGuidelines -and Code Conventions: +as well as the Code Conventions, which includes details of how to commit: https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md diff --git a/LICENSE b/LICENSE index e3474e33dd81a..929a2e97de7bf 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2013, 2014 Damien P. George +Copyright (c) 2013-2025 Damien P. George Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -19,3 +19,70 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +Unless specified otherwise (see below), the above license and copyright applies +to all files in this repository. + +Individual files may include additional copyright holders. + +The various ports of MicroPython may include third-party software that is +licensed under different terms. These licenses are summarised in the tree +below, please refer to these files and directories for further license and +copyright information. Note that (L)GPL-licensed code listed below is only +used during the build process and is not part of the compiled source code. + +/ (MIT) + /drivers + /cc3100 (BSD-3-clause) + /lib + /asf4 (Apache-2.0) + /axtls (BSD-3-clause) + /config + /scripts + /config (GPL-2.0-or-later) + /Rules.mak (GPL-2.0) + /berkeley-db-1xx (BSD-4-clause) + /btstack (See btstack/LICENSE) + /cmsis (BSD-3-clause) + /crypto-algorithms (NONE) + /libhydrogen (ISC) + /libmetal (BSD-3-clause) + /littlefs (BSD-3-clause) + /lwip (BSD-3-clause) + /mynewt-nimble (Apache-2.0) + /nrfx (BSD-3-clause) + /nxp_driver (BSD-3-Clause) + /oofatfs (BSD-1-clause) + /open-amp (BSD-3-clause) + /pico-sdk (BSD-3-clause) + /re15 (BSD-3-clause) + /stm32lib (BSD-3-clause) + /tinyusb (MIT) + /uzlib (Zlib) + /wiznet5k (MIT) + /logo (uses OFL-1.1) + /ports + /cc3200 + /hal (BSD-3-clause) + /simplelink (BSD-3-clause) + /FreeRTOS (GPL-2.0 with FreeRTOS exception) + /esp32 + /ppp_set_auth.* (Apache-2.0) + /rp2 + /mutex_extra.c (BSD-3-clause) + /clocks_extra.c (BSD-3-clause) + /stm32 + /usbd*.c (MCD-ST Liberty SW License Agreement V2) + /stm32_it.* (MIT + BSD-3-clause) + /system_stm32*.c (MIT + BSD-3-clause) + /boards + /startup_stm32*.s (BSD-3-clause) + /*/stm32*.h (BSD-3-clause) + /usbdev (MCD-ST Liberty SW License Agreement V2) + /usbhost (MCD-ST Liberty SW License Agreement V2) + /zephyr + /src (Apache-2.0) + /tools + /dfu.py (LGPL-3.0-only) diff --git a/README.md b/README.md index 44d062e8771c2..c78a2384604fe 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -[![Build Status](https://travis-ci.org/micropython/micropython.png?branch=master)](https://travis-ci.org/micropython/micropython) [![Coverage Status](https://coveralls.io/repos/micropython/micropython/badge.png?branch=master)](https://coveralls.io/r/micropython/micropython?branch=master) +[![Unix CI badge](https://github.com/micropython/micropython/actions/workflows/ports_unix.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![STM32 CI badge](https://github.com/micropython/micropython/actions/workflows/ports_stm32.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![Docs CI badge](https://github.com/micropython/micropython/actions/workflows/docs.yml/badge.svg)](https://docs.micropython.org/) [![codecov](https://codecov.io/gh/micropython/micropython/branch/master/graph/badge.svg?token=I92PfD05sD)](https://codecov.io/gh/micropython/micropython) The MicroPython project ======================= @@ -15,160 +15,133 @@ code-base, including project-wide name changes and API changes. MicroPython implements the entire Python 3.4 syntax (including exceptions, `with`, `yield from`, etc., and additionally `async`/`await` keywords from -Python 3.5). The following core datatypes are provided: `str` (including -basic Unicode support), `bytes`, `bytearray`, `tuple`, `list`, `dict`, `set`, -`frozenset`, `array.array`, `collections.namedtuple`, classes and instances. -Builtin modules include `sys`, `time`, and `struct`, etc. Select ports have -support for `_thread` module (multithreading). Note that only a subset of -Python 3 functionality is implemented for the data types and modules. - -MicroPython can execute scripts in textual source form or from precompiled -bytecode, in both cases either from an on-device filesystem or "frozen" into -the MicroPython executable. - -See the repository http://github.com/micropython/pyboard for the MicroPython -board (PyBoard), the officially supported reference electronic circuit board. - -Major components in this repository: -- py/ -- the core Python implementation, including compiler, runtime, and - core library. -- mpy-cross/ -- the MicroPython cross-compiler which is used to turn scripts - into precompiled bytecode. -- ports/unix/ -- a version of MicroPython that runs on Unix. -- ports/stm32/ -- a version of MicroPython that runs on the PyBoard and similar - STM32 boards (using ST's Cube HAL drivers). -- ports/minimal/ -- a minimal MicroPython port. Start with this if you want - to port MicroPython to another microcontroller. -- tests/ -- test framework and test scripts. -- docs/ -- user documentation in Sphinx reStructuredText format. Rendered - HTML documentation is available at http://docs.micropython.org (be sure - to select needed board/port at the bottom left corner). - -Additional components: -- ports/bare-arm/ -- a bare minimum version of MicroPython for ARM MCUs. Used - mostly to control code size. -- ports/teensy/ -- a version of MicroPython that runs on the Teensy 3.1 - (preliminary but functional). -- ports/pic16bit/ -- a version of MicroPython for 16-bit PIC microcontrollers. -- ports/cc3200/ -- a version of MicroPython that runs on the CC3200 from TI. -- ports/esp8266/ -- an experimental port for ESP8266 WiFi modules. -- extmod/ -- additional (non-core) modules implemented in C. -- tools/ -- various tools, including the pyboard.py module. -- examples/ -- a few example Python scripts. - -The subdirectories above may include READMEs with additional info. +Python 3.5 and some select features from later versions). The following core +datatypes are provided: `str`(including basic Unicode support), `bytes`, +`bytearray`, `tuple`, `list`, `dict`, `set`, `frozenset`, `array.array`, +`collections.namedtuple`, classes and instances. Builtin modules include +`os`, `sys`, `time`, `re`, and `struct`, etc. Some ports have support for +`_thread` module (multithreading), `socket` and `ssl` for networking, and +`asyncio`. Note that only a subset of Python 3 functionality is implemented +for the data types and modules. + +MicroPython can execute scripts in textual source form (.py files) or from +precompiled bytecode (.mpy files), in both cases either from an on-device +filesystem or "frozen" into the MicroPython executable. + +MicroPython also provides a set of MicroPython-specific modules to access +hardware-specific functionality and peripherals such as GPIO, Timers, ADC, +DAC, PWM, SPI, I2C, CAN, Bluetooth, and USB. + +Getting started +--------------- + +See the [online documentation](https://docs.micropython.org/) for the API +reference and information about using MicroPython and information about how +it is implemented. + +We use [GitHub Discussions](https://github.com/micropython/micropython/discussions) +as our forum, and [Discord](https://discord.gg/RB8HZSAExQ) for chat. These +are great places to ask questions and advice from the community or to discuss your +MicroPython-based projects. + +For bugs and feature requests, please [raise an issue](https://github.com/micropython/micropython/issues/new/choose) +and follow the templates there. + +For information about the [MicroPython pyboard](https://store.micropython.org/pyb-features), +the officially supported board from the +[original Kickstarter campaign](https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers), +see the [schematics and pinouts](http://github.com/micropython/pyboard) and +[documentation](https://docs.micropython.org/en/latest/pyboard/quickref.html). -"make" is used to build the components, or "gmake" on BSD-based systems. -You will also need bash, gcc, and Python (at least 2.7 or 3.3). +Contributing +------------ -The Unix version ----------------- +MicroPython is an open-source project and welcomes contributions. To be +productive, please be sure to follow the +[Contributors' Guidelines](https://github.com/micropython/micropython/wiki/ContributorGuidelines) +and the [Code Conventions](https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md). +Note that MicroPython is licenced under the MIT license, and all contributions +should follow this license. -The "unix" port requires a standard Unix environment with gcc and GNU make. -x86 and x64 architectures are supported (i.e. x86 32- and 64-bit), as well -as ARM and MIPS. Making full-featured port to another architecture requires -writing some assembly code for the exception handling and garbage collection. -Alternatively, fallback implementation based on setjmp/longjmp can be used. +About this repository +--------------------- -To build (see section below for required dependencies): +This repository contains the following components: +- [py/](py/) -- the core Python implementation, including compiler, runtime, and + core library. +- [mpy-cross/](mpy-cross/) -- the MicroPython cross-compiler which is used to turn scripts + into precompiled bytecode. +- [ports/](ports/) -- platform-specific code for the various ports and architectures that MicroPython runs on. +- [lib/](lib/) -- submodules for external dependencies. +- [tests/](tests/) -- test framework and test scripts. +- [docs/](docs/) -- user documentation in Sphinx reStructuredText format. This is used to generate the [online documentation](http://docs.micropython.org). +- [extmod/](extmod/) -- additional (non-core) modules implemented in C. +- [tools/](tools/) -- various tools, including the pyboard.py module. +- [examples/](examples/) -- a few example Python scripts. - $ git submodule update --init - $ cd ports/unix - $ make axtls +"make" is used to build the components, or "gmake" on BSD-based systems. +You will also need bash, gcc, and Python 3.3+ available as the command `python3` +(if your system only has Python 2.7 then invoke make with the additional option +`PYTHON=python2`). Some ports (rp2 and esp32) additionally use CMake. + +Supported platforms & architectures +----------------------------------- + +MicroPython runs on a wide range of microcontrollers, as well as on Unix-like +(including Linux, BSD, macOS, WSL) and Windows systems. + +Microcontroller targets can be as small as 256kiB flash + 16kiB RAM, although +devices with at least 512kiB flash + 128kiB RAM allow a much more +full-featured experience. + +The [Unix](ports/unix) and [Windows](ports/windows) ports allow both +development and testing of MicroPython itself, as well as providing +lightweight alternative to CPython on these platforms (in particular on +embedded Linux systems). + +The ["minimal"](ports/minimal) port provides an example of a very basic +MicroPython port and can be compiled as both a standalone Linux binary as +well as for ARM Cortex M4. Start with this if you want to port MicroPython to +another microcontroller. Additionally the ["bare-arm"](ports/bare-arm) port +is an example of the absolute minimum configuration, and is used to keep +track of the code size of the core runtime and VM. + +In addition, the following ports are provided in this repository: + - [cc3200](ports/cc3200) -- Texas Instruments CC3200 (including PyCom WiPy). + - [esp32](ports/esp32) -- Espressif ESP32 SoC (including ESP32S2, ESP32S3, ESP32C3, ESP32C6). + - [esp8266](ports/esp8266) -- Espressif ESP8266 SoC. + - [mimxrt](ports/mimxrt) -- NXP m.iMX RT (including Teensy 4.x). + - [nrf](ports/nrf) -- Nordic Semiconductor nRF51 and nRF52. + - [pic16bit](ports/pic16bit) -- Microchip PIC 16-bit. + - [powerpc](ports/powerpc) -- IBM PowerPC (including Microwatt) + - [qemu](ports/qemu) -- QEMU-based emulated target (for testing) + - [renesas-ra](ports/renesas-ra) -- Renesas RA family. + - [rp2](ports/rp2) -- Raspberry Pi RP2040 (including Pico and Pico W). + - [samd](ports/samd) -- Microchip (formerly Atmel) SAMD21 and SAMD51. + - [stm32](ports/stm32) -- STMicroelectronics STM32 family (including F0, F4, F7, G0, G4, H7, L0, L4, WB) + - [webassembly](ports/webassembly) -- Emscripten port targeting browsers and NodeJS. + - [zephyr](ports/zephyr) -- Zephyr RTOS. + +The MicroPython cross-compiler, mpy-cross +----------------------------------------- + +Most ports require the [MicroPython cross-compiler](mpy-cross) to be built +first. This program, called mpy-cross, is used to pre-compile Python scripts +to .mpy files which can then be included (frozen) into the +firmware/executable for a port. To build mpy-cross use: + + $ cd mpy-cross $ make -Then to give it a try: - - $ ./micropython - >>> list(5 * x + y for x in range(10) for y in [4, 2, 1]) - -Use `CTRL-D` (i.e. EOF) to exit the shell. -Learn about command-line options (in particular, how to increase heap size -which may be needed for larger applications): - - $ ./micropython --help - -Run complete testsuite: - - $ make test - -Unix version comes with a builtin package manager called upip, e.g.: - - $ ./micropython -m upip install micropython-pystone - $ ./micropython -m pystone - -Browse available modules on -[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython). -Standard library modules come from -[micropython-lib](https://github.com/micropython/micropython-lib) project. - External dependencies --------------------- -Building MicroPython ports may require some dependencies installed. +The core MicroPython VM and runtime has no external dependencies, but a given +port might depend on third-party drivers or vendor HALs. This repository +includes [several submodules](lib/) linking to these external dependencies. +Before compiling a given port, use -For Unix port, `libffi` library and `pkg-config` tool are required. On -Debian/Ubuntu/Mint derivative Linux distros, install `build-essential` -(includes toolchain and make), `libffi-dev`, and `pkg-config` packages. + $ cd ports/name + $ make submodules -Other dependencies can be built together with MicroPython. This may -be required to enable extra features or capabilities, and in recent -versions of MicroPython, these may be enabled by default. To build -these additional dependencies, first fetch git submodules for them: - - $ git submodule update --init - -Use the same command to get the latest versions of dependencies, as -they are updated from time to time. After that, in the port directory -(e.g. `ports/unix/`), execute: - - $ make deplibs - -This will build all available dependencies (regardless whether they -are used or not). If you intend to build MicroPython with additional -options (like cross-compiling), the same set of options should be passed -to `make deplibs`. To actually enable/disable use of dependencies, edit -`ports/unix/mpconfigport.mk` file, which has inline descriptions of the options. -For example, to build SSL module (required for `upip` tool described above, -and so enabled by dfeault), `MICROPY_PY_USSL` should be set to 1. - -For some ports, building required dependences is transparent, and happens -automatically. They still need to be fetched with the git submodule command -above. - -The STM32 version ------------------ - -The "stm32" port requires an ARM compiler, arm-none-eabi-gcc, and associated -bin-utils. For those using Arch Linux, you need arm-none-eabi-binutils, -arm-none-eabi-gcc and arm-none-eabi-newlib packages. Otherwise, try here: -https://launchpad.net/gcc-arm-embedded - -To build: - - $ git submodule update --init - $ cd ports/stm32 - $ make - -You then need to get your board into DFU mode. On the pyboard, connect the -3V3 pin to the P1/DFU pin with a wire (on PYBv1.0 they are next to each other -on the bottom left of the board, second row from the bottom). - -Then to flash the code via USB DFU to your device: - - $ make deploy - -This will use the included `tools/pydfu.py` script. If flashing the firmware -does not work it may be because you don't have the correct permissions, and -need to use `sudo make deploy`. -See the README.md file in the ports/stm32/ directory for further details. - -Contributing ------------- - -MicroPython is an open-source project and welcomes contributions. To be -productive, please be sure to follow the -[Contributors' Guidelines](https://github.com/micropython/micropython/wiki/ContributorGuidelines) -and the [Code Conventions](https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md). -Note that MicroPython is licenced under the MIT license, and all contributions -should follow this license. +to ensure that all required submodules are initialised. diff --git a/docs/Makefile b/docs/Makefile index e9c128e900788..766a669a500dc 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -3,7 +3,7 @@ # You can set these variables from the command line. PYTHON = python3 -SPHINXOPTS = +SPHINXOPTS = -W --keep-going -j auto SPHINXBUILD = sphinx-build PAPER = BUILDDIR = build/$(MICROPY_PORT) diff --git a/docs/README.md b/docs/README.md index faf386710a0d5..892726ba17ffe 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,20 +21,33 @@ preferably in a virtualenv: In `micropython/docs`, build the docs: - make MICROPY_PORT= html + make html -Where `` can be `unix`, `pyboard`, `wipy` or `esp8266`. +You'll find the index page at `micropython/docs/build/html/index.html`. -You'll find the index page at `micropython/docs/build//html/index.html`. +Having readthedocs.org build the documentation +---------------------------------------------- + +If you would like to have docs for forks/branches hosted on GitHub, GitLab or +BitBucket an alternative to building the docs locally is to sign up for a free +https://readthedocs.org account. The rough steps to follow are: +1. sign-up for an account, unless you already have one +2. in your account settings: add GitHub as a connected service (assuming +you have forked this repo on github) +3. in your account projects: import your forked/cloned micropython repository +into readthedocs +4. in the project's versions: add the branches you are developing on or +for which you'd like readthedocs to auto-generate docs whenever you +push a change PDF manual generation --------------------- This can be achieved with: - make MICROPY_PORT= latexpdf + make latexpdf -but require rather complete install of LaTeX with various extensions. On -Debian/Ubuntu, try (500MB+ download): +but requires a rather complete install of LaTeX with various extensions. On +Debian/Ubuntu, try (1GB+ download): - apt-get install texlive-latex-recommended texlive-latex-extra + apt install texlive-latex-recommended texlive-latex-extra texlive-xetex texlive-fonts-extra cm-super xindy diff --git a/docs/conf.py b/docs/conf.py index f9c3ecdb7acdf..eb61487582024 100755 --- a/docs/conf.py +++ b/docs/conf.py @@ -19,78 +19,59 @@ # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath('.')) - -# Work out the port to generate the docs for -from collections import OrderedDict -micropy_port = os.getenv('MICROPY_PORT') or 'pyboard' -tags.add('port_' + micropy_port) -ports = OrderedDict(( - ('unix', 'unix'), - ('pyboard', 'the pyboard'), - ('wipy', 'the WiPy'), - ('esp8266', 'the ESP8266'), -)) +sys.path.insert(0, os.path.abspath(".")) + +# The MICROPY_VERSION env var should be "vX.Y.Z" (or unset). +micropy_version = os.getenv("MICROPY_VERSION") or "latest" +micropy_all_versions = (os.getenv("MICROPY_ALL_VERSIONS") or "latest").split(",") +url_pattern = "%s/en/%%s" % (os.getenv("MICROPY_URL_PREFIX") or "/",) # The members of the html_context dict are available inside topindex.html -micropy_version = os.getenv('MICROPY_VERSION') or 'latest' -micropy_all_versions = (os.getenv('MICROPY_ALL_VERSIONS') or 'latest').split(',') -url_pattern = '%s/en/%%s/%%s' % (os.getenv('MICROPY_URL_PREFIX') or '/',) html_context = { - 'port':micropy_port, - 'port_name':ports[micropy_port], - 'port_version':micropy_version, - 'all_ports':[ - (port_id, url_pattern % (micropy_version, port_id)) - for port_id, port_name in ports.items() - ], - 'all_versions':[ - (ver, url_pattern % (ver, micropy_port)) - for ver in micropy_all_versions - ], - 'downloads':[ - ('PDF', url_pattern % (micropy_version, 'micropython-%s.pdf' % micropy_port)), + "cur_version": micropy_version, + "all_versions": [(ver, url_pattern % ver) for ver in micropy_all_versions], + "downloads": [ + ("PDF", url_pattern % micropy_version + "/micropython-docs.pdf"), ], + "is_release": micropy_version != "latest", } +# Authors used in various parts of the documentation. +micropy_authors = "MicroPython authors and contributors" -# Specify a custom master document based on the port name -master_doc = micropy_port + '_' + 'index' # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' +# needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.doctest', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', - 'sphinx_selective_exclude.modindex_exclude', - 'sphinx_selective_exclude.eager_only', - 'sphinx_selective_exclude.search_auto_exclude', + "sphinx.ext.autodoc", + "sphinx.ext.doctest", + "sphinx.ext.intersphinx", + "sphinx.ext.todo", + "sphinx.ext.coverage", + "sphinxcontrib.jquery", ] # Add any paths that contain templates here, relative to this directory. -templates_path = ['templates'] +templates_path = ["templates"] # The suffix of source filenames. -source_suffix = '.rst' +source_suffix = ".rst" # The encoding of source files. -#source_encoding = 'utf-8-sig' +# source_encoding = 'utf-8-sig' # The master toctree document. -#master_doc = 'index' +master_doc = "index" # General information about the project. -project = 'MicroPython' -copyright = '2014-2017, Damien P. George, Paul Sokolovsky, and contributors' +project = "MicroPython" +copyright = "- The MicroPython Documentation is Copyright © 2014-2025, " + micropy_authors # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -98,45 +79,45 @@ # # We don't follow "The short X.Y version" vs "The full version, including alpha/beta/rc tags" # breakdown, so use the same version identifier for both to avoid confusion. -version = release = '1.9.3' +version = release = micropy_version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. -#language = None +# language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: -#today = '' +# today = '' # Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' +# today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = ['build', '.venv'] +exclude_patterns = ["build", ".venv"] # The reST default role (used for this markup: `text`) to use for all # documents. -default_role = 'any' +default_role = "any" # If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True +# add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). -#add_module_names = True +# add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. -#show_authors = False +# show_authors = False # The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' +pygments_style = "sphinx" # A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] +# modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False +# keep_warnings = False # Global include files. Sphinx docs suggest using rst_epilog in preference # of rst_prolog, so we follow. Absolute paths below mean "from the base @@ -148,153 +129,168 @@ # -- Options for HTML output ---------------------------------------------- # on_rtd is whether we are on readthedocs.org -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' +on_rtd = os.environ.get("READTHEDOCS", None) == "True" if not on_rtd: # only import and set the theme if we're building docs locally try: import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path(), '.'] + + html_theme = "sphinx_rtd_theme" + html_theme_path = [sphinx_rtd_theme.get_html_theme_path(), "."] except: - html_theme = 'default' - html_theme_path = ['.'] + html_theme = "default" + html_theme_path = ["."] else: - html_theme_path = ['.'] + html_theme_path = ["."] # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. -#html_theme_options = {} +# html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. # html_theme_path = ['.'] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". -#html_title = None +# html_title = None # A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None +# html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. -#html_logo = '../../logo/trans-logo.png' +# html_logo = '../../logo/trans-logo.png' # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. -html_favicon = 'favicon.ico' +html_favicon = "static/favicon.ico" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['static'] +html_static_path = ["static"] +# Add a custom CSS file for HTML generation +html_css_files = [ + "custom.css", +] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. -#html_extra_path = [] +# html_extra_path = [] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. -html_last_updated_fmt = '%d %b %Y' +html_last_updated_fmt = "%d %b %Y" # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. -#html_use_smartypants = True +# html_use_smartypants = True # Custom sidebar templates, maps document names to template names. -#html_sidebars = {} +# html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. html_additional_pages = {"index": "topindex.html"} # If false, no module index is generated. -#html_domain_indices = True +# html_domain_indices = True # If false, no index is generated. -#html_use_index = True +# html_use_index = True # If true, the index is split into individual pages for each letter. -#html_split_index = False +# html_split_index = False # If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True +# html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True +# html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True +# html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. -#html_use_opensearch = '' +# html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None +# html_file_suffix = None # Output file base name for HTML help builder. -htmlhelp_basename = 'MicroPythondoc' +htmlhelp_basename = "MicroPythondoc" # -- Options for LaTeX output --------------------------------------------- latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -#'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -#'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -#'preamble': '', -# Include 3 levels of headers in PDF ToC -'preamble': '\setcounter{tocdepth}{2}', + # The paper size ('letterpaper' or 'a4paper'). + #'papersize': 'letterpaper', + # The font size ('10pt', '11pt' or '12pt'). + #'pointsize': '10pt', + # Additional stuff for the LaTeX preamble. + #'preamble': '', + # Include 3 levels of headers in PDF ToC + "preamble": r"\setcounter{tocdepth}{2}", } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'MicroPython.tex', 'MicroPython Documentation', - 'Damien P. George, Paul Sokolovsky, and contributors', 'manual'), + ( + master_doc, + "MicroPython.tex", + "MicroPython Documentation", + micropy_authors, + "manual", + ), ] # The name of an image file (relative to this directory) to place at the top of # the title page. -#latex_logo = None +# latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. -#latex_use_parts = False +# latex_use_parts = False # If true, show page references after internal links. -#latex_show_pagerefs = False +# latex_show_pagerefs = False # If true, show URL addresses after external links. -#latex_show_urls = False +# latex_show_urls = False # Documents to append as an appendix to all manuals. -#latex_appendices = [] +# latex_appendices = [] # If false, no module index is generated. -#latex_domain_indices = True +# latex_domain_indices = True +# Enable better Unicode support so that `make latexpdf` doesn't fail +latex_engine = "xelatex" # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - ('index', 'micropython', 'MicroPython Documentation', - ['Damien P. George, Paul Sokolovsky, and contributors'], 1), + ( + "index", + "micropython", + "MicroPython Documentation", + [micropy_authors], + 1, + ), ] # If true, show URL addresses after external links. -#man_show_urls = False +# man_show_urls = False # -- Options for Texinfo output ------------------------------------------- @@ -303,44 +299,29 @@ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'MicroPython', 'MicroPython Documentation', - 'Damien P. George, Paul Sokolovsky, and contributors', 'MicroPython', 'One line description of project.', - 'Miscellaneous'), + ( + master_doc, + "MicroPython", + "MicroPython Documentation", + micropy_authors, + "MicroPython", + "One line description of project.", + "Miscellaneous", + ), ] # Documents to append as an appendix to all manuals. -#texinfo_appendices = [] +# texinfo_appendices = [] # If false, no module index is generated. -#texinfo_domain_indices = True +# texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' +# texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False +# texinfo_no_detailmenu = False # Example configuration for intersphinx: refer to the Python standard library. -intersphinx_mapping = {'python': ('http://docs.python.org/3.5', None)} - -# Append the other ports' specific folders/files to the exclude pattern -exclude_patterns.extend([port + '*' for port in ports if port != micropy_port]) - -modules_port_specific = { - 'pyboard': ['pyb'], - 'wipy': ['wipy'], - 'esp8266': ['esp'], -} - -modindex_exclude = [] - -for p, l in modules_port_specific.items(): - if p != micropy_port: - modindex_exclude += l - -# Exclude extra modules per port -modindex_exclude += { - 'esp8266': ['cmath', 'select'], - 'wipy': ['cmath'], -}.get(micropy_port, []) +intersphinx_mapping = {"python": ("https://docs.python.org/3.5", None)} diff --git a/docs/develop/cmodules.rst b/docs/develop/cmodules.rst new file mode 100644 index 0000000000000..c5aa919b7645a --- /dev/null +++ b/docs/develop/cmodules.rst @@ -0,0 +1,287 @@ +.. _cmodules: + +MicroPython external C modules +============================== + +When developing modules for use with MicroPython you may find you run into +limitations with the Python environment, often due to an inability to access +certain hardware resources or Python speed limitations. + +If your limitations can't be resolved with suggestions in :ref:`speed_python`, +writing some or all of your module in C (and/or C++ if implemented for your port) +is a viable option. + +If your module is designed to access or work with commonly available +hardware or libraries please consider implementing it inside the MicroPython +source tree alongside similar modules and submitting it as a pull request. +If however you're targeting obscure or proprietary systems it may make +more sense to keep this external to the main MicroPython repository. + +This chapter describes how to compile such external modules into the +MicroPython executable or firmware image. Both Make and CMake build +tools are supported, and when writing an external module it's a good idea to +add the build files for both of these tools so the module can be used on all +ports. But when compiling a particular port you will only need to use one +method of building, either Make or CMake. + +An alternative approach is to use :ref:`natmod` which allows writing custom C +code that is placed in a .mpy file, which can be imported dynamically in to +a running MicroPython system without the need to recompile the main firmware. + + +Structure of an external C module +--------------------------------- + +A MicroPython user C module is a directory with the following files: + +* ``*.c`` / ``*.cpp`` / ``*.h`` source code files for your module. + + These will typically include the low level functionality being implemented and + the MicroPython binding functions to expose the functions and module(s). + + Currently the best reference for writing these functions/modules is + to find similar modules within the MicroPython tree and use them as examples. + +* ``micropython.mk`` contains the Makefile fragment for this module. + + ``$(USERMOD_DIR)`` is available in ``micropython.mk`` as the path to your + module directory. As it's redefined for each c module, is should be expanded + in your ``micropython.mk`` to a local make variable, + eg ``EXAMPLE_MOD_DIR := $(USERMOD_DIR)`` + + Your ``micropython.mk`` must add your modules source files to the + ``SRC_USERMOD_C`` or ``SRC_USERMOD_LIB_C`` variables. The former will be + processed for ``MP_QSTR_`` and ``MP_REGISTER_MODULE`` definitions, the latter + will not (e.g. helpers and library code that isn't MicroPython-specific). + These paths should include your expanded copy of ``$(USERMOD_DIR)``, e.g.:: + + SRC_USERMOD_C += $(EXAMPLE_MOD_DIR)/modexample.c + SRC_USERMOD_LIB_C += $(EXAMPLE_MOD_DIR)/utils/algorithm.c + + Similarly, use ``SRC_USERMOD_CXX`` and ``SRC_USERMOD_LIB_CXX`` for C++ + source files. If you want to include assembly files use ``SRC_USERMOD_LIB_ASM``. + + If you have custom compiler options (like ``-I`` to add directories to search + for header files), these should be added to ``CFLAGS_USERMOD`` for C code + and to ``CXXFLAGS_USERMOD`` for C++ code. + +* ``micropython.cmake`` contains the CMake configuration for this module. + + In ``micropython.cmake``, you may use ``${CMAKE_CURRENT_LIST_DIR}`` as the path to + the current module. + + Your ``micropython.cmake`` should define an ``INTERFACE`` library and associate + your source files, compile definitions and include directories with it. + The library should then be linked to the ``usermod`` target. + + .. code-block:: cmake + + add_library(usermod_cexample INTERFACE) + + target_sources(usermod_cexample INTERFACE + ${CMAKE_CURRENT_LIST_DIR}/examplemodule.c + ) + + target_include_directories(usermod_cexample INTERFACE + ${CMAKE_CURRENT_LIST_DIR} + ) + + target_link_libraries(usermod INTERFACE usermod_cexample) + + + See below for full usage example. + + +Basic example +------------- + +The ``cexample`` module provides examples for a function and a class. The +``cexample.add_ints(a, b)`` function adds two integer args together and returns +the result. The ``cexample.Timer()`` type creates timers that can be used to +measure the elapsed time since the object is instantiated. + +The module can be found in the MicroPython source tree +`in the examples directory `_ +and has a source file and a Makefile fragment with content as described above:: + + micropython/ + └──examples/ + └──usercmodule/ + └──cexample/ + ├── examplemodule.c + ├── micropython.mk + └── micropython.cmake + + +Refer to the comments in these files for additional explanation. +Next to the ``cexample`` module there's also ``cppexample`` which +works in the same way but shows one way of mixing C and C++ code +in MicroPython. + + +Compiling the cmodule into MicroPython +-------------------------------------- + +To build such a module, compile MicroPython (see `getting started +`_), +applying 2 modifications: + +1. Set the build-time flag ``USER_C_MODULES`` to point to the modules + you want to include. For ports that use Make this variable should be a + directory which is searched automatically for modules. For ports that + use CMake this variable should be a file which includes the modules to + build. See below for details. + +2. Enable the modules by setting the corresponding C preprocessor macro to + 1. This is only needed if the modules you are building are not + automatically enabled. + +For building the example modules which come with MicroPython, +set ``USER_C_MODULES`` to the ``examples/usercmodule`` directory for Make, +or to ``examples/usercmodule/micropython.cmake`` for CMake. + +For example, here's how the to build the unix port with the example modules: + +.. code-block:: bash + + cd micropython/ports/unix + make USER_C_MODULES=../../examples/usercmodule + +You may need to run ``make clean`` once at the start when including new +user modules in the build. The build output will show the modules found:: + + ... + Including User C Module from ../../examples/usercmodule/cexample + Including User C Module from ../../examples/usercmodule/cppexample + ... + +For a CMake-based port such as rp2, this will look a little different (note +that CMake is actually invoked by ``make``): + +.. code-block:: bash + + cd micropython/ports/rp2 + make USER_C_MODULES=../../examples/usercmodule/micropython.cmake + +Again, you may need to run ``make clean`` first for CMake to pick up the +user modules. The CMake build output lists the modules by name:: + + ... + Including User C Module(s) from ../../examples/usercmodule/micropython.cmake + Found User C Module(s): usermod_cexample, usermod_cppexample + ... + +The contents of the top-level ``micropython.cmake`` can be used to control which +modules are enabled. + +For your own projects it's more convenient to keep custom code out of the main +MicroPython source tree, so a typical project directory structure will look +like this:: + + my_project/ + ├── modules/ + │ ├── example1/ + │ │ ├── example1.c + │ │ ├── micropython.mk + │ │ └── micropython.cmake + │ ├── example2/ + │ │ ├── example2.c + │ │ ├── micropython.mk + │ │ └── micropython.cmake + │ └── micropython.cmake + └── micropython/ + ├──ports/ + ... ├──stm32/ + ... + +When building with Make set ``USER_C_MODULES`` to the ``my_project/modules`` +directory. For example, building the stm32 port: + +.. code-block:: bash + + cd my_project/micropython/ports/stm32 + make USER_C_MODULES=../../../modules + +When building with CMake the top level ``micropython.cmake`` -- found directly +in the ``my_project/modules`` directory -- should ``include`` all of the modules +you want to have available: + + .. code-block:: cmake + + include(${CMAKE_CURRENT_LIST_DIR}/example1/micropython.cmake) + include(${CMAKE_CURRENT_LIST_DIR}/example2/micropython.cmake) + +Then build with: + +.. code-block:: bash + + cd my_project/micropython/ports/esp32 + make USER_C_MODULES=../../../../modules/micropython.cmake + +Note that the esp32 port needs the extra ``..`` for relative paths due to the +location of its main ``CMakeLists.txt`` file. You can also specify absolute +paths to ``USER_C_MODULES``. + +All modules specified by the ``USER_C_MODULES`` variable (either found in this +directory when using Make, or added via ``include`` when using CMake) will be +compiled, but only those which are enabled will be available for importing. +User modules are usually enabled by default (this is decided by the developer +of the module), in which case there is nothing more to do than set ``USER_C_MODULES`` +as described above. + +If a module is not enabled by default then the corresponding C preprocessor macro +must be enabled. This macro name can be found by searching for the ``MP_REGISTER_MODULE`` +line in the module's source code (it usually appears at the end of the main source file). +This macro should be surrounded by a ``#if X`` / ``#endif`` pair, and the configuration +option ``X`` must be set to 1 using ``CFLAGS_EXTRA`` to make the module available. If +there is no ``#if X`` / ``#endif`` pair then the module is enabled by default. + +For example, the ``examples/usercmodule/cexample`` module is enabled by default so +has the following line in its source code: + + .. code-block:: c + + MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule); + +Alternatively, to make this module disabled by default but selectable through +a preprocessor configuration option, it would be: + + .. code-block:: c + + #if MODULE_CEXAMPLE_ENABLED + MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule); + #endif + +In this case the module is enabled by adding ``CFLAGS_EXTRA=-DMODULE_CEXAMPLE_ENABLED=1`` +to the ``make`` command, or editing ``mpconfigport.h`` or ``mpconfigboard.h`` to add + + .. code-block:: c + + #define MODULE_CEXAMPLE_ENABLED (1) + +Note that the exact method depends on the port as they have different +structures. If not done correctly it will compile but importing will +fail to find the module. + + +Module usage in MicroPython +--------------------------- + +Once built into your copy of MicroPython, the module +can now be accessed in Python just like any other builtin module, e.g. + +.. code-block:: python + + import cexample + print(cexample.add_ints(1, 3)) + # should display 4 + +.. code-block:: python + + from cexample import Timer + from time import sleep_ms + + watch = Timer() + sleep_ms(1000) + print(watch.time()) + # should display approximately 1000 diff --git a/docs/develop/compiler.rst b/docs/develop/compiler.rst new file mode 100644 index 0000000000000..0c25ad3a01ef1 --- /dev/null +++ b/docs/develop/compiler.rst @@ -0,0 +1,324 @@ +.. _compiler: + +The Compiler +============ + +The compilation process in MicroPython involves the following steps: + +* The lexer converts the stream of text that makes up a MicroPython program into tokens. +* The parser then converts the tokens into an abstract syntax (parse tree). +* Then bytecode or native code is emitted based on the parse tree. + +For purposes of this discussion we are going to add a simple language feature ``add1`` +that can be use in Python as: + +.. code-block:: bash + + >>> add1 3 + 4 + >>> + +The ``add1`` statement takes an integer as argument and adds ``1`` to it. + +Adding a grammar rule +---------------------- + +MicroPython's grammar is based on the `CPython grammar `_ +and is defined in `py/grammar.h `_. +This grammar is what is used to parse MicroPython source files. + +There are two macros you need to know to define a grammar rule: ``DEF_RULE`` and ``DEF_RULE_NC``. +``DEF_RULE`` allows you to define a rule with an associated compile function, +while ``DEF_RULE_NC`` has no compile (NC) function for it. + +A simple grammar definition with a compile function for our new ``add1`` statement +looks like the following: + +.. code-block:: c + + DEF_RULE(add1_stmt, c(add1_stmt), and(2), tok(KW_ADD1), rule(testlist)) + +The second argument ``c(add1_stmt)`` is the corresponding compile function that should be implemented +in ``py/compile.c`` to turn this rule into executable code. + +The third required argument can be ``or`` or ``and``. This specifies the number of nodes associated +with a statement. For example, in this case, our ``add1`` statement is similar to ADD1 in assembly +language. It takes one numeric argument. Therefore, the ``add1_stmt`` has two nodes associated with it. +One node is for the statement itself, i.e the literal ``add1`` corresponding to ``KW_ADD1``, +and the other for its argument, a ``testlist`` rule which is the top-level expression rule. + +.. note:: + The ``add1`` rule here is just an example and not part of the standard + MicroPython grammar. + +The fourth argument in this example is the token associated with the rule, ``KW_ADD1``. This token should be +defined in the lexer by editing ``py/lexer.h``. + +Defining the same rule without a compile function is achieved by using the ``DEF_RULE_NC`` macro +and omitting the compile function argument: + +.. code-block:: c + + DEF_RULE_NC(add1_stmt, and(2), tok(KW_ADD1), rule(testlist)) + +The remaining arguments take on the same meaning. A rule without a compile function must +be handled explicitly by all rules that may have this rule as a node. Such NC-rules are usually +used to express sub-parts of a complicated grammar structure that cannot be expressed in a +single rule. + +.. note:: + The macros ``DEF_RULE`` and ``DEF_RULE_NC`` take other arguments. For an in-depth understanding of + supported parameters, see `py/grammar.h `_. + +Adding a lexical token +---------------------- + +Every rule defined in the grammar should have a token associated with it that is defined in ``py/lexer.h``. +Add this token by editing the ``_mp_token_kind_t`` enum: + +.. code-block:: c + :emphasize-lines: 12 + + typedef enum _mp_token_kind_t { + ... + MP_TOKEN_KW_OR, + MP_TOKEN_KW_PASS, + MP_TOKEN_KW_RAISE, + MP_TOKEN_KW_RETURN, + MP_TOKEN_KW_TRY, + MP_TOKEN_KW_WHILE, + MP_TOKEN_KW_WITH, + MP_TOKEN_KW_YIELD, + MP_TOKEN_KW_ADD1, + ... + } mp_token_kind_t; + +Then also edit ``py/lexer.c`` to add the new keyword literal text: + +.. code-block:: c + :emphasize-lines: 12 + + static const char *const tok_kw[] = { + ... + "or", + "pass", + "raise", + "return", + "try", + "while", + "with", + "yield", + "add1", + ... + }; + +Notice the keyword is named depending on what you want it to be. For consistency, maintain the +naming standard accordingly. + +.. note:: + The order of these keywords in ``py/lexer.c`` must match the order of tokens in the enum + defined in ``py/lexer.h``. + +Parsing +------- + +In the parsing stage the parser takes the tokens produced by the lexer and converts them to an abstract syntax tree (AST) or +*parse tree*. The implementation for the parser is defined in `py/parse.c `_. + +The parser also maintains a table of constants for use in different aspects of parsing, similar to what a +`symbol table `_ +does. + +Several optimizations like `constant folding `_ +on integers for most operations e.g. logical, binary, unary, etc, and optimizing enhancements on parenthesis +around expressions are performed during this phase, along with some optimizations on strings. + +It's worth noting that *docstrings* are discarded and not accessible to the compiler. +Even optimizations like `string interning `_ are +not applied to *docstrings*. + +Compiler passes +--------------- + +Like many compilers, MicroPython compiles all code to MicroPython bytecode or native code. The functionality +that achieves this is implemented in `py/compile.c `_. +The most relevant method you should know about is this: + +.. code-block:: c + + mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, bool is_repl) { + // Create a context for this module, and set its globals dict. + mp_module_context_t *context = m_new_obj(mp_module_context_t); + context->module.globals = mp_globals_get(); + + // Compile the input parse_tree to a raw-code structure. + mp_compiled_module_t cm; + cm.context = context; + mp_compile_to_raw_code(parse_tree, source_file, is_repl, &cm); + + // Create and return a function object that executes the outer module. + return mp_make_function_from_proto_fun(cm.rc, cm.context, NULL); + } + +The compiler compiles the code in four passes: scope, stack size, code size and emit. +Each pass runs the same C code over the same AST data structure, with different things +being computed each time based on the results of the previous pass. + +First pass +~~~~~~~~~~ + +In the first pass, the compiler learns about the known identifiers (variables) and +their scope, being global, local, closed over, etc. In the same pass the emitter +(bytecode or native code) also computes the number of labels needed for the emitted +code. + +.. code-block:: c + + // Compile pass 1. + comp->emit = emit_bc; + comp->emit_method_table = &emit_bc_method_table; + + uint max_num_labels = 0; + for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) { + if (s->emit_options == MP_EMIT_OPT_ASM) { + compile_scope_inline_asm(comp, s, MP_PASS_SCOPE); + } else { + compile_scope(comp, s, MP_PASS_SCOPE); + + // Check if any implicitly declared variables should be closed over. + for (size_t i = 0; i < s->id_info_len; ++i) { + id_info_t *id = &s->id_info[i]; + if (id->kind == ID_INFO_KIND_GLOBAL_IMPLICIT) { + scope_check_to_close_over(s, id); + } + } + } + ... + } + +Second and third passes +~~~~~~~~~~~~~~~~~~~~~~~ + +The second and third passes involve computing the Python stack size and code size +for the bytecode or native code. After the third pass the code size cannot change, +otherwise jump labels will be incorrect. + +.. code-block:: c + + for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) { + ... + + // Pass 2: Compute the Python stack size. + compile_scope(comp, s, MP_PASS_STACK_SIZE); + + // Pass 3: Compute the code size. + if (comp->compile_error == MP_OBJ_NULL) { + compile_scope(comp, s, MP_PASS_CODE_SIZE); + } + + ... + } + +Just before pass two there is a selection for the type of code to be emitted, which can +either be native or bytecode. + +.. code-block:: c + + // Choose the emitter type. + switch (s->emit_options) { + case MP_EMIT_OPT_NATIVE_PYTHON: + case MP_EMIT_OPT_VIPER: + if (emit_native == NULL) { + emit_native = NATIVE_EMITTER(new)(&comp->compile_error, &comp->next_label, max_num_labels); + } + comp->emit_method_table = NATIVE_EMITTER_TABLE; + comp->emit = emit_native; + break; + + default: + comp->emit = emit_bc; + comp->emit_method_table = &emit_bc_method_table; + break; + } + +The bytecode option is the default but something unique to note for the native +code option is that there is another option via ``VIPER``. See the +:ref:`Emitting native code ` section for more details on +viper annotations. + +There is also support for *inline assembly code*, where assembly instructions are +written as Python function calls but are emitted directly as the corresponding +machine code. This assembler has only three passes (scope, code size, emit) +and uses a different implementation, not the ``compile_scope`` function. +See the `inline assembler tutorial `_ +for more details. + +Fourth pass +~~~~~~~~~~~ + +The fourth pass emits the final code that can be executed, either bytecode in +the virtual machine, or native code directly by the CPU. + +.. code-block:: c + + for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) { + ... + + // Pass 4: Emit the compiled bytecode or native code. + if (comp->compile_error == MP_OBJ_NULL) { + compile_scope(comp, s, MP_PASS_EMIT); + } + } + +Emitting bytecode +----------------- + +Statements in Python code usually correspond to emitted bytecode, for example ``a + b`` +generates "push a" then "push b" then "binary op add". Some statements do not emit +anything but instead affect other things like the scope of variables, for example +``global a``. + +The implementation of a function that emits bytecode looks similar to this: + +.. code-block:: c + + void mp_emit_bc_unary_op(emit_t *emit, mp_unary_op_t op) { + emit_write_bytecode_byte(emit, 0, MP_BC_UNARY_OP_MULTI + op); + } + +We use the unary operator expressions for an example here but the implementation +details are similar for other statements/expressions. The method ``emit_write_bytecode_byte()`` +is a wrapper around the main function ``emit_get_cur_to_write_bytecode()`` that all +functions must call to emit bytecode. + +.. _emitting_native_code: + +Emitting native code +--------------------- + +Similar to how bytecode is generated, there should be a corresponding function in ``py/emitnative.c`` for each +code statement: + +.. code-block:: c + + static void emit_native_unary_op(emit_t *emit, mp_unary_op_t op) { + vtype_kind_t vtype; + emit_pre_pop_reg(emit, &vtype, REG_ARG_2); + if (vtype == VTYPE_PYOBJ) { + emit_call_with_imm_arg(emit, MP_F_UNARY_OP, op, REG_ARG_1); + emit_post_push_reg(emit, VTYPE_PYOBJ, REG_RET); + } else { + adjust_stack(emit, 1); + EMIT_NATIVE_VIPER_TYPE_ERROR(emit, + MP_ERROR_TEXT("unary op %q not implemented"), mp_unary_op_method_name[op]); + } + } + +The difference here is that we have to handle *viper typing*. Viper annotations allow +us to handle more than one type of variable. By default all variables are Python objects, +but with viper a variable can also be declared as a machine-typed variable like a native +integer or pointer. Viper can be thought of as a superset of Python, where normal Python +objects are handled as usual, while native machine variables are handled in an optimised +way by using direct machine instructions for the operations. Viper typing may break +Python equivalence because, for example, integers become native integers and can overflow +(unlike Python integers which extend automatically to arbitrary precision). diff --git a/docs/develop/extendingmicropython.rst b/docs/develop/extendingmicropython.rst new file mode 100644 index 0000000000000..b4029c421c89a --- /dev/null +++ b/docs/develop/extendingmicropython.rst @@ -0,0 +1,18 @@ +.. _extendingmicropython: + +Extending MicroPython in C +========================== + +This chapter describes options for implementing additional functionality in C, but from code +written outside of the main MicroPython repository. The first approach is useful for building +your own custom firmware with some project-specific additional modules or functions that can +be accessed from Python. The second approach is for building modules that can be loaded at runtime. + +Please see the :ref:`library section ` for more information on building core modules that +live in the main MicroPython repository. + +.. toctree:: + :maxdepth: 3 + + cmodules.rst + natmod.rst diff --git a/docs/develop/gettingstarted.rst b/docs/develop/gettingstarted.rst new file mode 100644 index 0000000000000..fed632ea1ac15 --- /dev/null +++ b/docs/develop/gettingstarted.rst @@ -0,0 +1,330 @@ +.. _gettingstarted: + +Getting Started +=============== + +This guide covers a step-by-step process on setting up version control, obtaining and building +a copy of the source code for a port, building the documentation, running tests, and a description of the +directory structure of the MicroPython code base. + +Source control with git +----------------------- + +MicroPython is hosted on `GitHub `_ and uses +`Git `_ for source control. The workflow is such that +code is pulled and pushed to and from the main repository. Install the respective version +of Git for your operating system to follow through the rest of the steps. + +.. note:: + For a reference on the installation instructions, please refer to + the `Git installation instructions `_. + Learn about the basic git commands in this `Git Handbook `_ + or any other sources on the internet. + +.. note:: + A .git-blame-ignore-revs file is included which avoids the output of git blame getting cluttered + by commits which are only for formatting code but have no functional changes. See `git blame documentation + `_ on how to use this. + +Get the code +------------ + +It is recommended that you maintain a fork of the MicroPython repository for your development purposes. +The process of obtaining the source code includes the following: + +#. Fork the repository https://github.com/micropython/micropython +#. You will now have a fork at /micropython>. +#. Clone the forked repository using the following command: + +.. code-block:: bash + + $ git clone https://github.com//micropython + +Then, `configure the remote repositories `_ to be able to +collaborate on the MicroPython project. + +Configure remote upstream: + +.. code-block:: bash + + $ cd micropython + $ git remote add upstream https://github.com/micropython/micropython + +It is common to configure ``upstream`` and ``origin`` on a forked repository +to assist with sharing code changes. You can maintain your own mapping but +it is recommended that ``origin`` maps to your fork and ``upstream`` to the main +MicroPython repository. + +After the above configuration, your setup should be similar to this: + +.. code-block:: bash + + $ git remote -v + origin https://github.com//micropython (fetch) + origin https://github.com//micropython (push) + upstream https://github.com/micropython/micropython (fetch) + upstream https://github.com/micropython/micropython (push) + +You should now have a copy of the source code. By default, you are pointing +to the master branch. To prepare for further development, it is recommended +to work on a development branch. + +.. code-block:: bash + + $ git checkout -b dev-branch + +You can give it any name. You will have to compile MicroPython whenever you change +to a different branch. + +Compile and build the code +-------------------------- + +When compiling MicroPython, you compile a specific :term:`port`, usually +targeting a specific :ref:`board `. Start by installing the required dependencies. +Then build the MicroPython cross-compiler before you can successfully compile and build. +This applies specifically when using Linux to compile. +The Windows instructions are provided in a later section. + +.. _required_dependencies: + +Required dependencies +~~~~~~~~~~~~~~~~~~~~~ + +Install the required dependencies for Linux: + +.. code-block:: bash + + $ sudo apt-get install build-essential libffi-dev git pkg-config + +For the stm32 port, the ARM cross-compiler is required: + +.. code-block:: bash + + $ sudo apt-get install gcc-arm-none-eabi libnewlib-arm-none-eabi + +See the `ARM GCC +toolchain `_ +for the latest details. + +Python is also required. Python 2 is supported for now, but we recommend using Python 3. +Check that you have Python available on your system: + +.. code-block:: bash + + $ python3 + Python 3.5.0 (default, Jul 17 2020, 14:04:10) + [GCC 5.4.0 20160609] on linux + Type "help", "copyright", "credits" or "license" for more information. + >>> + +All supported ports have different dependency requirements, see their respective +`readme files `_. + +Building the MicroPython cross-compiler +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Almost all ports require building ``mpy-cross`` first to perform pre-compilation +of Python code that will be included in the port firmware: + +.. code-block:: bash + + $ cd mpy-cross + $ make + +.. note:: + Note that, ``mpy-cross`` must be built for the host architecture + and not the target architecture. + +If it built successfully, you should see a message similar to this: + +.. code-block:: bash + + LINK mpy-cross + text data bss dec hex filename + 279328 776 880 280984 44998 mpy-cross + +.. note:: + + Use ``make -C mpy-cross`` to build the cross-compiler in one statement + without moving to the ``mpy-cross`` directory otherwise, you will need + to do ``cd ..`` for the next steps. + +Building the Unix port of MicroPython +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Unix port is a version of MicroPython that runs on Linux, macOS, and other Unix-like operating systems. +It's extremely useful for developing MicroPython as it avoids having to deploy your code to a device to test it. +In many ways, it works a lot like CPython's python binary. + +To build for the Unix port, make sure all Linux related dependencies are installed as detailed in the +required dependencies section. See the :ref:`required_dependencies` +to make sure that all dependencies are installed for this port. Also, make sure you have a working +environment for ``gcc`` and ``GNU make``. Ubuntu 20.04 has been used for the example +below but other unixes ought to work with little modification: + +.. code-block:: bash + + $ gcc --version + gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0 + Copyright (C) 2019 Free Software Foundation, Inc. + This is free software; see the source for copying conditions. There is NO + warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.then build: + +.. code-block:: bash + + $ cd ports/unix + $ make submodules + $ make + +If MicroPython built correctly, you should see the following: + +.. code-block:: bash + + LINK micropython + text data bss dec hex filename + 412033 5680 2496 420209 66971 micropython + +Now run it: + +.. code-block:: bash + + $ ./micropython + MicroPython v1.13-38-gc67012d-dirty on 2020-09-13; linux version + Use Ctrl-D to exit, Ctrl-E for paste mode + >>> print("hello world") + hello world + >>> + +Building the Windows port +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Windows port includes a Visual Studio project file micropython.vcxproj that you can use to build micropython.exe. +It can be opened in Visual Studio or built from the command line using msbuild. Alternatively, it can be built using mingw, +either in Windows with Cygwin, or on Linux. +See `windows port documentation `_ for more information. + +Building the STM32 port +~~~~~~~~~~~~~~~~~~~~~~~ + +Like the Unix port, you need to install some required dependencies +as detailed in the :ref:`required_dependencies` section, then build: + +.. code-block:: bash + + $ cd ports/stm32 + $ make submodules + $ make + +Please refer to the `stm32 documentation `_ +for more details on flashing the firmware. + +.. note:: + See the :ref:`required_dependencies` to make sure that all dependencies are installed for this port. + The cross-compiler is needed. ``arm-none-eabi-gcc`` should also be in the $PATH or specified manually + via CROSS_COMPILE, either by setting the environment variable or in the ``make`` command line arguments. + +You can also specify which board to use: + +.. code-block:: bash + + $ cd ports/stm32 + $ make BOARD= submodules + $ make BOARD= + +See `ports/stm32/boards `_ +for the available boards. e.g. "PYBV11" or "NUCLEO_WB55". + +Building the documentation +-------------------------- + +MicroPython documentation is created using ``Sphinx``. If you have already +installed Python, then install ``Sphinx`` using ``pip``. It is recommended +that you use a virtual environment: + +.. code-block:: bash + + $ python3 -m venv env + $ source env/bin/activate + $ pip install -r docs/requirements.txt + +Navigate to the ``docs`` directory: + +.. code-block:: bash + + $ cd docs + +Build the docs: + +.. code-block:: bash + + $ make html + +Open ``docs/build/html/index.html`` in your browser to view the docs locally. Refer to the +documentation on `importing your documentation +`_ to use Read the Docs. + +Running the tests +----------------- + +To run all tests in the test suite on the Unix port use: + +.. code-block:: bash + + $ cd ports/unix + $ make test + +To run a selection of tests on a board/device connected over USB use: + +.. code-block:: bash + + $ cd tests + $ ./run-tests.py -t /dev/ttyACM0 + +See also :ref:`writingtests`. + +Folder structure +---------------- + +There are a couple of directories to take note of in terms of where certain implementation details +are. The following is a break down of the top-level folders in the source code. + +py + + Contains the compiler, runtime, and core library implementation. + +mpy-cross + + Has the MicroPython cross-compiler which pre-compiles the Python scripts to bytecode. + +ports + + Code for all the versions of MicroPython for the supported ports. + +lib + + Low-level C libraries used by any port which are mostly 3rd-party libraries. + +drivers + + Has drivers for specific hardware and intended to work across multiple ports. + +extmod + + Contains a C implementation of more non-core modules. + +docs + + Has the standard documentation found at https://docs.micropython.org/. + +tests + + An implementation of the test suite. + +tools + + Contains scripts used by the build and CI process, as well as user tools such + as ``pyboard.py`` and ``mpremote``. + +examples + + Example code for building MicroPython as a library as well as native modules. diff --git a/docs/develop/img/bitmap.png b/docs/develop/img/bitmap.png new file mode 100644 index 0000000000000..87de81d769431 Binary files /dev/null and b/docs/develop/img/bitmap.png differ diff --git a/docs/develop/img/collision.png b/docs/develop/img/collision.png new file mode 100644 index 0000000000000..a67ddd6137c0f Binary files /dev/null and b/docs/develop/img/collision.png differ diff --git a/docs/develop/img/linprob.png b/docs/develop/img/linprob.png new file mode 100644 index 0000000000000..c28818908464d Binary files /dev/null and b/docs/develop/img/linprob.png differ diff --git a/docs/develop/index.rst b/docs/develop/index.rst new file mode 100644 index 0000000000000..327038f1978bd --- /dev/null +++ b/docs/develop/index.rst @@ -0,0 +1,26 @@ +MicroPython Internals +===================== + +This chapter covers a tour of MicroPython from the perspective of a developer, contributing +to MicroPython. It acts as a comprehensive resource on the implementation details of MicroPython +for both novice and expert contributors. + +Development around MicroPython usually involves modifying the core runtime, porting or +maintaining a new library. This guide describes at great depth, the implementation +details of MicroPython including a getting started guide, compiler internals, porting +MicroPython to a new platform and implementing a core MicroPython library. + +.. toctree:: + :maxdepth: 3 + + gettingstarted.rst + writingtests.rst + compiler.rst + memorymgt.rst + library.rst + optimizations.rst + qstr.rst + maps.rst + publiccapi.rst + extendingmicropython.rst + porting.rst diff --git a/docs/develop/library.rst b/docs/develop/library.rst new file mode 100644 index 0000000000000..830211d81c862 --- /dev/null +++ b/docs/develop/library.rst @@ -0,0 +1,86 @@ +.. _internals_library: + +Implementing a Module +===================== + +This chapter details how to implement a core module in MicroPython. +MicroPython modules can be one of the following: + +- Built-in module: A general module that is be part of the MicroPython repository. +- User module: A module that is useful for your specific project that you maintain + in your own repository or private codebase. +- Dynamic module: A module that can be deployed and imported at runtime to your device. + +A module in MicroPython can be implemented in one of the following locations: + +- py/: A core library that mirrors core CPython functionality. +- extmod/: A CPython or MicroPython-specific module that is shared across multiple ports. +- ports//: A port-specific module. + +.. note:: + This chapter describes modules implemented in ``py/`` or core modules. + See :ref:`extendingmicropython` for details on implementing an external module. + For details on port-specific modules, see :ref:`porting_to_a_board`. + +Implementing a core module +-------------------------- + +Like CPython, MicroPython has core builtin modules that can be accessed through import statements. +An example is the ``gc`` module discussed in :ref:`memorymanagement`. + +.. code-block:: bash + + >>> import gc + >>> gc.enable() + >>> + +MicroPython has several other builtin standard/core modules like ``io``, ``array`` etc. +Adding a new core module involves several modifications. + +First, create the ``C`` file in the ``py/`` directory. In this example we are adding a +hypothetical new module ``subsystem`` in the file ``modsubsystem.c``: + +.. code-block:: c + + #include "py/builtin.h" + #include "py/runtime.h" + + #if MICROPY_PY_SUBSYSTEM + + // info() + static mp_obj_t py_subsystem_info(void) { + return MP_OBJ_NEW_SMALL_INT(42); + } + MP_DEFINE_CONST_FUN_OBJ_0(subsystem_info_obj, py_subsystem_info); + + static const mp_rom_map_elem_t mp_module_subsystem_globals_table[] = { + { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_subsystem) }, + { MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&subsystem_info_obj) }, + }; + static MP_DEFINE_CONST_DICT(mp_module_subsystem_globals, mp_module_subsystem_globals_table); + + const mp_obj_module_t mp_module_subsystem = { + .base = { &mp_type_module }, + .globals = (mp_obj_dict_t *)&mp_module_subsystem_globals, + }; + + MP_REGISTER_MODULE(MP_QSTR_subsystem, mp_module_subsystem); + + #endif + +The implementation includes a definition of all functions related to the module and adds the +functions to the module's global table in ``mp_module_subsystem_globals_table``. It also +creates the module object with ``mp_module_subsystem``. The module is then registered with +the wider system via the ``MP_REGISTER_MODULE`` macro. + +After building and running the modified MicroPython, the module should now be importable: + +.. code-block:: bash + + >>> import subsystem + >>> subsystem.info() + 42 + >>> + +Our ``info()`` function currently returns just a single number but can be extended +to do anything. Similarly, more functions can be added to this new module. diff --git a/docs/develop/maps.rst b/docs/develop/maps.rst new file mode 100644 index 0000000000000..8f899fa1d3462 --- /dev/null +++ b/docs/develop/maps.rst @@ -0,0 +1,63 @@ +.. _maps: + +Maps and Dictionaries +===================== + +MicroPython dictionaries and maps use techniques called open addressing and linear probing. +This chapter details both of these methods. + +Open addressing +--------------- + +`Open addressing `_ is used to resolve collisions. +Collisions are very common occurrences and happen when two items happen to hash to the same +slot or location. For example, given a hash setup as this: + +.. image:: img/collision.png + +If there is a request to fill slot ``0`` with ``70``, since the slot ``0`` is not empty, open addressing +finds the next available slot in the dictionary to service this request. This sequential search for an alternate +location is called *probing*. There are several sequence probing algorithms but MicroPython uses +linear probing that is described in the next section. + +Linear probing +-------------- + +Linear probing is one of the methods for finding an available address or slot in a dictionary. In MicroPython, +it is used with open addressing. To service the request described above, unlike other probing algorithms, +linear probing assumes a fixed interval of ``1`` between probes. The request will therefore be serviced by +placing the item in the next free slot which is slot ``4`` in our example: + +.. image:: img/linprob.png + +The same methods i.e open addressing and linear probing are used to search for an item in a dictionary. +Assume we want to search for the data item ``33``. The computed hash value will be 2. Looking at slot 2 +reveals ``33``, at this point, we return ``True``. Searching for ``70`` is quite different as there was a +collision at the time of insertion. Therefore computing the hash value is ``0`` which is currently +holding ``44``. Instead of simply returning ``False``, we perform a sequential search starting at point +``1`` until the item ``70`` is found or we encounter a free slot. This is the general way of performing +look-ups in hashes: + +.. code-block:: c + + // not yet found, keep searching in this table + pos = (pos + 1) % set->alloc; + + if (pos == start_pos) { + // search got back to starting position, so index is not in table + if (lookup_kind & MP_MAP_LOOKUP_ADD_IF_NOT_FOUND) { + if (avail_slot != NULL) { + // there was an available slot, so use that + set->used++; + *avail_slot = index; + return index; + } else { + // not enough room in table, rehash it + mp_set_rehash(set); + // restart the search for the new element + start_pos = pos = hash % set->alloc; + } + } + } else { + return MP_OBJ_NULL; + } diff --git a/docs/develop/memorymgt.rst b/docs/develop/memorymgt.rst new file mode 100644 index 0000000000000..5b1690cc827b3 --- /dev/null +++ b/docs/develop/memorymgt.rst @@ -0,0 +1,141 @@ +.. _memorymanagement: + +Memory Management +================= + +Unlike programming languages such as C/C++, MicroPython hides memory management +details from the developer by supporting automatic memory management. +Automatic memory management is a technique used by operating systems or applications to automatically manage +the allocation and deallocation of memory. This eliminates challenges such as forgetting to +free the memory allocated to an object. Automatic memory management also avoids the critical issue of using memory +that is already released. Automatic memory management takes many forms, one of them being +garbage collection (GC). + +The garbage collector usually has two responsibilities; + +#. Allocate new objects in available memory. +#. Free unused memory. + +There are many GC algorithms but MicroPython uses the +`Mark and Sweep `_ +policy for managing memory. This algorithm has a mark phase that traverses the heap marking all +live objects while the sweep phase goes through the heap reclaiming all unmarked objects. + +Garbage collection functionality in MicroPython is available through the ``gc`` built-in +module: + +.. code-block:: bash + + >>> x = 5 + >>> x + 5 + >>> import gc + >>> gc.enable() + >>> gc.mem_alloc() + 1312 + >>> gc.mem_free() + 2071392 + >>> gc.collect() + 19 + >>> gc.disable() + >>> + +Even when ``gc.disable()`` is invoked, collection can be triggered with ``gc.collect()``. + +The object model +---------------- + +All MicroPython objects are referred to by the ``mp_obj_t`` data type. +This is usually word-sized (i.e. the same size as a pointer on the target architecture), +and can be typically 32-bit (STM32, nRF, ESP32, Unix x86) or 64-bit (Unix x64). +It can also be greater than a word-size for certain object representations, for +example ``OBJ_REPR_D`` has a 64-bit sized ``mp_obj_t`` on a 32-bit architecture. + +An ``mp_obj_t`` represents a MicroPython object, for example an integer, float, type, dict or +class instance. Some objects, like booleans and small integers, have their value stored directly +in the ``mp_obj_t`` value and do not require additional memory. Other objects have their value +store elsewhere in memory (for example on the garbage-collected heap) and their ``mp_obj_t`` contains +a pointer to that memory. A portion of ``mp_obj_t`` is the tag which tells what type of object it is. + +See ``py/mpconfig.h`` for the specific details of the available representations. + +**Pointer tagging** + +Because pointers are word-aligned, when they are stored in an ``mp_obj_t`` the +lower bits of this object handle will be zero. For example on a 32-bit architecture +the lower 2 bits will be zero: + +``********|********|********|******00`` + +These bits are reserved for purposes of storing a tag. The tag stores extra information as +opposed to introducing a new field to store that information in the object, which may be +inefficient. In MicroPython the tag tells if we are dealing with a small integer, interned +(small) string or a concrete object, and different semantics apply to each of these. + +For small integers the mapping is this: + +``********|********|********|*******1`` + +Where the asterisks hold the actual integer value. For an interned string or an immediate +object (e.g. ``True``) the layout of the ``mp_obj_t`` value is, respectively: + +``********|********|********|*****010`` + +``********|********|********|*****110`` + +While a concrete object that is none of the above takes the form: + +``********|********|********|******00`` + +The stars here correspond to the address of the concrete object in memory. + +Allocation of objects +---------------------- + +The value of a small integer is stored directly in the ``mp_obj_t`` and will be +allocated in-place, not on the heap or elsewhere. As such, creation of small +integers does not affect the heap. Similarly for interned strings that already have +their textual data stored elsewhere, and immediate values like ``None``, ``False`` +and ``True``. + +Everything else which is a concrete object is allocated on the heap and its object structure is such that +a field is reserved in the object header to store the type of the object. + +.. code-block:: bash + + +++++++++++ + + + + + type + object header + + + + +++++++++++ + + + object items + + + + + + + +++++++++++ + +The heap's smallest unit of allocation is a block, which is four machine words in +size (16 bytes on a 32-bit machine, 32 bytes on a 64-bit machine). +Another structure also allocated on the heap tracks the allocation of +objects in each block. This structure is called a *bitmap*. + +.. image:: img/bitmap.png + +The bitmap tracks whether a block is "free" or "in use" and use two bits to track this state +for each block. + +The mark-sweep garbage collector manages the objects allocated on the heap, and also +utilises the bitmap to mark objects that are still in use. +See `py/gc.c `_ +for the full implementation of these details. + +**Allocation: heap layout** + +The heap is arranged such that it consists of blocks in pools. A block +can have different properties: + +- *ATB(allocation table byte):* If set, then the block is a normal block +- *FREE:* Free block +- *HEAD:* Head of a chain of blocks +- *TAIL:* In the tail of a chain of blocks +- *MARK :* Marked head block +- *FTB(finaliser table byte):* If set, then the block has a finaliser diff --git a/docs/develop/natmod.rst b/docs/develop/natmod.rst new file mode 100644 index 0000000000000..2ccd83288565c --- /dev/null +++ b/docs/develop/natmod.rst @@ -0,0 +1,240 @@ +.. _natmod: + +Native machine code in .mpy files +================================= + +This section describes how to build and work with .mpy files that contain native +machine code from a language other than Python. This allows you to +write code in a language like C, compile and link it into a .mpy file, and then +import this file like a normal Python module. This can be used for implementing +functionality which is performance critical, or for including an existing +library written in another language. + +One of the main advantages of using native .mpy files is that native machine code +can be imported by a script dynamically, without the need to rebuild the main +MicroPython firmware. This is in contrast to :ref:`cmodules` which also allows +defining custom modules in C but they must be compiled into the main firmware image. + +The focus here is on using C to build native modules, but in principle any +language which can be compiled to stand-alone machine code can be put into a +.mpy file. + +A native .mpy module is built using the ``mpy_ld.py`` tool, which is found in the +``tools/`` directory of the project. This tool takes a set of object files +(.o files) and links them together to create a native .mpy files. It requires +CPython 3 and the library pyelftools v0.25 or greater. + +Supported features and limitations +---------------------------------- + +A .mpy file can contain MicroPython bytecode and/or native machine code. If it +contains native machine code then the .mpy file has a specific architecture +associated with it. Current supported architectures are (these are the valid +options for the ``ARCH`` variable, see below): + +* ``x86`` (32 bit) +* ``x64`` (64 bit x86) +* ``armv6m`` (ARM Thumb, eg Cortex-M0) +* ``armv7m`` (ARM Thumb 2, eg Cortex-M3) +* ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7) +* ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7) +* ``xtensa`` (non-windowed, eg ESP8266) +* ``xtensawin`` (windowed with window size 8, eg ESP32, ESP32S3) +* ``rv32imc`` (RISC-V 32 bits with compressed instructions, eg ESP32C3, ESP32C6) + +When compiling and linking the native .mpy file the architecture must be chosen +and the corresponding file can only be imported on that architecture. For more +details about .mpy files see :ref:`mpy_files`. + +Native code must be compiled as position independent code (PIC) and use a global +offset table (GOT), although the details of this varies from architecture to +architecture. When importing .mpy files with native code the import machinery +is able to do some basic relocation of the native code. This includes +relocating text, rodata and BSS sections. + +Supported features of the linker and dynamic loader are: + +* executable code (text) +* read-only data (rodata), including strings and constant data (arrays, structs, etc) +* zeroed data (BSS) +* pointers in text to text, rodata and BSS +* pointers in rodata to text, rodata and BSS + +The known limitations are: + +* data sections are not supported; workaround: use BSS data and initialise the + data values explicitly + +* static BSS variables are not supported; workaround: use global BSS variables + +So, if your C code has writable data, make sure the data is defined globally, +without an initialiser, and only written to within functions. + +The native module is not automatically linked against the standard static libraries +like ``libm.a`` and ``libgcc.a``, which can lead to ``undefined symbol`` errors. +You can link the runtime libraries by setting ``LINK_RUNTIME = 1`` +in your Makefile. Custom static libraries can also be linked by adding +``MPY_LD_FLAGS += -l path/to/library.a``. Note that these are linked into +the native module and will not be shared with other modules or the system. + +Linker limitation: the native module is not linked against the symbol table of the +full MicroPython firmware. Rather, it is linked against an explicit table of exported +symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware +build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system +function, for example, unless that resides at a fixed address. In that case, the path +of a linkerscript containing a series of symbol names and their fixed address can be +passed to ``mpy_ld.py`` via the ``--externs`` command line argument. That way symbols +appearing in the linkerscript will take precedence over what is provided from object +files, but at the moment the object files' implementation will still reside in the +final MPY file. The linkerscript parser is limited in its capabilities, and is +currently used only for parsing the ESP8266 port ROM symbols list (see +``ports/esp8266/boards/eagle.rom.addr.v6.ld``). + +New symbols can be added to the end of the table and the firmware rebuilt. +The symbols also need to be added to ``tools/mpy_ld.py``'s ``fun_table`` dict in the +same location. This allows ``mpy_ld.py`` to be able to pick the new symbols up and +provide relocations for them when the mpy is imported. Finally, if the symbol is a +function, a macro or stub should be added to ``py/dynruntime.h`` to make it easy to +call the function. + +Defining a native module +------------------------ + +A native .mpy module is defined by a set of files that are used to build the .mpy. +The filesystem layout consists of two main parts, the source files and the Makefile: + +* In the simplest case only a single C source file is required, which contains all + the code that will be compiled into the .mpy module. This C source code must + include the ``py/dynruntime.h`` file to access the MicroPython dynamic API, and + must at least define a function called ``mpy_init``. This function will be the + entry point of the module, called when the module is imported. + + The module can be split into multiple C source files if desired. Parts of the + module can also be implemented in Python. All source files should be listed in + the Makefile, by adding them to the ``SRC`` variable (see below). This includes + both C source files as well as any Python files which will be included in the + resulting .mpy file. + +* The ``Makefile`` contains the build configuration for the module and list the + source files used to build the .mpy module. It should define ``MPY_DIR`` as the + location of the MicroPython repository (to find header files, the relevant Makefile + fragment, and the ``mpy_ld.py`` tool), ``MOD`` as the name of the module, ``SRC`` + as the list of source files, optionally specify the machine architecture via ``ARCH``, + and then include ``py/dynruntime.mk``. + +Minimal example +--------------- + +This section provides a fully working example of a simple module named ``factorial``. +This module provides a single function ``factorial.factorial(x)`` which computes the +factorial of the input and returns the result. + +Directory layout:: + + factorial/ + ├── factorial.c + └── Makefile + +The file ``factorial.c`` contains: + +.. code-block:: c + + // Include the header file to get access to the MicroPython API + #include "py/dynruntime.h" + + // Helper function to compute factorial + static mp_int_t factorial_helper(mp_int_t x) { + if (x == 0) { + return 1; + } + return x * factorial_helper(x - 1); + } + + // This is the function which will be called from Python, as factorial(x) + static mp_obj_t factorial(mp_obj_t x_obj) { + // Extract the integer from the MicroPython input object + mp_int_t x = mp_obj_get_int(x_obj); + // Calculate the factorial + mp_int_t result = factorial_helper(x); + // Convert the result to a MicroPython integer object and return it + return mp_obj_new_int(result); + } + // Define a Python reference to the function above + static MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial); + + // This is the entry point and is called when the module is imported + mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) { + // This must be first, it sets up the globals dict and other things + MP_DYNRUNTIME_INIT_ENTRY + + // Make the function available in the module's namespace + mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj)); + + // This must be last, it restores the globals dict + MP_DYNRUNTIME_INIT_EXIT + } + +The file ``Makefile`` contains: + +.. code-block:: make + + # Location of top-level MicroPython directory + MPY_DIR = ../../.. + + # Name of module + MOD = factorial + + # Source files (.c or .py) + SRC = factorial.c + + # Architecture to build for (x86, x64, armv6m, armv7m, xtensa, xtensawin, rv32imc) + ARCH = x64 + + # Include to get the rules for compiling and linking the module + include $(MPY_DIR)/py/dynruntime.mk + +Compiling the module +-------------------- + +The prerequisite tools needed to build a native .mpy file are: + +* The MicroPython repository (at least the ``py/`` and ``tools/`` directories). +* CPython 3, and the library pyelftools (eg ``pip install 'pyelftools>=0.25'``). +* GNU make. +* A C compiler for the target architecture (if C source is used). +* Optionally ``mpy-cross``, built from the MicroPython repository (if .py source is used). + +Be sure to select the correct ``ARCH`` for the target you are going to run on. +Then build with:: + + $ make + +Without modifying the Makefile you can specify the target architecture via:: + + $ make ARCH=armv7m + +Module usage in MicroPython +--------------------------- + +Once the module is built there should be a file called ``factorial.mpy``. Copy +this so it is accessible on the filesystem of your MicroPython system and can be +found in the import path. The module can now be accessed in Python just like any +other module, for example:: + + import factorial + print(factorial.factorial(10)) + # should display 3628800 + +Further examples +---------------- + +See ``examples/natmod/`` for further examples which show many of the available +features of native .mpy modules. Such features include: + +* using multiple C source files +* including Python code alongside C code +* rodata and BSS data +* memory allocation +* use of floating point +* exception handling +* including external C libraries diff --git a/docs/develop/optimizations.rst b/docs/develop/optimizations.rst new file mode 100644 index 0000000000000..533dd64116f03 --- /dev/null +++ b/docs/develop/optimizations.rst @@ -0,0 +1,74 @@ +.. _optimizations: + +Optimizations +============= + +MicroPython uses several optimizations to save RAM but also ensure the efficient +execution of programs. This chapter discusses some of these optimizations. + +.. note:: + :ref:`qstr` and :ref:`maps` details other optimizations on strings and + dictionaries. + +Frozen bytecode +--------------- + +When MicroPython loads Python code from the filesystem, it first has to parse the file into +a temporary in-memory representation, and then generate bytecode for execution, both of which +are stored in the heap (in RAM). This can lead to significant amounts of memory being used. +The MicroPython cross compiler can be used to generate +a ``.mpy`` file, containing the pre-compiled bytecode for a Python module. This will still +be loaded into RAM, but it avoids the additional overhead of the parsing stage. + +As a further optimisation, the pre-compiled bytecode from a ``.mpy`` file can be "frozen" +into the firmware image as part of the main firmware compilation process, which means that +the bytecode will be executed from ROM. This can lead to a significant memory saving, and +reduce heap fragmentation. + +See :ref:`manifest` for more information. + +Variables +--------- + +MicroPython processes local and global variables differently. Global variables +are stored and looked up from a global dictionary that is allocated on the heap +(note that each module has its own separate dict, so separate namespace). +Local variables on the other hand are stored on the Python value stack, which may +live on the C stack or on the heap. They are accessed directly by their offset +within the Python stack, which is more efficient than a global lookup in a dict. + +The length of global variable names also affects how much RAM is used as identifiers +are stored in RAM. The shorter the identifier, the less memory is used. + +The other aspect is that ``const`` variables that start with an underscore are treated as +proper constants and are not allocated or added in a dictionary, hence saving some memory. +These variables use ``const()`` from the MicroPython library. Therefore: + +.. code-block:: python + + from micropython import const + + X = const(1) + _Y = const(2) + foo(X, _Y) + +Compiles to: + +.. code-block:: python + + X = 1 + foo(1, 2) + +Allocation of memory +-------------------- + +Most of the common MicroPython constructs are not allocated on the heap. +However the following are: + +- Dynamic data structures like lists, mappings, etc; +- Functions, classes and object instances; +- imports; and +- First-time assignment of global variables (to create the slot in the global dict). + +For a detailed discussion on a more user-centric perspective on optimization, +see `Maximising MicroPython speed `_ diff --git a/docs/develop/porting.rst b/docs/develop/porting.rst new file mode 100644 index 0000000000000..99725e1000bef --- /dev/null +++ b/docs/develop/porting.rst @@ -0,0 +1,305 @@ +.. _porting_to_a_board: + +Porting MicroPython +=================== + +The MicroPython project contains several ports to different microcontroller families and +architectures. The project repository has a `ports `_ +directory containing a subdirectory for each supported port. + +A port will typically contain definitions for multiple "boards", each of which is a specific piece of +hardware that that port can run on, e.g. a development kit or device. + +The `minimal port `_ is +available as a simplified reference implementation of a MicroPython port. It can run on both the +host system and an STM32F4xx MCU. + +In general, starting a port requires: + +- Setting up the toolchain (configuring Makefiles, etc). +- Implementing boot configuration and CPU initialization. +- Initialising basic drivers required for development and debugging (e.g. GPIO, UART). +- Performing the board-specific configurations. +- Implementing the port-specific modules. + +Minimal MicroPython firmware +---------------------------- + +The best way to start porting MicroPython to a new board is by integrating a minimal +MicroPython interpreter. For this walkthrough, create a subdirectory for the new +port in the ``ports`` directory: + +.. code-block:: bash + + $ cd ports + $ mkdir example_port + +The basic MicroPython firmware is implemented in the main port file, e.g ``main.c``: + +.. code-block:: c + + #include "py/builtin.h" + #include "py/compile.h" + #include "py/gc.h" + #include "py/mperrno.h" + #include "py/stackctrl.h" + #include "shared/runtime/gchelper.h" + #include "shared/runtime/pyexec.h" + + // Allocate memory for the MicroPython GC heap. + static char heap[4096]; + + int main(int argc, char **argv) { + // Initialise the MicroPython runtime. + mp_stack_ctrl_init(); + gc_init(heap, heap + sizeof(heap)); + mp_init(); + + // Start a normal REPL; will exit when ctrl-D is entered on a blank line. + pyexec_friendly_repl(); + + // Deinitialise the runtime. + gc_sweep_all(); + mp_deinit(); + return 0; + } + + // Handle uncaught exceptions (should never be reached in a correct C implementation). + void nlr_jump_fail(void *val) { + for (;;) { + } + } + + // Do a garbage collection cycle. + void gc_collect(void) { + gc_collect_start(); + gc_helper_collect_regs_and_stack(); + gc_collect_end(); + } + + // There is no filesystem so stat'ing returns nothing. + mp_import_stat_t mp_import_stat(const char *path) { + return MP_IMPORT_STAT_NO_EXIST; + } + + // There is no filesystem so opening a file raises an exception. + mp_lexer_t *mp_lexer_new_from_file(qstr filename) { + mp_raise_OSError(MP_ENOENT); + } + +We also need a Makefile at this point for the port: + +.. code-block:: Makefile + + # Include the core environment definitions; this will set $(TOP). + include ../../py/mkenv.mk + + # Include py core make definitions. + include $(TOP)/py/py.mk + include $(TOP)/extmod/extmod.mk + + # Set CFLAGS and libraries. + CFLAGS += -I. -I$(BUILD) -I$(TOP) + LIBS += -lm + + # Define the required source files. + SRC_C = \ + main.c \ + mphalport.c \ + shared/readline/readline.c \ + shared/runtime/gchelper_generic.c \ + shared/runtime/pyexec.c \ + shared/runtime/stdout_helpers.c \ + + # Define source files containung qstrs. + SRC_QSTR += shared/readline/readline.c shared/runtime/pyexec.c + + # Define the required object files. + OBJ = $(PY_CORE_O) $(addprefix $(BUILD)/, $(SRC_C:.c=.o)) + + # Define the top-level target, the main firmware. + all: $(BUILD)/firmware.elf + + # Define how to build the firmware. + $(BUILD)/firmware.elf: $(OBJ) + $(ECHO) "LINK $@" + $(Q)$(CC) $(LDFLAGS) -o $@ $^ $(LIBS) + $(Q)$(SIZE) $@ + + # Include remaining core make rules. + include $(TOP)/py/mkrules.mk + +Remember to use proper tabs to indent the Makefile. + +MicroPython Configurations +-------------------------- + +After integrating the minimal code above, the next step is to create the MicroPython +configuration files for the port. The compile-time configurations are specified in +``mpconfigport.h`` and additional hardware-abstraction functions, such as time keeping, +in ``mphalport.h``. + +The following is an example of an ``mpconfigport.h`` file: + +.. code-block:: c + + #include + + // Python internal features. + #define MICROPY_ENABLE_GC (1) + #define MICROPY_HELPER_REPL (1) + #define MICROPY_ERROR_REPORTING (MICROPY_ERROR_REPORTING_TERSE) + #define MICROPY_FLOAT_IMPL (MICROPY_FLOAT_IMPL_FLOAT) + + // Fine control over Python builtins, classes, modules, etc. + #define MICROPY_PY_ASYNC_AWAIT (0) + #define MICROPY_PY_BUILTINS_SET (0) + #define MICROPY_PY_ATTRTUPLE (0) + #define MICROPY_PY_COLLECTIONS (0) + #define MICROPY_PY_MATH (0) + #define MICROPY_PY_IO (0) + #define MICROPY_PY_STRUCT (0) + + // Type definitions for the specific machine. + + typedef intptr_t mp_int_t; // must be pointer size + typedef uintptr_t mp_uint_t; // must be pointer size + typedef long mp_off_t; + + // We need to provide a declaration/definition of alloca(). + #include + + // Define the port's name and hardware. + #define MICROPY_HW_BOARD_NAME "example-board" + #define MICROPY_HW_MCU_NAME "unknown-cpu" + + #define MP_STATE_PORT MP_STATE_VM + +This configuration file contains machine-specific configurations including aspects like if different +MicroPython features should be enabled e.g. ``#define MICROPY_ENABLE_GC (1)``. Making this Setting +``(0)`` disables the feature. + +Other configurations include type definitions, root pointers, board name, microcontroller name +etc. + +Similarly, an minimal example ``mphalport.h`` file looks like this: + +.. code-block:: c + + static inline void mp_hal_set_interrupt_char(char c) {} + +Support for standard input/output +--------------------------------- + +MicroPython requires at least a way to output characters, and to have a REPL it also +requires a way to input characters. Functions for this can be implemented in the file +``mphalport.c``, for example: + +.. code-block:: c + + #include + #include "py/mpconfig.h" + + // Receive single character, blocking until one is available. + int mp_hal_stdin_rx_chr(void) { + unsigned char c = 0; + int r = read(STDIN_FILENO, &c, 1); + (void)r; + return c; + } + + // Send the string of given length. + void mp_hal_stdout_tx_strn(const char *str, mp_uint_t len) { + int r = write(STDOUT_FILENO, str, len); + (void)r; + } + +These input and output functions have to be modified depending on the +specific board API. This example uses the standard input/output stream. + +Building and running +-------------------- + +At this stage the directory of the new port should contain:: + + ports/example_port/ + ├── main.c + ├── Makefile + ├── mpconfigport.h + ├── mphalport.c + └── mphalport.h + +The port can now be built by running ``make`` (or otherwise, depending on your system). + +If you are using the default compiler settings in the Makefile given above then this +will create an executable called ``build/firmware.elf`` which can be executed directly. +To get a functional REPL you may need to first configure the terminal to raw mode: + +.. code-block:: bash + + $ stty raw opost -echo + $ ./build/firmware.elf + +That should give a MicroPython REPL. You can then run commands like: + +.. code-block:: bash + + MicroPython v1.13 on 2021-01-01; example-board with unknown-cpu + >>> import sys + >>> sys.implementation + ('micropython', (1, 13, 0)) + >>> + +Use Ctrl-D to exit, and then run ``reset`` to reset the terminal. + +Adding a module to the port +--------------------------- + +To add a custom module like ``myport``, first add the module definition in a file +``modmyport.c``: + +.. code-block:: c + + #include "py/runtime.h" + + static mp_obj_t myport_info(void) { + mp_printf(&mp_plat_print, "info about my port\n"); + return mp_const_none; + } + static MP_DEFINE_CONST_FUN_OBJ_0(myport_info_obj, myport_info); + + static const mp_rom_map_elem_t myport_module_globals_table[] = { + { MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_myport) }, + { MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&myport_info_obj) }, + }; + static MP_DEFINE_CONST_DICT(myport_module_globals, myport_module_globals_table); + + const mp_obj_module_t myport_module = { + .base = { &mp_type_module }, + .globals = (mp_obj_dict_t *)&myport_module_globals, + }; + + MP_REGISTER_MODULE(MP_QSTR_myport, myport_module); + +You will also need to edit the Makefile to add ``modmyport.c`` to the ``SRC_C`` list, and +a new line adding the same file to ``SRC_QSTR`` (so qstrs are searched for in this new file), +like this: + +.. code-block:: Makefile + + SRC_C = \ + main.c \ + modmyport.c \ + mphalport.c \ + ... + + SRC_QSTR += modmyport.c + +If all went correctly then, after rebuilding, you should be able to import the new module: + +.. code-block:: bash + + >>> import myport + >>> myport.info() + info about my port + >>> diff --git a/docs/develop/publiccapi.rst b/docs/develop/publiccapi.rst new file mode 100644 index 0000000000000..132c7b136bc14 --- /dev/null +++ b/docs/develop/publiccapi.rst @@ -0,0 +1,25 @@ +.. _publiccapi: + +The public C API +================ + +The public C-API comprises functions defined in all C header files in the ``py/`` +directory. Most of the important core runtime C APIs are exposed in ``runtime.h`` and +``obj.h``. + +The following is an example of public API functions from ``obj.h``: + +.. code-block:: c + + mp_obj_t mp_obj_new_list(size_t n, mp_obj_t *items); + mp_obj_t mp_obj_list_append(mp_obj_t self_in, mp_obj_t arg); + mp_obj_t mp_obj_list_remove(mp_obj_t self_in, mp_obj_t value); + void mp_obj_list_get(mp_obj_t self_in, size_t *len, mp_obj_t **items); + +At its core, any functions and macros in header files make up the public +API and can be used to access very low-level details of MicroPython. Static +inline functions in header files are fine too, such functions will be +inlined in the code when used. + +Header files in the ``ports`` directory are only exposed to the functionality +specific to a given port. diff --git a/docs/develop/qstr.rst b/docs/develop/qstr.rst new file mode 100644 index 0000000000000..5d1ac5bb4a49f --- /dev/null +++ b/docs/develop/qstr.rst @@ -0,0 +1,115 @@ +.. _qstr: + +MicroPython string interning +============================ + +MicroPython uses `string interning`_ to save both RAM and ROM. This avoids +having to store duplicate copies of the same string. Primarily, this applies to +identifiers in your code, as something like a function or variable name is very +likely to appear in multiple places in the code. In MicroPython an interned +string is called a QSTR (uniQue STRing). + +A QSTR value (with type ``qstr``) is a index into a linked list of QSTR pools. +QSTRs store their length and a hash of their contents for fast comparison during +the de-duplication process. All bytecode operations that work with strings use +a QSTR argument. + +Compile-time QSTR generation +---------------------------- + +In the MicroPython C code, any strings that should be interned in the final +firmware are written as ``MP_QSTR_Foo``. At compile time this will evaluate to +a ``qstr`` value that points to the index of ``"Foo"`` in the QSTR pool. + +A multi-step process in the ``Makefile`` makes this work. In summary this +process has three parts: + +1. Find all ``MP_QSTR_Foo`` tokens in the code. + +2. Generate a static QSTR pool containing all the string data (including lengths + and hashes). + +3. Replace all ``MP_QSTR_Foo`` (via the preprocessor) with their corresponding + index. + +``MP_QSTR_Foo`` tokens are searched for in two sources: + +1. All files referenced in ``$(SRC_QSTR)``. This is all C code (i.e. ``py``, + ``extmod``, ``ports/stm32``) but not including third-party code such as + ``lib``. + +2. Additional ``$(QSTR_GLOBAL_DEPENDENCIES)`` (which includes ``mpconfig*.h``). + +*Note:* ``frozen_mpy.c`` (generated by mpy-tool.py) has its own QSTR generation +and pool. + +Some additional strings that can't be expressed using the ``MP_QSTR_Foo`` syntax +(e.g. they contain non-alphanumeric characters) are explicitly provided in +``qstrdefs.h`` and ``qstrdefsport.h`` via the ``$(QSTR_DEFS)`` variable. + +Processing happens in the following stages: + +1. ``qstr.i.last`` is the concatenation of putting every single input file + through the C pre-processor. This means that any conditionally disabled code + will be removed, and macros expanded. This means we don't add strings to the + pool that won't be used in the final firmware. Because at this stage (thanks + to the ``NO_QSTR`` macro added by ``QSTR_GEN_CFLAGS``) there is no + definition for ``MP_QSTR_Foo`` it passes through this stage unaffected. This + file also includes comments from the preprocessor that include line number + information. Note that this step only uses files that have changed, which + means that ``qstr.i.last`` will only contain data from files that have + changed since the last compile. + +2. ``qstr.split`` is an empty file created after running ``makeqstrdefs.py split`` + on qstr.i.last. It's just used as a dependency to indicate that the step ran. + This script outputs one file per input C file, ``genhdr/qstr/...file.c.qstr``, + which contains only the matched QSTRs. Each QSTR is printed as ``Q(Foo)``. + This step is necessary to combine the existing files with the new data + generated from the incremental update in ``qstr.i.last``. + +3. ``qstrdefs.collected.h`` is the output of concatenating ``genhdr/qstr/*`` + using ``makeqstrdefs.py cat``. This is now the full set of ``MP_QSTR_Foo``'s + found in the code, now formatted as ``Q(Foo)``, one-per-line, with duplicates. + This file is only updated if the set of qstrs has changed. A hash of the QSTR + data is written to another file (``qstrdefs.collected.h.hash``) which allows + it to track changes across builds. + +4. Generate an enumeration, each entry of which maps a ``MP_QSTR_Foo`` to it's corresponding index. + It concatenates ``qstrdefs.collected.h`` with ``qstrdefs*.h``, then it transforms + each line from ``Q(Foo)`` to ``"Q(Foo)"`` so they pass through the preprocessor + unchanged. Then the preprocessor is used to deal with any conditional + compilation in ``qstrdefs*.h``. Then the transformation is undone back to + ``Q(Foo)``, and saved as ``qstrdefs.preprocessed.h``. + +5. ``qstrdefs.generated.h`` is the output of ``makeqstrdata.py``. For each + ``Q(Foo)`` in qstrdefs.preprocessed.h (plus some extra hard-coded ones), it outputs + ``QDEF(MP_QSTR_Foo, (const byte*)"hash" "Foo")``. + +Then in the main compile, two things happen with ``qstrdefs.generated.h``: + +1. In qstr.h, each QDEF becomes an entry in an enum, which makes ``MP_QSTR_Foo`` + available to code and equal to the index of that string in the QSTR table. + +2. In qstr.c, the actual QSTR data table is generated as elements of the + ``mp_qstr_const_pool->qstrs``. + +.. _`string interning`: https://en.wikipedia.org/wiki/String_interning + +Run-time QSTR generation +------------------------ + +Additional QSTR pools can be created at runtime so that strings can be added to +them. For example, the code:: + + foo[x] = 3 + +Will need to create a QSTR for the value of ``x`` so it can be used by the +"load attr" bytecode. + +Also, when compiling Python code, identifiers and literals need to have QSTRs +created. Note: only literals shorter than 10 characters become QSTRs. This is +because a regular string on the heap always takes up a minimum of 16 bytes (one +GC block), whereas QSTRs allow them to be packed more efficiently into the pool. + +QSTR pools (and the underlying "chunks" that store the string data) are allocated +on-demand on the heap with a minimum size. diff --git a/docs/develop/writingtests.rst b/docs/develop/writingtests.rst new file mode 100644 index 0000000000000..a7d033f17e9b9 --- /dev/null +++ b/docs/develop/writingtests.rst @@ -0,0 +1,70 @@ +.. _writingtests: + +Writing tests +============= + +Tests in MicroPython are located at the path ``tests/``. The following is a listing of +key directories and the run-tests.py runner script: + +.. code-block:: bash + + . + ├── basics + ├── extmod + ├── float + ├── micropython + ├── run-tests.py + ... + +There are subfolders maintained to categorize the tests. Add a test by creating a new file in one of the +existing folders or in a new folder. It's also possible to make custom tests outside this tests folder, +which would be recommended for a custom port. + +For example, add the following code in a file ``print.py`` in the ``tests/unix/`` subdirectory: + +.. code-block:: python + + def print_one(): + print(1) + + print_one() + +If you run your tests, this test should appear in the test output: + +.. code-block:: bash + + $ cd ports/unix + $ make tests + skip unix/extra_coverage.py + pass unix/ffi_callback.py + pass unix/ffi_float.py + pass unix/ffi_float2.py + pass unix/print.py + pass unix/time.py + pass unix/time2.py + +Tests are run by comparing the output from the test target against the output from CPython. +So any test should use print statements to indicate test results. + +For tests that can't be compared to CPython (i.e. micropython-specific functionality), +you can provide a ``.py.exp`` file which will be used as the truth for comparison. + +The other way to run tests, which is useful when running on targets other than the Unix port, is: + +.. code-block:: bash + + $ cd tests + $ ./run-tests.py + +Then to run on a board: + +.. code-block:: bash + + $ ./run-tests.py -t /dev/ttyACM0 + +And to run only a certain set of tests (eg a directory): + +.. code-block:: bash + + $ ./run-tests.py -d basics + $ ./run-tests.py float/builtin*.py diff --git a/docs/differences/index_template.txt b/docs/differences/index_template.txt index eb8b3ba6409db..1bbfbfedce921 100644 --- a/docs/differences/index_template.txt +++ b/docs/differences/index_template.txt @@ -3,7 +3,22 @@ MicroPython differences from CPython ==================================== -The operations listed in this section produce conflicting results in MicroPython when compared to standard Python. +MicroPython implements Python 3.4 and some select features of Python 3.5 and +above. The sections below describe the current status of these features. + +.. toctree:: + + ../differences/python_35.rst + ../differences/python_36.rst + ../differences/python_37.rst + ../differences/python_38.rst + ../differences/python_39.rst + ../differences/python_310.rst + +For the features of Python that are implemented by MicroPython, there are +sometimes differences in their behaviour compared to standard Python. The +operations listed in the sections below produce conflicting results in +MicroPython when compared to standard Python. .. toctree:: :maxdepth: 2 diff --git a/docs/differences/modules_preamble.txt b/docs/differences/modules_preamble.txt new file mode 100644 index 0000000000000..1958f0084d2db --- /dev/null +++ b/docs/differences/modules_preamble.txt @@ -0,0 +1,33 @@ +.. Preamble section inserted into generated output + +Positional-only Parameters +-------------------------- + +To save code size, many functions that accept keyword arguments in CPython only accept positional arguments in MicroPython. + +MicroPython marks positional-only parameters in the same way as CPython, by inserting a ``/`` to mark the end of the positional parameters. Any function whose signature ends in ``/`` takes *only* positional arguments. For more details, see `PEP 570 `_. + +Example +~~~~~~~ + +For example, in CPython 3.4 this is the signature of the constructor ``socket.socket``:: + + socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None) + +However, the signature documented in :func:`MicroPython` is:: + + socket(af=AF_INET, type=SOCK_STREAM, proto=IPPROTO_TCP, /) + +The ``/`` at the end of the parameters indicates that they are all positional-only in MicroPython. The following code works in CPython but not in most MicroPython ports:: + + import socket + s = socket.socket(type=socket.SOCK_DGRAM) + +MicroPython will raise an exception:: + + TypeError: function doesn't take keyword arguments + +The following code will work in both CPython and MicroPython:: + + import socket + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) diff --git a/docs/differences/python_310.rst b/docs/differences/python_310.rst new file mode 100644 index 0000000000000..f9d92db655372 --- /dev/null +++ b/docs/differences/python_310.rst @@ -0,0 +1,238 @@ +.. _python_310: + +Python 3.10 +=========== + +Python 3.10.0 (final) was released on the 4 October 2021. The Features for 3.10 are +defined in `PEP 619 `_ +and a detailed description of the changes can be found in +`What's New in Python 3.10 `_. + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **New syntax features** | **Status** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 634 `_ | Structural Pattern Matching: Specification | [#spm]_ | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 635 `_ | Structural Pattern Matching: Motivation and | [#spm]_ | + | | Rationale | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 636 `_ | Structural Pattern Matching: Tutorial | [#spm]_ | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `bpo-12782 | Parenthesized context managers are now officially | | + | `_ | allowed | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **New features in the standard library** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 618 `_ | Add Optional Length-Checking To zip | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **Interpreter improvements** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 626 `_ | Precise line numbers for debugging and other tools | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **New typing features** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 604 `_ | Allow writing union types as X | Y | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 613 `_ | Explicit Type Aliases | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 612 `_ | Parameter Specification Variables | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **Important deprecations, removals or restrictions** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 644 `_ | Require OpenSSL 1.1.1 or newer | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 632 `_ | Deprecate distutils module. | Not relevant | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 623 `_ | Deprecate and prepare for the removal of the wstr | Not relevant | + | | member in PyUnicodeObject. | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 624 `_ | Remove Py_UNICODE encoder APIs | Not relevant | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 597 `_ | Add optional EncodingWarning | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-------------------------------------------------------------------------------------------------------------+---------------+ + | The :class:`int` type has a new method :meth:`int.bit_count`, returning the | | + | number of ones in the binary expansion of a given integer, also known | | + | as the population count. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | The views returned by :meth:`dict.keys`, :meth:`dict.values` and | | + | :meth:`dict.items` now all have a ``mapping`` attribute that gives a | | + | :class:`types.MappingProxyType` object wrapping the original | | + | dictionary. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | :pep:`618`: The :func:`zip` function now has an optional ``strict`` flag, used | | + | to require that all the iterables have an equal length. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Builtin and extension functions that take integer arguments no longer accept | | + | :class:`~decimal.Decimal`\ s, :class:`~fractions.Fraction`\ s and other | | + | objects that can be converted to integers only with a loss (e.g. that have | | + | the :meth:`~object.__int__` method but do not have the | | + | :meth:`~object.__index__` method). | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | If :func:`object.__ipow__` returns :const:`NotImplemented`, the operator will | | + | correctly fall back to :func:`object.__pow__` and :func:`object.__rpow__` as expected. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Assignment expressions can now be used unparenthesized within set literals | | + | and set comprehensions, as well as in sequence indexes (but not slices). | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Functions have a new ``__builtins__`` attribute which is used to look for | | + | builtin symbols when a function is executed, instead of looking into | | + | ``__globals__['__builtins__']``. The attribute is initialized from | | + | ``__globals__["__builtins__"]`` if it exists, else from the current builtins. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Two new builtin functions -- :func:`aiter` and :func:`anext` have been added | | + | to provide asynchronous counterparts to :func:`iter` and :func:`next`, | | + | respectively. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Static methods (:func:`@staticmethod `) and class methods | | + | (:func:`@classmethod `) now inherit the method attributes | | + | (``__module__``, ``__name__``, ``__qualname__``, ``__doc__``, | | + | ``__annotations__``) and have a new ``__wrapped__`` attribute. | | + | Moreover, static methods are now callable as regular functions. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Annotations for complex targets (everything beside ``simple name`` targets | | + | defined by :pep:`526`) no longer cause any runtime effects with ``from __future__ import annotations``. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Class and module objects now lazy-create empty annotations dicts on demand. | | + | The annotations dicts are stored in the object’s ``__dict__`` for | | + | backwards compatibility. This improves the best practices for working | | + | with ``__annotations__``. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Annotations consist of ``yield``, ``yield from``, ``await`` or named expressions | | + | are now forbidden under ``from __future__ import annotations`` due to their side | | + | effects. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Usage of unbound variables, ``super()`` and other expressions that might | | + | alter the processing of symbol table as annotations are now rendered | | + | effectless under ``from __future__ import annotations``. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Hashes of NaN values of both :class:`float` type and | | + | :class:`decimal.Decimal` type now depend on object identity. Formerly, they | | + | always hashed to ``0`` even though NaN values are not equal to one another. | | + | This caused potentially quadratic runtime behavior due to excessive hash | | + | collisions when creating dictionaries and sets containing multiple NaNs. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | A :exc:`SyntaxError` (instead of a :exc:`NameError`) will be raised when deleting | | + | the :const:`__debug__` constant. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | :exc:`SyntaxError` exceptions now have ``end_lineno`` and | | + | ``end_offset`` attributes. They will be ``None`` if not determined. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `asyncio `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add missing :meth:`~asyncio.events.AbstractEventLoop.connect_accepted_socket` | | + | method. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `array `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The :meth:`~array.array.index` method of :class:`array.array` now has | | + | optional *start* and *stop* parameters. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `gc `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add audit hooks for :func:`gc.get_objects`, :func:`gc.get_referrers` and | | + | :func:`gc.get_referents`. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `hashlib `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The hashlib module requires OpenSSL 1.1.1 or newer. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The hashlib module has preliminary support for OpenSSL 3.0.0. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The pure-Python fallback of :func:`~hashlib.pbkdf2_hmac` is deprecated. In | | + | the future PBKDF2-HMAC will only be available when Python has been built with | | + | OpenSSL support. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `os `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :func:`os.cpu_count()` support for VxWorks RTOS. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add a new function :func:`os.eventfd` and related helpers to wrap the | | + | ``eventfd2`` syscall on Linux. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :func:`os.splice()` that allows to move data between two file | | + | descriptors without copying between kernel address space and user | | + | address space, where one of the file descriptors must refer to a | | + | pipe. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :data:`~os.O_EVTONLY`, :data:`~os.O_FSYNC`, :data:`~os.O_SYMLINK` | | + | and :data:`~os.O_NOFOLLOW_ANY` for macOS. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `platform `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :func:`platform.freedesktop_os_release()` to retrieve operation system | | + | identification from `freedesktop.org os-release | | + | `_ standard file. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `socket `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The exception :exc:`socket.timeout` is now an alias of :exc:`TimeoutError`. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add option to create MPTCP sockets with ``IPPROTO_MPTCP``. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add ``IP_RECVTOS`` option to receive the type of service (ToS) or DSCP/ECN fields. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `ssl `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module requires OpenSSL 1.1.1 or newer. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module has preliminary support for OpenSSL 3.0.0 and new option | | + | :data:`~ssl.OP_IGNORE_UNEXPECTED_EOF`. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Deprecated function and use of deprecated constants now result in | | + | a :exc:`DeprecationWarning`. :attr:`ssl.SSLContext.options` has | | + | :data:`~ssl.OP_NO_SSLv2` and :data:`~ssl.OP_NO_SSLv3` set by default and | | + | therefore cannot warn about setting the flag again. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module now has more secure default settings. Ciphers without forward | | + | secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits | | + | weak RSA, DH, and ECC keys with less than 112 bits of security. | | + | :class:`~ssl.SSLContext` defaults to minimum protocol version TLS 1.2. | | + | Settings are based on Hynek Schlawack's research. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer | | + | officially supported. Python does not block them actively. However | | + | OpenSSL build options, distro configurations, vendor patches, and cipher | | + | suites may prevent a successful handshake. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add a *timeout* parameter to the :func:`ssl.get_server_certificate` function. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module uses heap-types and multi-phase initialization. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | A new verify flag :data:`~ssl.VERIFY_X509_PARTIAL_CHAIN` has been added. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `sys `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :data:`sys.orig_argv` attribute: the list of the original command line | | + | arguments passed to the Python executable. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :data:`sys.stdlib_module_names`, containing the list of the standard library | | + | module names. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `_thread `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | :func:`_thread.interrupt_main` now takes an optional signal number to | | + | simulate (the default is still :data:`signal.SIGINT`). | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + +.. rubric:: Notes + +.. [#spm] The structural pattern matching feature is discussed in `issue #7847 `_. + diff --git a/docs/differences/python_35.rst b/docs/differences/python_35.rst new file mode 100644 index 0000000000000..0fdc6121a1dd0 --- /dev/null +++ b/docs/differences/python_35.rst @@ -0,0 +1,193 @@ +.. _python_35: + +Python 3.5 +========== + +Below is a list of finalised/accepted PEPs for Python 3.5 grouped into their impact to MicroPython. + +.. table:: + :widths: 30 50 20 + + +--------------------------------------------------------------------------------------------------------------+--------------------+ + | **Extensions to the syntax** | **Status** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 448 `_ | Additional unpacking generalizations | Partial | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 465 `_ | A new matrix multiplication operator | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 492 `_ | Coroutines with ``async`` and ``await`` syntax | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | **Extensions and changes to runtime** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 461 `_ | % formatting for binary strings | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 475 `_ | Retrying system calls that fail with ``EINTR`` | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 479 `_ | Change ``StopIteration`` handling inside generators | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | **Standard library changes** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 471 `_ | ``os.scandir()`` | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 485 `_ | ``math.isclose()``, a function for testing | Complete | + | | approximate equality | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | **Miscellaneous changes** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 441 `_ | Improved Python zip application support | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 486 `_ | Make the Python Launcher aware of virtual | Not relevant | + | | environments | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 484 `_ | Type hints (advisory only) | Complete [#fth]_ | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 488 `_ | Elimination of PYO files | Not relevant | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 489 `_ | Redesigning extension module loading | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Added the *namereplace* error handlers. The *backslashreplace* error handlers now work with decoding and | | + | translating. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Property docstrings are now writable. This is especially useful for collections.namedtuple() docstrings | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Circular imports involving relative imports are now supported. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + + +New Modules: + +* `typing `_ + +* `zipzap `_ + + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `collections `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *OrderedDict* class is now implemented in C, which makes it 4 to 100 times faster. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | *OrderedDict.items()* , *OrderedDict.keys()* , *OrderedDict.values()* views now support reversed() | | + | iteration. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The deque class now defines *index()*, *insert()*, and *copy()*, and supports the + and * operators. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Docstrings produced by namedtuple() can now be updated. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The UserString class now implements the *__getnewargs__()*, *__rmod__()*, *casefold()*, *format_map()*, | | + | *isprintable()*, and *maketrans()* methods to match the corresponding methods of str. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `heapq `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Element comparison in *merge()* can now be customized by passing a key function in a new optional key | | + | keyword argument, and a new optional *reverse* keyword argument can be used to reverse element comparison | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `io `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *BufferedIOBase.readinto1()* method, that uses at most one call to the underlying raw stream's | | + | *RawIOBase.read()* or *RawIOBase.readinto()* methods | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `json `_ | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | JSON decoder now raises JSONDecodeError instead of ValueError to provide better context information about | | + | the error. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `math `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Two new constants have been added to the math module: *inf* and *nan*. | Complete | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new function *isclose()* provides a way to test for approximate equality. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *gcd()* function has been added. The *fractions.gcd()* function is now deprecated. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `os `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The new *scandir()* function returning an iterator of DirEntry objects has been added. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *urandom()* function now uses the *getrandom()* syscall on Linux 3.17 or newer, and *getentropy()* on | | + | OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file | | + | descriptor exhaustion. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | New *get_blocking()* and *set_blocking()* functions allow getting and setting a file descriptor's blocking| | + | mode (O_NONBLOCK.) | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | There is a new *os.path.commonpath()* function returning the longest common sub-path of each passed | | + | pathname | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `re `_ | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | References and conditional references to groups with fixed length are now allowed in lookbehind assertions| | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The number of capturing groups in regular expressions is no longer limited to 100. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *sub()* and *subn()* functions now replace unmatched groups with empty strings instead of raising an | | + | exception. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *re.error* exceptions have new attributes, msg, pattern, pos, lineno, and colno, that provide better | | + | context information about the error | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `socket `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Functions with timeouts now use a monotonic clock, instead of a system clock. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *socket.sendfile()* method allows sending a file over a socket by using the high-performance | | + | *os.sendfile()* function on UNIX, resulting in uploads being from 2 to 3 times faster than when using | | + | plain *socket.send()* | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *socket.sendall()* method no longer resets the socket timeout every time bytes are received or sent. | | + | The socket timeout is now the maximum total duration to send all data. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The backlog argument of the *socket.listen()* method is now optional. By default it is set to SOMAXCONN or| Complete | + | to 128, whichever is less. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `ssl `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Memory BIO Support | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Application-Layer Protocol Negotiation Support | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | There is a new *SSLSocket.version()* method to query the actual protocol version in use. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The SSLSocket class now implements a *SSLSocket.sendfile()* method. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *SSLSocket.send()* method now raises either the *ssl.SSLWantReadError* or *ssl.SSLWantWriteError* | | + | exception on a non-blocking socket if the operation would block. Previously, it would return 0. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *cert_time_to_seconds()* function now interprets the input time as UTC and not as local time, per RFC | | + | 5280. Additionally, the return value is always an int. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | New *SSLObject.shared_ciphers()* and *SSLSocket.shared_ciphers()* methods return the list of ciphers sent | | + | by the client during the handshake. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *SSLSocket.do_handshake()*, *SSLSocket.read()*, *SSLSocket.shutdown()*, and *SSLSocket.write()* | | + | methods of the SSLSocket class no longer reset the socket timeout every time bytes are received or sent. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *match_hostname()* function now supports matching of IP addresses. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `sys `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *set_coroutine_wrapper()* function allows setting a global hook that will be called whenever a | | + | coroutine object is created by an async def function. A corresponding *get_coroutine_wrapper()* can be | | + | used to obtain a currently set wrapper. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *is_finalizing()* function can be used to check if the Python interpreter is shutting down. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `time `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *monotonic()* function is now always available | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + +.. rubric:: Notes + +.. [#fth] The MicroPython parser correct ignores all type hints. However, the ``typing`` module is not built-in. diff --git a/docs/differences/python_36.rst b/docs/differences/python_36.rst new file mode 100644 index 0000000000000..3315b0594dafc --- /dev/null +++ b/docs/differences/python_36.rst @@ -0,0 +1,200 @@ +.. _python_36: + +Python 3.6 +========== + +Python 3.6 beta 1 was released on 12 Sep 2016, and a summary of the new features can be found here: + +.. table:: + :widths: 30 50 20 + + +-----------------------------------------------------------------------------------------------------------+-----------------+ + | **New Syntax Features** | **Status** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 498 `_ | Literal String Formatting | Complete | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 515 `_ | Underscores in Numeric Literals | Complete | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 525 `_ | Asynchronous Generators | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 526 `_ | Syntax for Variable Annotations (provisional) | Complete | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 530 `_ | Asynchronous Comprehensions | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **New Built-in Features** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 468 `_ | Preserving the order of *kwargs* in a function | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 487 `_ | Simpler customization of class creation | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 520 `_ | Preserving Class Attribute Definition Order | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **Standard Library Changes** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 495 `_ | Local Time Disambiguation | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 506 `_ | Adding A Secrets Module To The Standard Library | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 519 `_ | Adding a file system path protocol | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **CPython Internals** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 509 `_ | Add a private version to dict | Won't do | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 523 `_ | Adding a frame evaluation API to CPython | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **Linux/Window Changes** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 524 `_ | Make ``os.urandom()`` blocking on Linux | | + | | (during system startup) | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 528 `_ | Change Windows console encoding to UTF-8 | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 529 `_ | Change Windows filesystem encoding to UTF-8 | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-------------------------------------------------------------------------------------------------------------+---------------+ + | A *global* or *nonlocal* statement must now textually appear before the first use of the affected name in | | + | the same scope. Previously this was a SyntaxWarning. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | It is now possible to set a special method to None to indicate that the corresponding operation is not | | + | available. For example, if a class sets *__iter__()* to *None* , the class is not iterable. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Long sequences of repeated traceback lines are now abbreviated as *[Previous line repeated {count} more | | + | times]* | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Import now raises the new exception *ModuleNotFoundError* when it cannot find a module. Code that currently | | + | checks for ImportError (in try-except) will still work. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Class methods relying on zero-argument *super()* will now work correctly when called from metaclass methods | | + | during class creation. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `array `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Exhausted iterators of *array.array* will now stay exhausted even if the iterated array is extended. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `binascii `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline | Complete | + | character is appended to the return value | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `cmath `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new cmath.tau (τ) constant has been added | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | New constants: *cmath.inf* and *cmath.nan* to match *math.inf* and *math.nan* , and also *cmath.infj* and | | + | *cmath.nanj* to match the format used by complex repr | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `collections `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new Collection abstract base class has been added to represent sized iterable container classes | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *Reversible* abstract base class represents iterable classes that also provide the *__reversed__()* | | + | method. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *AsyncGenerator* abstract base class represents asynchronous generators. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *namedtuple()* function now accepts an optional keyword argument module, which, when specified, is used | | + | for the *__module__* attribute of the returned named tuple class. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The verbose and rename arguments for *namedtuple()* are now keyword-only. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Recursive *collections.deque* instances can now be pickled. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `hashlib `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | BLAKE2 hash functions were added to the module. *blake2b()* and *blake2s()* are always available and support | | + | the full feature set of BLAKE2. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The SHA-3 hash functions *sha3_224()*, *sha3_256()*, *sha3_384()*, *sha3_512()*, and *SHAKE* hash functions | | + | *shake_128()* and *shake_256()* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The password-based key derivation function *scrypt()* is now available with OpenSSL 1.1.0 and newer. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `json `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | *json.load()* and *json.loads()* now support binary input. Encoded JSON should be represented using either | | + | UTF-8, UTF-16, or UTF-32. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `math `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new math.tau (τ) constant has been added | Complete | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `os `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | A new *close()* method allows explicitly closing a *scandir()* iterator. The *scandir()* iterator now | | + | supports the context manager protocol. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | On Linux, *os.urandom()* now blocks until the system urandom entropy pool is initialized to increase the | | + | security. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The Linux *getrandom()* syscall (get random bytes) is now exposed as the new *os.getrandom()* function. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `re `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Added support of modifier spans in regular expressions. Examples: *'(?i:p)ython'* matches 'python' and | | + | 'Python', but not 'PYTHON'; *'(?i)g(?-i:v)r'* matches *'GvR'* and *'gvr'*, but not *'GVR'*. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Match object groups can be accessed by *__getitem__*, which is equivalent to *group()*. So *mo['name']* is | | + | now equivalent to *mo.group('name')*. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Match objects now support index-like objects as group indices. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `socket `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *ioctl()* function now supports the *SIO_LOOPBACK_FAST_PATH* control code. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *getsockopt()* constants *SO_DOMAIN* , *SO_PROTOCOL*, *SO_PEERSEC* , and *SO_PASSSEC* are now supported. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *setsockopt()* now supports the *setsockopt(level, optname, None, optlen: int)* form. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The socket module now supports the address family *AF_ALG* to interface with Linux Kernel crypto API. | | + | *ALG_*, *SOL_ALG* and *sendmsg_afalg()* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | New Linux constants *TCP_USER_TIMEOUT* and *TCP_CONGESTION* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `ssl `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | 3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | *SSLContext* has better default configuration for options and ciphers. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | SSL session can be copied from one client-side connection to another with the new *SSLSession* class. TLS | | + | session resumption can speed up the initial handshake, reduce latency and improve performance. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *get_ciphers()* method can be used to get a list of enabled ciphers in order of cipher priority. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | All constants and flags have been converted to *IntEnum* and *IntFlags*. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Server and client-side specific TLS protocols for *SSLContext* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Added *SSLContext.post_handshake_auth* to enable and *ssl.SSLSocket.verify_client_post_handshake()* to | | + | initiate TLS 1.3 post-handshake authentication. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `struct `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | now supports IEEE 754 half-precision floats via the 'e' format specifier. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `sys `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *getfilesystemencodeerrors()* function returns the name of the error mode used to convert between | | + | Unicode filenames and bytes filenames. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `zlib `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *compress()* and *decompress()* functions now accept keyword arguments | | + +--------------------------------------------------------------------------------------------------------------+----------------+ diff --git a/docs/differences/python_37.rst b/docs/differences/python_37.rst new file mode 100644 index 0000000000000..2b9a089217f6d --- /dev/null +++ b/docs/differences/python_37.rst @@ -0,0 +1,108 @@ +.. _python_37: + +Python 3.7 +========== + +New Features: + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | **Feature** | **Status** | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 538 `_ | Coercing the legacy C locale to a UTF-8 based | | + | | locale | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 539 `_ | A New C-API for Thread-Local Storage in CPython | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 540 `_ | UTF-8 mode | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 552 `_ | Deterministic pyc | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 553 `_ | Built-in ``breakpoint()`` | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 557 `_ | Data Classes | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 560 `_ | Core support for typing module and generic types | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 562 `_ | Module ``__getattr__`` and ``__dir__`` | Partial | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 563 `_ | Postponed Evaluation of Annotations | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 564 `_ | Time functions with nanosecond resolution | Partial [#ftimenanosec]_ | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 565 `_ | Show DeprecationWarning in ``__main__`` | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 567 `_ | Context Variables | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``async`` and ``await`` are now reserved keywords | Complete | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``dict`` objects must preserve insertion-order | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | More than 255 arguments can now be passed to a function; a function can now have more than 255 parameters | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``bytes.fromhex()`` and ``bytearray.fromhex()`` now ignore all ASCII whitespace, not only spaces | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``str``, ``bytes``, and ``bytearray`` gained support for the new ``isascii()`` method, which can be used to | | + | test if a string or bytes contain only the ASCII characters | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``ImportError`` now displays module name and module ``__file__`` path when ``from ... import ...`` fails | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | Circular imports involving absolute imports with binding a submodule to a name are now supported | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than ``format(str(self), '')`` | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | In order to better support dynamic creation of stack traces, ``types.TracebackType`` can now be | | + | instantiated from Python code, and the ``tb_next`` attribute on tracebacks is now writable | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | When using the ``-m`` switch, ``sys.path[0]`` is now eagerly expanded to the full starting directory path, | | + | rather than being left as the empty directory (which allows imports from the current working directory | | + | at the time when an import occurs) | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | The new ``-X importtime`` option or the ``PYTHONPROFILEIMPORTTIME`` environment variable can be used to | | + | show the timing of each module import | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +------------------------------------------------------------------------------------------------------------+----------------+ + | `asyncio `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | Too many to list | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `gc `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | New features include *gc.freeze()*, *gc.unfreeze()*, *gc-get_freeze_count* | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `math `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | math.remainder() added to implement IEEE 754-style remainder | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `re `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | A number of tidy up features including better support for splitting on empty strings and copy support for | | + | compiled expressions and match objects | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `sys `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | sys.breakpointhook() added. sys.get(/set)_coroutine_origin_tracking_depth() added | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `time `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | Mostly updates to support nanosecond resolution in PEP564, see above | | + +------------------------------------------------------------------------------------------------------------+----------------+ + +.. rubric:: Notes + +.. [#ftimenanosec] Only :func:`time.time_ns` is implemented. diff --git a/docs/differences/python_38.rst b/docs/differences/python_38.rst new file mode 100644 index 0000000000000..32fd68d2caeac --- /dev/null +++ b/docs/differences/python_38.rst @@ -0,0 +1,127 @@ +.. _python_38: + +Python 3.8 +========== + +Python 3.8.0 (final) was released on the 14 October 2019. The Features for 3.8 +are defined in `PEP 569 `_ and +a detailed description of the changes can be found in `What's New in Python +3.8. `_ + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | **Features** | **Status** | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 570 `_ | Positional-only arguments | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 572 `_ | Assignment Expressions | Complete | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 574 `_ | Pickle protocol 5 with out-of-band data | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 578 `_ | Runtime audit hooks | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 587 `_ | Python Initialization Configuration | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 590 `_ | Vectorcall: a fast calling protocol for CPython | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | **Miscellaneous** | + +------------------------------------------------------------------------------------------------------------+---------------+ + | f-strings support = for self-documenting expressions and debugging | Complete | + +------------------------------------------------------------------------------------------------------------+---------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +------------------------------------------------------------------------------------------------------------+-------------+ + | A *continue* statement was illegal in the *finally* clause due to a problem with the implementation. In | Complete | + | Python 3.8 this restriction was lifted | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The *bool*, *int* , and *fractions.Fraction* types now have an *as_integer_ratio()* method like that found | | + | in *float* and *decimal.Decimal* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Constructors of *int*, *float* and *complex* will now use the *__index__()* special method, if available | | + | and the corresponding method *__int__()*, *__float__()* or *__complex__()* is not available | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added support of *\N{name}* escapes in regular expressions | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Dict and dictviews are now iterable in reversed insertion order using *reversed()* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The syntax allowed for keyword names in function calls was further restricted. In particular, | | + | f((keyword)=arg) is no longer allowed | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Generalized iterable unpacking in yield and return statements no longer requires enclosing parentheses | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | When a comma is missed in code such as [(10, 20) (30, 40)], the compiler displays a SyntaxWarning with a | | + | helpful suggestion | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Arithmetic operations between subclasses of *datetime.date* or *datetime.datetime* and *datetime.timedelta*| | + | objects now return an instance of the subclass, rather than the base class | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | When the Python interpreter is interrupted by *Ctrl-C (SIGINT)* and the resulting *KeyboardInterrupt* | | + | exception is not caught, the Python process now exits via a SIGINT signal or with the correct exit code | | + | such that the calling process can detect that it died due to a *Ctrl-C* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Some advanced styles of programming require updating the *types.CodeType* object for an existing function | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | For integers, the three-argument form of the pow() function now permits the exponent to be negative in the | | + | case where the base is relatively prime to the modulus | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Dict comprehensions have been synced-up with dict literals so that the key is computed first and the value | | + | second | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The *object.__reduce__()* method can now return a tuple from two to six elements long | | + +------------------------------------------------------------------------------------------------------------+-------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +------------------------------------------------------------------------------------------------------------+-------------+ + | `asyncio `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | *asyncio.run()* has graduated from the provisional to stable API | Complete | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Running *python -m asyncio* launches a natively async REPL | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The exception *asyncio.CancelledError* now inherits from *BaseException* rather than *Exception* and no | Complete | + | longer inherits from *concurrent.futures.CancelledError* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added *asyncio.Task.get_coro()* for getting the wrapped coroutine within an *asyncio.Task* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Asyncio tasks can now be named, either by passing the name keyword argument to *asyncio.create_task()* or | | + | the *create_task()* event loop method, or by calling the *set_name()* method on the task object | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added support for Happy Eyeballs to *asyncio.loop.create_connection()*. To specify the behavior, two new | | + | parameters have been added: *happy_eyeballs_delay* and interleave. | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | `gc `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | *get_objects()* can now receive an optional generation parameter indicating a generation to get objects | | + | from. (Note, though, that while *gc* is a built-in, *get_objects()* is not implemented for MicroPython) | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | `math `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added new function *math.dist()* for computing Euclidean distance between two points | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Expanded the *math.hypot()* function to handle multiple dimensions | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added new function, *math.prod()*, as analogous function to *sum()* that returns the product of a "start" | | + | value (default: 1) times an iterable of numbers | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added two new combinatoric functions *math.perm()* and *math.comb()* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added a new function *math.isqrt()* for computing accurate integer square roots without conversion to | | + | floating point | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The function *math.factorial()* no longer accepts arguments that are not int-like | Complete | + +------------------------------------------------------------------------------------------------------------+-------------+ + | `sys `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Add new *sys.unraisablehook()* function which can be overridden to control how "unraisable exceptions" | | + | are handled | | + +------------------------------------------------------------------------------------------------------------+-------------+ diff --git a/docs/differences/python_39.rst b/docs/differences/python_39.rst new file mode 100644 index 0000000000000..f68abc50740ac --- /dev/null +++ b/docs/differences/python_39.rst @@ -0,0 +1,134 @@ +.. _python_39: + +Python 3.9 +========== + +Python 3.9.0 (final) was released on the 5th October 2020. The Features for 3.9 are +defined in `PEP 596 `_ +and a detailed description of the changes can be found in +`What's New in Python 3.9 `_ + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | **Features** | | **Status** | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 573 `_ | Fast access to module state from methods of C | Not relevant | + | | extension types | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 584 `_ | Union operators added to dict | Complete [#pep584]_ | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 585 `_ | Type hinting generics in standard collections | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 593 `_ | Flexible function and variable annotations | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 602 `_ | CPython adopts an annual release cycle. Instead of | Not relevant | + | | annual, aiming for two month release cycle | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 614 `_ | Relaxed grammar restrictions on decorators | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 615 `_ | The IANA Time Zone Database is now present in the | | + | | standard library in the zoneinfo module | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 616 `_ | String methods to remove prefixes and suffixes | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 617 `_ | CPython now uses a new parser based on PEG | Not relevant | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-------------------------------------------------------------------------------------------------------------+---------------+ + | *__import__()* now raises *ImportError* instead of *ValueError* | Complete | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Python now gets the absolute path of the script filename specified on the command line (ex: *python3* | | + | *script.py*): the *__file__* attribute of the *__main__* module became an absolute path, rather than a | | + | relative path | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | By default, for best performance, the errors argument is only checked at the first encoding/decoding error | | + | and the encoding argument is sometimes ignored for empty strings | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | *"".replace("", s, n)* now returns *s* instead of an empty string for all non-zero n. It is now consistent | | + | with *"".replace("", s)* | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Any valid expression can now be used as a decorator. Previously, the grammar was much more restrictive | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Parallel running of *aclose()* / *asend()* / *athrow()* is now prohibited, and *ag_running* now reflects | | + | the actual running status of the async generator | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Unexpected errors in calling the *__iter__* method are no longer masked by TypeError in the in operator and | | + | functions contains(), indexOf() and countOf() of the operator module | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Unparenthesized lambda expressions can no longer be the expression part in an if clause in comprehensions | | + | and generator expressions | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `asyncio `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Due to significant security concerns, the reuse_address parameter of *asyncio.loop.create_datagram_endpoint()*| | + | is no longer supported | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new coroutine *shutdown_default_executor()* that schedules a shutdown for the default executor that | | + | waits on the *ThreadPoolExecutor* to finish closing. Also, *asyncio.run()* has been updated to use the new | | + | coroutine. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *asyncio.PidfdChildWatcher*, a Linux-specific child watcher implementation that polls process file | | + | descriptors | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | added a new *coroutine asyncio.to_thread()* | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | When cancelling the task due to a timeout, *asyncio.wait_for()* will now wait until the cancellation is | | + | complete also in the case when timeout is <= 0, like it does with positive timeouts | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | *asyncio* now raises *TyperError* when calling incompatible methods with an *ssl.SSLSocket* socket | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `gc `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Garbage collection does not block on resurrected objects | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new function *gc.is_finalized()* to check if an object has been finalized by the garbage collector | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `math `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Expanded the *math.gcd()* function to handle multiple arguments. Formerly, it only supported two arguments | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *math.lcm()*: return the least common multiple of specified arguments | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *math.nextafter()*: return the next floating-point value after x towards y | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *math.ulp()*: return the value of the least significant bit of a float | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `os `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Exposed the Linux-specific *os.pidfd_open()* and *os.P_PIDFD* | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The *os.unsetenv()* function is now also available on Windows | Complete | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The *os.putenv()* and *os.unsetenv()* functions are now always available | Complete | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *os.waitstatus_to_exitcode()* function: convert a wait status to an exit code | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `random `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new *random.Random.randbytes* method: generate random bytes | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `sys `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new *sys.platlibdir* attribute: name of the platform-specific library directory | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Previously, *sys.stderr* was block-buffered when non-interactive. Now stderr defaults to always being | | + | line-buffered | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + +.. rubric:: Notes + +.. [#pep584] PEP 584 ``dict`` union operator is only available on MicroPython builds with ``MICROPY_CPYTHON_COMPAT`` enabled. diff --git a/docs/esp32/general.rst b/docs/esp32/general.rst new file mode 100644 index 0000000000000..8137c042d9214 --- /dev/null +++ b/docs/esp32/general.rst @@ -0,0 +1,64 @@ +.. _esp32_general: + +General information about the ESP32 port +======================================== + +The ESP32 is a popular WiFi and Bluetooth enabled System-on-Chip (SoC) by +Espressif Systems. + +Multitude of boards +------------------- + +There is a multitude of modules and boards from different sources which carry +the ESP32 chip. MicroPython tries to provide a generic port which would run on +as many boards/modules as possible, but there may be limitations. Espressif +development boards are taken as reference for the port (for example, testing is +performed on them). For any board you are using please make sure you have a +datasheet, schematics and other reference materials so you can look up any +board-specific functions. + +To make a generic ESP32 port and support as many boards as possible the +following design and implementation decision were made: + +* GPIO pin numbering is based on ESP32 chip numbering. Please have the manual/pin + diagram of your board at hand to find correspondence between your board pins and + actual ESP32 pins. +* All pins are supported by MicroPython but not all are usable on any given board. + For example pins that are connected to external SPI flash should not be used, + and a board may only expose a certain selection of pins. + + +Technical specifications and SoC datasheets +------------------------------------------- + +The datasheets and other reference material for ESP32 chip are available +from the vendor site: https://www.espressif.com/en/support/download/documents?keys=esp32 . +They are the primary reference for the chip technical specifications, capabilities, +operating modes, internal functioning, etc. + +For your convenience, some of technical specifications are provided below: + +* Architecture: Xtensa Dual-Core 32-bit LX6 +* CPU frequency: up to 240MHz +* Total RAM available: 528KB (part of it reserved for system) +* BootROM: 448KB +* Internal FlashROM: none +* External FlashROM: code and data, via SPI Flash; usual size 4MB +* GPIO: 34 (GPIOs are multiplexed with other functions, including + external FlashROM, UART, etc.) +* UART: 3 RX/TX UART (no hardware handshaking), one TX-only UART +* SPI: 4 SPI interfaces (one used for FlashROM) +* I2C: 2 I2C (bitbang implementation available on any pins) +* I2S: 2 +* ADC: 12-bit SAR ADC up to 18 channels +* DAC: 2 8-bit DACs +* RMT: 8 channels allowing accurate pulse transmit/receive +* Programming: using BootROM bootloader from UART - due to external FlashROM + and always-available BootROM bootloader, the ESP32 is not brickable + +For more information see the ESP32 datasheet: https://www.espressif.com/sites/default/files/documentation/esp32_datasheet_en.pdf + +MicroPython is implemented on top of the ESP-IDF, Espressif's development +framework for the ESP32. This is a FreeRTOS based system. See the +`ESP-IDF Programming Guide `_ +for details. diff --git a/docs/esp32/img/esp32.jpg b/docs/esp32/img/esp32.jpg new file mode 100644 index 0000000000000..a96ddcbe6a39a Binary files /dev/null and b/docs/esp32/img/esp32.jpg differ diff --git a/docs/esp32/quickref.rst b/docs/esp32/quickref.rst new file mode 100644 index 0000000000000..c394414a76f51 --- /dev/null +++ b/docs/esp32/quickref.rst @@ -0,0 +1,962 @@ +.. _esp32_quickref: + +Quick reference for the ESP32 +============================= + +.. image:: img/esp32.jpg + :alt: ESP32 board + :width: 640px + +The Espressif ESP32 Development Board (image attribution: Adafruit). + +Below is a quick reference for ESP32-based boards. If it is your first time +working with this board it may be useful to get an overview of the microcontroller: + +.. toctree:: + :maxdepth: 1 + + general.rst + tutorial/index.rst + +Note that there are several varieties of ESP32 -- ESP32, ESP32C3, ESP32C6, ESP32S2, ESP32S3 -- +supported by MicroPython, with some differences in functionality between them. + +Installing MicroPython +---------------------- + +See the corresponding section of tutorial: :ref:`esp32_intro`. It also includes +a troubleshooting subsection. + +General board control +--------------------- + +The MicroPython REPL is on UART0 (GPIO1=TX, GPIO3=RX) at baudrate 115200. +Tab-completion is useful to find out what methods an object has. +Paste mode (ctrl-E) is useful to paste a large slab of Python code into +the REPL. + +The :mod:`machine` module:: + + import machine + + machine.freq() # get the current frequency of the CPU + machine.freq(240000000) # set the CPU frequency to 240 MHz + +The :mod:`esp` module:: + + import esp + + esp.osdebug(None) # turn off vendor O/S debugging messages + esp.osdebug(0) # redirect vendor O/S debugging messages to UART(0) + + # low level methods to interact with flash storage + esp.flash_size() + esp.flash_user_start() + esp.flash_erase(sector_no) + esp.flash_write(byte_offset, buffer) + esp.flash_read(byte_offset, buffer) + +The :mod:`esp32` module:: + + import esp32 + + esp32.raw_temperature() # read the internal temperature of the MCU, in Fahrenheit + esp32.ULP() # access to the Ultra-Low-Power Co-processor, not on ESP32C3/C6 + +Note that the temperature sensor in the ESP32 will typically read higher than +ambient due to the IC getting warm while it runs. This effect can be minimised +by reading the temperature sensor immediately after waking up from sleep. + +ESP32C3, ESP32C6, ESP32S2, and ESP32S3 also have an internal temperature sensor available. +It is implemented a bit differently to the ESP32 and returns the temperature in +Celsius:: + + esp32.mcu_temperature() # read the internal temperature of the MCU, in Celsius + +Networking +---------- + +WLAN +^^^^ + +The :class:`network.WLAN` class in the :mod:`network` module:: + + import network + + wlan = network.WLAN() # create station interface (the default, see below for an access point interface) + wlan.active(True) # activate the interface + wlan.scan() # scan for access points + wlan.isconnected() # check if the station is connected to an AP + wlan.connect('ssid', 'key') # connect to an AP + wlan.config('mac') # get the interface's MAC address + wlan.ipconfig('addr4') # get the interface's IPv4 addresses + + ap = network.WLAN(network.WLAN.IF_AP) # create access-point interface + ap.config(ssid='ESP-AP') # set the SSID of the access point + ap.config(max_clients=10) # set how many clients can connect to the network + ap.active(True) # activate the interface + +A useful function for connecting to your local WiFi network is:: + + def do_connect(): + import machine, network + wlan = network.WLAN() + wlan.active(True) + if not wlan.isconnected(): + print('connecting to network...') + wlan.connect('ssid', 'key') + while not wlan.isconnected(): + machine.idle() + print('network config:', wlan.ipconfig('addr4')) + +Once the network is established the :mod:`socket ` module can be used +to create and use TCP/UDP sockets as usual, and the ``requests`` module for +convenient HTTP requests. + +After a call to ``wlan.connect()``, the device will by default retry to connect +**forever**, even when the authentication failed or no AP is in range. +``wlan.status()`` will return ``network.STAT_CONNECTING`` in this state until a +connection succeeds or the interface gets disabled. This can be changed by +calling ``wlan.config(reconnects=n)``, where n are the number of desired reconnect +attempts (0 means it won't retry, -1 will restore the default behaviour of trying +to reconnect forever). + +.. _esp32_network_lan: + +LAN +^^^ + +Built-in MAC (original ESP32) +""""""""""""""""""""""""""""" + +The original ESP32 SoC has a built-in Ethernet MAC. Using this MAC requires an +external Ethernet PHY to be wired to the chip's EMAC pins. Most of the EMAC pin +assignments are fixed, consult the ESP32 datasheet for details. + +If the PHY is connected, the internal Ethernet MAC can be configured via +the :class:`network.LAN` constructor:: + + import network + + lan = network.LAN(mdc=PIN_MDC, ...) # Set the pin and mode configuration + lan.active(True) # activate the interface + lan.ipconfig('addr4') # get the interface's IPv4 addresses + + +Required keyword arguments for the constructor: + +- ``mdc`` and ``mdio`` - :class:`machine.Pin` objects (or integers) specifying + the MDC and MDIO pins. +- ``phy_type`` - Select the PHY device type. Supported devices are + ``PHY_GENERIC``, + ``PHY_LAN8710``, ``PHY_LAN8720``, ``PHY_IP101``, ``PHY_RTL8201``, + ``PHY_DP83848``, ``PHY_KSZ8041`` and ``PHY_KSZ8081``. These values are all + constants defined in the ``network`` module. +- ``phy_addr`` - The address number of the PHY device. Must be an integer in the + range 0x00 to 0x1f, inclusive. Common values are ``0`` and ``1``. + +All of the above keyword arguments must be present to configure the interface. + +Optional keyword arguments: + +- ``reset`` - :class:`machine.Pin` object (or integer) specifying the PHY reset pin. +- ``power`` - :class:`machine.Pin` object (or integer) specifying a pin which + switches the power of the PHY device. +- ``ref_clk`` - :class:`machine.Pin` object (or integer) specifying the pin used + for the EMAC ``ref_clk`` signal. If not specified, the board default is used + (typically GPIO 0, but may be different if a particular board has Ethernet.) +- ``ref_clk_mode`` - Defines whether the EMAC ``ref_clk`` pin of the ESP32 + should be an input or an output. Suitable values are ``machine.Pin.IN`` and + ``machine.Pin.OUT``. If not specified, the board default is used + (typically input, but may be different if a particular board has Ethernet.) + +These are working configurations for LAN interfaces of some popular ESP32 boards:: + + # Olimex ESP32-GATEWAY: power controlled by Pin(5) + # Olimex ESP32 PoE and ESP32-PoE ISO: power controlled by Pin(12) + + lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), power=machine.Pin(5), + phy_type=network.PHY_LAN8720, phy_addr=0, + ref_clk=machine.Pin(17), ref_clk_mode=machine.Pin.OUT) + + # Wireless-Tag's WT32-ETH01 + + lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), + phy_type=network.PHY_LAN8720, phy_addr=1, power=None) + + # Wireless-Tag's WT32-ETH01 v1.4 + + lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), + phy_type=network.PHY_LAN8720, phy_addr=1, + power=machine.Pin(16)) + + # Espressif ESP32-Ethernet-Kit_A_V1.2 + + lan = network.LAN(id=0, mdc=Pin(23), mdio=Pin(18), power=Pin(5), + phy_type=network.PHY_IP101, phy_addr=1) + + +.. _esp32_spi_ethernet: + +SPI Ethernet Interface +"""""""""""""""""""""" + +All ESP32 SoCs support external SPI Ethernet interface chips. These are Ethernet +interfaces that connect via a SPI bus, rather than an Ethernet RMII interface. + +.. note:: The only exception is the ESP32 ``d2wd`` variant, where this feature is disabled + to save code size. + +SPI Ethernet uses the same :class:`network.LAN` constructor, with a different +set of keyword arguments:: + + import machine, network + + spi = machine.SPI(1, sck=SCK_PIN, mosi=MOSI_PIN, miso=MISO_PIN) + lan = network.LAN(spi=spi, cs=CS_PIN, ...) # Set the pin and mode configuration + lan.active(True) # activate the interface + lan.ipconfig('addr4') # get the interface's IPv4 addresses + +Required keyword arguments for the constructor: + +- ``spi`` - Should be a :class:`machine.SPI` object configured for this + connection. Note that any clock speed configured on the SPI object is ignored, + the SPI Ethernet clock speed is configured at compile time. +- ``cs`` - :class:`machine.Pin` object (or integer) specifying the CS pin + connected to the interface. +- ``int`` - :class:`machine.Pin` object (or integer) specifying the INT pin + connected to the interface. +- ``phy_type`` - Select the SPI Ethernet interface type. Supported devices are + ``PHY_KSZ8851SNL``, ``PHY_DM9051``, ``PHY_W5500``. These values are all + constants defined in the ``network`` module. +- ``phy_addr`` - The address number of the PHY device. Must be an integer in the + range 0x00 to 0x1f, inclusive. This is usually ``0`` for SPI Ethernet devices. + +All of the above keyword arguments must be present to configure the interface. + +Optional keyword arguments for the constructor: + +- ``reset`` - :class:`machine.Pin` object (or integer) specifying the SPI Ethernet + interface reset pin. +- ``power`` - :class:`machine.Pin` object (or integer) specifying a pin which + switches the power of the SPI Ethernet interface. + +Here is a sample configuration for a WIZNet W5500 chip connected to pins on +an ESP32-S3 development board:: + + import machine, network + from machine import Pin, SPI + + spi = SPI(1, sck=Pin(12), mosi=Pin(13), miso=Pin(14)) + lan = network.LAN(spi=spi, phy_type=network.PHY_W5500, phy_addr=0, + cs=Pin(10), int=Pin(11)) + +.. note:: WIZnet W5500 Ethernet is also supported on some other MicroPython + ports, but using a :ref:`different software interface + `. + +Delay and timing +---------------- + +Use the :mod:`time