8000 PEP 783: Emscripten packaging (#4328) · python/peps@0cdfff8 · GitHub
[go: up one dir, main page]
Skip to content

Commit 0cdfff8

Browse files
hoodmaneAA-Turnerjuntyrhugovk
authored
PEP 783: Emscripten packaging (#4328)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Juniper Tyree <50025784+juntyr@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
1 parent 77031b7 commit 0cdfff8

File tree

2 files changed

+223
-0
lines changed

2 files changed

+223
-0
lines changed

‎.github/CODEOWNERS

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -663,6 +663,8 @@ peps/pep-0781.rst @methane
663663
peps/pep-0782.rst @vstinner
664664
peps/pep-0784.rst @gpshead
665665
# ...
666+
peps/pep-0783.rst @hoodmane @ambv
667+
# ...
666668
peps/pep-0789.rst @njsmith
667669
# ...
668670
peps/pep-0801.rst @warsaw

‎peps/pep-0783.rst

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
PEP: 783
2+
Title: Emscripten Packaging
3+
Author: Hood Chatham <roberthoodchatham at gmail.com>
4+
Sponsor: Łukasz Langa <lukasz at python.org>
5+
Discussions-To: https://discuss.python.org/t/86862
6+
Status: Draft
7+
Type: Standards Track
8+
Topic: Packaging
9+
Created: 28-Mar-2025
10+
Post-History: `02-Apr-2025 <https://discuss.python.org/t/86862>`__,
11+
`18-Mar-2025 <https://discuss.python.org/t/84996>`__,
12+
13+
Abstract
14+
========
15+
16+
This PEP proposes a new platform tag series ``pyodide`` for binary Python package
17+
distributions for the Pyodide Python runtime.
18+
19+
`Emscripten <https://emscripten.org/>`__ is a complete open-source compiler
20+
toolchain. It compiles C/C++ code into WebAssembly/JavaScript executables, for
21+
use in JavaScript runtimes, including browsers and Node.js. The Rust language
22+
also maintains an Emscripten target. :pep:`776` specifies Python's support for
23+
Emscripten.
24+
25+
26+
Motivation
27+
==========
28+
29+
Pyodide is a CPython distribution for use in the browser. A web browser is a
30+
universal computing platform, available on Windows, macOS, Linux, and every
31+
smartphone. Hundreds of thousands of students have learned Python through
32+
Pyodide via projects like `Capytale
33+
<https://web.archive.org/web/20241211090946/https://cfp.jupytercon.com/2023/talk/TJ9YEV/>`__
34+
and `PyodideU <https://stanford.edu/~cpiech/bio/papers/pyodideU.pdf>`__. Pyodide
35+
is also increasingly being used by Python packages to provide interactive
36+
documentation.
37+
38+
Pyodide currently maintains ports of 255 different packages at the time of this
39+
writing, including major scientific Python packages like NumPy, SciPy, pandas,
40+
Polars, scikit-learn, OpenCV, PyArrow, and Pillow as well as general purpose
41+
packages like aiohttp, Requests, Pydantic, cryptography, and orjson.
42+
43+
About 60 packages are also testing against Pyodide in their CI, including NumPy,
44+
pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3.
45+
46+
Python package projects cannot deploy binary distributions for Pyodide on PyPI.
47+
Instead they must use other options like ``anaconda.org`` or ``jsdelivr.com``.
48+
This creates friction both for package maintainers and for users.
49+
50+
51+
Rationale
52+
=========
53+
54+
Emscripten uses a variant of musl libc. The Emscripten compiler makes no ABI
55+
stability guarantees between versions. Many Emscripten updates are ABI
56+
compatible by chance, and the Rust Emscripten target behaves as if the ABI were
57+
stable with only `occasional negative consequences
58+
<https://github.com/rust-lang/rust/issues/131467>`__.
59+
60+
There are several linker flags that adjust the Emscripten ABI, so Python
61+
packages built to run with Emscripten must make sure to match the ABI-sensitive
62+
linker flags used to compile the interpreter to avoid load-time or run-time
63+
errors. The Emscripten compiler continuously fixes bugs and adds support for new
64+
web platform features. Thus, there is significant benefit to being able to
65+
update the ABI.
66+
67+
In order to balance the ABI stability needs of package maintainers with the ABI
68+
flexibility to allow the platform to move forward, Pyodide plans to adopt a new
69+
ABI for each feature release of Python.
70+
71+
The Pyodide team also coordinates the ABI flags that Pyodide uses with the
72+
Emscripten ABI that Rust supports in order to ensure that we have support for
73+
the many popular Rust packages. Historically, most of the work for this has
74+
been related to unwinding ABIs. See for instance `this Rust Major Change
75+
Proposal <https://github.com/rust-lang/compiler-team/issues/801>`__.
76+
77+
The ``pyodide`` platform tags only apply to Python interpreters compiled and
78+
linked with the same version of Emscripten as Pyodide, with the same
79+
ABI-sensitive flags.
80+
81+
82+
Specification
83+
=============
84+
85+
The platform tags will take the form:
86+
87+
.. code-block:: text
88+
89+
pyodide_${YEAR}_${PATCH}_wasm32
90+
91+
Each one of these will be used with a specified Python version. For example, the
92+
platform tag ``pyodide_2025_0`` will be used with Python 3.13.
93+
94+
Emscripten Wheel ABI
95+
--------------------
96+
97+
The specification of the ``pyodide_<abi>`` platform includes:
98+
99+
* Which version of the Emscripten compiler is used
100+
* What libraries are statically linked with the interpreter
101+
* What stack unwinding ABI is to be used
102+
* How the loader handles dependency lookup
103+
* That libraries cannot use ``-pthread``
104+
* That libraries should be linked with ``-sWASM_BIGINT``
105+
106+
The ABI is selected by choosing the appropriate version of the Emscripten
107+
compiler and passing appropriate compiler and linker flags. It is possible for
108+
other people to build their own Python interpreter that is compatible with the
109+
Pyodide ABI, it is not necessary to use the Pyodide distribution itself.
110+
111+
THe Pyodide ABIs are fully specified in the `Pyodide Platform ABI
112+
<https://pyodide.org/en/stable/development/abi.html>`__ documentation.
113+
114+
The ``pyodide build`` tool knows how to create wheels that match the Pyodide
115+
ABI. Unlike with manylinux wheels, there is no need for a Docker container to
116+
build the ``pyodide_<abi>`` wheels. All that is needed is a Linux machine and
117+
appropriate versions of Python, Node.js, and Emscripten.
118+
119+
It is possible to validate a wheel by installing and importing it into the
120+
Pyodide runtime. Because Pyodide can run in an environment with strong
121+
sandboxing guarantees, doing this produces no security risks.
122+
123+
Determining the ABI version
124+
---------------------------
125+
126+
The Pyodide ABI version is stored in the ``PYODIDE_ABI_VERSION`` config variable
127+
and can be determined via:
128+
129+
.. code-block:: python
130+
131+
pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION")
132+
133+
134+
To generate the list of compatible tags, one can use the following code:
135+
136+
.. code-block:: python
137+
138+
from packaging.tags import cpython_tags, _generic_platforms
139+
140+
def _emscripten_platforms() -> Iterator[str]:
141+
pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION")
142+
if pyodide_abi_version:
143+
yield f"pyodide_{pyodide_abi_version}_wasm32"
144+
yield from _generic_platforms()
145+
146+
emscripten_tags = cpython_tags(platforms=_emscripten_platforms())
147+
148+
This code will be added to `pypa/packaging
149+
<https://github.com/pypa/packaging/pull/804>`__.
150+
151+
152+
Package Installers
153+
------------------
154+
155+
Installers should use the ``_emscripten_platforms()`` function shown above to
156+
determine which platforms are compatible with an Emscripten build of CPython. In
157+
particular, the Pyodide ABI version is exposed via
158+
``sysconfig.get_config_var("PYODIDE_ABI_VERSION")``.
159+
160+
Package Indexes
161+
---------------
162+
163+
Package indexes SHOULD accept any wheel whose platform tag matches
164+
the regular expression ``pyodide_[0-9]+_[0-9]+_wasm32``.
165+
166+
167+
Dependency Specifier Markers
168+
----------------------------
169+
170+
According to :pep:`776#platform-identification`, in Emscripten Python
171+
``sys.platform`` returns ``"emscripten"``. To check for the Emscripten platform in a
172+
dependency specifier, one can use ``sys_platform == "emscripten"`` (or its
173+
negation).
174+
175+
176+
Trove Classifier
177+
----------------
178+
179+
Packages that build and test Emscripten wheels can declare this by adding the
180+
``Environment :: WebAssembly :: Emscripten`` classifier. PyPI already accepts uploads of
181+
`packages with this classifier
182+
<https://pypi.org/search/?q=&o=&c=Environment+%3A%3A+WebAssembly+%3A%3A+Emscripten>`__.
183+
184+
185+
Backwards Compatibility
186+
=======================
187+
188+
There are no backwards compatibility concerns in this PEP.
189+
190+
191+
Security Implications
192+
=====================
193+
194+
There are no security implications in this PEP.
195+
196+
How to Teach This
197+
=================
198+
199+
For Pyodide users, we recommend the `Pyodide documentation on installing
200+
packages <https://pyodide.org/en/stable/usage/loading-packages.html>`__.
201+
202+
For package maintainers, we recommend the `Pyodide documentation on building and
203+
testing packages
204+
<https://pyodide.org/en/stable/development/building-and-testing-packages.html>`__.
205+
206+
Reference Implementation
207+
========================
208+
209+
For building packages, `pyodide build
210+
<https://github.com/pyodide/pyodide-build>`__ and `cibuildwheel
211+
<https://github.com/pypa/cibuildwheel/>`__.
212+
213+
For installers to decide whether a wheel tag is compatible with a Pyodide
214+
interpreter, `pypa/packaging#804
215+
<https://github.com/pypa/packaging/pull/804>`__.
216+
217+
Copyright
218+
=========
219+
220+
This document is placed in the public domain or under the
221+
CC0-1.0-Universal license, whichever is more permissive.

0 commit comments

Comments
 (0)
0