8000 docs/reference/packages: Add chapter on distribution packages and dep… · lvgl/lv_micropython@c60fc67 · GitHub
[go: up one dir, main page]
Skip to content

Commit c60fc67

Browse files
pfalconpfalcon
committed
docs/reference/packages: Add chapter on distribution packages and deployment.
A long overdue overview of preparing packages, installing them with upip, freezing, dealing with resources. Initial version, more iterations required.
1 parent e7fc765 commit c60fc67

File tree

2 files changed

+275
-0
lines changed

2 files changed

+275
-0
lines changed

‎docs/reference/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ implementation and the best practices to use them.
2424
isr_rules.rst
2525
speed_python.rst
2626
constrained.rst
27+
packages.rst
2728

2829
.. only:: port_pyboard
2930

‎docs/reference/packages.rst

Lines changed: 274 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
Distribution packages, package management, and deploying applications
2+
=====================================================================
3+
4+
Just as the "big" Python, MicroPython supports creation of "third party"
5+
packages, distributing them, and easily installing them in each user's
6+
environment. This chapter discusses how these actions are achieved.
7+
Some familiarity with Python packaging is recommended.
8+
9+
Overview
10+
--------
11+
12+
Steps below represent a high-level workflow when creating and consuming
13+
packages:
14+
15+
1. Python modules and packages are turned into distribution package
16+
archives, and published at the Python Package Index (PyPI).
17+
2. `upip` package manager can be used to install a distribution package
18+
on a `MicroPython port` with networking capabilities (for example,
19+
on the Unix port).
20+
3. For ports without networking capabilities, an "installation image"
21+
can be prepared on the Unix port, and transferred to a device by
22+
suitable means.
23+
4. For low-memory ports, the installation image can be frozen as the
24+
bytecode into MicroPython executable, thus minimizing the memory
25+
storage overheads.
26+
27+
The sections below describe this process in details.
28+
29+
Distribution packages
30+
---------------------
31+
32+
Python modules and packages can be packaged into archives suitable for
33+
transfer between systems, storing at the well-known location (PyPI),
34+
and downloading on demand for deployment. These archives are known as
35+
*distribution packages* (to differentiate them from Python packages
36+
(means to organize Python source code)).
37+
38+
The MicroPython distribution package format is a well-known tar.gz
39+
format, with some adaptations however. The Gzip compressor, used as
40+
an external wrapper for TAR archives, by default uses 32KB dictionary
41+
size, which means that to uncompress a compressed stream, 32KB of
42+
contguous memory needs to be allocated. This requirement may be not
43+
satisfiable on low-memory devices, which may have total memory available
44+
less than that amount, and even if not, a contiguous block like that
45+
may be hard to allocate due to `memory fragmentation`. To accommodate
46+
these constraints, MicroPython distribution packages use Gzip compression
47+
with the dictionary size of 4K, which should be a suitable compromise
48+
with still achieving some compression while being able to uncompressed
49+
even by the smallest devices.
50+
51+
Besides the small compression dictionary size, MicroPython distribution
52+
packages also have other optimizations, like removing any files from
53+
the archive which aren't used by the installation process. In particular,
54+
`upip` package manager doesn't execute ``setup.py`` during installation
55+
(see below), and thus that file is not included in the archive.
56+
57+
At the same time, these optimizations make MicroPython distribution
58+
packages not compatible with `CPython`'s package manager, ``pip``.
59+
This isn't considered a big problem, because:
60+
61+
1. Packages can be installed with `upip`, and then can be used with
62+
CPython (if they are compatible with it).
63+
2. In the other direction, majority of CPython packages would be
64+
incompatible with MicroPython by various reasons, first of all,
65+
the reliance on features not implemented by MicroPython.
66+
67+
Summing up, the MicroPython distribution package archives are highly
68+
optimized for MicroPython's target environments, which are highly
69+
resource constrained devices.
70+
71+
72+
``upip`` package manager
73+
------------------------
74+
75+
MicroPython distribution packages are intended to be installed using
76+
the `upip` package manager. `upip` is a Python application which is
77+
usually distributed (as frozen bytecode) with network-enabled
78+
`MicroPython ports <MicroPython port>`. At the very least,
79+
`upip` is available in the `MicroPython Unix port`.
80+
81+
On any `MicroPython port` providing `upip`, it can be accessed as
82+
following::
83+
84+
import upip
85+
upip.help()
86+
upip.install(package_or_package_list, [path])
87+
88+
Where *package_or_package_list* is the name of a distribution
89+
package to install, or a list of such names to install multiple
90+
packages. Optional *path* parameter specifies filesystem
91+
location to install under and defaults to the standard library
92+
location (see below).
93+
94+
An example of installing a specific package and then using it::
95+
96+
>>> import upip
97+
>>> upip.install("micropython-pystone_lowmem")
98+
[...]
99+
>>> import pystone_lowmem
100+
>>> pystone_lowmem.main()
101+
102+
Note that the name of Python package and the name of distribution
103+
package for it in general don't have to match, and oftentimes they
104+
don't. This is because PyPI provides a central package repository
105+
for all different Python implementations and versions, and thus
106+
distribution package names may need to be namespaced for a particular
107+
implementation. For example, all packages from `micropython-lib`
108+
follow this naming convention: for a Python module or package named
109+
``foo``, the distribution package name is ``micropython-foo``.
110+
111+
For the ports which run MicroPython executable from the OS command
112+
prompts (like the Unix port), `upip` can be (and indeed, usually is)
113+
run from the command line instead of MicroPython's own REPL. The
114+
commands which corresponds to the example above are::
115+
116+
micropython -m upip -h
117+
micropython -m upip install [-p <path>] <packages>...
118+
micropython -m upip install micropython-pystone_lowmem
119+
120+
[TODO: Describe installation path.]
121+
122+
123+
Cross-installing packages
124+
-------------------------
125+
126+
For `MicroPython ports <MicroPython port>` without native networking
127+
capabilities, the recommend process is "cross-installing" them into a
128+
"directory image" using the `MicroPython Unix port`, and then
129+
transferring this image to a device by suitable means.
130+
131+
Installing to a directory image involves using ``-p`` switch to `upip`::
132+
133+
micropython -m upip install -p install_image micropython-pystone_lowmem
134+
135+
After this command, the package content (and contents of every depenency
136+
packages) will be available in the ``install_image/`` subdirectory. You
137+
would need to transfer contents of this directory (without the
138+
``install_image/`` prefix) to the device, at the suitable location, where
139+
it can be found by the Python ``import`` statement (see discussion of
140+
the `upip` installation path above).
141+
142+
143+
Cross-installing packages with freezing
144+
---------------------------------------
145+
146+
For the low-memory `MicroPython ports <MicroPython port>`, the process
147+
described in the previous section does not provide the most efficient
148+
resource usage,because the packages are installed in the source form,
149+
so need to be compiled to the bytecome on each import. This compilation
150+
requires RAM, and the resulting bytecode is also stored in RAM, reducing
151+
its amount available for storing application data. Moreover, the process
152+
above requires presence of the filesystem on a device, and the most
153+
resource-constrained devices may not even have it.
154+
155+
The bytecode freezing is a process which resolves all the issues
156+
mentioned above:
157+
158+
* The source code is pre-compiled into bytecode and store as such.
159+
* The bytecode is stored in ROM, not RAM.
160+
* Filesystem is not required for frozen packages.
161+
162+
Using frozen bytecode requires building the executable (firmware)
163+
for a given `MicroPython port` from the C source code. Consequently,
164+
the process is:
165+
166+
1. Follow the instructions for a particular port on setting up a
167+
toolchain and building the port. For example, for ESP8266 port,
168+
study instructions in ``ports/esp8266/README.md`` and follow them.
169+
Make sure you can build the port and deploy the resulting
170+
executable/firmware successfully before proceeding to the next steps.
171+
2. Build `MicroPython Unix port` and make sure it is in your PATH and
172+
you can execute ``micropython``.
173+
3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266).
174+
4. Run ``make clean-frozen``. This step cleans up any previous
175+
modules which were installed for freezing (consequently, you need
176+
to skip this step to add additional modules, instead of starting
177+
from scratch).
178+
5. Run ``micropython -m upip install -p modules <packages>...`` to
179+
install packages you want to freeze.
180+
6. Run ``make clean``.
181+
7. Run ``make``.
182+
183+
After this, you should have the executable/firmware with modules as
184+
the bytecode inside, which you can deploy the usual way.
185+
186+
Few notes:
187+
188+
1. Step 5 in the sequence above assumes that the distribution package
189+
is available from PyPI. If that is not the case, you would need
190+
to copy Python source files manually to ``modules/`` subdirectory
191+
of the port port directory. (Note that upip does not support
192+
installing from e.g. version control repositories).
193+
2. The firmware for baremetal devices usually has size restrictions,
194+
so adding too many frozen modules may overflow it. Usually, you
195+
would get a linking error if this happens. However, in some cases,
196+
an image may be produced, which is not runnable on a device. Such
197+
cases are in general bugs, and should be reported and further
198+
investigated. If you face such a situation, as an initial step,
199+
you may want to decrease the amount of frozen modules included.
200+
201+
202+
Application resources
203+
---------------------
204+
205+
A complete application, besides the source code, oftentimes also consists
206+
of data files, e.g. web page templates, game images, etc. It's clear how
207+
to deal with those when application is installed manually - you just put
208+
those data files in the filesystem at some location and use the normal
209+
file access functions.
210+
211+
The situation is different when deploying applications from packages - this
212+
is more advanced, streamlined and flexible way, but also requires more
213+
advanced approach to accessing data files. This approach is treating
214+
the data files as "resources", and abstracting away access to them.
215+
216+
Python supports resource access using its "setuptools" library, using
217+
``pkg_resources`` module. MicroPython, following its usual approach,
218+
implements subset of the functionality of that module, specifically
219+
`pkg_resources.resource_stream(package, resource)` function.
220+
The idea is that an application calls this function, passing a
221+
resource identifier, which is a relative path to data file within
222+
the specified package (usually top-level application package). It
223+
returns a stream object which can be used to access resource contents.
224+
Thus, the ``resource_stream()`` emulates interface of the standard
225+
`open()` function.
226+
227+
Implementation-wise, ``resource_stream()`` uses file operations
228+
underlyingly, if distribution package is install in the filesystem.
229+
However, it also supports functioning without the underlying filesystem,
230+
e.g. if the package is frozen as the bytecode. This however requires
231+
an extra intermediate step when packaging application - creation of
232+
"Python resource module".
233+
234+
The idea of this module is to convert binary data to a Python bytes
235+
object, and put it into the dictionary, indexed by the resource name.
236+
This conversion is done using ``tools/mpy_bin2res.py`` script from
237+
the MicroPython distribution.
238+
239+
Let's trace the complete process using the following example. Suppose
240+
your application has the following structure::
241+
242+
my_app/
243+
__main__.py
244+
utils.py
245+
data/
246+
page.html
247+
image.png
248+
249+
``__main__.py`` and ``utils.py`` should access resources using the
250+
following calls::
251+
252+
import pkg_resources
253+
254+
pkg_resources.resource_stream(__name__, "data/page.html")
255+
pkg_resources.resource_stream(__name__, "data/image.png")
256+
257+
You can develop and debug using the `MicroPython Unix port` as usual.
258+
When times come to make a distribution package out of it, you would
259+
need to run following command, with ``my_app/`` being the current
260+
directory (and assuming ``mpy_bin2res.py`` is in your path)::
261+
262+
mpy_bin2res.py data/page.html data/image.png
263+
264+
This will produce a Python resource module named ``R.py``. Afterwards,
265+
you package the project for distribution as usual (using ``setup.py sdist``).
266+
Prepared like this, your application will work both when deployed to
267+
filesystem and as frozen bytecode.
268+
269+
References
270+
----------
271+
272+
* Python Packaging User Guide: https://packaging.python.org/
273+
* Setuptools documentation: https://setuptools.readthedocs.io/
274+
* Distutils documentation: https://docs.python.org/3/library/distutils.html

0 commit comments

Comments
 (0)
0