8000 upip: Support for baremetal systems · Issue #105 · micropython/micropython-lib · GitHub
[go: up one dir, main page]
Skip to content

upip: Support for baremetal systems #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pfalcon opened this issue Oct 5, 2016 · 5 comments
Closed

upip: Support for baremetal systems #105

pfalcon opened this issue Oct 5, 2016 · 5 comments

Comments

@pfalcon
Copy link
Contributor
pfalcon commented Oct 5, 2016

I've already done bunch of changes towards supporting baremetal systems. More things to discuss/do:

  1. Upstream pip doesn't seem to have easy programmatical interface to its functionality (it has stuff like pip.commands.install.InstallCommand, which is definitely not what we need). So, proposed interface is upip.install(package_name_or_list, path=None).
  2. If path above is None, it will be deduced, specifically sys.path[1] taken for baremetal system (/lib for esp8266 e.g., others should be updated to have preferred-to-install path as sys.path[1]).
  3. The idea is to use 4K dictionary space for gzip decompression. This will require repacking and re-pushing all micropython-lib packages to PyPI.

@dpgeorge , let's discuss.
@dpgeorge
Copy link
Member
dpgeorge commented Oct 7, 2016

Upstream pip doesn't seem to have easy programmatical interface to its functionality (it has stuff like pip.commands.install.InstallCommand, which is definitely not what we need). So, proposed interface is upip.install(package_name_or_list, path=None).

I'd say that the top-level function name and signature should closely reflect what you type on the cmd-line. What you propose is perfect.

If path above is None, it will be deduced, specifically sys.path[1] taken for baremetal system (/lib for esp8266 e.g., others should be updated to have preferred-to-install path as sys.path[1]).

That's how the cmd-line version works, so +1 agin.

The idea is to use 4K dictionary space for gzip decompression. This will require repacking and re-pushing all micropython-lib packages to PyPI.

Did you check that 4k dict gives decent compression? Maybe 2k would suffice? Or maybe we need 8k? Would be interesting to run some tests to see.

It looks like standard gzip doesn't allow to specify the dict size, so we'd need to write a Python gzip.py that does it.

@pfalcon
Copy link
Contributor Author
pfalcon commented Oct 7, 2016

Did you check that 4k dict gives decent compression? Maybe 2k would suffice? Or maybe we need 8k? Would be interesting to run some tests to see.

Yes, 4K dictionary already visibly deteriorates compression ratio (e.g. 2456 bytes vs 1952 bytes normal gzip). 8K would be better, but that's gigantic continuous block of memory for esp8266 heap. With 4K for zlib dictionary, 4K for TLS record buffer, 0.5K copy buffer(s), and ~4K to read in JSON metadata (micropython/micropython#2489), it already barely works (like, 50% chance of memory failure - that's after sprinkling gc.collect()'s around).

@pfalcon
Copy link
Contributor Author
pfalcon commented Oct 7, 2016

It looks like standard gzip doesn't allow to specify the dict size, so we'd need to write a Python gzip.py that does it.

I already have that, and due to the way setuptools works, it should be a setuptools plugin, not just a standalone script. Next big "it won't work" risk is TLS buffer - that's got to be 16K, or if less (like 4K), won't accommodate a file bigger than that. So, I'm thinking about minimizing e.g. setup.py, then removing all the cruft from .tar.gz, then switching to more frugal file format, e.g. .cpio, if not completely custom format.

@pfalcon
Copy link
Contributor Author
pfalcon commented Oct 8, 2016

I'm also applying following simplification: currently upip parses MICROPYPATH envvar, but micropython itself have already parsed that. If MICROPYPATH is not available, upip falls back to sys.path[1], and micropython have already set that based on MICROPYPATH. So, I'm just going to use sys.path[1].

@pfalcon
Copy link
Contributor Author
pfalcon commented Oct 29, 2016

Support has been implemented, initial version is available with esp8266 1.8.5 release.

@pfalcon pfalcon closed this as completed Oct 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pfalcon @dpgeorge
0