8000 Importing a new package that doesn't exist in `Pyodide` · Issue #85 · pyscript/pyscript · GitHub
[go: up one dir, main page]
Skip to content

Importing a new package that doesn't exist in Pyodide #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ma7dev opened this issue May 1, 2022 · 20 comments
Closed

Importing a new package that doesn't exist in Pyodide #85

ma7dev opened this issue May 1, 2022 · 20 comments
Labels
tag: pyodide Related to issues with pyodide

Comments

@ma7dev
Copy link
Contributor
ma7dev commented May 1, 2022 •
edited
Loading

I have been trying to look into ways to import torch but I couldn't manage to do so because it isn't in Pyodide's list of supported packages.

From my research on this, there are two ways to have any new package be utilized:

  1. Find a pure Python 3 wheel which means the package could be installed without the need of hard specifications.
  2. Add torch and its dependencies to Pyodide's list of support packages. This will require doing the following:
    1. Obtaining `Source Distribution of the package.
    2. Compiling the package locally.
    3. Building Pyodide from the source.
    4. Add the package to Pyodide's list of supported packages by following the documentation
    5. Importing the newer version of Pyodide to Pyscript I don't know how this can be done

Questions:

  1. How can I add a newer version of Pyodide which was built locally to Pyscript?
  2. Isn't there a way to import a python environment that has all of the packages already installed?
  3. Why do we need to have pure Python 3 wheel (aka needs to be compiled on all hardware)? Is it because it is actually being compiled on the browser-side which should be able to run all hardware?
  4. From looking into the logs, it seems that every time the page has loaded the packages listed to be used get installed by micropip, is this correct? If yes, why can't we store already installed packages to load them?
  5. Why PyScript is super slow? I am assuming this because of the installation of packages from scratch (if my assumption is true).

Relevant resource:
@ma7dev ma7dev changed the title Importing new package that don't exist in Pyodide Importing a new package that doesn't exist in Pyodide May 1, 2022
@verhulstm
Copy link
Contributor

hi, i am excited to see your interest in this project. :)

i have been working on getting torch into Pyodide. it's pretty challenging.

we do have a alpha workflow to get non-pure python packages with c/c++/cython extensions into the pyscript/Pyodide ecosystem, but torch is extra complicated.

i plan on posting the build process for non-pure python packages as soon as i can

@robotixdevteam
Copy link

hi, what about tensorflow as it has the javacript package in it. will it be able to work with pyscript.

@verhulstm
Copy link
Contributor
verhulstm commented May 2, 2022 •
edited
Loading

currently, you should be able to use all the packages here

https://github.com/pyodide/pyodide/tree/main/packages

you can build your own packages and add them into pyscript. but some packages are very difficult to build. torch and tensorflow are hard to build into WASM and the WASM compiler tooling. i am working on porting torch now and tensorflow after that. i hope to make some new posts as i progress on this task.

@verhulstm
Copy link
Contributor

i see more clearly now, sorry. i expect TensorFlow.js will work yes.

good idea. i will try that out

@josephrocca
Copy link
josephrocca commented May 2, 2022 •
edited
Loading

@verhulstm I'm super excited to hear about the progress you make on torch and tensorflow. Even more exciting from my perspective would be JAX. Are you working on this in a public repo? I'd love to follow along and help where I can.

@MohamedAliRashad
Copy link

@verhulstm I would suggest you take a look at onnx. They have a JavaScript library for running PyTorch models into browser.

https://cloudblogs.microsoft.com/opensource/2021/09/02/onnx-runtime-web-running-your-machine-learning-model-in-browser/

@rth
Copy link
Contributor
rth commented May 2, 2022

Even more exciting from my perspective would be JAX. Are you working on this in a public repo?

There was a discussion about JAX in pyodide/pyodide#2198, tensorflow in pyodide/pyodide#50 and torch in pyodide/pyodide#1625.
It would be great to have them but they are all fairly large and complex packages, so it would take a lot of work to make that happen.

I would suggest you take a look at onnx

For ML wrapping JS libraries (such as ONNX Runtime Web, Tensorflow.js etc) from Python via the Pyodide's FFI is indeed another solution. Though your API is then not going to be exactly the same as when using e.g. Tensorflow from Python, so one cannot reuse code without changes.

@josephrocca
Copy link
josephrocca commented May 2, 2022 •
edited
Loading

I would suggest you take a look at onnx

Yep, ONNX web runtime, tfjs-tflite, etc. are fine for inference - but JAX/torch/tensorflow can do much more than model inference. It would be a dream come true to be able to play with JAX's auto-differentiation in the browser!

@sterlinm
Copy link
sterlinm commented May 4, 2022 •
edited
Loading

you can build your own packages and add them into pyscript.

@verhulstm Are there any examples of how you would do this? I'd love to experiment with using this with some pure python libraries that aren't on pypi.

I'm assuming you'd follow the steps on how to build a Pyodide package but I'm not clear on how you'd add them into pyscript.

Thanks!

@verhulstm
Copy link
Contributor

i am prepping a better guide for this, but until that is ready.

get a pyodide dev env ready. in docker or vagrant

git clone https://github.com/pyodide/pyodide.git
git checkout 0.20.0a1
source pyodide_env.sh

then follow https://pyodide.org/en/stable/development/new-packages.html

@verhulstm
Copy link
Contributor

i did this in ubuntu 20.04

with these packages

apt install -y build-essential
apt install -y linux-headers-$(uname -r)
apt install -y libgdbm6 libdbus-glib-1-2 libtool
apt install -y libssl-dev libyaml-dev libreadline6-dev libncurses5-dev libffi-dev libgdbm-dev libdb-dev libltdl-dev
apt install -y zlib1g-dev
apt install -y clang-format-6.0
apt install -y prelink cmake patch automake autoconf bzip2 bison ccache pkg-config swig xz-utils autotools-dev
apt install -y texinfo dejagnu gnupg2
apt install -y git
apt install -y unzip
apt install -y wget
apt install -y gfortran
apt install -y g++
apt install -y f2c
apt install -y default-jre
apt install -y nodejs
apt install -y npm

and

https://repo.anaconda.com/miniconda/Miniconda3-py39_4.11.0-Linux-x86_64.sh

@verhulstm
Copy link
Contributor

if you have a pure python file that should already work. include a link to the file in your "py-env" tag

@ma7dev
Copy link
Contributor Author
ma7dev commented May 4, 2022 •
edited
Loading

You can link pure Python 3 wheels directly in Pyscript without the need of the package to be in Pyodide. I have tested with different packages where I have found pure Python 3 wheels.

import asyncio
import micropip

await micropip.install(URL)

I don't know if py-env supports external URLs.

@sterlinm
Copy link
sterlinm commented May 4, 2022

Thanks @verhulstm and @sudomaze!

@ma7dev
Copy link
Contributor Author
ma7dev commented May 4, 2022 •
edited
Loading

@verhulstm let me know if I can help, I will be happy to contribute.

The PyTorch team is aware of your contribution to import PyTorch into Pyodide, so they should be in contact with you

cc: @msaroufim

@verhulstm
Copy link
Contributor

great, i would love to chat in the next few days. email me :)

@fpliger fpliger added the tag: pyodide Related to issues with pyodide label May 4, 2022
@fpliger
Copy link
Contributor
fpliger commented May 4, 2022

Joining late to the party... a bunch of the questions in the original post was already addressed so I'll focus on the ones that I think were left out:

  • How can I add a newer version of Pyodide which was built locally to Pyscript?

Currently, you can't. Opened #172 to address this

  • Isn't there a way to import a python environment that has all of the packages already installed?

I think the question here means to load a packaged "bundle" with pyodide and all (and only) applications require packages once. Right? If so, no, this is not possible today. This is actually an upstream issue that we recently discussed with the pyodide team and we have some ideas on.. but not in the short term. /cc @rth

  • Why do we need to have pure Python 3 wheel (aka needs to be compiled on all hardware)? Is it because it is actually being compiled on the browser-side which should be able to run all hardware?

Not entirely sure I understand question but, in any way I could interpret it, it is an upstream question. @rth @hoodmane do you want to chime in?

From looking into the logs, it seems that every time the page has loaded the packages listed to be used get installed by micropip, is this correct? If yes, why can't we store already installed packages to load them?

If the packages are already included in Pyodide, it will only load them and not fetch them from PyPI and install them. I'd think it's to optimize size and performance (the more you load/add the bigger and slower it is).

Why PyScript is super slow? I am assuming this because of the installation of packages from scratch (if my assumption is true).

We are trying to investigate in the next few days but it's very very likely to be an upstream issue.

Thank you for the questions! As we roll out better documentation, we'll add some of these to the FAQ. 👍

@verhulstm verhulstm mentioned this issue May 4, 2022
Closed
This was referenced May 5, 2022
Closed
Closed
@josephrocca josephrocca mentioned this issue May 14, 2022
Open
@rth
Copy link
Contributor
rth commented May 18, 2022 •
edited
Loading

Why do we need to have pure Python 3 wheel (aka needs to be compiled on all hardware)? Is it because it is actually being compiled on the browser-side which should be able to run all hardware?

Packages that don't have pure Python wheels,

  • require to run setup.py with setuptools which currently doesn't work due to unsupported features in the browser Use wheel package to create .whl if not available? pyodide/pyodide#1501
  • even if it setuptools could be made to work, there is no guarantee that setup.py would run without error. At least with pure Python wheels it if exists, there is a guarantee that it would be installed correctly.
  • finally packages with binary extensions require to be built with Emscripen (currently using the Pyodide build system) and so those cannot be installed in the browser from .tar.gz in any case.

We have added a FAQ entry recently on what to do when a package fails to install https://pyodide.org/en/latest/usage/faq.html#micropip-can-t-find-a-pure-python-wheel

it seems that every time the page has loaded the packages listed to be used get installed by micropip, is this correct? If yes, why can't we store already installed packages to load them?

Yes, they will be re-installed each time. However the browser would cache them, so they don't have to be re-downloaded and it's mostly a matter of unzipping them to the virtual file system (which is fast) and pre-loading dynamic libraries contained in the package (see below).

If the packages are already included in Pyodide, it will only load them and not fetch them from PyPI and install them.

Yes, in particular for packages included in Pyodide we unvendor tests, so they are a bit smaller. Otherwise, for pure Python packages, there is little difference from where to install them.

Why PyScript is super slow? I am assuming this because of the installation of packages from scratch

If you profile a typical Python application say that loads pandas results would be roughly the following:

  • loading CPython in Pyodide: 25%
  • download the package files: depends on your network connexion but it's usually not the bottleneck
  • unpack package files to the virtual file system: 3%
  • pre-load dynamic libraries: 55%
  • import pandas: 15%

A large part of the time would be spent compiling .wasm code objects by the browser. This situation is made worse by the fact that we cannot load dynamic libraries synchronously on demand when Python modules are imported and have to-preload/compile all the libraries in a package when the package is loaded. Libraries such as numpy, pandas etc have a lot of binary extensions which take a while to compile (the same goes for CPython itself). Some browsers would cache WASM compilation https://v8.dev/blog/wasm-code-caching but it's a step that takes a while the first time. There is active work happening on how to reduce load time in Pyodide but it's still a challenging topic.

@fpliger
Copy link
Contributor
fpliger commented Jun 28, 2022

closing as upstream, seems that questions have been answered and there's not much else to do. Happy to be convinced otherwise :)

@fpliger fpliger closed this as completed Jun 28, 2022
@josephrocca josephrocca mentioned this issue Jul 5, 2022
Open
@upupbo
Copy link
upupbo commented Oct 20, 2022

Would you like to ask if it supports loading torch models now？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tag: pyodide Related to issues with pyodide
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@rth @fpliger @josephrocca @sterlinm @verhulstm @ma7dev @MohamedAliRashad @upupbo @robotixdevteam
0