DEV add missing dep to lock-file script docstring #30574
Conversation
There was a problem hiding this comment.
Thanks, I think it was noted by @adrinjalali at one point indeed ...
I have moved to micromamba not too long ago and I may have a look at one point but to be honest this is very unlikely to be at the top of my priorities ...
|
just wait and hope that someone sets pixi up! |
Honestly I'd like to but this seems like may take a while ... I mean if someone was setting up something like https://github.com/rgommers/pixi-dev-scipystack for scikit-learn I'd be willing to give it a go in my day to day scikit-learn work ... @lucascolley are you using something like this for scipy out of curiosity? For scikit-learn to completely switch to pixi I would say one of the technical hurdle is pixi lock-file size prefix-dev/pixi#1509. since lock-files are in our repo and we don't want to incur the CI churn that goes with updating lock-files every week. See #22448 (comment) where I did some quick and dirty estimates of the repo growth due to CI lock files updates. For completeness, I just did a quick comparison and pixi lock files seems about 5x bigger in our use case. I generated and # environment.yml
channels:
- conda-forge
dependencies:
- python
- numpy
- scipy
- cython
- joblib
- threadpoolctl
- matplotlib
- pandas
- pyamg
- pytest
- pytest-xdist
- pillow
- pip
- ninja
- meson-python
- pytest-cov
- coverage
- ccache
- polars |
|
yes, I use rgommers/pixi-dev-scipystack for 100% of my SciPy work and it seems stable. It also seems quite far off to integrate pixi into the main repo for SciPy. Perhaps using it for CI environments is a little more within reach. Out of interest, which version of pixi are you using? I know that the lock-file size reduced in a relatively recent update, but maybe you already have that. |
Good to know, maybe it's just a matter of getting the motivation of putting together a scikit-learn setup then!
Thanks, I was using a somewhat older version but I just tried with the latest pixi version ( |
|
Looks like Isn't CI such a wonderful world 🙃 ... Side-comment Azure does not care about |
I took a brief look. Is scikit-learn set-up for non-editable meson installs? Editable installs do not work with meson-python |
Nice thanks for investigating this! I guess that's the kind of problem that can happen with shiny new tools, you end-up alone in weird corners that are a bit outside the beaten path 😅 ... scikit-learn is not set-up for non-editable meson install. Depending on how complicated the issue seems to be on the pixi/uv side, I would be open to add More context: historically scikit-learn moved to meson late and meson editable install existed and kind of almost worked 1 . The meson editable install looked similar to what we were used to and works better inside an IDE according to scipy doc. I did not see any reason to have two ways to do a similar things and complicate the situation further. Also I wanted to avoid coupling Meson (already quite tricky) to adopting spin. Footnotes
|
|
@lucascolley apparently @glemaitre has a pixi setup that works for him and should be similar enough to your setup: https://github.com/glemaitre/scikit-learn-workspace. I would be curious to know how it is different to your setup that avoids the uv issue with meson editable install, but I don't have time to look into this 😅 ... |
|
So I guess you are saying this is the key: [feature.build.pypi-options]
no-build-isolation = ["scikit-learn"] |
|
Yes, sorry if that wasn't obvious haha! I wasn't aware that that |
|
What problem does pixi (or in general having a lockfile) for scikit-learn development solve? AFAIK we have a good guide that contains just a handful of commands (https://scikit-learn.org/dev/developers/advanced_installation.html#building-from-source) to get setup and running. If we couldn't achieve this, say you'd have to have pinned versions of the dependencies or a much longer list of them, then I could see what pixi offers. My assumption is that using pixi is mostly about having lockfiles? |
|
those handful commands don't replace a pixi run -e ci-sklearn12 tests -svl where On top of that, I often do pixi shell -e ci-sklearn12to get a shell into that environment. In an IDE like vscode, The workflow is certainly not bug-free and has issues, but it has made my dev life much happier there. |
|
Another huge benefit is for reproducibility / helping new contributors. Rather than saying "follow these 10 commands" and having to worry about different system setups or mistakes following the commands, it's usually just 1) install pixi 2) run this one command. But perhaps the biggest benefit is when you need different envs for different tasks. It looks like Guillaume's setup isn't quite that complex yet, but take a look at https://github.com/rgommers/pixi-dev-scipystack/blob/main/scipy/pixi.toml. Managing all of that alternative backend development via conda environments had started to become a huge pain for me. |
hi @lesteve! closes gh-30571