8000 DOC Update developer documentation to describe meson builds (#28751) · patrickkwang/scikit-learn@30f4d9d · GitHub 8000
[go: up one dir, main page]
Skip to content

Commit 30f4d9d

Browse files
betatimlesteve
betatim
and
lesteve
authored
DOC Update developer documentation to describe meson builds (scikit-learn#28751)
Co-authored-by: Loïc Estève <loic.esteve@ymail.com>
1 parent 713caca commit 30f4d9d

File tree

4 files changed

+59
-145
lines changed

4 files changed

+59
-145
lines changed

‎doc/developers/advanced_installation.rst

Lines changed: 41 additions & 131 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,11 @@ feature, code or documentation improvement).
6464

6565
If you installed Python with conda, we recommend to create a dedicated
6666
`conda environment`_ with all the build dependencies of scikit-learn
67-
(namely NumPy_, SciPy_, and Cython_):
67+
(namely NumPy_, SciPy_, Cython_, meson-python_ and Ninja_):
6868

6969
.. prompt:: bash $
7070

71-
conda create -n sklearn-env -c conda-forge python=3.9 numpy scipy cython
71+
conda create -n sklearn-env -c conda-forge python=3.9 numpy scipy cython meson-python ninja
7272

7373
It is not always necessary but it is safer to open a new prompt before
7474
activating the newly created conda environment.
@@ -87,17 +87,19 @@ feature, code or documentation improvement).
8787

8888
python3 -m venv sklearn-env
8989
source sklearn-env/bin/activate
90-
pip install wheel numpy scipy cython
90+
pip install wheel numpy scipy cython meson-python ninja
9191

9292
#. Install a compiler with OpenMP_ support for your platform. See instructions
9393
for :ref:`compiler_windows`, :ref:`compiler_macos`, :ref:`compiler_linux`
9494
and :ref:`compiler_freebsd`.
9595

96-
#. Build the project with pip in :ref:`editable_mode`:
96+
#. Build the project with pip:
9797

9898
.. prompt:: bash $
9999

100-
pip install -v --no-use-pep517 --no-build-isolation -e .
100+
pip install --editable . \
101+
--verbose --no-build-isolation \
102+
--config-settings editable-verbose=true
101103

102104
#. Check that the installed scikit-learn has a version number ending with
103105
`.dev0`:
@@ -111,14 +113,14 @@ feature, code or documentation improvement).
111113

112114
.. note::
113115

114-
You will have to run the ``pip install -v --no-use-pep517 --no-build-isolation -e .``
115-
command every time the source code of a Cython file is updated
116-
(ending in `.pyx` or `.pxd`). This can happen when you edit them or when you
117-
use certain git commands such as `git pull`. Use the ``--no-build-isolation`` flag
118-
to avoid compiling the whole project each time, only the files you have
119-
modified. Include the ``--no-use-pep517`` flag because the ``--no-build-isolation``
120-
option might not work otherwise (this is due to a bug which will be fixed in the
121-
future).
116+
`--config-settings editable-verbose=true` is optional but recommended
117+
to avoid surprises when you import `sklearn`. `meson-python` implements
118+
editable installs by rebuilding `sklearn` when executing `import sklearn`.
119+
With the recommended setting you will see a message when this happens,
120+
rather than potentially waiting without feed-back and wondering
121+
what is taking so long. Bonus: this means you only have to run the `pip
122+
install` command once, `sklearn` will automatically be rebuilt when
123+
importing `sklearn`.
122124

123125
Dependencies
124126
------------
@@ -182,100 +184,6 @@ If you want to build a stable version, you can ``git checkout <VERSION>``
182184
to get the code for that particular version, or download an zip archive of
183185
the version from github.
184186

185-
.. _editable_mode:
186-
187-
Editable mode
188-
-------------
189-
190-
If you run the development version, it is cumbersome to reinstall the package
191-
each time you update the sources. Therefore it is recommended that you install
192-
in with the ``pip install -v --no-use-pep517 --no-build-isolation -e .`` command,
193-
which allows you to edit the code in-place. This builds the extension in place and
194-
creates a link to the development directory (see `the pip docs
195-
<https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs>`_).
196-
197-
As the doc above explains, this is fundamentally similar to using the command
198-
``python setup.py develop``. (see `the setuptool docs
199-
<https://setuptools.pypa.io/en/latest/userguide/development_mode.html>`_).
200-
It is however preferred to use pip.
201-
202-
On Unix-like systems, you can equivalently type ``make in`` from the top-level
203-
folder. Have a look at the ``Makefile`` for additional utilities.
204-
205-
.. _building_with_meson:
206-
207-
Building with Meson
208-
-------------------
209-
210-
Support for Meson is experimental, in scikit-learn 1.5.0.dev0.
211-
`Open an issue <https://github.com/scikit-learn/scikit-learn/issues/new>`__ if
212-
you encounter any problems!
213-
214-
Make sure you have `meson-python` and `ninja` installed, either with `conda`:
215-
216-
.. code-block:: bash
217-
218-
conda install -c conda-forge meson-python ninja -y
219-
220-
or with pip:
221-
222-
.. code-block:: bash
223-
224-
pip install meson-python ninja
225-
226-
Simplest way to build with Meson
227-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
228-
229-
To build scikit-learn, the simplest way is to run:
230-
231-
.. code-block:: bash
232-
233-
make dev-meson
234-
235-
You need to do it once after this you can run your code that imports `sklearn`
236-
and it will recompile as needed.
237-
238-
In case you want to go back to using setuptools:
239-
240-
.. code-block:: bash
241-
242-
make clean-meson
243-
244-
More advanced way to build with Meson
245-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
246-
247-
If you can not use `make`, want to do it yourself or understand what goes in
248-
behind the scenes, you can build scikit-learn with the following `pip` command:
249-
250-
.. code-block:: bash
251-
252-
pip install --editable . \
253-
--verbose --no-build-isolation \
254-
--config-settings editable-verbose=true
255-
256-
If you want to go back to using `setuptools`:
257-
258-
.. code-block:: bash
259-
260-
pip uninstall -y scikit-learn
261-
262-
Note `--config-settings editable-verbose=true` is advised to avoid surprises.
263-
meson-python implements editable install by recompiling when doing `import
264-
sklearn`. Even changing python files involves copying files to the Meson build
265-
directory. You will see the meson output when that happens, rather than
266-
potentially waiting a while and wondering what is taking so long. Bonus: that
267-
means you only have to do the `pip install` once, after that your code will
268-
recompile when doing `import sklearn`.
269-
270-
Other places that may be worth looking at:
271-
272-
- `pandas setup doc
273-
<https://pandas.pydata.org/docs/development/contributing_environment.html#step-3-build-and-install-pandas>`_:
274-
pandas has a similar setup as ours (no spin or dev.py)
275-
- `scipy Meson doc
276-
<https://scipy.github.io/devdocs/building/understanding_meson.html>`_ gives
277-
more background about how Meson works behind the scenes
278-
279187
.. _platform_specific_instructions:
280188

281189
Platform-specific instructions
@@ -324,11 +232,13 @@ Please be aware that the path above might be different from user to user. The
324232
aim is to point to the "vcvarsall.bat" file that will set the necessary
325233
environment variables in the current command prompt.
326234

327-
Finally, build scikit-learn from this command prompt:
235+
Finally, build scikit-learn with this command prompt:
328236

329237
.. prompt:: bash $
330238

331-
pip install -v --no-use-pep517 --no-build-isolation -e .
239+
pip install --editable . \
240+
--verbose --no-build-isolation \
241+
--config-settings editable-verbose=true
332242

333243
.. _compiler_macos:
334244

@@ -367,7 +277,7 @@ scikit-learn from source:
367277
.. prompt:: bash $
368278

369279
conda create -n sklearn-dev -c conda-forge python numpy scipy cython \
370-
joblib threadpoolctl pytest compilers llvm-openmp
280+
joblib threadpoolctl pytest compilers llvm-openmp meson-python ninja
371281

372282
It is not always necessary but it is safer to open a new prompt before
373283
activating the newly created conda environment.
@@ -376,7 +286,9 @@ activating the newly created conda environment.
376286

377287
conda activate sklearn-dev
378288
make clean
379-
pip install -v --no-use-pep517 --no-build-isolation -e .
289+
pip install --editable . \
290+
--verbose --no-build-isolation \
291+
--config-settings editable-verbose=true
380292

381293
.. note::
382294

@@ -450,7 +362,9 @@ Finally, build scikit-learn in verbose mode (to check for the presence of the
450362
.. prompt:: bash $
451363

452364
make clean
453-
pip install -v --no-use-pep517 --no-build-isolation -e .
365+
pip install --editable . \
366+
--verbose --no-build-isolation \
367+
--config-settings editable-verbose=true
454368

455369
.. _compiler_linux:
456370

@@ -476,7 +390,9 @@ then proceed as usual:
476390
.. prompt:: bash $
477391

478392
pip3 install cython
479-
pip3 install --verbose --editable .
393+
pip3 install --editable . \
394+
--verbose --no-build-isolation \
395+
--config-settings editable-verbose=true
480396

481397
Cython and the pre-compiled wheels for the runtime dependencies (numpy, scipy
482398
and joblib) should automatically be installed in
@@ -508,15 +424,17 @@ in the user folder using conda:
508424
.. prompt:: bash $
509425

510426
conda create -n sklearn-dev -c conda-forge python numpy scipy cython \
511-
joblib threadpoolctl pytest compilers
427+
joblib threadpoolctl pytest compilers meson-python ninja
512428

513429
It is not always necessary but it is safer to open a new prompt before
514430
activating the newly created conda environment.
515431

516432
.. prompt:: bash $
517433

518434
conda activate sklearn-dev
519-
pip install -v --no-use-pep517 --no-build-isolation -e .
435+
pip install --editable . \
436+
--verbose --no-build-isolation \
437+
--config-settings editable-verbose=true
520438

521439
.. _compiler_freebsd:
522440

@@ -545,13 +463,17 @@ Finally, build the package using the standard command:
545463

546464
.. prompt:: bash $
547465

548-
pip install -v --no-use-pep517 --no-build-isolation -e .
466+
pip install --editable . \
467+
--verbose --no-build-isolation \
468+
--config-settings editable-verbose=true
549469

550470
For the upcoming FreeBSD 12.1 and 11.3 versions, OpenMP will be included in
551471
the base system and these steps will not be necessary.
552472

553473
.. _OpenMP: https://en.wikipedia.org/wiki/OpenMP
554474
.. _Cython: https://cython.org
475+
.. _meson-python: https://mesonbuild.com/meson-python
476+
.. _Ninja: https://ninja-build.org/
555477
.. _NumPy: https://numpy.org
556478
.. _SciPy: https://www.scipy.org
557479
.. _Homebrew: https://brew.sh
@@ -566,7 +488,9 @@ The following command will build scikit-learn using your default C/C++ compiler.
566488

567489
.. prompt:: bash $
568490

569-
pip install --verbose --editable .
491+
pip install --editable . \
492+
--verbose --no-build-isolation \
493+
--config-settings editable-verbose=true
570494

571495
If you want to build scikit-learn with another compiler handled by ``setuptools``,
572496
use the following command:
@@ -597,17 +521,3 @@ When setting these environment variables, it is advised to first check their
597521

598522
In addition, since Scikit-learn uses OpenMP, you need to include the appropriate OpenMP
599523
flag of your compiler into the ``CFLAGS`` and ``CPPFLAGS`` environment variables.
600-
601-
Parallel builds
602-
===============
603-
604-
It is possible to build scikit-learn compiled extensions in parallel by setting
605-
and environment variable as follows before calling the ``pip install`` or
606-
``python setup.py build_ext`` commands::
607-
608-
export SKLEARN_BUILD_PARALLEL=3
609-
pip install -v --no-use-pep517 --no-build-isolation -e .
610-
611-
On a machine with 2 CPU cores, it can be beneficial to use a parallelism level
612-
of 3 to overlap IO bound tasks (reading and writing files on disk) with CPU
613-
bound tasks (actually compiling).

‎doc/developers/contributing.rst

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -334,18 +334,6 @@ The next steps now describe the process of modifying code and submitting a PR:
334334
email to the committers. You may want to consider sending an email to the
335335
mailing list for more visibility.
336336

337-
.. note::
338-
339-
If you are modifying a Cython module, you have to re-compile after
340-
modifications and before testing them:
341-
342-
.. prompt:: bash $
343-
344-
pip install -v --no-use-pep517 --no-build-isolation -e .
345-
346-
Use the ``--no-build-isolation`` flag to avoid compiling the whole project
347-
each time, only the files you have modified.
348-
349337
It is often helpful to keep your local feature branch synchronized with the
350338
latest changes of the main scikit-learn repository:
351339

‎doc/developers/tips.rst

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,3 +355,19 @@ point.
355355

356356
Then use pytest to run only the tests of the module you are interested in
357357
debugging.
358+
359+
.. _meson_build_backend:
360+
361+
The Meson Build Backend
362+
=======================
363+
364+
Since scikit-learn 1.5.0 we use meson-python as the build tool. Meson is
365+
a new tool for scikit-learn and the PyData ecosystem. It is used by several
366+
other packages that have written good guides about what it is and how it works.
367+
368+
- `pandas setup doc
369+
<https://pandas.pydata.org/docs/development/contributing_environment.html#step-3-build-and-install-pandas>`_:
370+
pandas has a similar setup as ours (no spin or dev.py)
371+
- `scipy Meson doc
372+
<https://scipy.github.io/devdocs/building/understanding_meson.html>`_ gives
373+
more background about how Meson works behind the scenes

‎doc/whats_new/v1.5.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,8 @@ See :ref:`array_api` for more details.
5050
Support for building with Meson
5151
-------------------------------
5252

53-
Meson is now supported as a build backend, see :ref:`Building with Meson
54-
<building_with_meson>` for more details.
53+
Meson is now supported as a build backend, see :ref:`Building from source
54+
<install_bleeding_edge>` for more details.
5555

5656
:pr:`28040` by :user:`Loïc Estève <lesteve>`
5757

0 commit comments

Comments
 (0)
0