8000 DOC: update wheel building procedure for release · masasin/numpy@449899b · GitHub
[go: up one dir, main page]

Skip to content

Commit 449899b

Browse files
committed
DOC: update wheel building procedure for release
Document the single repository wheel building system. Strip out the old Wine instructions.
1 parent 36508b2 commit 449899b

File tree

1 file changed

+83
-169
lines changed

1 file changed

+83
-169
lines changed

doc/HOWTO_RELEASE.rst.txt

Lines changed: 83 additions & 169 deletions
< 8000 /tr>
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
This file gives an overview of what is necessary to build binary releases for
2-
NumPy. Windows binaries are built here using Wine, they can of course also be
3-
built on Windows itself. Building OS X binaries on another platform is not
4-
possible, but our current OSX binary build procedure uses travis-ci virtual
5-
machines running OSX.
2+
NumPy.
63

74
Current build and release info
85
==============================
@@ -39,15 +36,15 @@ Release Scripts
3936
Supported platforms and versions
4037
================================
4138

42-
Python 2.6-2.7 and >=3.2 are the currently supported versions when building
43-
from source. We test numpy against all these versions every time we merge
44-
code to trunk. Binary installers may be available for a subset of these
45-
versions (see below).
39+
Python 2.7 and >=3.4 are the currently supported versions when building from
40+
source. We test numpy against all these versions every time we merge code to
41+
trunk. Binary installers may be available for a subset of these versions (see
42+
below).
4643

4744
OS X
4845
----
4946

50-
Python 2.7 and >=3.3 are the versions for which we provide binary installers.
47+
Python 2.7 and >=3.4 are the versions for which we provide binary installers.
5148
OS X versions >= 10.6 are supported. We build binary wheels for OSX that are
5249
compatible with Python.org Python, system Python, homebrew and macports - see
5350
this `OSX wheel building summary
@@ -56,47 +53,49 @@ this `OSX wheel building summary
5653
Windows
5754
-------
5855

59-
32-bit Python 2.7, 3.3, 3.4 are the versions for which we provide binary
60-
installers. Windows XP, Vista and 7 are supported. Our current windows mingw
61-
toolchain is not able to build 64-bit binaries of numpy. We are hoping to
62-
update to a `mingw-w64 toolchain
63-
<https://github.com/numpy/numpy/wiki/Mingw-w64-faq>`_ soon.
56+
32-bit Python 2.7, 3.4, 3.5 are the versions for which we provide binary
57+
installers. Windows XP, Vista, 7, 8 and 10 are supported. We build numpy
58+
using the MSVC compilers on Appveyor, but we are hoping to update to a
59+
`mingw-w64 toolchain <https://github.com/numpy/numpy/wiki/Mingw-w64-faq>`_
60+
soon.
6461

6562
Linux
6663
-----
6764

68-
Many distributions include NumPy. Building from source is also relatively
69-
straightforward. Only tarballs are created for Linux, no specific binary
70-
installers are provided (yet).
65+
We build and ship `manylinux1 <https://www.python.org/dev/peps/pep-0513>`_
66+
wheels for numpy. Many Linux distributions include their own binary builds
67+
of NumPy.
7168

7269
BSD / Solaris
7370
-------------
74 8000 -
No binaries are provided, but succesful builds on Solaris and BSD have been
71+
72+
No binaries are provided, but successful builds on Solaris and BSD have been
7573
reported.
7674

7775
Tool chain
7876
==========
7977

78+
We build all our wheels on cloud infrastructure - so this list of compilers is
79+
for information and debugging builds locally. See the ``.travis.yml`` and
80+
``appveyor.yml`` scripts in the `numpy wheels`_ repo for the definitive source
81+
of the build recipes.
82+
8083
Compilers
8184
---------
8285

8386
The same gcc version is used as the one with which Python itself is built on
8487
each platform. At the moment this means:
8588

8689
* OS X builds on travis currently use `clang`. It appears that binary wheels
87-
for OSX >= 10.6 can be safely built from from OSX 10.9 when building against
88-
the Python from the Python.org installers.
89-
* Windows builds use MinGW 3.4.5. Updating this to a more recent MinGW with
90-
GCC 4.x is desired, but there are still practical difficulties in building
91-
the binary installers.
90+
for OSX >= 10.6 can be safely built from from the travis-ci OSX 10.9 VMs
91+
when building against the Python from the Python.org installers;
92+
* Windows builds use the MSVC version corresponding to the Python being built
93+
against;
94+
* Manylinux1 wheels use the gcc provided on the Manylinux docker images.
9295

9396
You will need Cython for building the binaries. Cython compiles the ``.pyx``
9497
files in the numpy distribution to ``.c`` files.
9598

96-
Fortran: on OS X gfortran from `this site <http://r.research.att.com/tools/>`_
97-
is used. On Windows g77 (included in MinGW) is the current default, in the future
98-
this may shift to gfortran as well.
99-
10099
Python
101100
------
102101
* Python(s) from `python.org <http://python.org>`_
@@ -110,54 +109,6 @@ Building docs
110109
* Matplotlib
111110
* Texlive (or MikTeX on Windows)
112111

113-
Wine
114-
----
115-
For building Windows binaries on OS X Wine can be used. In Wine the following
116-
needs to be installed:
117-
118-
* Python 2.6-2.7 and 3.3
119-
* MakeNsis
120-
* CpuId plugin for MakeNsis : this can be found in the NumPy source tree under
121-
tools/win32build/cpucaps and has to be built with MinGW (see SConstruct file in
122-
that dir for details)
123-
* MinGW
124-
* ATLAS, 3x ([No SSE, SSE2, SSE3] for superpack installer) : ATLAS does not
125-
compile under wine or on Windows out of the box. Binaries for ATLAS can be
126-
found in the vendor repository on GitHub (http://github.com/numpy/vendor).
127-
128-
To install Wine on OS X Snow Leopard the current options are to compile a
129-
current unstable version ,`<http://wiki.winehq.org/MacOSX/Building>`_, or to use
130-
an install script from `here <http://code.google.com/p/osxwinebuilder/>`_. For
131-
me, the former option did not work (everything compiled, but after installing
132-
Python the command ``import tempfile`` resulted in an exception. The latter
133-
option did work.
134-
135-
After successful installation and an invocation of the wine executable, a
136-
~/.wine folder exists - new programs will be installed there in
137-
~/.wine/drive_c. Installing Windows programs with .exe executables is done by
138-
running
139-
140-
$ wine yourprog.exe
141-
142-
and MSI installers can be installed with
143-
144-
$ msiexec /i yourprog.msi
145-
146-
For the above to work you probably need to put the wine-1.x.x/bin directory in
147-
your PATH.
148-
149-
To install MinGW, the easiest option is to use the automated installer on the
150-
MinGW download page. This will give you (at this moment) GCC 3.4.5; GCC 4.x is
151-
still not supported officially by MinGW.
152-
153-
To be able to use gcc and MakeNsis in Wine, the locations of gcc.exe and
154-
makensis.exe should be added to the Windows environment variable PATH. This can
155-
easily be done by running
156-
157-
$ wine regedit
158-
159-
add adding a PATH variable in HKEY_CURRENT_USER/Environment.
160-
161112
Virtualenv
162113
----------
163114
Virtualenv is a very useful tool to keep several versions of packages around.
@@ -167,18 +118,17 @@ It is also used in the Paver script to build the docs.
167118
What is released
168119
================
169120

170-
Binaries
171-
--------
172-
173-
Windows binary installers in "superpack" form for Python 2.7/3.3/3.4. A
174-
superpack contains three builds, for SSE2, SSE3 and no SSE.
175-
176121
Wheels
177122
------
178123

179-
OSX wheels built via travis-ci : see - see `building OSX wheels`_.
124+
* Windows wheels for Python 2.7, 3.4, 3.5, for 32- and 64-bit, built using
125+
Appveyor;
126+
* Dual architecture OSX wheels built via travis-ci;
127+
* 32- and 64-bit Manylinux1 wheels built via travis-ci.
128+
129+
See the `numpy wheels`_ building repository for more detail.
180130

181-
.. _build OSX wheels: https://github.com/MacPython/numpy-wheels
131+
.. _numpy wheels : https://github.com/MacPython/numpy-wheels
182132

183133
Other
184134
-----
@@ -366,78 +316,36 @@ Increment the release number in setup.py. Release candidates should have "rc1"
366316
Also create a new version hash in cversions.txt and a corresponding version
367317
define NPY_x_y_API_VERSION in numpyconfig.h
368318

369-
Trigger the OSX builds on travis
370-
--------------------------------
319+
Trigger the wheel builds on travis-ci and Appveyor
320+
--------------------------------------------------
321+
322+
See the `numpy wheels` repository.
371323

372-
See `build OSX wheels`_.
324+
In that repository edit the files:
373325

374-
You may need to check the ``.travis.yml`` file of the
375-
https://github.com/MacPython/numpy-wheels repository.
326+
* ``.travis.yml``;
327+
* ``appveyor.yml``.
376328

377-
Make sure that the releast tag has been pushed, and that the ``.travis.yml``
378-
is set thusly::
329+
In both cases, set the ``BUILD_COMMIT`` variable to the current release tag -
330+
e.g. ``v1.11.1``.
379331

380-
- NP_COMMIT=latest-tag # comment out to build version in submodule
332+
Make sure that the release tag has been pushed.
381333

382-
Trigger a build by doing an empty (or otherwise) commit to the repository::
334+
Trigger a build by doing a commit of your edits to ``.travis.yml`` and
335+
``appveyor.yml`` to the repository::
383336

384337
cd /path/to/numpy-wheels
385-
git commit --allow-empty
338+
# Edit .travis.yml, appveyor.yml
339+
git commit
386340
git push
387341

388-
The wheels, once built, appear in http://wheels.scipy.org
389-
390-
Trigger Windows builds on Appveyor
391-
----------------------------------
392-
393-
See: `build Windows wheels`_
394-
395-
* Clone / update the https://github.com/numpy/windows-wheel-builder repository;
396-
* Check the ``appveyor.yml`` file in that repository;
397-
* Edit the line starting ``NP_VERSION:`` to give the numpy tag that you want
398-
to build;
399-
* Push up to github to trigger a build.
400-
401-
The wheels appear in a Rackspace CDN container at:
402-
403-
* http://58688808cd85529d4031-38dee5dca2544308e91131f21428d924.r12.cf2.rackcdn.com
404-
* https://84c1a9a06db6836f5a98-38dee5dca2544308e91131f21428d924.ssl.cf2.rackcdn.com
405-
406-
The contents via the HTTPS URL seems to get updated more slowly than via the
407-
HTTP URL, so if you need the binaries quickly, prefer the HTTP URL.
408-
409-
.. _build Windows wheels: https://github.com/numpy/windows-wheel-builder
342+
The wheels, once built, appear at a Rackspace container pointed at by:
410343

411-
Trigger Manylinux builds on travis-ci
412-
-------------------------------------
344+
* http://wheels.scipy.org
345+
* https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com
413346

414-
.. note::
415-
416-
Until we move the manylinux build scripts, you'll need to ask
417-
``@matthew-brett`` to make you a collaborator on the manylinux repos.
418-
419-
* Clone / update the repository at
420-
https://github.com/matthew-brett/manylinux-builds
421-
* Edit the line in ``.travis.yml`` starting ``NUMPY_VERSIONS=`` to set the
422-
numpy tag to build;
423-
* Push your edits to ``.travis.yml`` up to github to trigger a mass manylinux
424-
build;
425-
* Clone / update the repository at
426-
https://github.com/matthew-brett/manylinux-testing;
427-
* Push an empty commit to the ``manylinux-testing`` repo to trigger a test run
428-
of the newly-built numpy wheels with a range of dependent libraries, as well
429-
as numpy's own unit tests. The tests will take several hours.
430-
431-
The built wheels will be available from a Rackspace CDN container at:
432-
433-
* http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
434-
* https://d9a97980b71d47cde94d-aae005c4999d7244ac63632f8b80e089.ssl.cf2.rackcdn.com
435-
436-
As for the other Rackspace containers, the HTTP address may update first, and
437-
you should wait 15 minutes after the build finishes before fetching the
438-
binaries. For the manylinux wheels, the time to run ``manylinux-testing`` is
439-
much greater than 15 minutes, so waiting for the tests to pass will be enough
440-
time for the binaries to refresh on Rackspace.
347+
The HTTP address may update first, and you should wait 15 minutes after the
348+
build finishes before fetching the binaries.
441349

442350
Make the release
443351
----------------
@@ -471,30 +379,39 @@ downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI.
471379
::
472380

473381
cd ~/wheelhouse # local directory to cache wheel downloads
474-
MANYLINUX_URL=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
475-
WINDOWS_URL=http://58688808cd85529d4031-38dee5dca2544308e91131f21428d924.r12.cf2.rackcdn.com
476-
OSX_URL=http://wheels.scipy.org
477-
wheel-uploader -u $MANYLINUX_URL -v -s -t manylinux1 numpy 1.11.1rc1
478-
wheel-uploader -u $WINDOWS_URL -v -s -t win numpy 1.11.1rc1
479-
wheel-uploader -u $OSX_URL -v -s -t macosx numpy 1.11.1rc1
382+
CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com
383+
wheel-uploader -u $CDN_URL -w warehouse -v -s -t win numpy 1.11.1rc1
384+
wheel-uploader -u $CDN_URL -w warehouse -v -s -t macosx numpy 1.11.1rc1
385+
wheel-uploader -u $CDN_URL -w warehouse -v -s -t manylinux1 numpy 1.11.1rc1
480386

481387
The ``-v`` flag gives verbose feedback, ``-s`` causes the script to sign the
482-
wheels with your GPG key before upload.
483-
484-
You may well find that these uploads break at some point, with error messages
485-
from the PyPI server. In this case you'll have to continue the uploads by
486-
hand using `twine <https://pypi.python.org/pypi/twine>`_, using something
487-
like::
488-
489-
twine upload -s numpy-1.11.1rc1-cp34-*.whl
490-
491-
Do this for the wheel files that ``wheel-uploader`` downloaded, but for which
492-
the upload failed.
493-
494-
The ``warehouse`` PyPI server seems to be more reliable in receiving automated
495-
wheel uploads. You can set the repository to upload to with the ``-r`` flag
496-
to ``wheel-uploader`` and ``twine``. The warehouse repository URL for your
497-
``~/.pypirc`` file is https://upload.pypi.io/legacy/
388+
wheels with your GPG key before upload. ``-r warehouse`` causes the upload to
389+
use the Warehouse PyPI server. This is a good idea because the Warehouse
390+
server seems to be a lot more reliable in receiving automated wheel uploads.
391+
For this flag to work, you will need a ``warehouse`` section in your
392+
``~/.pypirc`` file, of form:
393+
394+
[distutils]
395+
index-servers =
396+
pypi
397+
warehouse
398+
399+
[pypi]
400+
username:your_user_name
401+
password:your_password
402+
403+
[warehouse]
404+
repository: https://upload.pypi.io/legacy/
405+
username: your_user_name
406+
password: your_password
407+
408+
[server-login]
409+
username:your_user_name
410+
password:your_password
411+
412+
Don't forget to upload the wheels before the source tarball, so there is no
413+
period for which people switch from an expected binary install to a source
414+
install from PyPI.
498415

499416
There are two ways to update the source release on PyPI, the first one is::
500417

@@ -509,9 +426,6 @@ The second way is to upload the PKG_INFO file inside the sdist dir in the
509426
web interface of PyPI. The source tarball can also be uploaded through this
510427
interface.
511428

512-
To push the travis-ci OSX wheels up to PyPI see :
513-
https://github.com/MacPython/numpy-wheels#uploading-the-built-wheels-to-pypi
514-
515429
.. _push-tag-and-commit:
516430

517431
Push the release tag and commit

0 commit comments

Comments
 (0)
0