8000 bpo-30450: Pull Windows dependencies from GitHub rather than svn (GH-… · python/cpython@04431c9 · GitHub
[go: up one dir, main page]
Skip to content

Commit 04431c9

Browse files
zwarezware
authored
bpo-30450: Pull Windows dependencies from GitHub rather than svn (GH-1783) (GH-2237)
The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio. Also fixes an outdated note about _lzma in PCbuild/readme.txt (cherry-picked from commit 51599e2)
1 parent 292b421 commit 04431c9

File tree

5 files changed

+250
-88
lines changed

5 files changed

+250
-88
lines changed

‎Misc/NEWS

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -286,7 +286,7 @@ Tools/Demos
286286
Tests
287287
-----
288288

289-
* bpo-30357: test_thread: setUp() now uses support.threading_setup() and
289+
- bpo-30357: test_thread: setUp() now uses support.threading_setup() and
290290
support.threading_cleanup() to wait until threads complete to avoid
291291
random side effects on following tests. Initial patch written by Grzegorz
292292
Grzywacz.
@@ -297,6 +297,14 @@ Tests
297297
if it doesn't exist) now will be assigned to the target of the "as" clause,
298298
if there is one.
299299

300+
Windows
301+
-------
302+
303+
- bpo-30450: The build process on Windows no longer depends on Subversion,
304+
instead pulling external code from GitHub via a Python script. If Python 3.6
305+
is not found on the system (via ``py -3.6``), NuGet is used to download a
306+
copy of 32-bit Python.
307+
300308

301309
What's New in Python 3.6.1?
302310
===========================

‎PCbuild/get_external.py

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/usr/bin/env python3
2+
3+
import argparse
4+
import os
5+
import pathlib
6+
import zipfile
7+
from urllib.request import urlretrieve
8+
9+
10+
def fetch_zip(commit_hash, zip_dir, *, org='python', binary=False, verbose):
11+
repo = f'cpython-{"bin" if binary else "source"}-deps'
12+
url = f'https://github.com/{org}/{repo}/archive/{commit_hash}.zip'
13+
reporthook = None
14+
if verbose:
15+
reporthook = print
16+
zip_dir.mkdir(parents=True, exist_ok=True)
17+
filename, headers = urlretrieve(
18+
url,
19+
zip_dir / f'{commit_hash}.zip',
20+
reporthook=reporthook,
21+
)
22+
return filename
23+
24+
25+
def extract_zip(externals_dir, zip_path):
26+
with zipfile.ZipFile(os.fspath(zip_path)) as zf:
27+
zf.extractall(os.fspath(externals_dir))
28+
return externals_dir / zf.namelist()[0].split('/')[0]
29+
30+
31+
def parse_args():
32+
p = argparse.ArgumentParser()
33+
p.add_argument('-v', '--verbose', action='store_true')
34+
p.add_argument('-b', '--binary', action='store_true',
35+
help='Is the dependency in the binary repo?')
36+
p.add_argument('-O', '--organization',
37+
help='Organization owning the deps repos', default='python')
38+
p.add_argument('-e', '--externals-dir', type=pathlib.Path,
39+
help='Directory in which to store dependencies',
40+
default=pathlib.Path(__file__).parent.parent / 'externals')
41+
p.add_argument('tag',
42+
help='tag of the dependency')
43+
return p.parse_args()
44+
45+
46+
def main():
47+
args = parse_args()
48+
zip_path = fetch_zip(
49+
args.tag,
50+
args.externals_dir / 'zips',
51+
org=args.organization,
52+
binary=args.binary,
53+
verbose=args.verbose,
54+
)
55+
final_name = args.externals_dir / args.tag
56+
extract_zip(args.externals_dir, zip_path).replace(final_name)
57+
58+
59+
if __name__ == '__main__':
60+
main()

‎PCbuild/get_externals.bat

Lines changed: 66 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -2,58 +2,59 @@
22
setlocal
33
rem Simple script to fetch source for external libraries
44

5-
if not exist "%~dp0..\externals" mkdir "%~dp0..\externals"
6-
pushd "%~dp0..\externals"
5+
if "%PCBUILD%"=="" (set PCBUILD=%~dp0)
6+
if "%EXTERNALS_DIR%"=="" (set EXTERNALS_DIR=%PCBUILD%\..\externals)
7+
if "%NUGET%"=="" (set NUGET=%EXTERNALS_DIR%\nuget.exe)
8+
if "%NUGET_URL%"=="" (set NUGET_URL=https://aka.ms/nugetclidl)
79

8-
if "%SVNROOT%"=="" set SVNROOT=http://svn.python.org/projects/external/
10+
set DO_FETCH=true
11+
set DO_CLEAN=false
912

10-
rem Optionally clean up first. Be warned that this can be very destructive!
11-
if not "%1"=="" (
12-
for %%c in (-c --clean --clean-only) do (
13-
if "%1"=="%%c" goto clean
14-
)
15-
goto usage
16-
)
17-
goto fetch
13+
:CheckOpts
14+
if "%~1"=="--no-tkinter" (set IncludeTkinter=false) & shift & goto CheckOpts
15+
if "%~1"=="--no-openssl" (set IncludeSSL=false) & shift & goto CheckOpts
16+
if "%~1"=="--python" (set PYTHON_FOR_BUILD=%2) & shift & shift & goto CheckOpts
17+
if "%~1"=="--organization" (set ORG=%2) & shift & shift & goto CheckOpts
18+
if "%~1"=="-c" (set DO_CLEAN=true) & shift & goto CheckOpts
19+
if "%~1"=="--clean" (set DO_CLEAN=true) & shift & goto CheckOpts
20+
if "%~1"=="--clean-only" (set DO_FETCH=false) & goto clean
21+
if "x%~1" NEQ "x" goto usage
1822

23+
if "%DO_CLEAN%"=="false" goto fetch
1924
:clean
2025
echo.Cleaning up external libraries.
21-
for /D %%d in (
22-
bzip2-*
23-
db-*
24-
nasm-*
25-
openssl-*
26-
tcl-*
27-
tcltk*
28-
tk-*
29-
tix-*
30-
sqlite-*
31-
xz-*
32-
) do (
33-
echo.Removing %%d
34-
rmdir /s /q %%d
35-
)
36-
if "%1"=="--clean-only" (
37-
goto end
26+
if exist "%EXTERNALS_DIR%" (
27+
rem Sometimes this fails the first time; try it twice
28+
rmdir /s /q "%EXTERNALS_DIR%" || rmdir /s /q "%EXTERNALS_DIR%"
3829
)
3930

31+
if "%DO_FETCH%"=="false" goto end
4032
:fetch
41-
rem Fetch current versions
42-
43-
svn --version > nul 2>&1
44-
if ERRORLEVEL 9009 (
45-
echo.svn.exe must be on your PATH.
46-
echo.Try TortoiseSVN (http://tortoisesvn.net/^) and be sure to check the
47-
echo.command line tools option.
48-
popd
49-
exit /b 1
33+
34+
if "%ORG%"=="" (set ORG=python)
35+
36+
if "%PYTHON_FOR_BUILD%"=="" (
37+
echo Checking for installed python...
38+
py -3.6 -V >nul 2>&1 && (set PYTHON_FOR_BUILD=py -3.6)
39+
)
40+
if "%PYTHON_FOR_BUILD%"=="" (
41+
if NOT exist "%EXTERNALS_DIR%" mkdir "%EXTERNALS_DIR%"
42+
if NOT exist "%NUGET%" (
43+
echo Downloading nuget...
44+
rem NB: Must use single quotes around NUGET here, NOT double!
45+
rem Otherwise, a space in the path would break things
46+
powershell.exe -Command Invoke-WebRequest %NUGET_URL% -OutFile '%NUGET%'
47+
)
48+
echo Installing Python via nuget...
49+
"%NUGET%" install pythonx86 -ExcludeVersion -OutputDirectory "%EXTERNALS_DIR%"
50+
rem Quote it here; it's not quoted later because "py -3.6" wouldn't work
51+
set PYTHON_FOR_BUILD="%EXTERNALS_DIR%\pythonx86\tools\python.exe"
5052
)
5153

5254
echo.Fetching external libraries...
5355

5456
set libraries=
5557
set libraries=%libraries% bzip2-1.0.6
56-
if NOT "%IncludeSSL%"=="false" set libraries=%libraries% nasm-2.11.06
5758
if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2k
5859
set libraries=%libraries% sqlite-3.14.2.0
5960
if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tcl-core-8.6.6.0
@@ -62,43 +63,48 @@ if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tix-8.4.3.6
6263
set libraries=%libraries% xz-5.2.2
6364

6465
for %%e in (%libraries%) do (
65-
if exist %%e (
66+
if exist "%EXTERNALS_DIR%\%%e" (
6667
echo.%%e already exists, skipping.
6768
) else (
6869
echo.Fetching %%e...
69-
svn export -q %SVNROOT%%%e
70+
%PYTHON_FOR_BUILD% "%PCBUILD%get_external.py" -O %ORG% %%e
71+
)
72+
)
73+
74+
echo.Fetching external binaries...
75+
76+
set binaries=
77+
set binaries=%binaries%
78+
if NOT "%IncludeSSL%"=="false" set binaries=%binaries% nasm-2.11.06
79+
80+
for %%b in (%binaries%) do (
81+
if exist "%EXTERNALS_DIR%\%%b" (
82+
echo.%%b already exists, skipping.
83+
) else (
84+
echo.Fetching %%b...
85+
%PYTHON_FOR_BUILD% "%PCBUILD%get_external.py" -b -O %ORG% %%b
7086
)
7187
)
7288

89+
echo Finished.
7390
goto end
7491

7592
:usage
76-
echo.invalid argument: %1
77-
echo.usage: %~n0 [[ -c ^| --clean ] ^| --clean-only ]
93+
echo.Valid options: -c, --clean, --clean-only, --organization, --python,
94+
echo.--no-tkinter, --no-openssl
7895
echo.
79-
echo.Pull all sources necessary for compiling optional extension modules
80-
echo.that rely on external libraries. Requires svn.exe to be on your PATH
81-
echo.and pulls sources from %SVNROOT%.
96+
echo.Pull all sources and binaries necessary for compiling optional extension
97+
echo.modules that rely on external libraries.
8298
echo.
83-
echo.Use the -c or --clean option to clean up all external library sources
84-
echo.before pulling in the current versions.
99+
echo.The --organization option determines which github organization to download
100+
echo.from, the --python option determines which Python 3.6+ interpreter to use
101+
echo.with PCbuild\get_external.py.
102+
echo.
103+
echo.Use the -c or --clean option to remove the entire externals directory.
85104
echo.
86105
echo.Use the --clean-only option to do the same cleaning, without pulling in
87106
echo.anything new.
88107
echo.
89-
echo.Only the first argument is checked, all others are ignored.
90-
echo.
91-
echo.**WARNING**: the cleaning options unconditionally remove any directory
92-
echo.that is a child of
93-
echo. %CD%
94-
echo.and matches wildcard patterns beginning with bzip2-, db-, nasm-, openssl-,
95-
echo.tcl-, tcltk, tk-, tix-, sqlite-, or xz-, and as such has the potential
96-
echo.to be very destructive if you are not aware of what it is doing. Use with
97-
echo.caution!
98-
popd
99108
exit /b -1
100109

101-
102110
:end
103-
echo Finished.
104-
popd

‎PCbuild/readme.txt

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@ Quick Start Guide
22
-----------------
33

44
1. Install Microsoft Visual Studio 2015, any edition.
5-
2. Install Subversion, and make sure 'svn.exe' is on your PATH.
6-
3. Run "build.bat -e" to build Python in 32-bit Release configuration.
7-
4. (Optional, but recommended) Run the test suite with "rt.bat -q".
5+
1a. Optionally install Python 3.6 or later. If not installed,
6+
get_externals.bat (build.bat -e) will download and use Python via
7+
NuGet.
8+
2. Run "build.bat -e" to build Python in 32-bit Release configuration.
9+
3. (Optional, but recommended) Run the test suite with "rt.bat -q".
810

911

1012
Building Python using Microsoft Visual C++
@@ -164,8 +166,7 @@ _bz2
164166
Homepage:
165167
http://www.bzip.org/
166168
_lzma
167-
Python wrapper for the liblzma compression library, using pre-built
168-
binaries of XZ Utils version 5.0.5
169+
Python wrapper for version 5.2.2 of the liblzma compression library
169170
Homepage:
170171
http://tukaani.org/xz/
171172
_ssl
@@ -236,9 +237,16 @@ order to download the relevant source files for each project before they
236237
can be built. However, a simple script is provided to make this as
237238
painless as possible, called "get_externals.bat" and located in this
238239
directory. This script extracts all the external sub-projects from
239-
http://svn.python.org/projects/external
240-
via Subversion (so you'll need svn.exe on your PATH) and places them
241-
in ..\externals (relative to this directory).
240+
https://github.com/python/cpython-source-deps
241+
and
242+
https://github.com/python/cpython-bin-deps
243+
via a Python script called "get_external.py", located in this directory.
244+
If Python 3.6 or later is not available via the "py.exe" launcher, the
245+
path or command to use for Python can be provided in the PYTHON_FOR_BUILD
246+
environment variable, or get_externals.bat will download the latest
247+
version of NuGet and use it to download the latest "pythonx86" package
248+
for use with get_external.py. Everything downloaded by these scripts is
249+
stored in ..\externals (relative to this directory).
242250

243251
It is also possible to download sources from each project's homepage,
244252
though you may have to change folder names or pass the names to MSBuild

0 commit comments

Comments
 (0)
0