8000 Merge pull request #14518 from mattip/hide-config-probe · numpy/numpy@815061c · GitHub
[go: up one dir, main page]
Skip to content

Commit 815061c

Browse files
rgommersrgommers
authored
Merge pull request #14518 from mattip/hide-config-probe
BUILD: Hide platform configuration probe behind --debug-configure
2 parents 3096f1a + 492fdab commit 815061c

15 files changed

+71
-54
lines changed

‎azure-pipelines.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
python3 -m pip install --user -r test_requirements.txt && \
3030
python3 -m pip install . && \
3131
F77=gfortran-5 F90=gfortran-5 \
32-
CFLAGS='-UNDEBUG -std=c99' python3 runtests.py -n --mode=full -- -rsx --junitxml=junit/test-results.xml && \
32+
CFLAGS='-UNDEBUG -std=c99' python3 runtests.py -n --debug-configure --mode=full -- -rsx --junitxml=junit/test-results.xml && \
3333
python3 tools/openblas_support.py --check_version $(OpenBLAS_version)"
3434
displayName: 'Run 32-bit Ubuntu Docker Build / Tests'
3535
- task: PublishTestResults@2
@@ -94,7 +94,7 @@ jobs:
9494
displayName: 'Check for unreachable code paths in Python modules'
9595
# prefer usage of clang over gcc proper
9696
# to match likely scenario on many user mac machines
97-
- script: python setup.py build -j 4 install
97+
- script: python setup.py build -j 4 build_src -v install
9898
displayName: 'Build NumPy'
9999
env:
100100
BLAS: None

‎doc/DISTUTILS.rst.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ in writing setup scripts:
243243
after processing all source generators, no extension module will
244244
be built. This is the recommended way to conditionally define
245245
extension modules. Source generator functions are called by the
246-
``build_src`` command of ``numpy.distutils``.
246+
``build_src`` sub-command of ``numpy.distutils``.
247247

248248
For example, here is a typical source generator function::
249249

‎doc/source/dev/development_environment.rst

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,11 @@ installs a ``.egg-link`` file into your site-packages as well as adjusts the
9696
Other build options
9797
-------------------
9898

99+
Build options can be discovered by running any of::
100+
101+
$ python setup.py --help
102+
$ python setup.py --help-commands
103+
99104
It's possible to do a parallel build with ``numpy.distutils`` with the ``-j`` option;
100105
see :ref:`parallel-builds` for more details.
101106

@@ -106,6 +111,16 @@ source tree is to use::
106111
$ export PYTHONPATH=/some/owned/folder/lib/python3.4/site-packages
107112

108113

114+
NumPy uses a series of tests to probe the compiler and libc libraries for
115+
funtions. The results are stored in ``_numpyconfig.h`` and ``config.h`` files
116+
using ``HAVE_XXX`` definitions. These tests are run during the ``build_src``
117+
phase of the ``_multiarray_umath`` module in the ``generate_config_h`` and
118+
``generate_numpyconfig_h`` functions. Since the output of these calls includes
119+
many compiler warnings and errors, by default it is run quietly. If you wish
120+
to see this output, you can run the ``build_src`` stage verbosely::
121+
122+
$ python build build_src -v
123+
109124
Using virtualenvs
110125
-----------------
111126

‎numpy/core/setup.py

Lines changed: 6 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -497,10 +497,10 @@ def generate_config_h(ext, build_dir):
497497
#endif
498498
"""))
499499

500-
print('File:', target)
500+
log.info('File: %s' % target)
501501
with open(target) as target_f:
502-
print(target_f.read())
503-
print('EOF')
502+
log.info(target_f.read())
503+
log.info('EOF')
504504
else:
505505
mathlibs = []
506506
with open(target) as target_f:
@@ -587,10 +587,10 @@ def generate_numpyconfig_h(ext, build_dir):
587587
"""))
588588

589589
# Dump the numpyconfig.h header to stdout
590-
print('File: %s' % target)
590+
log.info('File: %s' % target)
591591
with open(target) as target_f:
592-
print(target_f.read())
593-
print('EOF')
592+
log.info(target_f.read())
593+
log.info('EOF')
594594
config.add_data_files((header_dir, target))
595595
return target
596596

@@ -638,23 +638,6 @@ def generate_api(ext, build_dir):
638638
join(codegen_dir, 'genapi.py'),
639639
]
640640

641-
#######################################################################
642-
# dummy module #
643-
#######################################################################
644-
645-
# npymath needs the config.h and numpyconfig.h files to be generated, but
646-
# build_clib cannot handle generate_config_h and generate_numpyconfig_h
647-
# (don't ask). Because clib are generated before extensions, we have to
648-
# explicitly add an extension which has generate_config_h and
649-
# generate_numpyconfig_h as sources *before* adding npymath.
650-
651-
config.add_extension('_dummy',
652-
sources=[join('src', 'dummymodule.c'),
653-
generate_config_h,
654-
generate_numpyconfig_h,
655-
generate_numpy_api]
656-
)
657-
658641
#######################################################################
659642
# npymath library #
660643
#######################################################################

‎numpy/distutils/__init__.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ def customized_fcompiler(plat=None, compiler=None):
2828
c.customize()
2929
return c
3030

31-
def customized_ccompiler(plat=None, compiler=None):
32-
c = ccompiler.new_compiler(plat=plat, compiler=compiler)
31+
def customized_ccompiler(plat=None, compiler=None, verbose=1):
32+
c = ccompiler.new_compiler(plat=plat, compiler=compiler, verbose=verbose)
3333
c.customize('')
3434
return c

‎numpy/distutils/ccompiler.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,10 @@ def CCompiler_spawn(self, cmd, display=None):
140140
display = ' '.join(list(display))
141141
log.info(display)
142142
try:
143-
subprocess.check_output(cmd)
143+
if self.verbose:
144+
subprocess.check_output(cmd)
145+
else:
146+
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
144147
except subprocess.CalledProcessError as exc:
145148
o = exc.output
146149
s = exc.returncode
@@ -162,7 +165,8 @@ def CCompiler_spawn(self, cmd, display=None):
162165
if is_sequence(cmd):
163166
cmd = ' '.join(list(cmd))
164167

165-
forward_bytes_to_stdout(o)
168+
if self.verbose:
169+
forward_bytes_to_stdout(o)
166170

167171
if re.search(b'Too many open files', o):
168172
msg = '\nTry rerunning setup command until build succeeds.'
@@ -727,10 +731,12 @@ def CCompiler_cxx_compiler(self):
727731
_distutils_new_compiler = new_compiler
728732
def new_compiler (plat=None,
729733
compiler=None,
730-
verbose=0,
734+
verbose=None,
731735
dry_run=0,
732736
force=0):
733737
# Try first C compilers from numpy.distutils.
738+
if verbose is None:
739+
verbose = log.get_threshold() <= log.INFO
734740
if plat is None:
735741
plat = os.name
736742
try:
@@ -763,6 +769,7 @@ def new_compiler (plat=None,
763769
raise DistutilsModuleError(("can't compile C/C++ code: unable to find class '%s' " +
764770
"in module '%s'") % (class_name, module_name))
765771
compiler = klass(None, dry_run, force)
772+
compiler.verbose = verbose
766773
log.debug('new_compiler returns %s' % (klass))
767774
return compiler
768775

‎numpy/distutils/command/build.py

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,6 @@ class build(old_build):
1616
user_options = old_build.user_options + [
1717
('fcompiler=', None,
1818
"specify the Fortran compiler type"),
19-
('parallel=', 'j',
20-
"number of parallel jobs"),
2119
]
2220

2321
help_options = old_build.help_options + [
@@ -28,14 +26,8 @@ class build(old_build):
2826
def initialize_options(self):
2927
old_build.initialize_options(self)
3028
self.fcompiler = None
31-
self.parallel = None
3229

3330
def finalize_options(self):
34-
if self.parallel:
35-
try:
36-
self.parallel = int(self.parallel)
37-
except ValueError:
38-
raise ValueError("--parallel/-j argument must be an integer")
3931
build_scripts = self.build_scripts
4032
old_build.finalize_options(self)
4133
plat_specifier = ".{}-{}.{}".format(get_platform(), *sys.version_info[:2])

‎numpy/distutils/command/build_src.py

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,12 @@ class build_src(build_ext.build_ext):
5353
('inplace', 'i',
5454
"ignore build-lib and put compiled extensions into the source " +
5555
"directory alongside your pure Python modules"),
56+
('verbose', 'v',
57+
"change logging level from WARN to INFO which will show all " +
58+
"compiler output")
5659
]
5760

58-
boolean_options = ['force', 'inplace']
61+
boolean_options = ['force', 'inplace', 'verbose']
5962

6063
help_options = []
6164

@@ -76,6 +79,7 @@ def initialize_options(self):
7679
self.swig_opts = None
7780
self.swig_cpp = None
7881
self.swig = None
82+
self.verbose = False
7983

8084
def finalize_options(self):
8185
self.set_undefined_options('build',
@@ -365,6 +369,13 @@ def generate_sources(self, sources, extension):
365369
build_dir = os.path.join(*([self.build_src]
366370
+name.split('.')[:-1]))
367371
self.mkpath(build_dir)
372+
373+
if self.verbose:
374+
new_level = log.INFO
375+
else:
376+
new_level = log.WARN
377+
old_level = log.set_threshold(new_level)
378+
368379
for func in func_sources:
369380
source = func(extension, build_dir)
370381
if not source:
@@ -375,7 +386,7 @@ def generate_sources(self, sources, extension):
375386
else:
376387
log.info(" adding '%s' to sources." % (source,))
377388
new_sources.append(source)
378-
389+
log.set_threshold(old_level)
379390
return new_sources
380391

381392
def filter_py_files(self, sources):

‎numpy/distutils/log.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,8 @@ def set_threshold(level, force=False):
6767
' %s to %s' % (prev_level, level))
6868
return prev_level
6969

70+
def get_threshold():
71+
return _global_log.threshold
7072

7173
def set_verbosity(v, force=False):
7274
prev_level = _global_log.threshold

‎numpy/distutils/system_info.py

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@
146146
from distutils.errors import DistutilsError
147147
from distutils.dist import Distribution
148148
import distutils.sysconfig
149-
from distutils import log
149+
from numpy.distutils import log
150150
from distutils.util import get_platform
151151

152152
from numpy.distutils.exec_command import (
@@ -550,15 +550,13 @@ class system_info(object):
550550
dir_env_var = None
551551
search_static_first = 0 # XXX: disabled by default, may disappear in
552552
# future unless it is proved to be useful.
553-
verbosity = 1
554553
saved_results = {}
555554

556555
notfounderror = NotFoundError
557556

558557
def __init__(self,
559558
default_lib_dirs=default_lib_dirs,
560559
default_include_dirs=default_include_dirs,
561-
verbosity=1,
562560
):
563561
self.__class__.info = {}
564562
self.local_prefixes = []
@@ -704,7 +702,7 @@ def get_info(self, notfound_action=0):
704702
log.info(' FOUND:')
705703

706704
res = self.saved_results.get(self.__class__.__name__)
707-
if self.verbosity > 0 and flag:
705+
if log.get_threshold() <= log.INFO and flag:
708706
for k, v in res.items():
709707
v = str(v)
710708
if k in ['sources', 'libraries'] and len(v) > 270:
@@ -914,7 +912,7 @@ def combine_paths(self, *args):
914912
"""Return a list of existing paths composed by all combinations
915913
of items from the arguments.
916914
"""
917-
return combine_paths(*args, **{'verbosity': self.verbosity})
915+
return combine_paths(*args)
918916

919917

920918
class fft_opt_info(system_info):
@@ -1531,12 +1529,12 @@ def get_atlas_version(**config):
15311529
try:
15321530
s, o = c.get_output(atlas_version_c_text,
15331531
libraries=libraries, library_dirs=library_dirs,
1534-
use_tee=(system_info.verbosity > 0))
1532+
)
15351533
if s and re.search(r'undefined reference to `_gfortran', o, re.M):
15361534
s, o = c.get_output(atlas_version_c_text,
15371535
libraries=libraries + ['gfortran'],
15381536
library_dirs=library_dirs,
1539-
use_tee=(system_info.verbosity > 0))
1537+
)
15401538
if not s:
15411539
warnings.warn(textwrap.dedent("""
15421540
*****************************************************

‎runtests.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,10 @@ def main(argv):
6767
parser = ArgumentParser(usage=__doc__.lstrip())
6868
parser.add_argument("--verbose", "-v", action="count", default=1,
6969
help="more verbosity")
70+
parser.add_argument("--debug-configure", action="store_true",
71+
help=("add -v to build_src to show compiler "
72+
"configuration output while creating "
73+
"_numpyconfig.h and config.h"))
7074
parser.add_argument("--no-build", "-n", action="store_true", default=False,
7175
help="do not build the project (use system installed version)")
7276
parser.add_argument("--build-only", "-b", action="store_true", default=False,
@@ -366,6 +370,8 @@ def build_project(args):
366370
cmd += ["build"]
367371
if args.parallel > 1:
368372
cmd += ["-j", str(args.parallel)]
373+
if args.debug_configure:
374+
cmd += ["build_src", "--verbose"]
369375
# Install; avoid producing eggs so numpy can be imported from dst_dir.
370376
cmd += ['install', '--prefix=' + dst_dir,
371377
'--single-version-externally-managed',

‎setup.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -215,6 +215,7 @@ def __exit__(self, exception_type, exception_value, traceback):
215215

216216

217217
from distutils.command.sdist import sdist
218+
from numpy.distutils.command.build_src import build_src
218219
class sdist_checked(sdist):
219220
""" check submodules on sdist to prevent incomplete tarballs """
220221
def run(self):
@@ -263,7 +264,7 @@ def parse_setuppy_commands():
263264
# below and not standalone. Hence they're not added to good_commands.
264265
good_commands = ('develop', 'sdist', 'build', 'build_ext', 'build_py',
265266
'build_clib', 'build_scripts', 'bdist_wheel', 'bdist_rpm',
266-
'bdist_wininst', 'bdist_msi', 'bdist_mpkg')
267+
'bdist_wininst', 'bdist_msi', 'bdist_mpkg', 'build_src')
267268

268269
for command in good_commands:
269270
if command in args:
@@ -403,7 +404,9 @@ def setup_package():
403404
classifiers=[_f for _f in CLASSIFIERS.split('\n') if _f],
404405
platforms = ["Windows", "Linux", "Solaris", "Mac OS-X", "Unix"],
405406
test_suite='nose.collector',
406-
cmdclass={"sdist": sdist_checked},
407+
cmdclass={"sdist": sdist_checked,
408+
"build_src": build_src,
409+
},
407410
python_requires='>=3.5',
408411
zip_safe=False,
409412
entry_points={

‎shippable.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ build:
4848
# check OpenBLAS version
4949
- python tools/openblas_support.py --check_version 0.3.7
5050
# run the test suite
51-
- python runtests.py -- -rsx --junit-xml=$SHIPPABLE_REPO_DIR/shippable/testresults/tests.xml -n 2 --durations=10
51+
- python runtests.py --debug-configure --show-build-log -- -rsx --junit-xml=$SHIPPABLE_REPO_DIR/shippable/testresults/tests.xml -n 2 --durations=10
5252

5353
cache: true
5454
cache_dir_list:

‎tools/pypy-test.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ echo pypy3 version
3939
pypy3/bin/pypy3 -c "import sys; print(sys.version)"
4040
echo
4141

42-
pypy3/bin/pypy3 runtests.py --show-build-log -v -- -rsx \
42+
pypy3/bin/pypy3 runtests.py --debug-configure --show-build-log -v -- -rsx \
4343
--junitxml=junit/test-results.xml --durations 10
4444

4545
echo Make sure the correct openblas has been linked in

‎tools/travis-test.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ setup_base()
5252
else
5353
# Python3.5-dbg on travis seems to need this
5454
export CFLAGS=$CFLAGS" -Wno-maybe-uninitialized"
55-
$PYTHON setup.py build_ext --inplace 2>&1 | tee log
55+
$PYTHON setup.py build build_src -v build_ext --inplace 2>&1 | tee log
5656
fi
5757
grep -v "_configtest" log \
5858
| grep -vE "ld returned 1|no previously-included files matching|manifest_maker: standard file '-c'" \
@@ -151,7 +151,7 @@ if [ -n "$USE_WHEEL" ] && [ $# -eq 0 ]; then
151151
export F90='gfortran --coverage'
152152
export LDFLAGS='--coverage'
153153
fi
154-
$PYTHON setup.py bdist_wheel
154+
$PYTHON setup.py build build_src -v bdist_wheel
155155
# Make another virtualenv to install into
156156
virtualenv --python=`which $PYTHON` venv-for-wheel
157157
. venv-for-wheel/bin/activate

0 commit comments

Comments
 (0)
0