BUILD: Hide platform configuration probe behind --debug-configure #14518

mattip · 2019-09-15T17:33:54Z

As requested in #14470, this adds a --debug-configure option to runtests.py and setup.py build to hide all the compiler output during the configuration probe to create config.h and _numpyconfig.h.

This also converts some of the direct print to stdout so that it uses the logger instead. The lack of the flag simply raises the log level to WARN from INFO for the duration of the noisy function.

mattip · 2019-09-15T17:34:36Z

Should I add the --debug-configure flag to the CI runs?

rgommers · 2019-09-15T17:38:20Z

numpy/distutils/command/build.py

@@ -16,8 +16,8 @@ class build(old_build):
    user_options = old_build.user_options + [
        ('fcompiler=', None,
         "specify the Fortran compiler type"),
-        ('parallel=', 'j',


This change seems unrelated, why does parallel need to be removed here and in finalize_options?

It is redundant - the base build class has this option (on python 3)

Specifically it was added in CPython v3.5.0

Ah yes. We may have to add it back once we figure out why it's buggy right now, but let's remove it till someone gets annoyed enough to start looking into that.

rgommers · 2019-09-15T17:40:40Z

numpy/core/setup.py

-    # build_clib cannot handle generate_config_h and generate_numpyconfig_h
-    # (don't ask). Because clib are generated before extensions, we have to
-    # explicitly add an extension which has generate_config_h and
-    # generate_numpyconfig_h as sources *before* adding npymath.


what's going on here, did all this become obsolete somehow?

It should have been removed when we merged umath and multiarray. It causes the config file generation to be run twice, which I noticed when working on this PR.

rgommers · 2019-09-15T17:43:00Z

Should I add the --debug-configure flag to the CI runs?

Probably a good idea. The SDIST=1 build should use the new default (hide config output), in the other CI builds it's mostly just noise I think. Unless other people have a use for it, I can't remember many such cases ....

rgommers · 2019-09-15T17:43:37Z

Thanks for tackling this, will be very nice to have!

mattip · 2019-09-15T18:35:11Z

Comparing the logs from another PR's travis run to this one, the lines from

building library "npysort" sources to
building extension "numpy.core._multiarray_tests" sources now become from

building library "npysort" sources to
building extension "numpy.core._multiarray_tests" sources, removing about 1600 lines.

numpy/distutils/ccompiler.py

mattip · 2019-09-16T13:46:26Z

failing because of #14526

stefanv · 2019-09-18T18:03:37Z

runtests.py

@@ -67,6 +67,8 @@ def main(argv):
    parser = ArgumentParser(usage=__doc__.lstrip())
    parser.add_argument("--verbose", "-v", action="count", default=1,
                        help="more verbosity")
+    parser.add_argument("--debug-configure", action="store_true",
+                        help="add -v to build_src to show cconfiguration compiler output")


This is pretty vague; can we describe what the actual feature is that is enabled?

"adds -v to the build_src subcommand which will show compiler output while probing platform features used to create _numpyconfig.h and config.h" seems a bit too much.

Any ideas what would be more appropriate?

I think the problem is that there are several things being addressed here: -v is being added to build_src, but only for a certain type of build? Or is it always added? If always, then the latter part of the sentence is not necessary. It can simply be something like "Increase verbosity of build_src by adding -v flag". But the "configure" in --debug-configure makes me think it was intended for something more specific.

another option would be to not add it to runtests.py at all (which would always build without the flag), and in CI explicitly do setup.py build build_src -v; runtests.py.

and in CI explicitly do setup.py build build_src -v; runtests.py

That doesn't seem desirable at all, we want to run CI in the ways we expect humans to use things, not some arcane invocation. There's no real issue here, so why change anything?

Yes, I think just a slightly more descriptive help string would be sufficient.

rgommers · 2019-09-19T23:51:36Z

CI failure python3: can't open file 'build': [Errno 2] No such file or directory is related.

I'm -1 on the interface changes and promoting build_src usage, this was good before.

mattip · 2019-09-20T06:38:19Z

rebased and reverted the change. The help string now reads

...
optional arguments:
  -h, --help            show this help message and exit
  --verbose, -v         more verbosity
  --debug-configure     add -v to build_src to show compiler configuration
                        output while creating _numpyconfig.h and config.h
  --no-build, -n        do not build the project (use system installed
                        version)
...

clarifications happily accepted.

rgommers · 2019-09-20T08:25:39Z

That help description seems very clear to me, thanks @mattip.

rgommers · 2019-09-20T08:31:59Z

In it goes, very nice to have this!

rgommers · 2019-09-20T08:58:24Z

Hmm, one of the Windows builds on Azure for the merge commit failed, see https://dev.azure.com/numpy/numpy/_build/results?buildId=5577.

Related to gh-14464, which may have a bad interaction with this PR. Any thoughts @mattip?

================================== FAILURES ===================================
____________________ TestMatmul.test_vector_matrix_values _____________________

self = <numpy.core.tests.test_multiarray.TestMatmul object at 0x14B88330>

    def test_vector_matrix_values(self):
        vec = np.array([1, 2])
        mat1 = np.array([[1, 2], [3, 4]])
        mat2 = np.stack([mat1]*2, axis=0)
        tgt1 = np.array([7, 10])
        tgt2 = np.stack([tgt1]*2, axis=0)
        for dt in self.types[1:]:
            v = vec.astype(dt)
            m1 = mat1.astype(dt)
            m2 = mat2.astype(dt)
>           res = self.matmul(v, m1)
E           RuntimeWarning: invalid value encountered in matmul

dt         = 'F'
m1         = array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j]], dtype=complex64)
m2         = array([[[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]],

       [[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]]], dtype=complex64)
mat1       = array([[1, 2],
       [3, 4]])
mat2       = array([[[1, 2],
        [3, 4]],

       [[1, 2],
        [3, 4]]])
res        = array([[ 7., 10.],
       [ 7., 10.]], dtype=float64)
self       = <numpy.core.tests.test_multiarray.TestMatmul object at 0x14B88330>
tgt1       = array([ 7, 10])
tgt2       = array([[ 7, 10],
       [ 7, 10]])
v          = array([1.+0.j, 2.+0.j], dtype=complex64)
vec        = array([1, 2])

C:\hostedtoolcache\windows\Python\3.6.8\x86\lib\site-packages\numpy\core\tests\test_multiarray.py:6031: RuntimeWarning
____________________ TestMatmul.test_matrix_vector_values _____________________

self = <numpy.core.tests.test_multiarray.TestMatmul object at 0x14C75330>

    def test_matrix_vector_values(self):
        vec = np.array([1, 2])
        mat1 = np.array([[1, 2], [3, 4]])
        mat2 = np.stack([mat1]*2, axis=0)
        tgt1 = np.array([5, 11])
        tgt2 = np.stack([tgt1]*2, axis=0)
        for dt in self.types[1:]:
            v = vec.astype(dt)
            m1 = mat1.astype(dt)
            m2 = mat2.astype(dt)
>           res = self.matmul(m1, v)
E           RuntimeWarning: invalid value encountered in matmul

dt         = 'F'
m1         = array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j]], dtype=complex64)
m2         = array([[[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]],

       [[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]]], dtype=complex64)
mat1       = array([[1, 2],
       [3, 4]])
mat2       = array([[[1, 2],
        [3, 4]],

       [[1, 2],
        [3, 4]]])
res        = array([[ 5., 11.],
       [ 5., 11.]], dtype=float64)
self       = <numpy.core.tests.test_multiarray.TestMatmul object at 0x14C75330>
tgt1       = array([ 5, 11])
tgt2       = array([[ 5, 11],
       [ 5, 11]])
v          = array([1.+0.j, 2.+0.j], dtype=complex64)
vec        = array([1, 2])

C:\hostedtoolcache\windows\Python\3.6.8\x86\lib\site-packages\numpy\core\tests\test_multiarray.py:6058: RuntimeWarning
________________ TestMatmulOperator.test_vector_matrix_values _________________

self = <numpy.core.tests.test_multiarray.TestMatmulOperator object at 0x14C545B0>

    def test_vector_matrix_values(self):
        vec = np.array([1, 2])
        mat1 = np.array([[1, 2], [3, 4]])
        mat2 = np.stack([mat1]*2, axis=0)
        tgt1 = np.array([7, 10])
        tgt2 = np.stack([tgt1]*2, axis=0)
        for dt in self.types[1:]:
            v = vec.astype(dt)
            m1 = mat1.astype(dt)
            m2 = mat2.astype(dt)
>           res = self.matmul(v, m1)
E           RuntimeWarning: invalid value encountered in matmul

dt         = 'F'
m1         = array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j]], dtype=complex64)
m2         = array([[[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]],

       [[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]]], dtype=complex64)
mat1       = array([[1, 2],
       [3, 4]])
mat2       = array([[[1, 2],
        [3, 4]],

       [[1, 2],
        [3, 4]]])
res        = array([[ 7., 10.],
       [ 7., 10.]], dtype=float64)
self       = <numpy.core.tests.test_multiarray.TestMatmulOperator object at 0x14C545B0>
tgt1       = array([ 7, 10])
tgt2       = array([[ 7, 10],
       [ 7, 10]])
v          = array([1.+0.j, 2.+0.j], dtype=complex64)
vec        = array([1, 2])

C:\hostedtoolcache\windows\Python\3.6.8\x86\lib\site-packages\numpy\core\tests\test_multiarray.py:6031: RuntimeWarning
________________ TestMatmulOperator.test_matrix_vector_values _________________

self = <numpy.core.tests.test_multiarray.TestMatmulOperator object at 0x14B88470>

    def test_matrix_vector_values(self):
        vec = np.array([1, 2])
        mat1 = np.array([[1, 2], [3, 4]])
        mat2 = np.stack([mat1]*2, axis=0)
        tgt1 = np.array([5, 11])
        tgt2 = np.stack([tgt1]*2, axis=0)
        for dt in self.types[1:]:
            v = vec.astype(dt)
            m1 = mat1.astype(dt)
            m2 = mat2.astype(dt)
>           res = self.matmul(m1, v)
E           RuntimeWarning: invalid value encountered in matmul

dt         = 'F'
m1         = array([[1.+0.j, 2.+0.j],
       [3.+0.j, 4.+0.j]], dtype=complex64)
m2         = array([[[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]],

       [[1.+0.j, 2.+0.j],
        [3.+0.j, 4.+0.j]]], dtype=complex64)
mat1       = array([[1, 2],
       [3, 4]])
mat2       = array([[[1, 2],
        [3, 4]],

       [[1, 2],
        [3, 4]]])
res        = array([[ 5., 11.],
       [ 5., 11.]], dtype=float64)
self       = <numpy.core.tests.test_multiarray.TestMatmulOperator object at 0x14B88470>
tgt1       = array([ 5, 11])
tgt2       = array([[ 5, 11],
       [ 5, 11]])
v          = array([1.+0.j, 2.+0.j], dtype=complex64)
vec        = array([1, 2])

mattip · 2019-09-20T09:05:43Z

That is the infamous heisenbug that occurs 1/1000.

charris · 2019-09-21T20:32:15Z

This broke the appveyor wheel builds. See https://ci.appveyor.com/project/matthew-brett/numpy-wheels/builds/27571654/job/v5tm7ljl6htcfa0a. Probably the changes in setup.py put things out of order.

BLD, DOC: fix gh-14518, add release note

mattip added the component: build label Sep 15, 2019

rgommers reviewed Sep 15, 2019

View reviewed changes

eric-wieser reviewed Sep 16, 2019

View reviewed changes

numpy/distutils/ccompiler.py Outdated Show resolved Hide resolved

mattip force-pushed the hide-config-probe branch from 3c24f68 to 70dedf5 Compare September 16, 2019 07:19

eric-wieser reviewed Sep 16, 2019

View reviewed changes

numpy/distutils/ccompiler.py Outdated Show resolved Hide resolved

mattip force-pushed the hide-config-probe branch from 70dedf5 to aab36ae Compare September 16, 2019 08:28

mattip force-pushed the hide-config-probe branch from a039c5f to e05617f Compare September 16, 2019 16:18

mattip mentioned this pull request Sep 16, 2019

BLD: add warn-error option, adds -Werror to compiler #14527

Merged

stefanv reviewed Sep 18, 2019

View reviewed changes

mattip force-pushed the hide-config-probe branch from 47e6110 to 09e01c0 Compare September 19, 2019 20:59

mattip mentioned this pull request Sep 19, 2019

MAINT: remove unused functions, rearrange headers (from CC=clang) #14534

Merged

mattip force-pushed the hide-config-probe branch from 09e01c0 to c0c31f8 Compare September 20, 2019 06:31

mattip added 8 commits September 20, 2019 09:34

BUILD: add --debug-configure option to reduce output, use logger more

9852340

BUILD: only report these headers with --debug-configure

4729f3f

BUILD: fix runtests.py option, fix previous changeset

f548774

BUILD: mostly restore verbose output on CI (except for pip install)

a68c5b8

BUG: fixes from review

55df75a

MAINT: move the option to build_src, simplifies logic

cdf67fa

DOC, BUILD: make build_src more discoverable

2e4af96

DOC: improve help string

492fdab

mattip force-pushed the hide-config-probe branch from c0c31f8 to 492fdab Compare September 20, 2019 06:36

rgommers merged commit 815061c into numpy:master Sep 20, 2019

rgommers added this to the 1.18.0 release milestone Sep 20, 2019

mattip mentioned this pull request Sep 21, 2019

BLD, DOC: fix gh-14518, add release note #14568

Merged

charris added a commit that referenced this pull request Sep 22, 2019

Merge pull request #14568 from mattip/hide-config-probe2

c8fdb44

BLD, DOC: fix gh-14518, add release note

mattip mentioned this pull request Oct 3, 2019

BUILD: change to build_src --verbose-cfg, runtests.py --debug-info #14630

Merged

mattip deleted the hide-config-probe branch June 8, 2020 06:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUILD: Hide platform configuration probe behind --debug-configure #14518

BUILD: Hide platform configuration probe behind --debug-configure #14518

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BUILD: Hide platform configuration probe behind --debug-configure #14518

BUILD: Hide platform configuration probe behind --debug-configure #14518

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!