8000 Build Failure on Windows, Python 3.6 GCC Compilers: UnicodeDecodeError. · Issue #16668 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Build Failure on Windows, Python 3.6 GCC Compilers: UnicodeDecodeError. #16668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Qiyu8 opened this issue Jun 23, 2020 · 16 comments · Fixed by #16789
Closed

Build Failure on Windows, Python 3.6 GCC Compilers: UnicodeDecodeError. #16668

Qiyu8 opened this issue Jun 23, 2020 · 16 comments · Fixed by #16789

Comments

@Qiyu8
Copy link
Member
Qiyu8 commented Jun 23, 2020

updating to the latest master , an UnicodeDecodeError occurred even after execute git clean -xfd

Reproducing code example:

No code example.

Error message:

CCompilerOpt.cc_test_flags[989] : testing flags (/WX)
CCompilerOpt.__init__[1137] : feature 'AVX512_KNL' is disabled, MSVC compiler doesn't support it
CCompilerOpt.__init__[1137] : feature 'AVX512_KNM' is disabled, MSVC compiler doesn't support it
CCompilerOpt.__init__[1621] : check requested baseline
CCompilerOpt.cc_test_flags[989] : testing flags (/arch:SSE)
Traceback (most recent call last):
  File "setup.py", line 499, in <module>
    setup_package()
  File "setup.py", line 491, in setup_package
    setup(**metadata)
  File "D:\Source\numpy\numpy\distutils\core.py", line 169, in setup
    return old_setup(**new_attr)
  File "C:\Python36\lib\site-packages\setuptools\__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "C:\Python36\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Python36\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Python36\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "D:\Source\numpy\numpy\distutils\command\build_ext.py", line 111, in run
    build_clib.run()
  File "D:\Source\numpy\numpy\distutils\command\build_clib.py", line 126, in run
    cpu_dispatch=self.cpu_dispatch, cache_path=opt_cache_path
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 2451, in new_ccompiler_opt
    opt = CCompilerOpt(compiler, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 2055, in __init__
    _Parse.__init__(self, cpu_baseline, cpu_dispatch)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1624, in __init__
    baseline_names = self.feature_names(cpu_baseline)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1182, in feature_names
    if self.feature_is_supported(f, force_flags=force_flags):
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1471, in feature_is_supported
    if not self.feature_test(impl, force_flags):
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1432, in feature_test
    force_flags = self.feature_flags(name)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1411, in feature_flags
    if not f or not self.cc_test_flags(f):
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 991, in cc_test_flags
    test = self.dist_test(test_path, flags)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 556, in dist_test
    [source], flags, output_dir=self.conf_tmp_path
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 536, in dist_compile
    sources, extra_postargs=flags, **kwargs
  File "C:\Python36\lib\distutils\_msvccompiler.py", line 423, in compile
    self.spawn(args)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 668, in _dist_test_spawn_paths
    self._dist_test_spawn(cmd)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 686, in _dist_test_spawn
    o = o.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4: invalid continuation byte
CCompilerOpt._cache_write[786] : write cache to path -> D:\Source\numpy\build\temp.win-amd64-3.6\ccompiler_opt_cache_clib.py
@mattip
Copy link
Member
mattip commented Jun 23, 2020

Do you have non-ascii characters in your path? Could you stop at the error and try to work out what is going on?

@mattip
Copy link
Member
mattip commented Jun 23, 2020

does o have an encoding property o.encoding ?

@Qiyu8
Copy link
Member Author
Qiyu8 commented Jun 23, 2020

I'm sure that no non-ascii characters existed, It's seems like the exception is triggered when testing SSE flags:

CCompilerOpt.cc_test_flags[991] : testing flags (/arch:SSE)
decode:b'cl: \xc3\xfc\xc1\xee\xd0\xd0 warning D9002 :\xba\xf6\xc2\xd4\xce\xb4\xd6\xaa\xd1\xa1\xcf\xee\xa1\xb0/arch:SSE\xa1\xb1\r\ntest_flags.c\r\n'

o.decode() failed to decode

b'cl: \xc3\xfc\xc1\xee\xd0\xd0 warning D9002 :\xba\xf6\xc2\xd4\xce\xb4\xd6\xaa\xd1\xa1\xcf\xee\xa1\xb0/arch:SSE\xa1\xb1\r\ntest_flags.c\r\n'

@eric-wieser
Copy link
Member
eric-wieser commented Jun 23, 2020 •
edited
Loading

@Qiyu8, your compiler output is encoded in GB2312 rather than utf8 it seems. The string decodes to

cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE”
test_flags.c

@eric-wieser
Copy link
Member

https://bugs.python.org/issue27179#msg267091 seems slightly related, I haven't read the whole thread.

@Qiyu8
Copy link
Member Author
Qiyu8 commented Jun 23, 2020

Maybe a parallel problem, I always build numpy with python setup.py build_ext -i -j 8, turned to python setup.py build_ext -i and the build successfully.

@eric-wieser
Copy link
Member

My guess would be that the codepage of the parallel processes does not match the codepage of the parent process.

@Qiyu8
Copy link
Member Author
Qiyu8 commented Jul 9, 2020

This problem arose again in my local machine, even with non-parallel build.

@mattip
Copy link
Member
mattip commented Jul 9, 2020
8000

There was a long discussion and a PEP 597 about changing the default stdout encoding. Bottom line: what does
locale.getpreferredencoding() show for you? If you modify the offending line

  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 686, in _dist_test_spawn
    o = o.decode()

to use that encoding does it fix the problem?

@Qiyu8
Copy link
Member Author
Qiyu8 commented Jul 9, 2020

my encoding result is cp936, The build can succeed after modifed to o.decode(encoding="cp936"), But there are large amount of warning D9002 :忽略未知选项“/arch:SSE” produced during build.

@Qiyu8
Copy link
Member Author
Qiyu8 commented Jul 9, 2020

@mattip I want to provide a PR to solve this issue, modify decode() to decode(locale.getpreferredencoding()), what do you think?

@mattip
Copy link
Member
mattip commented Jul 9, 2020

modify decode() to decode(locale.getpreferredencoding())

I think the preferred spelling of this is when calling check_output to specify universal_newlines=True which will force the output to be text (already encoded). Search for "unversal_newlines" on the subprocess doc page. Could you try it (since we don't really test for non-ascii windows versions)?

As for the multiple warnings, can you show the part of the log that looks like the one in this comment ?

@Qiyu8
Copy link
Member Author
Qiyu8 commented Jul 9, 2020

The build can succeed after specify universal_newlines=True, see the build log of multiple warnings:

cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_s_lapack.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_s_lapack.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_lapack.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_lapack.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_blas.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_blas.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_config.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_config.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python36\libs /LIBPATH:C:\Python36\PCbuild\amd64 /LIBPATH:build\temp.win-amd64-3.6 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64 npymath.lib /EXPORT:PyInit__umath_linalg build\temp.win-amd64-3.6\Release\numpy\linalg\umath_linalg.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\python_xerbla.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_z_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_c_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_d_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_s_lapack.obj 
build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_blas.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_config.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c.obj /OUT:numpy\linalg\_umath_linalg.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\numpy\linalg\_umath_linalg.cp36-win_amd64.lib

@mattip
Copy link
Member
mattip commented Jul 9, 2020

Good to hear: so the correct fix for this issue is universal_newlines=True

As for the warnings: on 64-bit windows, you should not be using /arch:SSE2, since it is on by default. This is a separate issue, @seiko2plus might be able to help.

@eric-wieser
Copy link
Member

Note that universal_newlines=True is equivalent to decoding with locale.getpreferredencoding(False) according to https://docs.python.org/3/library/io.html#io.TextIOWrapper.

I still suspect there's a nastier issue here regarding the locale in which parallel vs non-parallel builds are executed in.

@seiko2plus
Copy link
Member

@mattip, This is a separate issue

yes, my bad. I didn't expect msvc to raise warning messages in another language than English.

@Qiyu8, well to solve this issue you have to replace:

numpy/numpy/distutils/ccompiler_opt.py

Line 676 in 86fcce6

"ignoring unknown option|" # msvc

with:

"warning D9002|"  # msvc, it should be work with any language.

And why?
Because microsoft and Intel compiler tending to raise warning messages rather than fatal errros for unkown or unsupported options.

@Qiyu8 Qiyu8 mentioned this issue Jul 9, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: fix decode error when building and get rid of warn Qiyu8/numpy
4 participants
@eric-wieser @mattip @seiko2plus @Qiyu8
0