8000 Windows CI builds broken · Issue #12460 · python/mypy · GitHub
[go: up one dir, main page]
Skip to content

Windows CI builds broken #12460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
JukkaL opened this issue Mar 26, 2022 · 14 comments · Fixed by #12477
Closed

Windows CI builds broken #12460

JukkaL opened this issue Mar 26, 2022 · 14 comments · Fixed by #12477
Labels
priority-0-high topic-developer Issues relevant to mypy developers

Comments

@JukkaL
Copy link
Collaborator
JukkaL commented Mar 26, 2022

Currently all Windows CI builds seem to be broken.

Example failure: https://github.com/python/mypy/actions/runs/2044190861

This might be caused by the switch to a new CI image with Windows Server 2022: actions/runner-images#4856

We could try falling back to Windows-2019 temporarily while we investigate what is going on.

Any help with investigating the failures would be appreciated, since I don't do much development in Windows.
@JukkaL JukkaL added priority-0-high topic-developer Issues relevant to mypy developers labels Mar 26, 2022
JukkaL added a commit that referenced this issue Mar 26, 2022
@JukkaL
Attempt to fix Windows CI by falling back to windows-2019
6a6c981 
See #12460 for context.
This was referenced Mar 26, 2022
Closed
Merged
@JelleZijlstra
Copy link
Member

The only suspicious thing I saw in the output is cl : Command line warning D9002 : ignoring unknown option '/O0'. https://docs.microsoft.com/en-us/cpp/build/reference/o-options-optimize-code?view=msvc-170 doesn't mention O0. Maybe it's a setuptools issue?

@JelleZijlstra JelleZijlstra mentioned this issue Mar 26, 2022
Closed
JelleZijlstra added a commit to JelleZijlstra/mypy that referenced this issue Mar 26, 2022
@JelleZijlstra
Try to fix Windows build
c3ffa48 
Fixes python#12460
@emmatyping
Copy link
Member

I can take a look at the issue a little later.

@KotlinIsland
Copy link
Contributor
KotlinIsland commented Mar 27, 2022 •
edited
Loading

last pass was https://github.com/python/mypy/runs/5694572863?check_suite_focus=true
first fail was https://github.com/python/mypy/runs/5700430887?check_suite_focus=true

The last pass was on windows server 2022, so I don't think that is the root cause.

Sample failed output 
__________________________________ testAsync __________________________________
[gw0] win32 -- Python 3.7.9 D:\a\mypy\mypy\.tox\py37\Scripts\python.EXE
data: D:\a\mypy\mypy\mypyc\test-data\run-misc.test:3:
D:\a\mypy\mypy\mypyc\test\test_run.py:137: in run_case
    self.run_case_inner(testcase)
D:\a\mypy\mypy\mypyc\test\test_run.py:153: in run_case_inner
    self.run_case_step(testcase, step)
D:\a\mypy\mypy\mypyc\test\test_run.py:266: in run_case_step
    assert glob.glob('native.*.{}'.format(suffix))
E   AssertionError: assert []
E    +  where [] = <function glob at 0x041B36A8>('native.*.pyd')
E    +    where <function glob at 0x041B36A8> = glob.glob
E    +    and   'native.*.pyd' = <built-in method format of str object at 0x05D9D1B0>('pyd')
E    +      where <built-in method format of str object at 0x05D9D1B0> = 'native.*.{}'.format
---------------------------- Captured stdout call -----------------------------
running build_ext
building 'native' extension
creating build\temp.win32-3.7
creating build\temp.win32-3.7\Release
creating build\temp.win32-3.7\Release\build
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x86\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\a\mypy\mypy\mypyc\lib-rt -ID:\a\mypy\mypy\.tox\py37\include -Ic:\hostedtoolcache\windows\python\3.7.9\x86\include -Ic:\hostedtoolcache\windows\python\3.7.9\x86\Include "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\inc
8000
lude" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" /Tcbuild\__native.c /Fobuild\temp.win32-3.7\Release\build\__native.obj /O0 /wd4102 /wd4101 /wd4146
__native.c

creating C:\Users\RUNNER~1\AppData\Local\Temp\mypy-test-wpgouzfj\tmp\build\lib.win32-3.7
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x86\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\a\mypy\mypy\.tox\py37\libs /LIBPATH:c:\hostedtoolcache\windows\python\3.7.9\x86\libs /LIBPATH:c:\hostedtoolcache\windows\python\3.7.9\x86 /LIBPATH:D:\a\mypy\mypy\.tox\py37\PCbuild\win32 "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\lib\x86" "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\lib\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x86" /EXPORT:PyInit_native build\temp.win32-3.7\Release\build\__native.obj /OUT:build\lib.win32-3.7\native.pyd /IMPLIB:build\temp.win32-3.7\Release\build\native.lib
   Creating library build\temp.win32-3.7\Release\build\native.lib and object build\temp.win32-3.7\Release\build\native.exp

Generating code

Finished generating code

copying build\lib.win32-3.7\native.pyd -> 
---------------------------- Captured stderr call -----------------------------
cl : Command line warning D9002 : ignoring unknown option '/O0'

Here are the details from "Set up job":

Successful job 
Current runner version: '2.289.1'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Virtual Environment
  Environment: windows-2022
  Version: 20220316.1
  Included Software: https://github.com/actions/virtual-environments/blob/win22/20220316.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win22%2F20220316.1
Virtual Environment Provisioner
  1.0.0.0-main-20220307-1
Failed job 
Current runner version: '2.289.1'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Virtual Environment
  Environment: windows-2022
  Version: 20220316.1
  Included Software: https://github.com/actions/virtual-environments/blob/win22/20220316.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win22%2F20220316.1
Virtual Environment Provisioner
  1.0.0.0-main-20220307-1

Additional analysis

unknown option '/O0'`

There is a warning: cl : Command line warning D9002 : ignoring unknown option '/O0'

/O0 is being passed, which is the MYPYC_OPT_LEVEL being set here:

mypy/mypyc/test/test_run.py

Line 246 in 1480344

opt_level = int(os.environ.get('MYPYC_OPT_LEVEL', 0))

And being used here:

mypy/mypyc/build.py

Lines 525 to 529 in 1480344

elif compiler.compiler_type == 'msvc':
if opt_level == '3':
opt_level = '2'
cflags += [
'/O{}'.format(opt_level),

#12462 Changed build to not pass the argument at all if the opt level was 0, which removed the warning but didn't resolve the failures, I still this this is a good change though and have re-raised the change here #12468, slightly modified to map 0 to 1

@JelleZijlstra
Copy link
Member

In #12462 I tried to fix the /O0 issue; unfortunately it didn't fix the build.

@pranavrajpal
Copy link
Contributor

setuptools had some releases around the time CI started failing, and since the bug seems to be related to the names of the generated extension modules and I think we use setuptools to handle the compilation, a setuptools change could cause a failure there.

The last pass in #12460 (comment) seems to be using setuptools 61.0.0, and the first fail seems to be using 61.1.0.

I haven't looked too much into what the cause of the failures is, but it seems worth trying to pin setuptools to 61.0.0 for now to see if that fixes the CI.

@emmatyping
Copy link
Member

I started looking into it last night, but it seems that the test suite hangs and I cannot Ctrl+C when it fails for me, which is unfortunate. I will try to run things with failfast and see if using an older setuptools fixes things. (Hopefully it fixes the test hanging as well, but that may be another issue).

@KotlinIsland KotlinIsland mentioned this issue Mar 28, 2022
Merged
@jhance
Copy link
Collaborator
jhance commented Mar 28, 2022

https://github.com/python/mypy/runs/5714129295?check_suite_focus=true

Doesn't seem like its setuptools?

@KotlinIsland
Copy link
Contributor

virtualenv==20.13.4 went to virtualenv==20.14.0?

@KotlinIsland
Copy link
Contributor

#12471 that didn't help either

@JukkaL JukkaL mentioned this issue Mar 28, 2022
Closed
@JukkaL
Copy link
Collaborator Author
JukkaL commented Mar 28, 2022

I'm doing some experiments in #12476. The first result is that a running single test case still fails (-k testRunDataclass), which makes it faster to iterate using a PR. I also disabled all the other builds in this PR to avoid using lots of CI capacity.

JukkaL added a commit that referenced this issue Mar 28, 2022
@JukkaL
Fix Windows test failures in CI
5443d52 
Allow extension file names such as `native.pyd` in addition to
`native.<python-version>.<ext>`. I'm not sure why the file names have
changed, but the new file names seem okay as well.

Fixes #12460.
@JukkaL JukkaL mentioned this issue Mar 28, 2022
Merged
JukkaL added a commit that referenced this issue Mar 28, 2022
@JukkaL
Fix Windows test failures in CI (#12477)
a66cb49 
Allow extension file names such as `native.pyd` in addition to
`native.<python-version>.<ext>`. I'm not sure why the file names have
changed, but the new file names seem okay as well.

Fixes #12460.
@JukkaL
Copy link
Collaborator Author
JukkaL commented Mar 28, 2022

Seems to be fixed now. Thanks for the help!

It would be great if somebody could install the latest Windows wheel from https://github.com/mypyc/mypy_mypyc-wheels (once it's ready) and check that it works and is compiled.

@pranavrajpal
Copy link
Contributor

It looks like this is an instance of pypa/setuptools#3219, and setuptools upgrading was what broke the CI. I think #12471 didn't help because the CI run still appears to be using setuptools version 61.2.0 because there are 2 pip install commands that try to install setuptools (here and here) and for some reason pip doesn't appear to be reinstalling setuptools for the second one.

@emmatyping
Copy link
Member

It would be great if somebody could install the latest Windows wheel from https://github.com/mypyc/mypy_mypyc-wheels (once it's ready) and check that it works and is compiled.

I will test this once I am back in front of my computer in about an hour.

@emmatyping
Copy link
Member

Things seem to be working as expected!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority-0-high topic-developer Issues relevant to mypy developers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Windows test failures in CI python/mypy
Try to fix Windows build JelleZijlstra/mypy
6 participants
@jhance @JelleZijlstra @JukkaL @emmatyping @KotlinIsland @pranavrajpal
0