CI failure - Windows unit tests #12372

kimadeline · 2020-06-16T16:34:19Z

https://dev.azure.com/ms/vscode-python/_build/results?buildId=88164&view=logs&j=e7ebb862-ce3e-5fd4-27ed-c97d7dbc698e&t=cdf7b4d7-ce31-509a-4cc5-d6c8b2fdc4b0&l=960

pythonFiles\tests\testing_tools\adapter\test_functional.py:163: AssertionError

kimadeline · 2020-06-16T17:47:18Z

Oldest failure seems to be https://dev.azure.com/ms/vscode-python/_build/results?buildId=88048&view=logs&j=c988df44-9d9a-569e-48be-286abc0c9c8f&t=e05b582e-afad-5edb-605e-ce81b7e22a0b for #12232 (the failure isn't related to the PR).

Seems like py==1.8.2 causes the failures, one of the last passing CI runs had py==1.8.1, and rolling back to py==1.8.1 makes the tests pass locally.

py changelog:

1.8.2 (2020-06-15)

On Windows, py.path.locals which differ only in case now have the same
Python hash value. Previously, such paths were considered equal but had
different hashes, which is not allowed and breaks the assumptions made by
dicts, sets and other users of hashes.

Maybe a related issue: https://github.com/pytest-dev/py/issues/245

The quickest fix would be to force py to be 1.8.1, because fixing the tests for windows (by changing the order of nodes in the array of expected results) is a Windows-specific fix that breaks Unix tests.

We could also try removing the offending test nodes and files 🙃

kimadeline · 2020-06-18T05:13:26Z

Windows unit tests are passing for the CI build for #12407: https://dev.azure.com/ms/vscode-python/_build/results?buildId=88808&view=results

nicoddemus · 2020-07-14T22:32:59Z

Hi 👋 ,

Can I get some feedback on this comment: pytest-dev/pytest#7477 (comment)? Seems to be directly related/cause for this issue, so it would be great to be able to figure out the problem/solve it before 6.0 is out.

Thanks!

kimadeline · 2020-07-15T00:41:59Z

Hi @nicoddemus,

What kind of specific feedback are you looking for? Do you want me to try out combinations of pytest 5.4.3 and py-1.8.1, py-1.8.2, and py-1.9.0?

nicoddemus · 2020-07-15T00:49:10Z

Hi @kimadeline,

It is just that the underlying issue is not clear to me: the output has changed in the sense that the name attribute of packages no longer contain a full path, but that is not related to py as far as I can figure, and it was an intended change.

It would be helpful to understand what caused the problem for vs code, as we would love to have this resolved.

kimadeline · 2020-07-16T04:45:16Z

Short explanation

The output of running pytest --rootdir vscode-python/pythonFiles vscode-python/pythonFiles/tests used to be the same on macOS, Linux and Windows, however the output on Windows changed with py==1.8.2, which broke our tests 😕

Bit more details

As part of our CI test suites we have some tests where we compare the output of pytest test discovery against an expected list of discovered tests (we maintain this list ourselves). Something in py 1.8.2 changed the order of tests in the test discovery output, but only on Windows, while Linux and macOS have the same output with py 1.8.1 and 1.8.2.

I noticed was that we had silently started using py 1.8.2 since it wasn't a pinned dependency, and our builds started failing. Pinning py to 1.8.1 solved our issue 📌

Given this folder structure:

The Windows output changed:

diff --git a/output_json_1.8.1.json b/output_json_1.8.2.json
index 87eec17..3dc6573 100644
--- a/output_json_1.8.1.json
+++ b/output_json_1.8.2.json
@@ -1094,20 +1094,6 @@
         "markers": [],
         "parentid": "./tests/v/test_spam.py"
       },
-      {
-        "id": "./tests/w/test_spam.py::test_simple",
-        "name": "test_simple",
-        "source": ".\\tests\\w\\test_spam.py:4",
-        "markers": [],
-        "parentid": "./tests/w/test_spam.py"
-      },
-      {
-        "id": "./tests/w/test_spam_ex.py::test_simple",
-        "name": "test_simple",
-        "source": ".\\tests\\w\\test_spam_ex.py:4",
-        "markers": [],
-        "parentid": "./tests/w/test_spam_ex.py"
-      },
       {
         "id": "./tests/x/y/z/test_ham.py::test_simple",
         "name": "test_simple",
@@ -1128,6 +1114,20 @@
         "source": ".\\tests\\x\\y\\z\\b\\test_spam.py:7",
         "markers": [],
         "parentid": "./tests/x/y/z/b/test_spam.py"
+      },
+      {
+        "id": "./tests/w/test_spam.py::test_simple",
+        "name": "test_simple",
+        "source": ".\\tests\\w\\test_spam.py:4",
+        "markers": [],
+        "parentid": "./tests/w/test_spam.py"
+      },
+      {
+        "id": "./tests/w/test_spam_ex.py::test_simple",
+        "name": "test_simple",
+        "source": ".\\tests\\w\\test_spam_ex.py:4",
+        "markers": [],
+        "parentid": "./tests/w/test_spam_ex.py"
       }
     ]
   }

Or in a UI-based diff checker:

nicoddemus · 2020-07-16T12:56:39Z

Thanks @kimadeline for writing a detailed description of the problem.

I can see a collection difference between py-1.8.1 vs py-1.8.2 with pytest 5.4.3:

--- out-1.8.1   2020-07-16 09:43:54.272329900 -0300
+++ out-1.8.2   2020-07-16 09:39:21.247834200 -0300
@@ -1,9 +1,8 @@
 ======================================= test session starts =======================================
-platform win32 -- Python 3.7.6, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
+platform win32 -- Python 3.7.6, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
 rootdir: d:\projects\pytest\.tmp, inifile: pytest.ini
 plugins: hypothesis-5.14.0, forked-1.1.3, xdist-1.33.0
 collected 61 items
-
 <Package D:\projects\pytest\.tmp\pythonFiles\tests>
   <Module test_normalize_for_interpreter.py>
     <Class TestNormalizationScript>
@@ -12,16 +11,13 @@
         <Function test_withHangingIndent>
         <Function test_clearOutExtraneousNewlines>
         <Function test_clearOutExtraLinesAndWhitespace>
-<Package D:\projects\pytest\.tmp\pythonFiles\tests\debug_adapter>
   <Module test_install_debugpy.py>
     <Function test_install_debugpy>
-<Package D:\projects\pytest\.tmp\pythonFiles\tests\ipython>
   <Module test_variables.py>
     <Function test_variable_list>
     <Function test_variable_value>
     <Function test_dataframe_info>
     <Function test_dataframe_rows>
-<Package D:\projects\pytest\.tmp\pythonFiles\tests\testing_tools\adapter>
   <Module test___main__.py>
     <UnitTestCase ParseGeneralTests>
       <TestCaseFunction test_unsupported_command>
@@ -71,7 +67,6 @@
       <TestCaseFunction test_one_arg>
       <TestCaseFunction test_quotation_marks>
       <TestCaseFunction test_whitespace>
-<Package D:\projects\pytest\.tmp\pythonFiles\tests\testing_tools\adapter\pytest>
   <Module test_cli.py>
     <UnitTestCase AddCLISubparserTests>
       <TestCaseFunction test_discover>

As we can see, 4 packages for some reason are not being collected in py-1.8.2 (same with py-1.9.0).

However, I get the same collection tree with pytest's master (soon to be released as 6.0.0) with py-1.8.1, py-1.8.2, and py-1.9.0:

--- out-1.8.1-master    2020-07-16 09:42:22.989084900 -0300
+++ out-1.8.2-master    2020-07-16 09:42:31.749859100 -0300
@@ -1,5 +1,5 @@
 ======================================= test session starts =======================================
-platform win32 -- Python 3.7.6, pytest-6.0.0rc2.dev45+g71ab6236a, py-1.8.1, pluggy-0.13.1
+platform win32 -- Python 3.7.6, pytest-6.0.0rc2.dev45+g71ab6236a, py-1.8.2, pluggy-0.13.1
 rootdir: d:\projects\pytest\.tmp, configfile: pytest.ini
 plugins: hypothesis-5.14.0, forked-1.1.3, xdist-1.33.0
 collected 61 items

So whatever the problem was it will be fixed in pytest-6.0.0.

While it is interesting to investigate further, we don't currently have plans for a pytest-5.4.4 release. My question for the VS code team is if it is OK to wait for 6.0.0 or do they feel it is important to find the cause and release a 5.4.4 with a fix.

kimadeline · 2020-07-16T15:12:12Z

Sadly we pinned pytest to 4.6.9 because we run some of our tests using Python 2.7. I don't think we are going to update to 5.4.4 or 6.0.0 soon.

nicoddemus · 2020-07-16T15:32:11Z

Wouldn't be possible to use 4.6.X only in Python 2?

kimadeline · 2020-07-16T15:43:26Z

Well, we could, but not only would we still need to pin pytest to 4.6.9 and py to 1.8.1 for Python 2, but we would also have to wait for the 6.0.0 release before we do that so that we modify our requirements only once.

Nonetheless I can bring it up to the team whenever pytest-6.0.0 gets released.

ghost removed the triage-needed Needs assignment to the proper sub-team label Jun 16, 2020

kimadeline added the xteam label Jun 16, 2020

DonJayamanne mentioned this issue Jun 16, 2020

Some of the CI tests are failing repeatedly #12380

Closed

kimadeline self-assigned this Jun 17, 2020

This was referenced Jun 17, 2020

Pin py to 1.8.1 (CI failure - Windows unit tests) #12407

Merged

Refactor our pytest unit tests #12416

Closed

mdmintz mentioned this issue Jun 18, 2020

Possible issue with collect-only package printing on Windows seleniumbase/SeleniumBase#598

Closed

kimadeline closed this as completed Jun 18, 2020

ghost removed the needs PR label Jun 18, 2020

mdmintz mentioned this issue Jul 12, 2020

py 1.8.2 breaks collect-only package printing on windows pytest-dev/pytest#7477

Closed

kimadeline mentioned this issue Aug 18, 2020

CI: Pin pytest to 4.6.9 and py to 1.8.1 for Python 2.7 only #13489

Closed

github-actions bot locked as resolved and limited conversation to collaborators Oct 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CI failure - Windows unit tests #12372

CI failure - Windows unit tests #12372

1.8.2 (2020-06-15)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CI failure - Windows unit tests #12372

CI failure - Windows unit tests #12372

Comments

Uh oh!

1.8.2 (2020-06-15)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Short explanation

Bit more details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!