False positives using when using pytest-xdist -f (--looponfail) #85

y2kbugger · 2020-04-12T19:29:03Z

When doing TDD, after the first mypy error, the plugin start emitting errors on collection.

It will continue normally until you hit an actual mypy fail, then in the follow round (after mypy error is fixed) this warning is emitted forever until pytest is killed and restarted.

Steps to Reproduce (assumes python 3.8 and pipenv available):

mkdir ~/tmp20200412
cd ~/tmp20200412
git clone git@bitbucket.org:y2kbugger/test_driven_development_by_example_tdd_follow_along.git
cd ./test*
git checkout b0a2f56
replicate versions exactly from Pipfile.lock pipenv sync
put venv on PATH pipenv shell
run pytest pytest --mypy -f money.py test_money.py

In another terminal:

cd ~/tmp20200412/test*
cause mypy error echo 'lol: str = 2' >> money.py
undo mypy error git checkout b0a2f56 -- .

###################################### waiting for changes #######################################
### Watching:   /home/y2k/devel/test_driven_development_by_example_tdd_follow_along
# MODIFIED /home/y2k/devel/test_driven_development_by_example_tdd_follow_along/money.py
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/y2k/devel/test_driven_development_by_example_tdd_follow_along
plugins: forked-1.1.3, mypy-0.6.1, xdist-1.31.0, flake8-1.0.4
collected 0 items / 1 error
collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/runner.py:244: in from_call
    result = func()
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/runner.py:264: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/main.py:498: in collect
    yield from self._collect(fspath, parts)
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/main.py:537: in _collect
    assert not names, "invalid arg {!r}".format((argpath, names))
E   AssertionError: invalid arg (local('/home/y2k/devel/test_driven_development_by_example_tdd_follow_along'), ['mypy'])
=========================== short test summary info ============================
ERROR  - AssertionError: invalid arg (local('/home/y2k/devel/test_driven_deve...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.22s ===============================
######################################### LOOPONFAILING ##########################################
money.py::mypy
test_money.py::test_multiplication
test_money.py::test_franc_multiplication
test_money.py::test_equality
test_money.py::test_currency
::mypy
###################################### waiting for changes #######################################
### Watching:   /home/y2k/devel/test_driven_development_by_example_tdd_follow_along

Note I kill with keyboard interrupt here:

^C

Then everything goes fine again:

(test_driven_development_by_example_tdd_follow_along) [y2k@15r test_driven_development_by_example_tdd_follow_along]
$ pytest --mypy -f money.py test_money.py 
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/y2k/devel/test_driven_development_by_example_tdd_follow_along
plugins: forked-1.1.3, mypy-0.6.1, xdist-1.31.0, flake8-1.0.4
collected 7 items
collected 7 items

money.py .                                                               [ 12%]
test_money.py ......                                                     [ 87%]
. .                                                                      [100%]. [100%]
===================================== mypy =====================================

Success: no issues found in 2 source files
============================== 9 passed in 0.34s ===============================
###################################### waiting for changes #######################################
### Watching:   /home/y2k/devel/test_driven_development_by_example_tdd_follow_along

The text was updated successfully, but these errors were encountered:

dmtucker · 2020-04-12T21:16:55Z

Thanks for the excellent repro steps! 💯

The problem seems to have been introduced in 0.6.0 (#79) 🤔

y2kbugger · 2020-04-13T00:50:10Z

I did notice that I was passing both the source file and the test file to pytest but I think I was able to reproduce when only passing the test file. I will get back to you on that.

…

On Sun, Apr 12, 2020, 17:17 David Tucker ***@***.***> wrote: Thanks for the excellent repro steps! 💯 The problem seems to have been introduced in 0.6.0 (#79 <#79>) 🤔 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABORVLM2V5ZGUMKNUNLAR63RMIVVFANCNFSM4MGQ6OJA> .

y2kbugger · 2020-04-13T23:14:14Z

Can confirm error occurs even if ran properly:

pytest -f --mypy ./test_money.py

dmtucker · 2020-08-16T21:03:38Z

When a test fails, pytest-xdist holds onto the node ID of the failing test. The MypyStatusItem that is injected in pytest_collection_modifyitems has parent=session which produces a node ID of ::mypy (where mypy is nodeid_name).
Running that node ID explicitly (like xdist does) reproduces this:

pytest --verbose --mypy ::mypy

================================================ test session starts =================================================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /home/dtux/Projects/pytest-mypy/venv/bin/python
cachedir: .pytest_cache
Using --randomly-seed=1597611273
rootdir: /home/dtux/Projects/pytest-mypy
plugins: xdist-2.0.0, forked-1.3.0, randomly-2.1.1, cov-2.5.1, mypy-0.6.3.dev21+gd052c4c.d20200816
collected 0 items / 1 error                                                                                          

======================================================= ERRORS =======================================================
___________________________________________ ERROR collecting test session ____________________________________________
venv/lib/python3.8/site-packages/_pytest/runner.py:294: in from_call
    result = func()  # type: Optional[_T]
venv/lib/python3.8/site-packages/_pytest/runner.py:324: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
venv/lib/python3.8/site-packages/_pytest/main.py:576: in collect
    yield from self._collect(fspath, parts)
venv/lib/python3.8/site-packages/_pytest/main.py:617: in _collect
    assert not names, "invalid arg {!r}".format((argpath, names))
E   AssertionError: invalid arg (local('/home/dtux/Projects/pytest-mypy'), ['mypy'])
============================================== short test summary info ===============================================
ERROR  - AssertionError: invalid arg (local('/home/dtux/Projects/pytest-mypy'), ['mypy'])
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================== 1 error in 0.08s ==================================================

offbyone · 2020-11-13T22:11:00Z

Would you mind cutting a release with this? I would love to have this working.

dmtucker · 2020-11-14T01:54:17Z

Fixed in https://pypi.org/project/pytest-mypy/0.8.0/

dmtucker mentioned this issue Aug 16, 2020

Injecting a Session-level Item pytest-dev/pytest#7655

Closed

dmtucker mentioned this issue Oct 15, 2020

Stop injecting MypyStatusItem in pytest_collection_modifyitems to fix --looponfail #109

Merged

dmtucker closed this as completed in #109 Oct 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

False positives using when using pytest-xdist -f (--looponfail) #85

False positives using when using pytest-xdist -f (--looponfail) #85

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

False positives using when using pytest-xdist -f (--looponfail) #85

False positives using when using pytest-xdist -f (--looponfail) #85

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!