[Bug]: pytest -s hangs #27176

Jacob-Stevens-Haas · 2023-10-23T15:52:50Z

Bug summary

In a development install, running pytest -s hangs indefinitely. This causes VSCode's python test explorer to also hang indefinitely as it tries to populate the list of tests (it seems like it ignores settings.json for the command to populate the list). The behavior occurs when running a single test, e.g.

Code for reproduction

pytest -s --co lib/matplotlib/tests/test_legend.py::test_text_nohandler_warning

Actual outcome

pytest-xvfb could not find Xvfb. You can install it to prevent windows from being shown.
======================================== test session starts ========================================
platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/xenophon/github/matplotlib
configfile: pytest.ini
plugins: xdist-3.3.1, rerunfailures-12.0, timeout-2.2.0, cov-4.1.0, xvfb-3.0.0
collecting ...

(keeps collecting indefinitely)

Expected outcome

Detects all tests

Additional information

No response

Operating system

Linux 6.2.0-34-generic; Ubuntu 22.04.1

Matplotlib Version

dev build on current master, fcd5bb1a2b

Matplotlib Backend

agg

Python version

3.10.12

Jupyter version

N/A

Installation

None

The text was updated successfully, but these errors were encountered:

ksunden · 2023-10-23T17:50:06Z

I cannot reproduce this, it seems to work fine for me, though it did perhaps take a little extra time in the collection phase it seemed. (tested with python 3.11.4 pytest 7.4.0/7.4.2)

Jacob-Stevens-Haas · 2023-10-24T08:48:02Z

Ok, thanks for trying. I'm working off my own install that took a few tries from the other issue; I'll try it on 3.11 in the codespaces later today

tacaswell · 2023-10-24T20:55:51Z

I also can not reproduce this as a single line, but I can reproduce that vscode does not find our tests right. I see it running ~/.virtualenvs/dd311/bin/python -m pytest -p vscode_pytest --collect-only. With -p vscode_pytest I can not run it at the CLI (I assume it is injecting a plugin?!), but

python -m pytest --collect-only

does take longer than I was willing to wait.

tacaswell · 2023-10-24T21:03:22Z

It is something about the parameterization in lib/matplotlib/tests/test_backends_interactive.py as if I remove that file the cli runs quickish for me.

tacaswell · 2023-10-24T21:13:50Z

It looks like there is a way to detect being run in --collect-only mode https://stackoverflow.com/questions/60152776/run-method-only-if-collect-only-was-used but it is not clear to me where would want to inject that or how we would make user of that in our fixture factory.

ksunden · 2023-10-24T21:35:05Z

https://github.com/microsoft/vscode-python/tree/main/pythonFiles

If I hack together the required files from the vscode-python plugin (the vscode_pytest __init__.py and the testing_tools.socket_manager module)and run with -p vscode_pytest, I get a lot of output to stdout and it is infinite looping here

Its not entirely surprising that things are breaking given that I'm not running it in vscode and dont have the relevant sockets listening, but it:

Doesn't increase counters on the codepath I'm looking at
Doesn't actually break out of the loop when when it says it will either (last line of the file)
- Only successfully connecting will escape the while True:
- Edit: It's not actually a while True:, so it _will break, but only if the socket exists in the first place, though narrower than I was originally thinking.
I suspect that the json they are trying to send (which is 4260543 bytes, 4.06 MB) is larger than their socket buffers and therefore needs to be sent in chunks, but the code does not handle that (I think 4 MB buffer size is relatively standard, and that is just a hair over that size). (This may also explain why getting rid of a file makes it better, especially one with a lot of tests/parameterization, though somewhat doubt it is anything particular about the file other than that it gets below 4MB output)

ksunden · 2023-10-24T21:46:58Z

I think this is the service at the other end, it is reading in 1MB chunks, but is appending to an unbounded bytestring, so I don't actually see anything that should be wrong about crossing the 4MB boundary, actually.

I think I would need to actually see what the error is, though I suspect the lack of break in the first file is why it is hanging forever (though does not explain why it is failing to send in the first place)

Jacob-Stevens-Haas · 2023-10-25T11:15:37Z

This may be more sophisticated than I'm really capable of troubleshooting, but I can grok enough, so I'm going to try to hang on. For my edification, could you share how you're identifying these things?

--co isn't a problem for me unless -s is also present. vscode's pythonFiles/testing_tools/adapter/pytest/_discovery.py without -s works, and it apparently hangs within pytest. So I guess that's why you're looking for a plugin?

I assume it is injecting a plugin?!

This line down to L30 add a plugin, not sure if that's the one though?

The local command that VSCode output runs when I try to populate the test explorer is

./env/bin/python ~/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear

Jacob-Stevens-Haas · 2023-10-25T11:21:31Z

Running .vscode/.../pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear in VSCode's debugger won't even pause it in debug mode. From a keyboard interrupt, pytest seems to be hanging when it looks up each attribute of the hook, IIUC?

Traceback from keyboard interrupt

  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 324, in _main
    config.hook.pytest_collection(session=session)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 335, in pytest_collection
    session.perform_collect()
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 675, in perform_collect
    self.items.extend(self.genitems(node))
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 845, in genitems
    yield from self.genitems(subnode)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 842, in genitems
    rep = collect_one_node(node)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/runner.py", line 547, in collect_one_node
    rep: CollectReport = ihook.pytest_make_collect_report(collector=collector)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/capture.py", line 859, in pytest_make_collect_report
    rep = outcome.get_result()
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/runner.py", line 391, in pytest_make_collect_report
    errorinfo = collector.repr_failure(call.excinfo)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/nodes.py", line 562, in repr_failure
    return self._repr_failure_py(excinfo, style=tbstyle)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/nodes.py", line 486, in _repr_failure_py
    return excinfo.getrepr(
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 701, in getrepr
    return fmt.repr_excinfo(self)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 989, in repr_excinfo
    reprtraceback = self.repr_traceback(excinfo_)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 913, in repr_traceback
    entries = [
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 914, in <listcomp>
    self.repr_traceback_entry(entry, excinfo if last == entry else None)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 852, in repr_traceback_entry
    source = self._getentrysource(entry)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 754, in _getentrysource
    source = entry.getsource(self.astcache)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 262, in getsource
    astnode, _, end = getstatementrange_ast(
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/source.py", line 187, in getstatementrange_ast
    start, end = get_statement_startend2(lineno, astnode)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/source.py", line 150, in get_statement_startend2
    for x in ast.walk(node):
  File "/usr/lib/python3.10/ast.py", line 389, in walk
    node = todo.popleft()
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xenophon/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/xenophon/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/testing_tools/adapter/__main__.py", line 99, in main
    parents, result = run(toolargs, **subargs)
  File "/home/xenophon/.vscode/extensions/ms-python.python-2023.18.0/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 30, in discover
    ec = _pytest_main(pytestargs, [_plugin])
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/main.py", line 285, in wrap_session
    config.hook.pytest_keyboard_interrupt(excinfo=excinfo)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/terminal.py", line 891, in pytest_keyboard_interrupt
    self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 701, in getrepr
    return fmt.repr_excinfo(self)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 989, in repr_excinfo
    reprtraceback = self.repr_traceback(excinfo_)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 913, in repr_traceback
    entries = [
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 914, in <listcomp>
    self.repr_traceback_entry(entry, excinfo if last == entry else None)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 852, in repr_traceback_entry
    source = self._getentrysource(entry)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 754, in _getentrysource
    source = entry.getsource(self.astcache)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/code.py", line 262, in getsource
    astnode, _, end = getstatementrange_ast(
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/source.py", line 187, in getstatementrange_ast
    start, end = get_statement_startend2(lineno, astnode)
  File "/home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/_pytest/_code/source.py", line 151, in get_statement_startend2
    if isinstance(x, (ast.stmt, ast.ExceptHandler)):
KeyboardInterrupt

ksunden · 2023-10-25T15:25:58Z

I was really frankensteining my environment by just copying individual files from the vscode-python repository into my working directory just to see if I could get anything similar and figure out if there was something happening there. (And running python -m pytest specifically (as opposed to just pytest) to pick up those files from my working directory)

When I also stood up a dummy server (to just listen, and drop the data it reads on the floor), the plugin actually succeeded.

But there is an infinite loop condition that is possible so that could be what is hanging. It happens when the listening service is offline. (Now why that would be offline, I do not know).

At this point I've kind of exhausted what I think I can look into, it seems like it is at least probably caused by VSCode rather than anything we specifically do, though I'm not 100% sure on that.

Jacob-Stevens-Haas · 2023-10-26T10:39:00Z

Ok cool, that makes sense. I think I'm catching up to seeing the socket issue, as I'm seeing from stdout redirect that it's identified all the tests.

How were you able to jump to the testing_tools.socket_manager module so quickly? I wrote the "collect all tests" command as a debug configuration, but VSCode won't let me pause when it's hanging.

Regardless, it looks like there's two issues: the --co one you guys have found (that's likely internal to vscode), and mine dealing with the -s flag. I'm going to try to follow up on @tacaswell's observation about parameterization in the testing interactive backends file. Maybe something about pytest not capturing output messes with something a backend expects, at least in my install.

QuLogic · 2023-12-05T22:48:00Z

Might this be related to (and fixed by) #27422?

Jacob-Stevens-Haas · 2023-12-06T11:41:43Z

Hot diggity! Yes it is fixed :) Thanks

pabloazurduy · 2024-02-17T18:08:46Z

hi, facing the same issue, is there an easy workaround ? should I install 3.9.0dev ? or is there an easy fix ?

Thanks in advance !
EDIT:
I open the matplotlib directory and simply copy and paste the matplotlib\tests\test_backends_interactive.py file from the PR. solved the problem

melissawm added the topic: testing label Oct 31, 2023

Jacob-Stevens-Haas closed this as completed Dec 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Bug]: pytest -s hangs #27176

[Bug]: pytest -s hangs #27176

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Bug]: pytest -s hangs #27176

[Bug]: pytest -s hangs #27176

Comments

Uh oh!

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!