IDLE: Failures when running tests by import (autotest) #75942
Comments
|
In Developer Guide says: "If you don’t have easy access to a command line, you can run the test suite from a Python or IDLE shell:
But I can't run test from IDLE: Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
from test import autotest
File "D:\repos\cpython\Lib\test\autotest.py", line 5, in <module>
main()
File "D:\repos\cpython\Lib\test\libregrtest\main.py", line 585, in main
Regrtest().main(tests=tests, **kwargs)
File "D:\repos\cpython\Lib\test\libregrtest\main.py", line 510, in main
self._main(tests, kwargs)
File "D:\repos\cpython\Lib\test\libregrtest\main.py", line 524, in _main
setup_tests(self.ns)
File "D:\repos\cpython\Lib\test\libregrtest\setup.py", line 18, in setup_tests
faulthandler.enable(all_threads=True)
io.UnsupportedOperation: filenoIf I understand it correct, this behavior is reasonable (issues 3003 and 25588). |
The devguide is wrong. You should not run the Python test suite in IDLE. It doesn't work and many tests fail just because of IDLE. Please run the test suite in a command line (like "cmd.exe" on Windows). |
|
Got it. Thank you for your help. |
|
Closing this issue, I opened python/devguide#280 so that Dev Guide can be updated. Thanks! |
|
See bpo-25588. Currently running tests from IDLE doesn't work, but it worked in the past and maybe we can make it working again. |
|
Victor, what is your opinion of running 'from test import autotest' in a standard python shell window, opened from an icon, as opposed to IDLE's shell?
We cannot know what to blame on IDLE without a proper control experiment. I updated my Win10 repository, rebuilt python 3.7, ran the tests from the command line, and all passed (thank you for making this routine). I ran autotest in a Python console and saw a new resource warning from one of the async tests and 3 failures. I cannot say what they were because when the test finished, the window closed (crashed), which is a metafailure. I will rerun and try to identify the error as they occur. Should we reopen this issue or open a new issue, to fix problems not due to IDLE, or remove the entire suggestion to use autotest, and remove autotest itself? |
|
0:00:28 [ 25/407] test_asyncio test_code_module - multiple errors test test_importlib failed -- Traceback (most recent call last):
File "F:\dev\3x\lib\test\test_importlib\test_locks.py", line 134, in test_all_locks
self.bootstrap._module_locks)
AssertionError: 0 != 1 : {'test.autotest': <weakref at 0x0395D620; to '_ModuleLock' at 0x03957E70>}test_warnings failed - multiple errors crash after summary printed. |
|
The Python console exits because test.libregrtest.main.Regrtest._main calls sys.exit. test.autotest consists of from test.libregrtest import main
main()The main call should be wrapped with try:...except sys.exit: pass The output also has this variance from the command line result. 0:15:47 [319/407/2] test_subprocess The list of tests skipped seems to be the same. By setting sys.argv before importing autotest, I got Traceback (most recent call last):
File "F:\dev\3x\lib\test\test_code_module.py", line 35, in test_ps1
self.assertEqual(self.sysmod.ps1, '>>> ')
AssertionError: <MagicMock name='sys.ps1' id='71914032'> != '>>> '====================================================================== Traceback (most recent call last):
File "F:\dev\3x\lib\test\test_code_module.py", line 40, in test_ps2
self.assertEqual(self.sysmod.ps2, '... ')
AssertionError: <MagicMock name='sys.ps2' id='72189080'> != '... '
----------------------------------------------------------------------
====================================================================== Traceback (most recent call last):
File "F:\dev\3x\lib\test\test_warnings\__init__.py", line 446, in test_missing_filename_main_with_argv
self.assertEqual(w[-1].filename, sys.argv[0])
AssertionError: '__main__' != ''
- __main__
+and ====================================================================== Traceback (most recent call last):
File "F:\dev\3x\lib\test\test_warnings\__init__.py", line 446, in test_missing_filename_main_with_argv
self.assertEqual(w[-1].filename, sys.argv[0])
AssertionError: '__main__' != ''
- __main__Wneh I start python with python.bat, sys.argv = [''], not ['__main__']. |
|
When I import autotest in IDLE's shell, test_code_module fails, test_importlib and test_warnings fail, as without IDLE, and test_gc and test_logging are new failures. (Results are easy to see because SystemExit is caught and ignored.) test test_gc failed -- Traceback (most recent call last):
File "F:\dev\3x\lib\test\support\__init__.py", line 1855, in wrapper
return func(*args, **kwargs)
File "F:\dev\3x\lib\test\test_gc.py", line 290, in test_get_count
self.assertLess(a, 5)
AssertionError: 8 not less than 5<a comes from gc.get_count() after gc.collect(). When I repeat outside of the test, a is 31. I think either gc or this test is buggy.> test test_logging failed -- Traceback (most recent call last):
File "F:\dev\3x\lib\test\test_logging.py", line 1960, in test_warnings
"dummy.py:42: UserWarning: Explicit\n Dummy line\n")
AssertionError: '\nWarning (from warnings module):\n File[57 chars]it\n' != 'dummy.py:42: UserWarning: Explicit\n Dummy line\n'
+ dummy.py:42: UserWarning: Explicit
-
- Warning (from warnings module):
- File "dummy.py", line 42
- Dummy line
? --
+ Dummy line
- UserWarning: ExplicitIDLE replaces warnings.show_warnings. In any case, there are 3 failures in Python's shell versus 4 in IDLE's shell. |
|
After starting Python from a command line instead of an icon, importing/running autotest results in the same three failures. |
|
I am re-opening because there is a bug, a conflict between doc and code comment and actual behavior, and one which has nothing to do with IDLE. The question is whether to fix it or to delete the invitation to invoke the bug. The reason the devguide gives an alternative for running tests is so that someone with only installed Python and no command line could still test their installation and even contribute a patch on the tracker. On 2000-08-23, Tim Peters add the following comment to test.autotest: The autotest code was later copied into test.__main__, and both are now: To test the equivalence in the setting where either import might be useful, I opened both python and IDLE from the 3.7.0a2 Windows start icons and imported autotest. The test deterministicly crashes after test_marshal. I call the stoppage a crash because wrapping the import in try: ... except BaseException has no effect. The Python console just vanishes. IDLE restarts. =============================== RESTART: Shell =========================
So the claimed equivalence in worse than just not true. There were, I believe, 3 failures in the Python console. The extra 2 in IDLE are due to the bpo-25588 fix guarding against sys.stderr being None is needed in a couple of other places. In any case, either running tests by importing should be made to work (again, presuming it once did) or deprecated and autotest removed. |
|
Victor, why do you persist in the nearly irrelevant focus on IDLE? As I reported above, failures and crashes happen ***without*** involving IDLE. Serhiy appears to report the same in bpo-31794. |
|
Sorry, I didn't read you comment. I just saw IDLE, I missed that you wrote |
|
Perhaps there are issues specific to IDLE, but we should first fix issues not related to IDLE in bpo-31794. |
|
python/devguide#700 removed the entire 'interactive test'sentence and import example. This leaves autotest as an undocumented module except in itself. I am thinking of adding a reference back to this issue. I also want to see if test.test_idle can be run either with subprocess or by adding a _main function and importing it. I have considered adding 'Run IDLE tests' to the Help menu. There is also the IDLE-only failure when importing autotest. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: