8000 gh-109276: libregrtest: WASM use filename for JSON by vstinner · Pull Request #109326 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-109276: libregrtest: WASM use filename for JSON #109326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

vstinner
Copy link
Member
@vstinner vstinner commented Sep 12, 2023

On Emscripten and WASI platforms, libregrtest now uses a filename for the JSON file. Passing a file descriptor to a child process doesn't work on these platforms.

@vstinner
Copy link
Member Author

!buildbot wasm32

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @vstinner for commit c45529b 🤖

The command will test the builders whose names match following regular expression: wasm32

The builders matched are:

  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@vstinner
Copy link
Member Author

Emscripten/WASI still use a file descriptor. It makes no sense to me :-( I'm not sure if buildbots got the right commit.

On Emscripten and WASI platforms, libregrtest now uses a filename for
the JSON file. Passing a file descriptor to a child process doesn't
work on these platforms.
@vstinner
Copy link
Member Author

!buildbot wasm32

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 9c052ba 🤖

The command will test the builders whose names match following regular expression: wasm32

The builders matched are:

  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@vstinner vstinner marked this pull request as draft September 12, 2023 14:08
@vstinner
Copy link
Member Author

!buildbot wasm32

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 9c052ba 🤖

The command will test the builders whose names match following regular expression: wasm32

The builders matched are:

  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@vstinner
Copy link
Member Author

wasm32-emscripten node (dynamic linking) PR failed with:

_PYTHON_HOSTRUNNER='/opt/emsdk/node/current/bin/node --experimental-wasm-bigint' _PYTHON_PROJECT_BASE=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/build_oot/host _PYTHON_HOST_PLATFORM=emscripten-wasm32 PYTHONPATH=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/build_oot/host/build/lib.emscripten-wasm32-3.13:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__emscripten_wasm32-emscripten ../build/python  ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j2 
== CPython 3.13.0a0 (heads/refs/pull/109326/merge-dirty:a70b894cdf, Sep 12 2023, 15:11:12) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== Python build: release
== cwd: /opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/build_oot/host/build/test_python_worker_432395æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 96764921
0:00:00 load avg: 2.54 Run 463 tests in parallel using 2 worker processes (timeout: 15 min, worker timeout: 20 min)
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 5
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 6
0:00:03 load avg: 2.54 [  1/463/1] test_optparse process crashed (Exit code 1)
worker: json_file type: <class 'int'>
worker: json_file: 5
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/Lib/test/libregrtest/worker.py", line 131, in <module>
    main()
  File "/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/Lib/test/libregrtest/worker.py", line 127, in main
    worker_process(worker_json)
  File "/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.emscripten-node-dl/build/Lib/test/libregrtest/worker.py", line 110, in worker_process
    with open(json_file, 'w', encoding='utf-8') as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 8] Bad file descriptor

wasm32-wasi PR failed with:

_PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib --mapdir /::../.. --' _PYTHON_PROJECT_BASE=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/lib.wasi-wasm32-3.13:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__wasi_wasm32-wasi ../build/python  ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j2 
== CPython 3.13.0a0 (heads/refs/pull/109326/merge-dirty:a70b894cdf, Sep 12 2023, 15:14:49) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== Python build: release
== cwd: /opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_worker_458151æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 76879218
0:00:00 load avg: 2.51 Run 463 tests in parallel using 2 worker processes (timeout: 15 min, worker timeout: 20 min)
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process is_emscripten: False
main process is_wasi: False
main process JSON_FILE_USE_FILENAME: False
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 5
main process: create TemporaryFile
main process json_type file: <class 'int'>
main process json_type: 6
0:00:00 load avg: 2.47 [  1/463/1] test_embed process crashed (Exit code 1)
/Lib/test/support/os_helper.py:531: RuntimeWarning: tests may fail, unable to create temporary directory '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ'
  with temp_dir(path=name, quiet=quiet) as temp_path:
/Lib/test/support/os_helper.py:532: RuntimeWarning: tests may fail, unable to change the current working directory to '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_48104æ'
  with change_cwd(temp_path, quiet=quiet) as cwd_dir:
worker: json_file type: <class 'int'>
worker: json_file: 6
test_embed skipped -- test module requires subprocess
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Lib/test/libregrtest/worker.py", line 131, in <module>
    main()
  File "/Lib/test/libregrtest/worker.py", line 127, in main
    worker_process(worker_json)
  File "/Lib/test/libregrtest/worker.py", line 110, in worker_process
    with open(json_file, 'w', encoding='utf-8') as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 8] Bad file descriptor

@vstinner
Copy link
Member Author

I just merged PR #109337 which collect more data in test.support and test.support sub-modules.

@vstinner
Copy link
Member Author

I just merged PR #109337 which collect more data in test.support and test.support sub-modules.

wasm32-emscripten node (dynamic linking) 3.x: test.pythoninfo, support.is_emscripten: True and sys.platform: emscripten as expected.

support.has_fork_support: False
support.has_socket_support: False
support.has_strftime_extensions: False
support.has_subprocess_support: False

support.is_android: False
support.is_emscripten: True
support.is_jython: False
support.is_wasi: False

support_threading_helper.can_start_thread: False

sys.platform: emscripten

wasm32-emscripten node (pthreads) 3.x: test.pythoninfo, support.is_emscripten: True and sys.platform: emscripten as expected

support.has_fork_support: False
support.has_socket_support: False
support.has_strftime_extensions: False
support.has_subprocess_support: False

support.is_android: False
support.is_emscripten: True
support.is_jython: False
support.is_wasi: False

support_threading_helper.can_start_thread: True

sys.platform: emscripten

wasm32-wasi 3.x: test.pythoninfo, support.is_wasi: True and sys.platform: wasi as expected

support.has_fork_support: False
support.has_socket_support: False
support.has_strftime_extensions: True
support.has_subprocess_support: False

support.is_android: False
support.is_emscripten: False
support.is_jython: False
support.is_wasi: True

support_threading_helper.can_start_thread: False

sys.platform: wasi

@vstinner
Copy link
Member Author

!buildbot wasm32

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @vstinner for commit c436370 🤖

The command will test the builders whose names match following regular expression: wasm32

The builders matched are:

  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@vstinner
Copy link
Member Author

!buildbot wasm32

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 4c8d5c8 🤖

The command will test the builders whose names match following regular expression: wasm32

The builders matched are:

  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@vstinner
Copy link
Member Author

!buildbot wasm32

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 4e2cfab 🤖

The command will test the builders whose names match following regular expression: wasm32

The builders matched are:

  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@vstinner
Copy link
Member Author

Ok, I wrote a clean fix from this debug PR: PR #109340.

@vstinner
Copy link
Member Author

wasm32-wasi PR failed: the WASI test worker process doesn't seem to have access to /tmp/test_python_json_7sbyggem of the Linuxparent process :-(

_PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib --mapdir /::../.. --' _PYTHON_PROJECT_BASE=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/lib.wasi-wasm32-3.13:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__wasi_wasm32-wasi ../build/python  ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j2 
1118966 JsonFileType: int | str
main process start sys.platform: linux
main process start is_emscripten: False
main process start is_wasi: False
main process python_cmd: ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
== CPython 3.13.0a0 (heads/refs/pull/109326/merge-dirty:0540fe293c, Sep 12 2023, 17:39:37) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== Python build: release
== cwd: /opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_worker_1118966æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 30180538
0:00:00 load avg: 1.66 Run 463 tests in parallel using 2 worker processes (timeout: 15 min, worker timeout: 20 min)
main process: json_file_use_filename? ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process: json_file_use_filename? ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process: create NamedTemporaryFile /tmp/test_python_json_7sbyggem
main process json_file type: <class 'str'>
main process json_file: /tmp/test_python_json_7sbyggem
main process: create NamedTemporaryFile /tmp/test_python_json_yq7jvz76
main process json_file type: <class 'str'>
main process json_file: /tmp/test_python_json_yq7jvz76
main process executable: ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process worker cmd: ['wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm', '-bb', '-Wdefault', '-u', '-m', 'test.libregrtest.worker', '{"tests": ["test_bigaddrspace"], "fail_fast": false, "fail_env_changed": true, "match_tests": null, "ignore_tests": null, "match_tests_dict": null, "rerun": null, "forever": false, "pgo": false, "pgo_extended": false, "output_on_failure": true, "timeout": 900.0, "verbose": 0, "quiet": false, "hunt_refleak": null, "test_dir": null, "use_junit": true, "memory_limit": null, "gc_threshold": null, "use_resources": ["audio", "curses", "largefile", "network", "decimal", "cpu", "subprocess", "urlfetch", "gui", "walltime"], "python_cmd": ["wasmtime", "run", "--env", "PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib", "--mapdir", "/::../..", "--", "python.wasm"], "randomize": true, "random_seed": 30180538, "json_file": "/tmp/test_python_json_7sbyggem", "__runtests__": true}']
create_worker_process() json_file: filename
main process executable: ('wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm')
main process worker cmd: ['wasmtime', 'run', '--env', 'PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib', '--mapdir', '/::../..', '--', 'python.wasm', '-bb', '-Wdefault', '-u', '-m', 'test.libregrtest.worker', '{"tests": ["test_venv"], "fail_fast": false, "fail_env_changed": true, "match_tests": null, "ignore_tests": null, "match_tests_dict": null, "rerun": null, "forever": false, "pgo": false, "pgo_extended": false, "output_on_failure": true, "timeout": 900.0, "verbose": 0, "quiet": false, "hunt_refleak": null, "test_dir": null, "use_junit": true, "memory_limit": null, "gc_threshold": null, "use_resources": ["audio", "curses", "largefile", "network", "decimal", "cpu", "subprocess", "urlfetch", "gui", "walltime"], "python_cmd": ["wasmtime", "run", "--env", "PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.13:/Lib", "--mapdir", "/::../..", "--", "python.wasm"], "randomize": true, "random_seed": 30180538, "json_file": "/tmp/test_python_json_yq7jvz76", "__runtests__": true}']
create_worker_process() json_file: filename
0:00:00 load avg: 1.66 [  1/463/1] test_bigaddrspace process crashed (Exit code 1)
42 JsonFileType: int | str
worker process sys.platform: wasi
worker process is_emscripten: False
worker process is_wasi: True
/Lib/test/support/os_helper.py:531: RuntimeWarning: tests may fail, unable to create temporary directory '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ'
  with temp_dir(path=name, quiet=quiet) as temp_path:
/Lib/test/support/os_helper.py:532: RuntimeWarning: tests may fail, unable to change the current working directory to '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ': [Errno 44] No such file or directory: '/opt/buildbot/bcannon-wasm/pull_request.bcannon-wasm.wasi/build/build_oot/host/build/test_python_230432æ'
  with change_cwd(temp_path, quiet=quiet) as cwd_dir:
worker: json_file type: <class 'str'>
worker: json_file: /tmp/test_python_json_7sbyggem
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Lib/test/libregrtest/worker.py", line 143, in <module>
    main()
  File "/Lib/test/libregrtest/worker.py", line 139, in main
    worker_process(worker_json)
  File "/Lib/test/libregrtest/worker.py", line 118, in worker_process
    with open(json_file, 'w', encoding='utf-8') as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 44] No such file or directory: '/tmp/test_python_json_7sbyggem'
Kill <WorkerThread #1 running test=test_venv pid=1118986 time=423 ms> process group

@vstinner
Copy link
Member Author

Ok, I wrote a clean fix from this debug PR: PR #109340.

I merged this PR instead of this DEBUG PR.

@vstinner vstinner closed this Sep 12, 2023
@vstinner vstinner deleted the regrtest_json_file2 branch September 12, 2023 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
0