8000 Update test suite (#2181) · pyscript/pyscript@06138bb · GitHub
[go: up one dir, main page]
Skip to content

Commit 06138bb

Browse files
ntollpre-commit-ci[bot]WebReflection
authored
Update test suite (#2181)
* pyscript.web tests pass with upytest. * Refactor of old integration tests to new Python tests. * Added comprehensive test suite for Python based `pyscript` module. * Add integration tests to Makefile (and CI) * Remove un-needed upload action. * Ensure fails are properly logged as an array. Remove the explicit test step, since this is already built into the build step. * Bump polyscript. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Andrea Giammarchi <andrea.giammarchi@gmail.com>
1 parent ae66d13 commit 06138bb

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+2390
-7707
lines changed

‎.github/workflows/test.yml

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -69,12 +69,7 @@ jobs:
6969
make setup
7070
7171
- name: Build
72-
run: make build
73-
74-
- name: Integration Tests
75-
#run: make test-integration-parallel
76-
run: |
77-
make test-integration
72+
run: make build # Integration tests run in the build step.
7873

7974
- uses: actions/upload-artifact@v4
8075
with:
@@ -83,10 +78,3 @@ jobs:
8378
pyscript.core/dist/
8479
if-no-files-found: error
8580
retention-days: 7
86-
87-
- uses: actions/upload-artifact@v4
88-
if: success() || failure()
89-
with:
90-
name: test_results
91-
path: test_results/
92-
if-no-files-found: error

‎Makefile

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ all:
1212
@echo "make clean - clean up auto-generated assets."
1313
@echo "make build - build PyScript."
1414
@echo "make precommit-check - run the precommit checks (run eslint)."
15-
@echo "make test-integration - run all integration tests sequentially."
15+
@echo "make test - run all automated tests in playwright."
1616
@echo "make fmt - format the code."
1717
@echo "make fmt-check - check the code formatting.\n"
1818

@@ -62,15 +62,9 @@ build:
6262
precommit-check:
6363
pre-commit run --all-files
6464

65-
# Run all integration tests sequentially.
66-
test-integration:
67-
mkdir -p test_results
68-
pytest -vv $(ARGS) pyscript.core/tests/integration/ --log-cli-level=warning --junitxml=test_results/integration.xml
69-
70-
# Run all integration tests in parallel.
71-
test-integration-parallel:
72-
mkdir -p test_results
73-
pytest --numprocesses auto -vv $(ARGS) pyscript.core/tests/integration/ --log-cli-level=warning --junitxml=test_results/integration.xml
65+
# Run all automated tests in playwright.
66+
test:
67+
cd pyscript.core && npm run test:integration
7468

7569
# Format the code.
7670
fmt: fmt-py

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,8 @@ Read the [contributing guide](https://docs.pyscript.net/latest/contributing/) to
7171

7272
Check out the [developing process](https://docs.pyscript.net/latest/developers/) documentation for more information on how to setup your development environment.
7373

74+
For technical details of the code, please see the [README](pyscript.core/README) in `pyscript.core`.
75+
7476
## Governance
7577

7678
The [PyScript organization governance](https://github.com/pyscript/governance) is documented in a separate repository.

‎pyscript.core/README.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,13 +37,25 @@ make setup
3737

3838
This will create a tests environment [in the root of the project, named `./env`]and install all the dependencies needed to run the tests.
3939

40-
After the command has completed and the tests environment has been created, you can run the **integration tests** with
40+
After the command has completed and the tests environment has been created, you can run the **automated tests** with
4141
the following command:
4242

4343
```
44-
make test-integration
44+
make test
4545
```
4646

47+
(This essentially runs the `npm run test:integration` command in the right place. This is defined in PyScript's `package.json` file.)
48+
49+
Tests are found in the `tests` directory. These are organised into three locations:
50+
51+
1. `python` - the Python based test suite to exercise Python code **within** PyScript.
52+
2. `javascript` - JavaScript tests to exercise PyScript itself, in the browser.
53+
3. `manual` - containing tests to run manually in a browser, due to the complex nature of the tests.
54+
55+
We use [Playwright](https://playwright.dev/) to automate the running of the Python and JavaScript test suites. We use [uPyTest](https://github.com/ntoll/upytest) as a test framework for the Python test suite. uPyTest is a "PyTest inspired" framework for running tests in the browser on both MicroPython and Pyodide.
56+
57+
The automated (Playwright) tests are specified in the `tests/integration.spec.js` file.
58+
4759
## `pyscript` python package
4860

4961
The `pyscript` package available in _Python_ lives in the folder `src/stdlib/pyscript/`.

‎pyscript.core/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@
5858
"@ungap/with-resolvers": "^0.1.0",
5959
"@webreflection/idb-map": "^0.3.1",
6060
"basic-devtools": "^0.1.6",
61-
"polyscript": "^0.15.7",
61+
"polyscript": "^0.15.8",
6262
"sabayon": "^0.5.2",
6363
"sticky-module": "^0.1.1",
6464
"to-json-callback": "^0.1.1",

‎pyscript.core/tests/README.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# PyScript Test Suite
2+
3+
There are three aspects to our test suite. These are reflected in the layout of
4+
the test directory:
5+
6+
1. `python` - contains the Python based test suite to exercise Python code
7+
**within** PyScript. These tests are run four differeng ways to ensure all
8+
combination of MicroPython/Pyodide and main thread/worker contexts are
9+
checked.
10+
2. `javascript` - contains JavaScript tests to exercise PyScript _itself_, in
11+
the browser.
12+
3. `manual` - contains tests to run manually in a browser, due to the complex
13+
nature of the tests.
14+
15+
We use [Playwright](https://playwright.dev/) to automate the running of the
16+
Python and JavaScript test suites. We use
17+
[uPyTest](https://github.com/ntoll/upytest) as a test framework for the Python
18+
test suite. uPyTest is a "PyTest inspired" framework for running tests in the
19+
browser on both MicroPython and Pyodide.
20+
21+
The automated (Playwright) tests are specified in the `integration.spec.js`
22+
file in this directory.
23+
24+
All automatic tests live in either the `python` or `javascript` folders. All
25+
the tests in these folder are run by CI or locally run by `make test` in the
26+
root of this project. Alternatively, run `npm run test:integration` in the
27+
PyScript source directory.
28+
29+
Similarly, some tests can only be run manually (due to their nature or
30+
underlying complexity). These are in the `manual` directory and are in the form
31+
of separate directories (each containing an `index.html`) or individual `*.html`
32+
files to which you point your browser. Each separate test may exercise
33+
JavaScript or Python code (or both), and the context for each separate test is
34+
kept carefully isolated.
35+
36+
Some rules of thumb:
37+
38+
* We don't test upstream projects: we assume they have their own test suites,
39+
and if we find bugs, we file an issue upstream with an example of how to
40+
recreate the problem.
41+
* We don't test browser functionality, we just have to trust that browsers work
42+
as advertised. Once again, if we find an issue, we report upstream.
43+
* All test cases should include commentary describing the **intent** and
44+
context of the test.
45+
* Tests in Python use [uPyTest](https://github.com/ntoll/upytest) (see the
46+
README for documentation), an "inspired by PyTest" test framework that works
47+
with both MicroPython and Pyodide in the browser. This means that all
48+
Python tests should work with both interpreters.
49+
* Tests in JavaScript... (Andrea to explain). ;-)

‎pyscript.core/tests/index.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,5 +14,5 @@
1414< 4E34 /td>
a:hover { opacity: 1; }
1515
</style>
1616
</head>
17-
<body><ul><li><strong><a href="./config/index.html">config</a></strong><ul><li><a href="./config/ambiguous-config.html">ambiguous-config<small>.html</small></a></li><li><a href="./config/same-config.html">same-config<small>.html</small></a></li><li><a href="./config/too-many-config.html">too-many-config<small>.html</small></a></li><li><a href="./config/too-many-py-config.html">too-many-py-config<small>.html</small></a></li></ul></li><li><strong><a href="./issue-7015/index.html">issue-7015</a></strong></li><li><strong><span>js-integration</span></strong><ul><li><a href="./js-integration/async-listener.html">async-listener<small>.html</small></a></li><li><a href="./js-integration/config-url.html">config-url<small>.html</small></a></li><li><a href="./js-integration/config_type.html">config_type<small>.html</small></a></li><li><strong><a href="./js-integration/fetch/index.html">fetch</a></strong></li><li><a href="./js-integration/ffi.html">ffi<small>.html</small></a></li><li><a href="./js-integration/hooks.html">hooks<small>.html</small></a></li><li><strong><a href="./js-integration/issue-2093/index.html">issue-2093</a></strong></li><li><a href="./js-integration/js-storage.html">js-storage<small>.html</small></a></li><li><a href="./js-integration/js_modules.html">js_modules<small>.html</small></a></li><li><strong><a href="./js-integration/loader/index.html">loader</a></strong></li><li><a href="./js-integration/mpy.html">mpy<small>.html</small></a></li><li><a href="./js-integration/py-terminal-main.html">py-terminal-main<small>.html</small></a></li><li><a href="./js-integration/py-terminal-worker.html">py-terminal-worker<small>.html</small></a></li><li><a href="./js-integration/py-terminal.html">py-terminal<small>.html</small></a></li><li><a href="./js-integration/py-terminals.html">py-terminals<small>.html</small>< F438 /a></li><li><a href="./js-integration/storage.html">storage<small>.html</small></a></li><li><strong><a href="./js-integration/workers/index.html">workers</a></strong><ul><li><a href="./js-integration/workers/named.html">named<small>.html</small></a></li></ul></li></ul></li><li><strong><a href="./manual/index.html">manual</a></strong><ul><li><a href="./manual/all-done.html">all-done<small>.html</small></a></li><li><a href="./manual/async.html">async<small>.html</small></a></li><li><a href="./manual/camera.html">camera<small>.html</small></a></li><li><a href="./manual/click.html">click<small>.html</small></a></li><li><a href="./manual/code-a-part.html">code-a-part<small>.html</small></a></li><li><a href="./manual/combo.html">combo<small>.html</small></a></li><li><a href="./manual/config.html">config<small>.html</small></a></li><li><a href="./manual/create-element.html">create-element<small>.html</small></a></li><li><a href="./manual/dialog.html">dialog<small>.html</small></a></li><li><a href="./manual/display.html">display<small>.html</small></a></li><li><a href="./manual/error.html">error<small>.html</small></a></li><li><a href="./manual/html-decode.html">html-decode<small>.html</small></a></li><li><a href="./manual/input.html">input<small>.html</small></a></li><li><a href="./manual/interpreter.html">interpreter<small>.html</small></a></li><li><a href="./manual/multi.html">multi<small>.html</small></a></li><li><a href="./manual/multiple-editors.html">multiple-editors<small>.html</small></a></li><li><a href="./manual/no-error.html">no-error<small>.html</small></a></li><li><a href="./manual/py-editor-failure.html">py-editor-failure<small>.html</small></a></li><li><a href="./manual/py-editor.html">py-editor<small>.html</small></a></li><li><a href="./manual/py_modules.html">py_modules<small>.html</small></a></li><li><a href="./manual/split-config.html">split-config<small>.html</small></a></li><li><a href="./manual/submit.html">submit<small>.html</small></a></li><li><a href="./manual/target.html">target<small>.html</small></a></li><li><a href="./manual/test_display_HTML.html">test_display_HTML<small>.html</small></a></li><li><a href="./manual/test_when.html">test_when<small>.html</small></a></li><li><a href="./manual/worker.html">worker<small>.html</small></a></li></ul></li><li><strong><a href="./no_sab/index.html">no_sab</a></strong></li><li><strong><a href="./piratical/index.html">piratical</a></strong></li><li><strong><a href="./py-editor/index.html">py-editor</a></strong><ul><li><a href="./py-editor/issue-2056.html">issue-2056<small>.html</small></a></li><li><a href="./py-editor/service-worker.html">service-worker<small>.html</small></a></li></ul></li><li><strong><a href="./py-terminals/index.html">py-terminals</a></strong><ul><li><a href="./py-terminals/no-repl.html">no-repl<small>.html</small></a></li><li><a href="./py-terminals/repl.html">repl<small>.html</small></a></li></ul></li><li><strong><a href="./pyscript_dom/index.html">pyscript_dom</a></strong></li><li><strong><a href="./service-worker/index.html">service-worker</a></strong></li><li><strong><a href="./ui/index.html">ui</a></strong><ul><li><a href="./ui/gallery.html">gallery<small>.html</small></a></li></ul></li></ul></body>
17+
<body><ul><li><strong><a href="./config/index.html">config</a></strong><ul><li><a href="./config/ambiguous-config.html">ambiguous-config<small>.html</small></a></li><li><a href="./config/same-config.html">same-config<small>.html</small></a></li><li><a href="./config/too-many-config.html">too-many-config<small>.html</small></a></li><li><a href="./config/too-many-py-config.html">too-many-py-config<small>.html</small></a></li></ul></li><li><strong><span>integration</span></strong><ul><li><strong><a href="./integration/python/index.html">python</a></strong></li></ul></li><li><strong><a href="./issue-7015/index.html">issue-7015</a></strong></li><li><strong><span>js-integration</span></strong><ul><li><a href="./js-integration/async-listener.html">async-listener<small>.html</small></a></li><li><a href="./js-integration/config-url.html">config-url<small>.html</small></a></li><li><a href="./js-integration/config_type.html">config_type<small>.html</small></a></li><li><strong><a href="./js-integration/fetch/index.html">fetch</a></strong></li><li><a href="./js-integration/ffi.html">ffi<small>.html</small></a></li><li><a href="./js-integration/hooks.html">hooks<small>.html</small></a></li><li><strong><a href="./js-integration/issue-2093/index.html">issue-2093</a></strong></li><li><a href="./js-integration/js-storage.html">js-storage<small>.html</small></a></li><li><a href="./js-integration/js_modules.html">js_modules<small>.html</small></a></li><li><strong><a href="./js-integration/loader/index.html">loader</a></strong></li><li><a href="./js-integration/mpy.html">mpy<small>.html</small></a></li><li><a href="./js-integration/py-terminal-main.html">py-terminal-main<small>.html</small></a></li><li><a href="./js-integration/py-terminal-worker.html">py-terminal-worker<small>.html</small></a></li><li><a href="./js-integration/py-terminal.html">py-terminal<small>.html</small></a></li><li><a href="./js-integration/py-terminals.html">py-terminals<small>.html</small></a></li><li><a href="./js-integration/storage.html">storage<small>.html</small></a></li><li><strong><a href="./js-integration/workers/index.html">workers</a></strong><ul><li><a href="./js-integration/workers/named.html">named<small>.html</small></a></li></ul></li></ul></li><li><strong><a href="./manual/index.html">manual</a></strong><ul><li><a href="./manual/all-done.html">all-done<small>.html</small></a></li><li><a href="./manual/async.html">async<small>.html</small></a></li><li><a href="./manual/camera.html">camera<small>.html</small></a></li><li><a href="./manual/click.html">click<small>.html</small></a></li><li><a href="./manual/code-a-part.html">code-a-part<small>.html</small></a></li><li><a href="./manual/combo.html">combo<small>.html</small></a></li><li><a href="./manual/config.html">config<small>.html</small></a></li><li><a href="./manual/create-element.html">create-element<small>.html</small></a></li><li><a href="./manual/dialog.html">dialog<small>.html</small></a></li><li><a href="./manual/display.html">display<small>.html</small></a></li><li><a href="./manual/error.html">error<small>.html</small></a></li><li><a href="./manual/html-decode.html">html-decode<small>.html</small></a></li><li><a href="./manual/input.html">input<small>.html</small></a></li><li><a href="./manual/interpreter.html">interpreter<small>.html</small></a></li><li><a href="./manual/multi.html">multi<small>.html</small></a></li><li><a href="./manual/multiple-editors.html">multiple-editors<small>.html</small></a></li><li><a href="./manual/no-error.html">no-error<small>.html</small></a></li><li><a href="./manual/py-editor-failure.html">py-editor-failure<small>.html</small></a></li><li><a href="./manual/py-editor.html">py-editor<small>.html</small></a></li><li><a href="./manual/py_modules.html">py_modules<small>.html</small></a></li><li><a href="./manual/split-config.html">split-config<small>.html</small></a></li><li><a href="./manual/submit.html">submit<small>.html</small></a></li><li><a href="./manual/target.html">target<small>.html</small></a></li><li><a href="./manual/test_display_HTML.html">test_display_HTML<small>.html</small></a></li><li><a href="./manual/test_when.html">test_when<small>.html</small></a></li><li><a href="./manual/worker.html">worker<small>.html</small></a></li></ul></li><li><strong><a href="./no_sab/index.html">no_sab</a></strong></li><li><strong><a href="./piratical/index.html">piratical</a></strong></li><li><strong><a href="./py-editor/index.html">py-editor</a></strong><ul><li><a href="./py-editor/issue-2056.html">issue-2056<small>.html</small></a></li><li><a href="./py-editor/service-worker.html">service-worker<small>.html</small></a></li></ul></li><li><strong><a href="./py-terminals/index.html">py-terminals</a></strong><ul><li><a href="./py-terminals/no-repl.html">no-repl<small>.html</small></a></li><li><a href="./py-terminals/repl.html">repl<small>.html</small></a></li></ul></li><li><strong><a href="./service-worker/index.html">service-worker</a></strong></li></ul></body>
1818
</html>

0 commit comments

Comments
 (0)
0