Run test262 tests in CI #408
Merged
+36
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
8ee7286 to
24e9e37
Compare
9c0da0d to
6960dd5
Compare
ab51af9 to
7ca7dc7
Compare
4a07bd7 to
44d52ea
Compare
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. QuickJS has its own test runner but unfortunately there is not automatic bootstraping and they run by hand. There is an upstream PR [2] to add these tests to run there but until that happens we can patch our own QuickJS with. This particular commit doesn't run the tests just patches the necessary updates for bootstrapping and some timeout tweaks to fix some flakes. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
0ced3d2 to
17475ee
Compare
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 20, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then we can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 21, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then we can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. For now run the test in the full CI only for Linux and skip s390x. Some architectures/os combinations show various discrepancies around TypedArrays or rounding errors. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
2b07a0b to
11325a5
Compare
8c30777 to
6f8b450
Compare
|
@bellard any interest in having the CI run test262 tests? I noticed you keep the test262 commit sha in the TODO file but I put it in the Makefile, not sure my approach is good. Maybe it could be a separate file? The idea is to have repeatability if the CI job is re-run and test262 is updated in the meantime. |
|
Your approach looks OK. I'll remove the test262 commit sha in TODO ASAP. |
6f8b450 to
3c38f29
Compare
|
Updated with latest test262 from yesterday (2025-09-14). Needed |
3c38f29 to
8b43bae
Compare
|
Updated it with the latest test262 and it seems to be all green 
What do you think, Fabrice, does it need any more changes or tweaks? |
19d8e52 to
0df111b
Compare
nickva
commented
Sep 26, 2025
| submodules: true | ||
| - name: Install gcc-multilib | ||
| run: | | ||
| sudo apt update |
There was a problem hiding this comment.
After ubuntu images receive an update for a about a day or two CI will start failing unless the CI jobs perform an apt update so that's why I added it here.
More info at actions/runner-images#12599
|
A last issue before committing: test262 should be run for at least one 32 bit architecture such as Linux x86. |
9551827 to
959a180
Compare
|
you should update the test262 to fix the remaining issue. |
Run test262 tests for Linux, macOS and Cosmopolitan Add a Makefile `test2-bootstrap` helper to clone and patch test262 tests at a particular commit. Running with the latest commit noticed a few tests were failing and added them to the errors list. The CI environment, especially for macOS, was a bit more unpredictable and was failing in atomics tests due to lower timeouts in the `atomicsHelper` so double the timeouts. The time didn't worsen too much. It still takes about 2-3 min.
959a180 to
fb14cc6
Compare
|
Updated, all tests pass, including on 32bit 
This is not test262 master though, it's an intermediate commit as there are quite a few changes on master, including the the harness files we patch |
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Sep 30, 2025
Upstream merged test262 PR so drop patches related to that: bellard/quickjs#408 Other improvements * Add set methods (union, difference, intersection, ...) bellard/quickjs@a1e073e * Add iterators bellard/quickjs@982b7aa bellard/quickjs@e924173 bellard/quickjs@b2ed2e9 bellard/quickjs@cf0e179 bellard/quickjs@2d99c32 * Add ArrayBuffer.prototype.transfer bellard/quickjs@1e19893 * Add resizable array buffers bellard/quickjs@f021d77 * Add Map and WeakMap upsert bellard/quickjs@456e016 * Add Map.sumPrecise() bellard/quickjs@bc753c6 bellard/quickjs@c3e5ae2 * Fix date parsing bellard/quickjs@0060876
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Sep 30, 2025
Upstream merged test262 PR so drop patches related to that: bellard/quickjs#408 Other improvements * Add set methods (union, difference, intersection, ...) bellard/quickjs@a1e073e * Add iterators bellard/quickjs@982b7aa bellard/quickjs@e924173 bellard/quickjs@b2ed2e9 bellard/quickjs@cf0e179 bellard/quickjs@2d99c32 * Add ArrayBuffer.prototype.transfer bellard/quickjs@1e19893 * Add resizable array buffers bellard/quickjs@f021d77 * Add Map and WeakMap upsert bellard/quickjs@456e016 * Add Map.sumPrecise() bellard/quickjs@bc753c6 bellard/quickjs@c3e5ae2 * Fix date parsing bellard/quickjs@0060876
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Sep 30, 2025
Upstream merged test262 PR so drop patches related to that: bellard/quickjs#408 Other improvements * Add set methods (union, difference, intersection, ...) bellard/quickjs@a1e073e * Add iterators bellard/quickjs@982b7aa bellard/quickjs@e924173 bellard/quickjs@b2ed2e9 bellard/quickjs@cf0e179 bellard/quickjs@2d99c32 * Add ArrayBuffer.prototype.transfer bellard/quickjs@1e19893 * Add resizable array buffers bellard/quickjs@f021d77 * Add Map and WeakMap upsert bellard/quickjs@456e016 * Add Map.sumPrecise() bellard/quickjs@bc753c6 bellard/quickjs@c3e5ae2 * Fix date parsing bellard/quickjs@0060876
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Run test262 tests for Linux, macOS and Cosmopolitan
Add a Makefile
test2-bootstraphelper to clone and patch test262 tests at a particular commit. Running with the latest commit noticed a few tests were failing and added them to the errors list.The CI environment, especially for macOS, was a bit more unpredictable and was failing in atomics tests due to lower timeouts in the
atomicsHelperso double the timeouts. The time didn't worsen too much. It still takes about 2-3 min.