Move ESM loaders off-thread#44710

Merged

aduh95 merged 126 commits intonodejs:mainfrom

JakobJingleheimer:feat/esm_off-thread-loaders

Apr 13, 2023

Commits on Apr 12, 2023

esm: move hook execution to separate thread

committed
esm: resolve optionally returns import assertions

GeoffreyBooth
authored and
aduh95
committed
port updates to `worker_thread`

JakobJingleheimer
authored and
aduh95
committed
move `HooksProxy` to `hooks.js`

JakobJingleheimer
authored and
aduh95
committed
`ESMLoader::addCustomLoaders` → `ESMLoader::registerCustomLoaders`

JakobJingleheimer
authored and
aduh95
committed
fix startup issues in hooks worker

JakobJingleheimer
authored and
aduh95
committed
WIP: move importing custom loaders back to main thread

JakobJingleheimer
authored and
aduh95
committed
🎉 `/test-esm-example-loader.mjs` is passing!!

JakobJingleheimer
authored and
aduh95
committed
Fix tests that are loading modules/esm/worker.js on the main thread

GeoffreyBooth
authored and
aduh95
committed
WIP: convert resolve to sync

JakobJingleheimer
authored and
aduh95
committed
Fix test

GeoffreyBooth
authored and
aduh95
committed
Restore parentURL validation to defaultResolve (and therefore also to import.meta.resolve)

GeoffreyBooth
authored and
aduh95
committed
Improve deserialization error (sometimes)

GeoffreyBooth
authored and
aduh95
committed
Rename #hooks to #chains

GeoffreyBooth
authored and
aduh95
committed
Subclass customized module loader

GeoffreyBooth
authored and
aduh95
committed
Fix import.meta being undefined (WIP)

GeoffreyBooth
authored and
aduh95
committed
Remove undocumented globalPreload setImportMetaCallback (WIP)

GeoffreyBooth
authored and
aduh95
committed
fs.write(1, message) works at least in Mac and Windows for printing from inside a worker

GeoffreyBooth
authored and
aduh95
committed
Standardize names, fix lint

GeoffreyBooth
authored and
aduh95
committed
Exceptions thrown inside loader hooks should be returned to the main thread and rethrown there so that they're printed in stdout

GeoffreyBooth
authored and
aduh95
committed
Rewrite test-esm-loader-hooks test to spawn child process and inspect stdout

GeoffreyBooth
authored and
aduh95
committed
Send loader initialization errors up to the main thread so they can be thrown and printed

GeoffreyBooth
authored and
aduh95
committed
Remove debugging code (fixes 3 tests!)

GeoffreyBooth
authored and
aduh95
committed
Catch another worker thread exception so that the process doesn't hang

GeoffreyBooth
authored and
aduh95
committed
Add error handling around hooks method responses being undefined or hooks method responses being too large when serialized

GeoffreyBooth
authored and
aduh95
committed
separate worker `data` & `lock` buffers

JakobJingleheimer
authored and
aduh95
committed
chunks working?? (code is very rough!)

JakobJingleheimer
authored and
aduh95
committed
Fix import.meta.resolve

GeoffreyBooth
authored and
aduh95
committed
More carefully pass errors back to the main thread

GeoffreyBooth
authored and
aduh95
committed
Some errors span multiple chunks, apparently

GeoffreyBooth
authored and
aduh95
committed
Fix chunking by separating metadata into separate buffers; fix error handling

GeoffreyBooth
authored and
aduh95
committed
unlock worker thread

targos
authored and
aduh95
committed
use syncCommPort to exchange data too

targos
authored and
aduh95
committed
fix --import

targos
authored and
aduh95
committed
fix doclint

targos
authored and
aduh95
committed
remove internal check

targos
authored and
aduh95
committed
refactor worker exception handling

targos
authored and
aduh95
committed
errors during load should just be rethrown

targos
authored and
aduh95
committed
run global preload scripts in the main thread

targos
authored and
aduh95
committed
nits
aduh95
committed
nits
aduh95
committed
fix linter errors
aduh95
committed
Add `HookProxy.prototype.makeAsyncRequest`
aduh95
committed
Wrap messages passed from worker to main thread so that we can carry along and restore the extra properties of Error objects

GeoffreyBooth
authored and
aduh95
committed
Emit experimental warning only once

GeoffreyBooth
authored and
aduh95
committed
Don't re-throw uncaught exceptions and then re-catch them ad infinitum

GeoffreyBooth
authored and
aduh95
committed
fix linter errors
aduh95
committed
transfer `ArrayBuffer`s
aduh95
committed
improve exception serialization
aduh95
committed
chaining loader hooks
aduh95
committed
make things more consistent
aduh95
committed
remove unnecessary parenthesis
aduh95
committed
simplify error cloning
aduh95
committed
small optimisation: read error properties only once when cloning
aduh95
committed
skip already cloned properties
aduh95
committed
DRY
aduh95
committed
fix `test-esm-loader-chaining`
aduh95
committed
update docs
aduh95
committed
fix `waitAsync` usage
aduh95
committed
simplify `ArrayBuffer` transfer
aduh95
committed
do not die waiting on worker to respond,
aduh95
committed
`import()` should not block the main thread, even for the resolve step
aduh95
committed
rework docs
aduh95
committed
do not keep the worker alive when awaiting hooks to resolve
aduh95
committed
keep the worker thread alive longer
aduh95
committed
let uncatcught exceptinnon be uncaught exceptions
aduh95
committed
use `serializeError` instead of custom logic
aduh95
committed
fix unserializable errors handling
aduh95
committed
only count async messages
aduh95
committed
remove global error handler only at the end of the sync messages
aduh95
committed
use `'beforeExit'` event to detect when threads are ready to die
aduh95
committed
ensure there are no pending messages before letting worker die
aduh95
committed
linter fixes
aduh95
committed
get rid of the tick counter timeout
aduh95
committed
Use `Atomics.load` for better results
aduh95
committed
add `shared_constants` internal module to make code clearer
aduh95
committed
rename `ModuleLoader` -> `createModuleLoader`
aduh95
committed
lazy load esmLoader
aduh95
committed
nits
aduh95
committed
fix test-esm-loader
aduh95
committed
unref worker more often
aduh95
committed
add more comments
aduh95
committed
fix main thread refusing to die waiting on a response that will never come
aduh95
committed
remove unrelated/unnecessary changes in `test/`
aduh95
committed
remove unrelated change in `src/`
aduh95
committed
restore changes related to sync resolve
aduh95
committed
Revert "unref worker more often"
aduh95
committed
nits
aduh95
committed
use `console.log` instead `fs.writeSync`
aduh95
committed
fix lint
aduh95
committed
add comment to describe the shared memory organisation
aduh95
committed
Remove unnecessary `Atomics.store` calls
aduh95
committed
s/BORED/IDLE/
aduh95
committed
add `SHARED_MEMORY_BYTE_LENGTH` to shared_constants
aduh95
committed
add definition for idleness
aduh95
committed
Update doc/api/esm.md

aduh95
and
targos
committed
Update lib/internal/modules/esm/shared_constants.js

aduh95
and
JakobJingleheimer
committed
add/fix comments + nits
aduh95
committed
for..of -> classic for
aduh95
committed
s/NUMBER_OF_INCOMING_MESSAGES/NUMBER_OF_MESSAGES_IN_TRANSIT/

aduh95
and
JakobJingleheimer
committed
add explanation for how `process.cwd()` can fail

JakobJingleheimer
and
aduh95
committed
Apply suggestions from code review

aduh95
and
JakobJingleheimer
committed
fix worker permissions for internal workers
aduh95
committed
fix dead lock
aduh95
committed
Apply suggestions from code review

aduh95
and
ljharb
committed
Apply suggestions from code review
aduh95
committed
`git mv lib/internal/modules/esm/worker.js lib/internal/main/loader_thread.js`
aduh95
committed
add comment
aduh95
committed
revert `git mv lib/internal/modules/esm/worker.js lib/internal/main/loader_thread.js`
aduh95
committed
never-settling `import.meta.resolve`
aduh95
committed
reuse the public port instead of creating an additional communication channel
aduh95
committed
fix: disable test's case concurrency to avoid deadlock ~1/10 times

JakobJingleheimer
authored and
aduh95
committed
fix rare deadlock

JakobJingleheimer
authored and
aduh95
committed
update Hooks entry in to note use of threads and caveat for async ops

JakobJingleheimer
authored and
aduh95
committed
chore: de-lint md

JakobJingleheimer
authored and
aduh95
committed
switch test fixtures from `console.log` → `writeSync(1, …)`

JakobJingleheimer
authored and
aduh95
committed
fix loaders with node:fs

targos
authored and
aduh95
committed
use more writeSync

targos
authored and
aduh95
committed
remove remaining `await` in front of `import.meta.resolve`
aduh95
committed
resolve TODOs for edge cases
aduh95
committed
fix race condition that could cause a dead lock when dealing with never settling hooks
aduh95
committed
reduce the size of the sab and introduce a `never-setlle` message status
aduh95
committed
fix race condition?
aduh95
committed
tidy code, comments, and docs
JakobJingleheimer
committed
Update lib/internal/worker.js
aduh95
authored
Update lib/internal/worker.js
aduh95
authored

Uh oh!

Move ESM loaders off-thread#44710aduh95 merged 126 commits intonodejs:mainnodejs/node:mainfrom JakobJingleheimer:feat/esm_off-thread-loadersJakobJingleheimer/node:feat/esm_off-thread-loadersCopy head branch name to clipboard

Commits

Commits on Apr 12, 2023

Move ESM loaders off-thread#44710
aduh95 merged 126 commits intonodejs:mainfrom
JakobJingleheimer:feat/esm_off-thread-loaders