8000 feat(executor): implement stack overflow protection by criticic · Pull Request #4559 · WasmEdge/WasmEdge · GitHub
[go: up one dir, main page]

Skip to content

Conversation

@criticic
Copy link

Add configurable call stack depth limit to prevent infinite recursion and stack overflow issues. Fixes #2079 and #4410.

Changes:

  • Add CallStackExhausted error code (0x041A) in execution phase
  • Add MaxCallDepth configuration option (default: 10000) to RuntimeConfigure
  • Add stack depth checks before pushing frames in enterFunction()
    • Checks added for host functions, compiled functions, and native functions
  • Implement assert_exhaustion handler in spec tests
  • Add C API functions: WasmEdge_ConfigureSetMaxCallDepth() and WasmEdge_ConfigureGetMaxCallDepth()
  • Add comprehensive unit tests for stack overflow scenarios

@criticic criticic requested a review from q82419 as a code owner January 26, 2026 22:48
Copilot AI review requested due to automatic review settings January 26, 2026 22:48
@github-project-automation github-project-automation bot moved this to Triage-required in WasmEdge Roadmap Jan 26, 2026
@github-actions github-actions bot added c-CAPI An issue related to the WasmEdge C API c-Test An issue/PR to enhance the test suite labels Jan 26, 2026
Copy link
Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a configurable max call stack depth in the runtime, wires it through the C API, and adds executor and test changes so recursive Wasm programs trap with a defined CallStackExhausted error instead of hanging or crashing (addressing #2079 and #4410).

Changes:

  • Add a new CallStackExhausted WASM error (0x041A) and a MaxCallDepth field to RuntimeConfigure with a default of 10000, plus C API getters/setters and VM behavior that checks depth in enterFunction for host, compiled, and native functions.
  • Implement assert_exhaustion handling in the spec tests to assert execution-phase CallStackExhausted errors, and add a dedicated stack overflow test module (stack_overflow.wat) plus GTest coverage for various depth limits and recursion patterns.
  • Extend the public C API (wasmedge_configure.h / wasmedge.cpp) and existing configuration tests to cover the new call depth limit behavior, including null-config safety.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
lib/executor/helper.cpp Adds pre-frame checks against RuntimeConfigure::getMaxCallDepth() in all enterFunction paths (host, compiled, native) and returns the new CallStackExhausted error on overflow to prevent unbounded recursion and stack faults.
include/runtime/stackmgr.h Introduces getFrameDepth() to expose the current number of frames so executor can enforce the configured max depth.
include/common/configure.h Extends RuntimeConfigure with an atomic MaxCallDepth (default 10000), copy-ctor initialization, and setMaxCallDepth/getMaxCallDepth accessors.
include/common/enum.inc Defines the new CallStackExhausted error code (0x041A, message "call stack exhausted") in the WASM execution error space.
include/api/wasmedge/wasmedge_configure.h Exposes WasmEdge_ConfigureSetMaxCallDepth / WasmEdge_ConfigureGetMaxCallDepth in the public C API with documentation describing the new depth limit.
lib/api/wasmedge.cpp Implements the new configure C API functions by forwarding to RuntimeConfigure::setMaxCallDepth / getMaxCallDepth, mirroring the existing max-memory-page pattern.
test/spec/spectest.cpp Implements CommandID::AssertExhaustion by invoking the action and asserting that failures are execution-phase CallStackExhausted errors; introduces one new failure-path assertion to force a test failure on unexpected success.
test/api/apiTestData/stack_overflow.wat Adds a small Wasm module with an infinite recursion export and a parameterized deep-recursion export used by the new API stack overflow tests.
test/api/APIUnitTest.cpp Extends APICoreTest.Configure to validate the new call depth configure APIs (including null-context behavior) and adds APICoreTest.StackOverflow to exercise default, low, and high call depth limits with the new Wasm test module and verify the 0x041A WASM error.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@hydai
Copy link
Member
hydai commented Jan 27, 2026

Hi @criticic,
Can you confirm if these tests are runnable? I think they may encounter timeout errors in several CI jobs.

@criticic criticic force-pushed the fix/stack-overflow-protection branch from cc4837e to 2b0f292 Compare January 27, 2026 05:46
@q82419
Copy link
Collaborator
q82419 commented Jan 27, 2026

In my opinion, I believe this implementation surely can prevent from the runtime overflow becuase it strickly limited the calling frame size.
But I'm still considering about the appropriate value of limited calling frame size. Maybe we should study the details in WASM tool conventions.
@criticic Your PR is meaningful but we should keep discussing with this. Thanks!

Add configurable call stack depth limit to prevent infinite recursion and
stack overflow issues. Fixes WasmEdge#2079 and WasmEdge#4410.

Changes:
- Add CallStackExhausted error code (0x041A) in execution phase
- Add MaxCallDepth configuration option (default: 10000) to RuntimeConfigure
- Add stack depth checks before pushing frames in enterFunction()
  - Checks added for host functions, compiled functions, and native functions
- Implement assert_exhaustion handler in spec tests
- Add C API functions: WasmEdge_ConfigureSetMaxCallDepth() and
  WasmEdge_ConfigureGetMaxCallDepth()
- Add comprehensive unit tests for stack overflow scenarios

Signed-off-by: Sagnik Mandal <acriticalcynic@outlook.com>
@criticic criticic force-pushed the fix/stack-overflow-protection branch from 2b0f292 to b441bfc Compare January 27, 2026 06:56
@criticic
Copy link
Author

appropriate value of limited calling frame size

I could not find anything in the spec about what the default call stack size should be. I'll also look at other implementations and see what they have done regarding this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c-CAPI An issue related to the WasmEdge C API c-Test An issue/PR to enhance the test suite

Projects

Status: Triage-required

Development

Successfully merging this pull request may close these issues.

assert_exhaustion is not implemented

3 participants

0