Jump to sections#

• EAGLE Decoding

  • EAGLE-2 Decoding

  • EAGLE-2 Decoding with torch.compile

  • EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling

  • EAGLE-3 Decoding

• Multi Token Prediction

• Standalone Speculative Decoding (Small Draft Model)

• Speculative Decoding V2 (Overlap Scheduler)

• Ngram Speculative Decoding

• Full Parameter Reference

• OOM Troubleshooting

• References

Quick guidance#

• Best speed/quality (recommended): Use EAGLE-3 with --speculative-algorithm EAGLE3.

• Strong default / broad compatibility: Use EAGLE-2 with --speculative-algorithm EAGLE.

• Lower lm_head overhead for EAGLE-2: Enable FR-Spec with --speculative-token-map.

• Model is MTP-enabled: Use MTP via speculative decoding (often with small speculative_num_steps/topk/num_draft_tokens, see the example section).

• You have a smaller draft LLM: Use STANDALONE (--speculative-algorithm STANDALONE).

• No extra model available: Use NGRAM (--speculative-algorithm NGRAM, CUDA-only).

• Want overlap scheduler (experimental): Enable SpecV2 with SGLANG_ENABLE_SPEC_V2=True (requires --speculative-eagle-topk 1).

Method comparison (mini table)#

Performance Highlights#

Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding. For further details please see the EAGLE3 paper.

EAGLE-2 Decoding#

You can enable EAGLE-2 Decoding by setting --speculative-algorithm EAGLE and choosing an appropriate model.

Launch the server:

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs 8 \
    --log-level warning

Send a request:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

EAGLE-2 Decoding with torch.compile#

You can optionally enable torch.compile to apply kernel-level optimizations (operator fusion, autotune) to the draft model. The actual speedup depends on your hardware, model architecture, and batch size. In some configurations (e.g., small draft models on H100 where cuBLAS is already optimal and CUDA graphs are enabled), the benefit may be negligible. We recommend benchmarking with and without this flag on your specific setup to verify whether it helps.

To enable it, add --enable-torch-compile and optionally set --torch-compile-max-bs:

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --enable-torch-compile \
    --torch-compile-max-bs 8 \
    --log-level warning

Send a request:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling#

By employing a truncated high-frequency token vocabulary in the draft model, EAGLE speculative decoding reduces lm_head computational overhead while accelerating the pipeline without quality degradation. For more details, check out the paper.

In our implementation, set --speculative-token-map to enable the optimization. You can get the high-frequency tokens in FR-Spec from this model. Or you can obtain high-frequency tokens by directly downloading these tokens from this repo.

Thanks for the contribution from Weilin Zhao and Zhousx.

python3 -m sglang.launch_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs 8 \
    --dtype float16 \
    --log-level warning

Send a request:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

EAGLE-3 Decoding#

You can enable EAGLE-3 decoding by setting --speculative-algorithm EAGLE3 and choosing an appropriate model.

python3 -m sglang.launch_server \
    --model meta-llama/Meta-Llama-3.1-8B-Instruct \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs 8 \
    --dtype float16 \
    --log-level warning

Send a request:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

Ngram-specific parameters#

Notes:

• Ngram speculative decoding only supports CUDA.

• It currently does not support --enable-dp-attention.

• It disables the overlap scheduler and mixed chunked prefill.

• If --speculative-ngram-max-bfs-breadth > 1 (thus speculative_eagle_topk > 1) and page_size > 1, use --attention-backend flashinfer; otherwise the server will error.

• Optional: set SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True to force greedy verification.

python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm NGRAM \
    --speculative-num-draft-tokens 16 \
    --speculative-ngram-max-match-window-size 12 \
    --speculative-ngram-max-bfs-breadth 10 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs 8 \
    --log-level warning

Send a request:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

Core parameters#

Ngram-specific parameters#

Environment variables#

Other related flags#

Step 1: Lower static memory fraction (most effective)#

--mem-fraction-static 0.5   # when omitted, this value is auto-computed

• --mem-fraction-static controls the memory budget for model weights + KV cache pool.

• Lowering it directly increases dynamic headroom for activations and CUDA graph buffers.

• If omitted, SGLang auto-estimates this value from other settings, and those auto settings can still be too aggressive for some workloads.

Step 2: Reduce CUDA graph batch size#

# Fewer CUDA graph captures = less memory reserved
--cuda-graph-max-bs 4   # or even 2 for tight memory situations

• If omitted, --cuda-graph-max-bs is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.

Step 3: Reduce draft tree size#

These three parameters directly control how much memory the draft tree consumes:

# Before (aggressive, high memory)
--speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64

# After (conservative, lower memory)
--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Step 4: Limit concurrent requests#

# Fewer concurrent requests lowers in-flight load and can reduce OOM risk
--max-running-requests 4

Quick OOM recovery recipe#

If you’re hitting OOM and just want something that works, start with this minimal configuration and scale up:

python3 -m sglang.launch_server \
    --model <your-model> \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path <your-draft-model> \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --cuda-graph-max-bs 2 \
    --mem-fraction-static 0.5 \
    --max-running-requests 4 \
    --log-level warning

Then gradually increase --speculative-num-draft-tokens, --speculative-eagle-topk, and --cuda-graph-max-bs. Increase --mem-fraction-static last, only after the run is stable.