8000 [tool call] Fix prev_tool_call_arr management in base_format_detector.py by CatherineSue · Pull Request #11367 · sgl-project/sglang · GitHub
[go: up one dir, main page]
Skip to content

[tool call] Fix prev_tool_call_arr management in base_format_detector.py #11367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

[tool call] Fix prev_tool_call_arr management in base_format_detector.py #11367

wants to merge 5 commits into from
+23 −17

Conversation

CatherineSue
Copy link
Collaborator
@CatherineSue CatherineSue commented Oct 9, 2025 •
edited
Loading

Motivation

There is a bug in the current prev_tool_call_arr management in base_format_detector.py.
When a tool call is complete at L258, self.current_tool_call_id increments by 1.

NOTE: the bug itself is not very harmless because it is hard to encounter (or notice).

1. Wrong current_tool_call_id

sglang/python/sglang/srt/function_call/base_format_detector.py

Lines 267 to 271 in 84768d1

if self.current_tool_id < len(self.prev_tool_call_arr):
self.prev_tool_call_arr[self.current_tool_id].clear()
self.current_tool_name_sent = False
self.streamed_args_for_tool[self.current_tool_id] = ""
self.current_tool_id += 1

However, at L301, current_tool_call_id is used to update in prev_tool_call_arr.

sglang/python/sglang/srt/function_call/base_format_detector.py

Lines 301 to 306 in 84768d1

# Update prev_tool_call_arr with current state
if self.current_tool_id >= 0:
# Ensure prev_tool_call_arr is large enough
while len(self.prev_tool_call_arr) <= self.current_tool_id:
self.prev_tool_call_arr.append({})
self.prev_tool_call_arr[self.current_tool_id] = current_tool_call

This would result a tool call that has completed, for instance, 0, be put in self.prev_tool_call_arr[1]. See the screenshot below. The current tool call 0 is completed, but the result is saved in self.prev_tool_call_arr[1].

Screenshot 2025-10-08 at 7 47 01 PM

This may cause a defensive check in _check_for_unstreamed_tool_args. As at this moment, the length of prev_tool_call_arr is inconsistent with streamed_args_for_tool

sglang/python/sglang/srt/entrypoints/openai/serving_chat.py

Lines 1116 to 1119 in 84768d1

# Get the last tool call that was being processed
tool_index = len(detector.prev_tool_call_arr) - 1
if tool_index < 0 or tool_index >= len(detector.streamed_args_for_tool):
return None

2. Empty prev_tool_call_arr

A second issue in the logic is that, when a tool call completes, base_format_detector.py#L267-L271 (see above) tries to clear the prev_tool_call_arr[current_tool_id]. However, prev_tool_call_arr is supposed to keep all the tool call data, so in _check_for_unstreamed_tool_args, it could do:

sglang/python/sglang/srt/entrypoints/openai/serving_chat.py

Lines 1121 to 1124 in 84768d1

# Get expected vs actual arguments
expected_args = detector.prev_tool_call_arr[tool_index].get("arguments", {})
expected_call = json.dumps(expected_args, ensure_ascii=False)
actual_call = detector.streamed_args_for_tool[tool_index]

If we clear the tool call when it is complete, this check above will always get {}.

3. Only update streamed_args_for_tool when is_complete

Same as the above reason, streamed_args_for_tool should be updated as well even when is_complete=True

Modifications

  • Always update self.prev_tool_call_arr[self.current_tool_id] with current_tool_call.
  • Doesn't clear self.prev_tool_call_arr[self.current_tool_id] when is_complete.

Accuracy Tests

Benchmarking and Profiling

Checklist

@gemini-code-assist Gemini Code Assist
Copy link
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@CatherineSue
Copy link
Collaborator Author

Need to wait for CI to complete

@CatherineSue
Copy link
Collaborator Author

Looking into failed tests

@CatherineSue
Copy link
Collaborator Author

function_call module passed.
Screenshot 2025-10-08 at 11 12 01 PM
Screenshot 2025-10-08 at 11 20 33 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JustinTong0323 JustinTong0323 Awaiting requested review from JustinTong0323 JustinTong0323 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
run-ci
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@CatherineSue @sglang-bot
0