Description
I'm noticing high latency (5-10 seconds) in below cases:
- Between a session is created & root agent 'before_model_callback' trigger - This is constant across all requests.
- Any agent making request/receiving response to/from tools. This is highly seen in many cases (3-6 seconds), but not 100% of the time.
- Response case - Between after_tool_callback of tool and before_model_callback of agent
- Request case - Between after_model_callback of agent and before_tool_callback of tool
My ADK is structured at high level as follows:
A root agent has 10 tools (8 direct API as Function tools, 2 SubAgents as tools using AgentAsTool). I won't describe internal structure of subagents, as I don't see it relevant to bug, let me know if it helps in debugging.
How I concluded on above points: I've added logs on root agent and subagent all 6 callbacks with timestamps. (after/before_tool/model/agent_callback). [I've noticed one more issue, if I'm using after_agent_callback for logging and returning 'None' on root_agent, it skips agent LLM all-together. I'll report it in another ticket]
Code -
Here's my event_generator() code:
async def event_generator():
content = types.Content(role='user', parts=[types.Part(text=query)])
existing_sessions = await session_service.list_sessions(app_name=APP_NAME, user_id=user_id)
existing_session_ids = [s.id for s in existing_sessions.sessions]
if not session_id or session_id not in existing_session_ids:
await session_service.create_session(app_name=APP_NAME, user_id=user_id, session_id=session_id, state={
TENANT_ID_KEY: tenant_id,
USER_ID_KEY: user_id,
REQUEST_ID: req.context.request_id,
"events": [{"event-name": "session_initialized-event", "time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "name": "session-create"}]
})
self.logger.debug(f"Session created and initial data has been set. Sessionid: {session_id}")
search_doc_ids = []
run_config = RunConfig(streaming_mode='sse')
async for event in runner.run_async(user_id=user_id, session_id=session_id, new_message=content, run_config=run_config):
print(f"{req.context.request_id} [Agent Event] Author: {event.author}, Type: {type(event).__name__}, Final: {event.is_final_response()}, Content: {event.content}")
for index, part in enumerate(event.content.parts):
function_call_present = part.function_call
function_response_present = part.function_response
-------
Here's the root_agent:
root_agent = LlmAgent(
model=LiteLlm(model=MODEL_GPT_4O),
name="XXX",
description=agent_description,
instruction=planner_instructions.format(TOOLS_DESCRIPTION=TOOLS_DESCRIPTION, TIME_NOW=datetime.now()),
tools=[....],
after_model_callback=after_model_callback,
before_model_callback=before_model_callback,
# before_agent_callback=before_agent_callback,
after_agent_callback=after_agent_callback,
before_tool_callback=before_tool_callback,
after_tool_callback=after_tool_callback,
)
For simplicity, any of the above callback code is like this: (tool_context for tools callback)
def log_agent_timing(callback_context: CallbackContext) -> Optional[types.Content]:
agent_name = callback_context.agent_name
reqId = callback_context.state[REQUEST_ID]
print(f"{reqId} [Callback] Before Agent call for agent: {agent_name}")
events = callback_context.state["events"]
events.append({"event-name": "before-agent-event", "time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "name": agent_name})
callback_context.state["events"] = events
return None
Results:
Events array in session state: (Notice 7 seconds delay in session create and root_agent before-model-event)
For Point 1:
[{
"name": "session-create",
"time": "2025-06-20 19:54:10",
"event-name": "session_initialized-event"
},
{
"name": "XXX",
"time": "2025-06-20 19:54:17",
"event-name": "before-model-event"
},
......
]
For Point 2:
-----
{
"name": "XXX",
"time": "2025-06-20 19:54:48",
"event-name": "after-model-event"
},
{
"name": "Tool-X",
"time": "2025-06-20 19:54:51",
"event-name": "before-tool-event"
},
----
DETAILS
OS: Mac
ADK version: 1.3.0
Python version: 3.10
Expected behavior:
- ADK Orchestration should not be a significant part of Agent latency.
Let me know what else is needed