Description
I have seen related discussion: #341
and a related PR: #654
But seems like function tools don't support returning images as outputs yet.
I wonder what's the best workaround we'd have around this, or whether including images in the outputs would make sense for my use case?
For context, I'm building a PagerDuty alert root cause analysis agent with access to tools like this:
agent = Agent("You are an expert SRE agent. Help me diagnose the root cause.",
tools = [search_logs_on_elasticsearch, check_panel_on_grafana]
)
For the check_panel_on_grafana tool, since time series data could be huge, I was thinking I'd first plot the data as an image, and then feed the image into LLM along with some descriptions (start time, end time, panel name, etc.).
I was thinking of just returning both the image and the text directly in the function output. Seems like that's not supported yet though.
Is my best workaround something like this? Call LLM directly and return the results?
@function_tool
def check_panel_on_grafana():
data = get_data_from_grafana()
graph = plot_graph(data)
description = "cool description"
prompt = "describe the image as thoroughly as possible"
result = call_chatgpt_directly(prompts, messages=[{graph, description}])
return result
but i guess call_chatgpt_directly won't have context to all the previous actions done by the agent thus far, and also, we only return text so all future actions won't get to see the actual image.