8000 [AI Evaluation] Quality evaluators should exclude tool call messages by default · Issue #6043 · dotnet/extensions · GitHub
[go: up one dir, main page]

Skip to content

[AI Evaluation] Quality evaluators should exclude tool call messages by default #6043

@shyamnamboodiripad

Description

@shyamnamboodiripad

See discussion here. Currently, all the evaluators included in the Microsoft.Extensions.AI.Evaluation.Quality package will render tool call messages as strings with prefix ToolCall and empty content.

Ideally, tool call messages should be excluded from the rendering by default since the Quality evaluators are designed to assess the quality / efficacy of the LLM responses in the user visible portion of the chat conversation. At the same time, the base classes in the Quality package can be used to author other (custom) derived evaluators that may want to take tool calls into account - so we should leave the door open for derived implementations to override the default rendering of messages.

FYI @peterwald

Metadata

Metadata

Labels

area-ai-evalMicrosoft.Extensions.AI.Evaluation and related

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0