8000 Update CachingChatClient with callbacks for if caching should happen · Issue #6030 · dotnet/extensions · GitHub
[go: up one dir, main page]
Skip to content

Update CachingChatClient with callbacks for if caching should happen #6030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
stephentoub opened this issue Mar 4, 2025 · 1 comment
Open

Update CachingChatClient with callbacks for if caching should happen #6030

stephentoub opened this issue Mar 4, 2025 · 1 comment
Labels
area-ai Microsoft.Extensions.AI libraries

Comments

@stephentoub
Copy link
Member
stephentoub commented Mar 4, 2025 •
edited
Loading

The caching chat client doesn't work great when dealing with an OpenAI Assistant-like entity, where the ConversationId is non-null and remains the same across all operations. In such cases, the provided chat messages and options are insufficient to uniquely identify state in the cache.

We should:

  1. Disable caching by default when ChatOptions.ConversationId is set
  2. Provide either virtual methods or delegate properties that allow determining on a per-call basis whether to check the cache or cache a result
@stephentoub stephentoub added untriaged area-ai Microsoft.Extensions.AI libraries and removed untriaged labels Mar 4, 2025
@stephentoub stephentoub self-assigned this May 8, 2025
@stephentoub stephentoub mentioned this issue May 8, 2025
Merged
@stephentoub
Copy link
Member Author
stephentoub commented May 8, 2025 •
edited
Loading
63F8

(1) is handled by #6400.

(2) is still open.

@stephentoub stephentoub removed their assignment May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-ai Microsoft.Extensions.AI libraries
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stephentoub
0