Translate OpenAI refusals to ErrorContent #6393
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it.
eiriktsarpalis
approved these changes
May 8, 2025
jeffhandley
pushed a commit
that referenced
this pull request
May 9, 2025
Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it.
jeffhandley
added a commit
that referenced
this pull request
May 9, 2025
* Update CHANGELOGS for M.E.AI libs (#6359) * Add project template build tests + CG reporting (#6355) Co-authored-by: Steve Sanderson <SteveSandersonMS@users.noreply.github.com> * Use ConversationId instead of ChatThreadId (#6367) * Update OpenTelemetryChatClient/EmbeddingGenerator to 1.33 (#6366) Also adds the Enable SensitiveData property to OpenTelemetryEmbeddingGenerators. This was missed when adding it to OpenTelemetryChatClient. * Add AsIEmbeddingGenerator for Azure.AI.Inference ImageEmbeddingsClient (#6363) * Add DataContent.Base64Data (#6365) * Fix AIFunctionFactory handling of default struct arguments (#6381) * Fix AIFunctionFactory handling of default struct arguments * Extend testing to custom structs * Use GetUninitializedObject instead of Activator * Add JS dependency update instructions to chat template README (#6376) * Add see also links to conceptual docs (#6368) * Translate OpenAI refusals to ErrorContent (#6393) Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it. * Rename useJsonSchema parameter (#6394) * Rename useJsonSchema parameter The `GetResponseAsync<T>` methods accept a `bool?` parameter currently called `useJsonSchema`. This is confusing, because the whole point of the method is to create and use a JSON schema from the `T`. The parameter actually controls _how_ that schema is used, whether it's included as part of the messages (false), as part of a ChatResponseFormat in the ChatOptions (true), or up to the system to decide. I've clarified it by renaming it from `useJsonSchema` to `useJsonSchemaResponseFormat`. It's wordier, but it disambiguates the intent. * Update src/Libraries/Microsoft.Extensions.AI/ChatCompletion/ChatClientStructuredOutputExtensions.cs Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> --------- Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> * Add JSON schema transformation functionality to `AIJsonUtilities` (#6383) * Add initial schema transformation functionality and incorporate into the OpenAI leaf client. * Update all leaf client implementions, improve naming, add testing. * Remove redundant suppressions * Address feedback. * Add ChatOptions.RawRepresentationFactory (#6319) * Look for OpenAI.ChatCompletionOptions in top-level additional properties and stop looking for individually specific additional properties * Add RawRepresentation to ChatOptions and use it in OpenAI and AzureAIInference * Remove now unused locals * Add [JsonIgnore] and update roundtrip tests * Overwirte properties only if the underlying model don't specify it already * Clone RawRepresentation * Reflection workaround for ToolChoice not being cloned * Style changes * AI.Inference: Bring back propagation of additional properties * Don't use 0.1f, it doesn't roundtrip properly in .NET Framework * Add RawRepresentationFactory instead of object? property * Augment remarks to discourage returning shared instances * Documentation feedback * AI.Inference: keep passing TopK as AdditionalProperty if not already there * Avoid caching in CachingChatClient when ConversationId is set (#6400) * Add comment LoggingChatClient et al trace-level logging (#6391) Also fixed the name of the LoggingSpeechToTextClientBuilderExtensions type to conform to patterns used elsewhere in the library. * Fix test validation of aggregate usage counts (#6401) * Add BinaryEmbedding (#6398) * Add BinaryEmbedding Also: - Renames the polymorphic discriminators to conform with typical lingo for these types. - Adds an Embedding.Dimensions virtual property. * Fix keys for EvaluationResult.Metrics dictionary to reflect the correct metric names for Safety evaluators (#6361) This was an unfortunate regression that was introduced during a recent refactoring. The metrics returned from the Azure AI Foundry Evaluation service have different names than the ones we use in the Safety library. We translate the EvaluationMetric.Name of the metrics returned by the service to the mo 8000 re display friendly names used in the library before returning the metrics to the caller. While the returned metrics were correctly patched up, the EvaluationResult.Metrics dictionary still stored these metrics by the original names returned from the service. Unfortunately, this meant EvaluationResult.Get would throw an exception when trying to fetch metric with name ViolenceEvaluator.ViolenceMetricName. The fix in this commit fixes the keys in the dictionary as well. This commit also updates tests to cover the case being fixed. Fixes #6360 * Update branding for Azure service used by Safety evaluators (#6362) `Azure AI Content Safety service` -> `Azure AI Foundry Evaluation service` * Remove CacheOptions from DiskBasedResponseCache (#6395) Details for this change are available in #6387. Fixes #6387 * Add back net9.0 version of the aieval dotnet tool (#6396) In #6148, we disabled net9.0 TFM for the aieval tool to work around the race described in dotnet/sdk#47696. The underlying issue was subsequently fixed in the SDK (via dotnet/sdk#47788). However, this fix has not been backported to the dotnet 9 SDK yet. The SDK team is working on backporting the fix (see discussion in dotnet/sdk#47788 (comment)). But in the meanwhile, we can add back the net9.0 TFM and continue to work around the race by disabling parallel build. This would help users of the aieval tool sidestep errors such as the ones described in #6388 when they dont have dotnet8 installed on the machine. We can remove this workaround, once the backported fix is available in the dotnet 9 SDK. Fixes #6388 * Add some additional documentation around usage of cache, and CSP properties on report (#6377) * Add documentation around proper usage of IDistributedCache * Add Content-Security-Policy to prevent page from calling into other sites. * Remove remark about IDistributedCache usage * Fix package-lock.json * Remove <remarks> start tag. * Some API related fixes for the evaluation libraries (#6402) * Rename IResultStore and IResponseCacheProvider IResultStore -> IEvaluationResultStore and IResponseCacheProvider -> IEvaluationResponseCacheProvider * Include missing EvaluationContextConverter in AzureStorageJsonUtilities Also use linked files to avoid the need to duplicate code. * Reorder enum members The new order goes from least desirable rating to most desirable. * Refactor extension method overloads Implement overloads that take ChatMessage by calling corresponding overloads that take ChatResponse. * Refactor AddTurnDetails to support adding details for multiple turns Adding single turns continues to be supported via a params array overload. * Add missing parameter for timeToLiveForCacheEntries to DiskBasedReportingConfiguration This was missed in an earlier PR that introduced the timeToLiveForCacheEntries on the constructor of DiskBasedResponseCacheProvider. Also reorder constructor parameters for AzureStorageReportingConfiguration so that the parameters for caching apear next to each other and so that the parameter ordering is aligned with DiskBasedReportingConfiguration. * Minor formatting changes * Bump vite from 6.2.6 to 6.3.4 in /src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/TypeScript (#6354) * Bump vite Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.2.6 to 6.3.4. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v6.3.4/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.3.4 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shyam Namboodiripad <gnamboo@microsoft.com> * Allow image rendering in evaluation report (#6407) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Stephen Toub <stoub@microsoft.com> Co-authored-by: Mackinnon Buck <mackinnon.buck@gmail.com> Co-authored-by: Steve Sanderson <SteveSandersonMS@users.noreply.github.com> Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> Co-authored-by: David Cantú <dacantu@microsoft.com> Co-authored-by: Shyam N <shyamnamboodiripad@users.noreply.github.com> Co-authored-by: Peter Waldschmidt <pewaldsc@microsoft.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shyam Namboodiripad <gnamboo@microsoft.com>
joperezr
pushed a commit
to joperezr/extensions
that referenced
this pull request
May 13, 2025
Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it.
jeffhandley
pushed a commit
that referenced
this pull request
May 15, 2025
Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it.
jeffhandley
added a commit
that referenced
this pull request
May 15, 2025
…es (#6451) * Translate OpenAI refusals to ErrorContent (#6393) Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it. * Add JSON schema transformation functionality to `AIJsonUtilities` (#6383) * Add initial schema transformation functionality and incorporate into the OpenAI leaf client. * Update all leaf client implementions, improve naming, add testing. * Remove redundant suppressions * Address feedback. * Add ChatOptions.RawRepresentationFactory (#6319) * Look for OpenAI.ChatCompletionOptions in top-level additional properties and stop looking for individually specific additional properties * Add RawRepresentation to ChatOptions and use it in OpenAI and AzureAIInference * Remove now unused locals * Add [JsonIgnore] and update roundtrip tests * Overwirte properties only if the underlying model don't specify it already * Clone RawRepresentation * Reflection workaround for ToolChoice not being cloned * Style changes * AI.Inference: Bring back propagation of additional properties * Don't use 0.1f, it doesn't roundtrip properly in .NET Framework * Add RawRepresentationFactory instead of object? property * Augment remarks to discourage returning shared instances * Documentation feedback * AI.Inference: keep passing TopK as AdditionalProperty if not already there * Fix streaming chat response example (#6408) * Move AIFunctionFactory down to M.E.AI.Abstractions (#6412) * Remove AIFunctionFactory dependency on M.E.DI This means reverting the recent changes to it that: - Special-cased KeyedServices - Special-cased IServiceProviderIsService - Used ActivatorUtilities.CreateInstance * Move AIFunctionFactory down to M.E.AI.Abstractions * Add CreateInstance delegate to AIFunctionFactoryOptions To enable use of ActivatorUtilities.CreateInstance or alternative. * Add some comments * Fix handling of tool calls with some OpenAI endpoints (#6405) * Fix handling of tool calls with some endpoints Most assistant messages containing tool calls don't contain text as well (though some can). In such a case, we were still creating the assistant with empty text. While OpenAI's service permits that, some other endpoints are more finicky about it. This avoids doing so. * Reduce to single iteration through assistant content * Delete Microsoft.Extensions.AI.Abstractions APIs marked [Obsolete] during preview (#6414) * Add WriteAsync overrides to stream helper in AIFunctionFactory (#6419) We use JsonSerializer.SerializeAsync but were missing the async overrides. As with MemoryStream, these don't need to queue. * Replace Type targetType AIFunctionFactory.Create parameter with a func (#6424) * Add an AIJsonSchemaTransformOptions property inside AIJsonSchemaCreateOptions and mark redundant properties in the latter as obsolete. (#6427) * Add an AIJsonSchemaTransformOptions property inside AIJsonSchemaCreateOptions and mark redundant properties in the latter as obsolete. * s/inferred/created * Eliminate ingestion cache from AI Chat Web template (#6428) * Begin updating to latest MEVD * Reimplement JsonVectorStore to match updated MEVD APIs * Remove ingestion cache and track ingestion status inside the vector DB * Track the document metadata in a separate collection so we don't have to fetch literally everything from the vector DB in order to update ingestion * Fix equality comparison issue with Qdrant connector * Tidying * More tidying * Update MEAI.Templates test snapshots --------- Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com> * Update the template test README with snapshot update instructions (#6431) * AI Chat Web template fixes for Azure AI Search (#6429) * AI Chat Web template fixes for Azure AI Search * Update snapshots * Add security comments for chat clients (#6386) * Remove unused select param (#6341) CreateRecordsForDocumentAsync includes `Select((pair, index) =>` but index is never used Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com> * Add RawRepresentationFactory to other options types (#6433) * Add RawRepresentationFactory to other options types * Undo changes in Azure.AI.Inference * Address documentation feedback * Ensure the type keyword is included when generating schemas for nullable enums. (#6440) * Remove obsolete members from AIJsonSchemaCreateOptions (#6432) * Use RawRepresentationFactory on AzureAIInference embedding generators (#6445) * Mark Microsoft.Extensions.AI and Microsoft.Extensions.AI.Abstractions as stable (#6446) * Bump ICSharpCode.Decompiler for record struct support in ApiChief tool Needed the fix for icsharpcode/ILSpy#3159 to fix "record struct" formatting (it was "recordstruct" before the fix). * Generate ApiChief baselines for MEAI libraries Ran .\scripts\MakeApiBaselines.ps1 and discarded other libraries' updates. * Hand-edit MEAI ApiChief baseline to fix params ReadOnlySpan Params collections are not yet supported in ICSharpCode.Decompiler: icsharpcode/ILSpy#829 The result is an emitted 'scoped' keyword instead of 'params'. This was edited by hand in the baseline MEAI file. * Mark Microsoft.Extensions.AI and Microsoft.Extensions.AI.Abstractions as stable * Update MEAI and MEAI.Abstractions NuGet package documentation * Update NuGet package documentation for MEAI implementation packages * Update MEAI.Templates package references, including SemanticKernel for a coherent build. * Lower OllamaSharp for integration tests to use version available on feed * Empty the ApiChief baselines for Ollama, AzureAIInference, and OpenAI adapters since they are not shipping stable * Apply code review feedback to the MEAI package READMEs * Update MEAI.Templates test snapshots for version bumps * Apply documentation review feedback to the MEAI package READMEs * Add comments to the MEAI API baseline file for the hand-editing required * Restore documentation blurb into Microsoft.Extensions.AI.AzureAIInference per other feedback * Use stable version for MEAI in templates * Mark Microsoft.Extensions.AI.Evaluation.* Libraries as stable (#6450) * Mark Microsoft.Extensions.AI packages stable All packages except Microsoft.Extensions.AI.Evaluation.Safety are being marked stable. * Remove primary constructors from API json files. * Remove more primary constructors from API Chief json --------- Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com> --------- Co-authored-by: Stephen Toub <stoub@microsoft.com> Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> Co-authored-by: David Cantú <dacantu@microsoft.com> Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Co-authored-by: Steve Sanderson <SteveSandersonMS@users.noreply.github.com> Co-authored-by: Mackinnon Buck <mackinnon.buck@gmail.com> Co-authored-by: Jon Galloway <jongalloway@gmail.com> Co-authored-by: Peter Waldschmidt <pewaldsc@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refusals in OpenAI are errors reported when the service can't generate an output that matches the requested schema. Translate refusals to ErrorContent now that we have it.
Microsoft Reviewers: Open in CodeFlow