8000 models - unsupported content types by pgrayy · Pull Request #144 · strands-agents/sdk-python · GitHub
[go: up one dir, main page]
Skip to content

models - unsupported content types #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 30, 2025
Merged

models - unsupported content types #144

merged 2 commits into from
May 30, 2025

Conversation

pgrayy
Copy link
Member
@pgrayy pgrayy commented May 29, 2025 •
edited
Loading

Description

Currently, if a model provider doesn't support a specific content type (e.g., video), we try to coerce it into a text block using json.dumps. This will fail if the content type includes binary data. As an alternative, we should probably not try to be opinionated on this and just throw an exception.

Note, throwing an exception now technically creates a backwards incompatible change. Where unsupported content blocks are successfully coerced into text, they will now result in an exception. However, consider the following...

Currently, strands itself can accepts messages with content blocks of the following types:

  • cachePoint
  • document
  • guardContent
  • image
  • reasoningContent
  • text
  • toolResult
  • toolUse
  • video

Here is the support across the model providers:

  • Bedrock: Supports all types
  • Anthropic: Does not support guardContent and video
  • OpenAI: Does not support cachePoint, guardContent, reasoningContent, and video
  • LiteLLM: Does not support cachePoint and guardContent
  • LlamaAPI: Does not support cachePoint, document, guardContent, reasoningContent, and video
  • Ollama: Does not support cachePoint, document, guardContent, reasoningContent, and video

For each type, it is important to note:

  • document, image, and video all include binary payloads and so currently, users who try to pass them through a model without support will encounter a json.dumps exception. The explicit exception we add here improves the experience.
  • reasoningContent is content generated by the model (assistant role) and so models that don't support it shouldn't be seeing these blocks anyway.
  • Prompt caching is configurable on Bedrock and Anthropic but automatic for other model providers. It is thus unlikely users have tried to pass in cachePoint where not supported.
  • guardContent is specific to Bedrock and so it is unlikely users have tried to pass it in where not supported.

In summary, even though this change is backwards incompatible, it is unlikely we will actually break any customers.

Related Issues

strands-agents/tools#53

For model providers that don't support binary document types (e.g., pdf), they are already seeing JSON dump exceptions. Now we will throw an explicit exception for more clarity.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update: Docstrings for now. Will follow up to add details on these changes in the model provider user guides.
  • Other (please describe):

Testing

  • hatch fmt --linter
  • hatch fmt --formatter
  • hatch test --all
  • Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli
  • hatch run test-lint

Checklist

  • I have read the CONTRIBUTING document
  • I have added tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@pgrayy pgrayy requested a review from a team as a code owner May 29, 2025 19:05
Unshure
Unshure reviewed May 29, 2025
View reviewed changes
@pgrayy pgrayy merged commit b8939e7 into strands-agents:main May 30, 2025
10 checks passed
jer96 pushed a commit to lukehau/sdk-python that referenced this pull request Jun 3, 2025
@pgrayy @jer96
models - unsupported content types (strands-agents#144)
837B
5d17dbe
@fede-dash fede-dash mentioned this pull request Jun 10, 2025
Draft
15 tasks
@pgrayy pgrayy deleted the models-unsupported-content branch June 10, 2025 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Unshure Unshure Unshure approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pgrayy @Unshure
0