models - unsupported content types #144
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently, if a model provider doesn't support a specific content type (e.g., video), we try to coerce it into a text block using
json.dumps
. This will fail if the content type includes binary data. As an alternative, we should probably not try to be opinionated on this and just throw an exception.Note, throwing an exception now technically creates a backwards incompatible change. Where unsupported content blocks are successfully coerced into text, they will now result in an exception. However, consider the following...
Currently, strands itself can accepts messages with content blocks of the following types:
Here is the support across the model providers:
For each type, it is important to note:
document
,image
, andvideo
all include binary payloads and so currently, users who try to pass them through a model without support will encounter ajson.dumps
exception. The explicit exception we add here improves the experience.reasoningContent
is content generated by the model (assistant role) and so models that don't support it shouldn't be seeing these blocks anyway.cachePoint
where not supported.guardContent
is specific to Bedrock and so it is unlikely users have tried to pass it in where not supported.In summary, even though this change is backwards incompatible, it is unlikely we will actually break any customers.
Related Issues
strands-agents/tools#53
For model providers that don't support binary document types (e.g., pdf), they are already seeing JSON dump exceptions. Now we will throw an explicit exception for more clarity.
Type of Change
Testing
hatch fmt --linter
hatch fmt --formatter
hatch test --all
hatch run test-lint
Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.