Add OpenRouterModel
#1870
Add OpenRouterModel
#1870
Conversation
6f299fb to
1341160
Compare
There was a problem hiding this comment.
@DanKing1903 Thanks Dan! In addition to the comments I left, we'll also need to update the docs. Can you please have a look at that? Note that this may not make sense anymore to do this under the "OpenAI compatible" section as it's now its own model, not just a provider to use with OpenAIModel.
| error: OpenRouterErrorResponse | None | ||
|
|
||
|
|
||
| class OpenRouterModel(OpenAIModel): |
There was a problem hiding this comment.
Can we please override __init__ to automatically use the OpenRouterProvider as well when this model is used?
There was a problem hiding this comment.
@DouweM I have added an __init__ method with narrowed type annotations for provider that does nothing apart from calling super().__init__. Is this acceptable or would you like me to add something like:
if isinstance(provider, str):
if provider != "openrouter":
error_msg = ...
raise ValueError(error_msg)
provider = OpenRouterProvider()
| """Extends OpenAIModel to capture extra metadata for Openrouter.""" | ||
|
|
||
| def _process_response(self, response: chat.ChatCompletion) -> ModelResponse: | ||
| response = cast(OpenRouterChatCompletion, response) |
There was a problem hiding this comment.
I don't think this does what we want, because response won't actually be an OpenRouterChatCompletion. Would response = OpenRouterChatCompletion.modal_validate(response) work to actually create the new object? Then we should be able to read response.error without checking hasattr.
There was a problem hiding this comment.
Thanks for the feedback. While testing your suggestion I ran into some issues. The problem is that when OpenRouter is relaying upstream errors, the response is not actually a valid ChatCompletion. The signature of the response as per their docs is:
type ErrorResponse = {
error: {
code: number;
message: string;
metadata?: Record<string, unknown>;
};
};
So the code should look more like this:
class OpenRouterErrorDetails(BaseModel):
code: int
message: str
metadata: dict[str, Any] | None
class OpenRouterErrorResponse(BaseModel):
error: OpenRouterErrorDetails
class OpenRouterChatCompletion(chat.ChatCompletion):
provider: str
class OpenRouterModel(OpenAIModel):
def _process_response(self, response: Union[OpenRouterChatCompletion, OpenRouterErrorResponse]) -> ModelResponse:
if error := getattr(response, 'error', None):
raise ModelHTTPError(status_code=error['code'], model_name=self.model_name, body=error)
else:
model_response = super()._process_response(response=response)
response = OpenRouterChatCompletion.model_validate(response.model_dump())
...
However this approach results in typechecking errors because I would be overriding the signature of _process_response:
PYRIGHT_PYTHON_IGNORE_WARNINGS=1 uv run pyright
/Users/danwork/Dev/pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openrouter.py
/Users/danwork/Dev/pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openrouter.py:52:9 - error: Method "_process_response" overrides class "OpenAIModel" in an incompatible manner
Parameter 2 type mismatch: base parameter is type "ChatCompletion", override parameter is type "OpenRouterChatCompletion | OpenRouterErrorResponse"
Type "ChatCompletion" is not assignable to type "OpenRouterChatCompletion | OpenRouterErrorResponse"
"ChatCompletion" is not assignable to "OpenRouterChatCompletion"
"ChatCompletion" is not assignable to "OpenRouterErrorResponse" (reportIncompatibleMethodOverride)
/Users/danwork/Dev/pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openrouter.py:56:65 - error: Argument of type "OpenRouterChatCompletion | OpenRouterErrorResponse" cannot be assigned to parameter "response" of type "ChatCompletion" in function "_process_response"
Type "OpenRouterChatCompletion | OpenRouterErrorResponse" is not assignable to type "ChatCompletion"
"OpenRouterErrorResponse" is not assignable to "ChatCompletion" (reportArgumentType)
2 errors, 0 warnings, 0 informations
make: *** [typecheck-pyright] Error 1
Would appreciate if you have any advice on how I could handle this! Thanks
There was a problem hiding this comment.
Ah I think the solution is to move the error handling into _completions_create...
|
Can't we not create a new model and handle this in the |
|
@Kludex I didn't want to clutter @DanKing1903 Can you please try reimplementing this on |
|
Thanks for the PR. I wanted to highlight that OpenRouter’s API offers much more than basic OpenAI compatibility. The provider routing and fallback logic are key differentiators that I and likely many others rely on. I understand that this PR primarily focuses on capturing provider metadata in responses, which is absolutely great. Would it be possible to also support OpenRouter-specific request parameters? Examples include:
See: https://openrouter.ai/docs/features/provider-routing for the full parameter list. These are typically passed via I haven’t checked if this is already supported. If it is, disregard this comment. Otherwise, please consider enabling this for the OpenRouter provider/model. This would enable some important routing use cases. |
|
@piiq Thanks for jumping in, I agree those are OpenRouter-specific features worth supporting that wouldn't make sense on @DanKing1903 Would you be up for seeing how those features could be implemented on the new |
This PR implements an OpenRouterModel class as discussed in #1849. The
OpenRouterModelis an extension ofOpenAIModelthat captures theprovidermetadata returned by OpenRouter in thevendor_detailsdictionary.Additionally adds error handling for the scenario when OpenRouter API returns a 200 OK, but the response relays an error from the upstream LLM provider as described in #1746 and #527
Changes
OpenRouterModelthat extendsOpenAIModelto handle OpenRouter-specific response fields and error handlingLinked Issues