How to have an “optional” field but if present required to conform to non None value? #1223
Comments
|
use a validator from pydantic import BaseModel, validator
class Foo(BaseModel):
count: int
size: Optional[float] = None
@validator('size')
def prevent_none(cls, v):
assert v is not None, 'size may not be None'
return v |
|
Just to add onto @samuelcolvin's answer, this works because, by default, validators aren't called on arguments that are not provided (though there is a keyword argument on (Another route would be to use a sentinel value (e.g., Note that the partial lack of idempotency may cause trouble with certain frameworks (like FastAPI) which may convert the model to dict form (which may have the None), and then call the initializer again on the dumped data. In particular, as of now you are likely to run into this issue if you specify the model as a There may be a way to achieve a similar pattern that can track whether the root source of the value was default initialization (defeating the idempotency issues), but you'd have to avoid the use of |
|
This is irking me at the moment. Passing in a field as None is fundamentally different from not passing in a field at all. Right now, pydantic conflates the two by using the
is different (with regards to the intentions of the programmer) from
There might be more use-cases, but my own and I think the most obvious one is that sometimes I want to do a partial update: pass in some values, validate them (with pydantic) and then update them elsewhere (say: in my datastore). But. I don't want the values to be null. I want them to be either some type or I don't want them to exist at all. Having to use a custom validator for this wherever I need it is a lot of extra effort. At a library level, it seems there are two ways of dealing with this. One is to allow from pydantic import Required
class Foo(BaseModel):
a: Required[int]
b: int
c: Optional[int]
d: Required[Optional[int]]
class Config:
require_by_default = False (default: True)This would result in these requirements:
The other option would be to add a custom type that supports from pydantic import NotNone
class Foo(BaseModel):
a: int
b: NotNone[int]
c: Optional[int]The problem with this solution is that it does not support use case However, in both cases, I would actually prefer it if the first option was the default behavior for pydantic, but at this point clearly that is not on the table. |
|
@samuelcolvin @dmontagu Would there be any willingness to add this functionality to pydantic? I would be willing to start a PR if so. I personally am a big fan of option 1's functionality, as it allows for all possible iterations of providing data to a pydantic class, and I think is a better reflection of what The set of fields actually passed into instantiation is already stored anyway, so doesn't seem like this would hurt much perf wise. Some alternatives to calling the "must be present" pydantic type |
|
I wouldn't necessarily have a problem adding such functionality, but I'll warn you that if your goal is to use this with FastAPI, you may well run into problems until FastAPI is updated to make less use of dump-then-reparse in various places. After some review of the specs, I think the approach described in your first bullet is a substantially better reflection of OpenAPI/JSON Schema semantics. Because of that, I'm more open to a config-flag-based approach than I might otherwise be. However, I see two major obstacles:
As a result, I think we should avoid the use of
As a result, I think we need to keep annotated fields required by default. Between the above two points, I think it could make sense to add
Note that adding another config flag, especially one that modifies fundamental behavior like whether a type is required/optional by default, is likely to introduce many subtle bugs, add a lot of maintenance burden, and make it difficult to refactor things as we discover better approaches to implementation. So despite the comment I made above that I am more open to a config-based approach than I would normally be, I think in this case we should really, really try to avoid it if possible. But I think maybe the approach using the Note that adding support for |
|
Thanks for the great thoughts @dmontagu! Need to go over in more detail, but I just had an idea that I thought I'd throw out there (and is somewhat inspired by your previous comment in the thread) before I go to bed and forget it: What if, instead of Pydantic loading in missing fields as Then, for the "can be missing or This would presumably alleviate some of the burden with regard to type checking with mypy and the like, no? That way you don't technically even need generic types like |
|
Technically that information is already tracked -- you can look inside The obvious challenges I see with adding a new type to represent
Note that adding additional generic types wouldn't require changing the behavior of any existing code. To the extent that we needed to change existing logic to avoid serializing |
|
I think this is a duplicate of #1223 (at least the latter part of this discussion). I'm inclined to stick to the current logic that: class Model(base):
a: Optional[int] # this field is required bit can be given None (to be CHANGED in v2)
b: Optional[int] = None # field is not required, can be given None or an int (current behaviour)
c: int = None # this field isn't required but must be an int if it is provided (current behaviour)The only other thing I would consider (but am currently opposed to having read through #1223) is
|
|
Oh, I see that If not, I think |
|
The issue is that The deserialization/serialization lib I used before pydantic (marshmallow) handled this by having a |
|
I disagree. Since I therefore continue to hold the opinion that "X is not required, but if it is supplied it may not be None" is not particularly common. I'll wait to be proved wrong by 👍 on this issue or duplicate issues. Since there are two workarounds (validators or a custom type), I'm not that interested in continuing this conversation or adding Let's wait to see if others agree with you. |
|
The only reason How common it is does not change the fact that explicitly passing in some field as There is clearly a lot of confusion around how
If pydantic actually returned a I recognize that arguably the real reason for all this is because Python's @dmontagu I'm not sure what you mean by concern (1): How is this not already covered by the functionality of type checkers? Pydantic would return either |
|
@acnebs I agree that it is unfortunate that both missing or specified-as-null are conflated. But for better or worse I think this is ultimately a fairly pythonic convention -- if you have a keyword argument with a default value, you can't tell whether the keyword argument provided was specified as the default value or just not provided. Some may see this as a flaw in the language's design (e.g. Rust users), but at this point it's certainly conventional/idiomatic python.
The way you are describing this makes me think you might not be aware that you can obtain the precise set of fields that were set during initialization by using
Personally I am inclined to agree that it might have resulted in less confusion to have
You seem to be approaching the problem from a very JSON-centric perspective, but I would argue pydantic should be somewhat more concerned with type safety than following JSON conventions, and the approach used by pydantic is the approach used by mypy. Also, I would contest the claim that "most other languages would call this concept Nullable" -- when type-checked with mypy, python's At any rate, not everyone is using pydantic strictly for JSON-oriented parsing, so I'm not sure it makes sense to prioritize those conventions here.
Yes, this is true, but the vast majority of existing pydantic code has been written to assume that missing While it could certainly be type-safe (arguably more so than the current approach) to use a fundamentally different type to represent an unspecified value, it would add a large amount of boilerplate any time you didn't want to handle the cases differently, which I would argue is the case for most real applications.
As I said above, this is possible now using |
|
Thank you for pointing out I'm not sure "the approach used by pydantic is the approach used by mypy". Mypy doesn't concern itself with things that don't exist at all, so it's not really comparable. Mypy can only type check things that exist – pydantic had to make a choice about how to handle missing fields which aren't even there. This is a bit different from idiomatic python as well, because in idiomatic python (at least in the past), the reason that unspecified optional kwargs defaulted to I think the real reason for my confusion is that to my mind it doesn't make much sense for a default to be arbitrarily chosen as I will admit that I am using pydantic for JSON-centric parsing, but I think I have made it clear that this is also more of a general obejction to conflation rather than a complaint that "this doesn't perfectly mirror JSON's behavior". I understand that this is a generic data library (builtin methods for dumping to JSON notwithstanding). I'm more looking at things like marshmallow (what I was using before in Python and which is very widely used), which is also not JSON-centric but easily allows for the behavior we are talking about (in fact it is the default). |
|
How about this idea? Create a singleton for a missing property, so that the property is allowed to be Actually, this kind of Even better, I think that it could be great if |
|
A better |
from pydantic import BaseModel
import inspect
def optional(*fields):
def dec(_cls):
for field in fields:
_cls.__fields__[field].required = False
return _cls
if fields and inspect.isclass(fields[0]) and issubclass(fields[0], BaseModel):
cls = fields[0]
fields = cls.__fields__
return dec(cls)
return dec
class Book(BaseModel):
author: str
available: bool
isbn: str
@optional
class BookUpdate(Book):
pass
@optional('val1', 'val2')
class Model(BaseModel):
val1: str
val2: str
captcha: str |
|
I've come to this link after reading the pydantic documentation and searching internet for answers about the same question from this issue. |
|
I am having trouble understanding the optional decorator above. If we use the @ decorator syntax on a pydantic BaseModel schema, isn't that the condition - if fields and inspect.isclass(fields[0]) and issubclass(fields[0], BaseModel) - will always be true ? |
|
@VianneyMI when using it as
|
|
I'm writing APIs where I usually use patch requests instead of update requests. Problem is, some fields are nullable in the database, in which case the client apps should be able to pass null as a value. I tend to use But with this situation I cannot be sure frontend apps won't give a None value and then I have to check it myself, which contradicts the purpose of a validation library in my opinion. How would you solve the use case I described above? Notes:
|
Both actually isn't in both cases the first argument will be the decorated function anyway ? @ar45 |
|
@samuelcolvin so what's the final, recommended solution for such a use case? I read the whole discussion but it still looks like there is no one, final solution for partial objects that pydantic would support out of the box. |
|
In reference to: #1223 (comment) def optional(*fields, deep: bool = True):
"""
Makes specified fields optional.
If no fields are specified, makes all fields optional.
To not recursively make all fields of nested models optional as well, pass deep=False
"""
# Work is done inside optionalize
def optionalize(_cls):
for field in fields:
subfield = _cls.__fields__[field]
if deep and inspect.isclass(subfield.type_) and issubclass(subfield.type_, BaseModel):
# Must pass through optional so that fields variable gets prepared
optional(subfield.type_, deep=deep)
subfield.required = False
return _cls
# Decorator (only used if parameters are passed to optional)
def decorator(_cls):
return optionalize(_cls)
# If no parameters are passed to optional, return the result of optionalize (which is a class callable)
if fields and inspect.isclass(fields[0]) and issubclass(fields[0], BaseModel):
cls = fields[0]
fields = cls.__fields__
return optionalize(cls)
# Else, return the generated decorator
return decoratorPlease do not hesitate to make suggestions for further improvement! |
|
Here's an improved version of @ar45 's decorator: def partial(*fields):
""" Make the object "partial": i.e. mark all fields as "skippable"
In Pydantic terms, this means that they're not nullable, but not required either.
Example:
@partial
class User(pd.BaseModel):
id: int
# `id` can be skipped, but cannot be `None`
User()
User(id=1)
Example:
@partial('id')
class User(pd.BaseModel):
id: int
login: str
# `id` can be skipped, but not `login`
User(login='johnwick')
User(login='johnwick', id=1)
"""
# Call pattern: @partial class Model(pd.BaseModel):
if len(fields) == 1 and lenient_issubclass(fields[0], pd.BaseModel):
Model = fields[0]
field_names = ()
# Call pattern: @partial('field_name') class Model(pd.BaseModel):
else:
Model = None
field_names = fields
# Decorator
def decorator(Model: type[pd.BaseModel] = Model, field_names: frozenset[str] = frozenset(field_names)):
# Iter fields, set `required=False`
for field in Model.__fields__.values():
# All fields, or specific named fields
if not field_names or field.name in field_names:
field.required = False
# Exclude unset
# Otherwise non-nullable fields would have `{'field': None}` which is unacceptable
dict_orig = Model.dict
def dict_excludes_unset(*args, exclude_unset: bool = None, **kwargs):
exclude_unset = True
return dict_orig(*args, **kwargs, exclude_unset=exclude_unset)
Model.dict = dict_excludes_unset
# Done
return Modeland a unit-test, if you wonder how it works: import pydantic as pd
import pytest
from typing import Optional
def test_partial():
# === Test: @partial() all
@partial
class UserPartial(pd.BaseModel):
# Skippable, not nullable
id: int
# Skippable, nullable
login: Optional[str]
# Test: no arguments
user = UserPartial()
assert user.dict() == {}
assert user.dict(exclude_unset=False) == {} # cannot override
# Test: skippable argument provided
user = UserPartial(id=1)
assert user.dict() == {'id': 1}
# Test: optional argument provided
user = UserPartial(login='qwerty')
assert user.dict() == {'login': 'qwerty'} # 'id' null skipped
# Test: fails on None
with pytest.raises(pd.ValidationError):
UserPartial(id=None)
# === Test: @partial() names
@partial('id')
class UserPartial(pd.BaseModel):
# Skippable, not nullable
id: int
# Skippable, nullable
login: Optional[str]
# Test: no arguments
user = UserPartial()
assert user.dict() == {}
# Test: skippable argument provided
user = UserPartial(id=1)
assert user.dict() == {'id': 1}
# Test: optional argument provided
user = UserPartial(login='qwerty')
assert user.dict() == {'login': 'qwerty'} |
|
Building on the work on @ar45, @sborovic, and @kolypto, I've a solution that enables "Partial" models, including recursive model fields, but does so in a threadsafe way without modifying other model classes unnecessarily (@sborovic's recursive optionalizing modifies other Model classes globally, which was undesirable for me). Changelog:
Here is the metaclass that Partial models should use: class PartialModelMetaclass(ModelMetaclass):
def __new__(
meta: Type["PartialModelMetaclass"], *args: Any, **kwargs: Any
) -> "PartialModelMetaclass":
cls = super(PartialModelMetaclass, meta).__new__(meta, *args, *kwargs)
cls_init = cls.__init__
# Because the class will be modified temporarily, need to lock __init__
init_lock = threading.Lock()
# To preserve identical hashes of temporary nested partial models,
# only one instance of each temporary partial class can exist
temporary_partial_classes: Dict[str, ModelMetaclass] = {}
def __init__(self: BaseModel, *args: Any, **kwargs: Any) -> None:
with init_lock:
fields = self.__class__.__fields__
fields_map: Dict[ModelField, Tuple[Any, bool]] = {}
def optionalize(
fields: Dict[str, ModelField], *, restore: bool = False
) -> None:
for _, field in fields.items():
if not restore:
assert not isinstance(field.required, UndefinedType)
fields_map[field] = (field.type_, field.required)
field.required = False
if (
inspect.isclass(field.type_)
and issubclass(field.type_, BaseModel)
and not field.type_.__name__.startswith(
"TemporaryPartial"
)
):
# Assign a temporary type to optionalize to avoid
# modifying *other* classes
class_name = f"TemporaryPartial{field.type_.__name__}"
if class_name in temporary_partial_classes:
field.type_ = temporary_partial_classes[class_name]
else:
field.type_ = ModelMetaclass(
class_name,
(field.type_,),
{},
)
temporary_partial_classes[class_name] = field.type_
field.populate_validators()
if field.sub_fields is not None:
for sub_field in field.sub_fields:
sub_field.type_ = field.type_
sub_field.populate_validators()
optionalize(field.type_.__fields__)
else:
# No need to recursively de-optionalize once original types
# are restored
field.type_, field.required = fields_map[field]
if field.sub_fields is not None:
for sub_field in field.sub_fields:
sub_field.type_ = field.type_
# Make fields and fields of nested model types optional
optionalize(fields)
# Transform kwargs that are PartialModels to their dict() forms. This
# will exclude `None` (see below) from the dictionary used to construct
# the temporarily-partial model field, avoiding ValidationErrors of
# type type_error.none.not_allowed.
for kwarg, value in kwargs.items():
if value.__class__.__class__ is PartialModelMetaclass:
kwargs[kwarg] = value.dict()
elif isinstance(value, (tuple, list)):
kwargs[kwarg] = value.__class__(
v.dict()
if v.__class__.__class__ is PartialModelMetaclass
else v
for v in value
)
# Validation is performed in __init__, for which all fields are now optional
cls_init(self, *args, **kwargs)
# Restore requiredness
optionalize(fields, restore=True)
setattr(cls, "__init__", __init__)
# Exclude unset (`None`) from dict(), which isn't allowed in the schema
# but will be the default for non-required fields. This enables
# PartialModel(**PartialModel().dict()) to work correctly.
cls_dict = cls.dict
def dict_exclude_unset(
self: BaseModel, *args: Any, exclude_unset: bool = None, **kwargs: Any
) -> Dict[str, Any]:
return cls_dict(self, *args, **kwargs, exclude_unset=True)
cls.dict = dict_exclude_unset
return clsThe intended usage of the metaclass is to define subclasses of your "real" models (where requiredness is desired) that can be partial. For me, this makes sense when composing, say, configurations from multiple sources, where each source defines incomplete configuration, but when combined/merged, they should validate successfully. An example: class InnerConfig(BaseModel):
foo: int
bar: int
class Config(BaseModel):
name: str
inner: InnerConfig
do_something: bool
class PartialConfig(Config, metaclass=PartialModelMetaclass):
pass
Config(**{}) # will raise ValidationError
PartialConfig(**{}) # OK
InnerConfig(**{}) # will raise ValidationError!
# This is important, because it shows that simply declaring
# a partial model does not cause other models to become
# partial themselves!
# But, when constructing partial models, nested model fields *are* partial
config_src_1 = PartialConfig(**{ "name": "Example", "inner": { "foo": 5 } })
config_src_2 = PartialConfig(**{ "inner": { "bar": 2 }, "do_something": False })
# Composing partial models to validate a complete model
# You will need a deep dictionary merging function:
# e.g. https://stackoverflow.com/questions/7204805/how-to-merge-dictionaries-of-dictionaries
config = Config(**merge(config_src_1.dict(), config_src_2.dict())) # OKI also patched the mypy plugin so that it knows about the metaclass and won't complain about required keyword arguments for models created with the Expand for mypy plugin detailsSince we can't patch the plugin itself without depending on a fork of pydantic, I created a new file in my project that is itself a plugin, extended from pydantic's plugin. IMPORTANT: You will have to modify the full name of the metaclass in this code! # An extension to pydantic's mypy plugin to support the `PartialModelMetaclass`
# introduced to support "partial" models to be merged into a full model at some future time
from typing import List, Type
from mypy.nodes import Argument, RefExpr
from mypy.plugin import ClassDefContext, Plugin
from pydantic.mypy import PydanticModelField, PydanticModelTransformer, PydanticPlugin
class PartialPydanticModelTransformer(PydanticModelTransformer):
def get_field_arguments(
self,
fields: List[PydanticModelField],
typed: bool,
force_all_optional: bool,
use_alias: bool,
) -> List[Argument]:
# Register all fields as optional if the model is declared with a metaclass of PartialModelMetaclass
if self._ctx.cls.metaclass is not None:
assert isinstance(self._ctx.cls.metaclass, RefExpr)
if self._ctx.cls.metaclass.fullname == "FULL.NAME.OF.PartialModelMetaclass":
return super().get_field_arguments(fields, typed, True, use_alias)
return super().get_field_arguments(fields, typed, force_all_optional, use_alias)
class PartialPydanticPlugin(PydanticPlugin):
def _pydantic_model_class_maker_callback(self, ctx: ClassDefContext) -> None:
transformer = PartialPydanticModelTransformer(ctx, self.plugin_config)
transformer.transform()
def plugin(version: str) -> Type[Plugin]:
return PartialPydanticPluginThen, modify your mypy config to use this plugin! |
|
There's a PR for this here: |
I have a frequent case when the field is optional, but you don't want None. I would like to be able to set the default value if None is sent to this field. |
|
In Pydantic v2, the approach provided by @ar45 might not be applicable due to changes in the newer version. However, the following solution worked for me: You can use the provided function as a decorator to make the fields of a child model optional. from typing import Optional
from pydantic import create_model
def optional(*fields):
def dec(cls):
fields_dict = {}
for field in fields:
field_info = cls.__annotations__.get(field)
if field_info is not None:
fields_dict[field] = (Optional[field_info], None)
OptionalModel = create_model(cls.__name__, **fields_dict)
OptionalModel.__module__ = cls.__module__
return OptionalModel
if fields and inspect.isclass(fields[0]) and issubclass(fields[0], BaseModel):
cls = fields[0]
fields = cls.__annotations__
return dec(cls)
return dec |
|
@mubtasimfuad |
Some combination of pydantic and fastapi (fastapi, I'm pretty sure, due to things like pydantic/pydantic#1223 ) conflate "null" and "not present" in weird ways. This means that in the health response pydantic class, - If you create robot_serial in the proper way to have a pydantic field that can be either a string or null but must always be passed in to the constructor - annotation Optional[str] and do not provide a default - then pydantic will say there's a field missing if you explicitly pass in null (the bug this fixes, since this 500s the route) - If you create robot_serial with a default value, which is _not correct_ because there is no default, then something (probably fastapi) will remove the _key_ from the model rather than issueing it with a null, which is terrible, but is better than the other way I guess. Updating fastapi might help with this maybe. I don't know.
Some combination of pydantic and fastapi (fastapi, I'm pretty sure, due to things like pydantic/pydantic#1223 ) conflate "null" and "not present" in weird ways. This means that in the health response pydantic class, - If you create robot_serial in the proper way to have a pydantic field that can be either a string or null but must always be passed in to the constructor - annotation Optional[str] and do not provide a default - then pydantic will say there's a field missing if you explicitly pass in null (the bug this fixes, since this 500s the route) - If you create robot_serial with a default value, which is _not correct_ because there is no default, then something (probably fastapi) will remove the _key_ from the model rather than issueing it with a null, which is terrible, but is better than the other way I guess. Updating fastapi might help with this maybe. I don't know.
|
My solution for partial update. from pydantic import BaseModel
class _UNSET:
def __bool__(self):
return False
def __repr__(self):
return "UNSET"
UNSET = _UNSET()
class UpdatePayload(BaseModel):
name: str = UNSET
description: str
F438
= UNSET>>> UpdatePayload(**{}).json(exclude_unset=True)
'{}'
>>> UpdatePayload(**{"name": "a"}).json(exclude_unset=True)
'{"name":"a"}'
>>> UpdatePayload(**{"name": "a"}).description
UNSET
>>> if not UpdatePayload(**{"name": "a"}).description:
... print("Empty")
Empty
>>> UpdatePayload(**{"name": None}).json(exclude_unset=True)
Will raise ValidationError |
|
@lyzohub but then mypy would complain about assigning wrong type as default |
|
@kamilglod Cast to the |
I just ran into a use case that (I think) is not well covered by your reasoning: GeoJSON specification defines a "bbox" field in its objects that is optional. It is allowed to be missing, but if present, it has to be a list of float values of length 4 (at least). Although the validation can be achieved with a validator function, it is not practical and allowing for optional values that cannot be null (None) if present would be nicer imho. |
How would you do that in Pydantic v2? @ar45 |
The JSON:API spec also does something similar. |
|
My pydantic v2 version import inspect
import typing
from typing import Optional, Annotated, Any
from typing import overload, Callable, Type
from pydantic import BaseModel
from pydantic import Field
from pydantic import create_model, BeforeValidator
from pydantic.fields import FieldInfo
from pydantic.json_schema import SkipJsonSchema
from pydantic_core import PydanticUndefined
T = typing.TypeVar('T')
@overload
def optional(*fields: str) -> Callable[[T], T]:
...
@overload
def optional(func: T) -> T:
...
def optional(*fields):
def dec(_cls):
fields_dict = {}
for field in fields:
field_info = _cls.model_fields.get(field)
if field_info is not None:
# if field_info.get_default() == PydanticUndefined:
# field_info.default = None
if field_info.is_required():
new_field_info = FieldInfo.merge_field_infos(field_info, FieldInfo.from_field(None))
fields_dict[field] = (Optional[field_info.annotation], new_field_info)
# create a new model only if any of the fields were notified
if fields_dict:
OptionalModel = create_model(_cls.__name__, __base__=_cls, **fields_dict) # noqa N806
OptionalModel.__module__ = _cls.__module__
return OptionalModel
return _cls
if fields and inspect.isclass(fields[0]) and issubclass(fields[0], BaseModel):
cls = fields[0]
fields = list(cls.model_fields.keys())
return dec(cls)
if not fields:
return optional
return dec
def exclude(*fields):
def dec(_cls: Type[BaseModel]):
fields_dict = {}
for field in fields:
field_info = _cls.model_fields.get(field)
if field_info is not None:
# if field_info.get_default() == PydanticUndefined:
# field_info.default = None
field_info.exclude = True
field_info.default = None
fields_dict[field] = (SkipJsonSchema[Annotated[Any, BeforeValidator(lambda x: PydanticUndefined)]], Field(None, exclude=True, repr=False, validation_alias='____________a'))
OptionalModel = create_model(_cls.__name__, __base__=_cls, **fields_dict) # noqa N806
OptionalModel.__module__ = _cls.__module__
return OptionalModel
if not fields:
return exclude
return dec |
|
@ar45 Could there be an error in your code? I would have expected an exception thrown in this code or at least that the None value is not included in the model_dump from pydantic import BaseModel, Field
from test.lib import optional
@optional
class Test(BaseModel):
name: str = Field(...)
description: str = Field(...)
a = Test(name=None, description='description')
print(a.model_dump(exclude_unset=True))
>>> {'name': None, 'description': 'description'} |
How to have an “optional” field but if present required to conform to non None value?
How can I have an optional field where None is not allowed? Meaning a field may be missing but if it is present it should not be None.
I cross posted SO:
https://stackoverflow.com/questions/60191270/python-pydantic-how-to-have-an-optional-field-but-if-present-required-to-con
The text was updated successfully, but these errors were encountered: