Change of API in v2 to declare required and optional fields #2295

PrettyWood · 2021-01-27T22:27:12Z

Hi everyone!

I'm playing a lot with the new builtin generic types in 3.9 and type union operator introduced in 3.10 (I even wrote a backport for 3.6+ since I love them so much 😆).
Playing with them and pydantic, I really feel like the API can be challenged for v2. The idea is:
All fields without a default value would be required and all fields with a default value would be optional.
For example

class Model(BaseModel):
    a: str  # required field
    b: str | None  # required nullable field
    c: str = ...  # optional field with no default value
    d: str = 'c'  # optional field with a default value
    e: str | None = ...  # optional nullable field with no default value

It would imply that Optional[str] does not mean "optional" but more "nullable" (which is what it is 😄)
To define an optional field without actually setting a default value, I suggest we use Ellipsis. I like the fact that we don't need to import a Undefined sentinel to say "no value at all".
It would be a bit confusing when switching from v1 to v2 since ... is currently used to define a required field but IMHO it makes more sense to write a model this way with a homogeneous API.

For Field it would just mean Field(description='') is required when Field(..., description='') is optional.
For create_model it's the same

create_model(
  'Model',
  my_required_field=str,
  my_optional_field=(str, ...),
  my_required_field_2=(str, Field(description='')),
  my_optional_field_2=(str, Field(..., description=''),
)

@samuelcolvin @StephenBrown2 @MrMrRobat others WDYT?

Partially related issue #990
Related issue #1223

The text was updated successfully, but these errors were encountered:

StephenBrown2 · 2021-01-28T16:00:10Z

I was initially thrown off by the union operator, thinking "That's not valid!" but, after looking at it further, I must say, it is highly intriguing. It does look like it would solve the "What is Optional" issue that many have had. Would we even need/use Optional? Since it currently means Union[type, None], I presume it would line up with b: str | None # required nullable field. Hopefully not breaking many uses currently in the wild.

If we added future-typing as a dependency for python<3.10, I think it would indeed make a lot of things simpler.

layday · 2021-01-28T16:11:28Z

What would it mean for an optional field to not have a default value? Would the attribute be omitted from model instances and how would type checkers be made aware?

The ellipsis is also only valid as a placeholder in functions:

def foo(
    # this type checks
    bar: str = ...,
) -> None: ...

class Foo:
    # this doesn't: 'Incompatible types in assignment (expression has type "ellipsis", variable has type "str")'
    bar: str = ...

Bobronium · 2021-01-28T16:26:31Z

I see potential for encountering a lot of AttributeErrors here 😁

AFAIK, you can't tell mypy that specific attr might be undefined on some instances. So we'll be forced to check if hasattr(model, "foo").

What are the benefits of having such fields?

PrettyWood · 2021-01-28T16:34:02Z

The ellipsis is also only valid as a placeholder in functions

Hmm that's already the case now when we force required fields with ... right?

AFAIK, you can't tell mypy that specific attr might be undefined on some instances

And indeed it would need some extra work on the mypy plugin to make it work but I feel like it should be doable ;)

My main focus is making models more explicit and the DX.
For me the main advantage is really differentiate required, nullable and optional fields where all the default values used in the code would be Undefined.

Bobronium · 2021-01-28T16:43:32Z

Hmm that's already the case now when we force required fields with ... right?

How's that?

So by saying «optional field with no default value», you mean that if field was not in data, attr will still be present on model and value would be Undefined instead of None?

PrettyWood · 2021-01-28T16:50:22Z

Sorry @MrMrRobat maybe I was not clear.
I just meant we already use ... in BaseModel so the "it's not understood by mypy" is already an issue now and it should be possible to tackle it by the plugin.

And yes I just mean the default value used behind the scene is always Undefined. And dict(), json(), ... methods would of course remove them. It would also make schema generation easier and closer to the model where you could have not required fields without a default value.

layday · 2021-01-28T17:16:04Z

Hmm that's already the case now when we force required fields with ... right?

'Forcing' required fields with the ellipsis is redundant which is why it hasn't been an issue so far, I guess.

And indeed it would need some extra work on the mypy plugin to make it work but I feel like it should be doable ;)

Well, there are other type checkers out there, some of which do not support plug-ins. Pyright in particular has been gaining in popularity owing to its integration in VS Code.

PrettyWood · 2021-01-28T17:25:08Z

Well, there are other type checkers out there, some of which do not support plug-ins

You're absolutely right. I only use mypy and tend to forget others may be a problem too 😅
Maybe the ... is not the best idea and we should go with something more robust like

from pydantic import BaseModel, Unset

class Model(BaseModel):
    a: int | None
    b: int | None | Unset  # or `Undefined` or something else...
    c: int | None | Unset

with class Unset: ... and a default value for every single field equal to Unset().
At least it would remove the type checking issues, be explicit and still quite concise

layday · 2021-01-29T16:14:45Z

That seems reasonable from a typing standpoint, but I'm not sure I fully understand the use case, if you could explain how this would be useful.

PrettyWood · 2021-01-29T16:37:34Z

Sorry if I'm not clear :(
Well for v2 there are already two forthcoming changes:

switch to "required optional fields" in Required Optional fields #990, which imo makes more sense since Optional[x] only means Union[x, None] (which is even more obvious when you use the 3.10 syntax)
support nullable in JSON schemas in JSON Schema Nullable Required Syntax? #1270

My main point is to be able to declare a field that may be omitted, but may not be null if the field is supplied
(which has already been discussed in #1761 for example). So we need a way to say "this field is not required but doesn't have a default value" hence the Unset or whatever else proposal

layday · 2021-01-29T18:55:17Z

I see, so this would be for fields that (a) can be omitted but (b) should also not have a default value in a JSON schema. The discussion of Omittable on typing-sig would be of some interest here - if the Omittable semantics could be extended to native classes in addition to TypedDict.

As an aside, if #990 would mean reverting to normal typing semantics and = None is required to make a typing.Optional field optional, that would resolve #1270 without having to add anything extra.

mpkocher · 2021-01-29T19:06:01Z

from pydantic import BaseModel, Unset

class Model(BaseModel):
    a: int | None
    b: int | None | Unset  # or `Undefined` or something else...
    c: int | None | Unset

I don't think this is a great idea to attempt to capture this at the type level by using Unset as a type. It might be better to capture the "unset" semantics as metadata using Python 3.9's Annotated feature.

https://www.python.org/dev/peps/pep-0593/

from typing import Annotated, TypeVar, Optional
from pydantic import BaseModel

class _Unset:
    pass

UNSET = _Unset()

class M1:
    a: Optional[int]
    b: Annotated[Optional[int], UNSET]

Or

T = TypeVar('T')
Unset = Annotated[T, UNSET]


class M2:
    a: Optional[int]
    b: Unset[Optional[int]]

Also, note that proper NoneType and EllipsisType types have been added to 3.10 and should be supported by mypy in a future release.

https://bugs.python.org/issue41810

python/typing#684 (comment)

layday · 2021-01-29T19:24:16Z

What would the value of M1(a=1).b be? If it is actually unset your type checker won't know. I assume the idea with Unset[] is that it would be a type-only descriptor and __get__ would return Optional[int] for instances of M2, meaning that it'd suffer from the same issue as M1.

PrettyWood · 2021-01-29T19:35:59Z

I see, so this would be for fields that (a) can be omitted but (b) should also not have a default value in a JSON schema. The discussion of Omittable on typing-sig would be of some interest here - if the Omittable semantics could be extended to native classes in addition to TypedDict.

Yes exactly! I also would like this for TypedDict (which I compare with TS interfaces). Thanks for sharing this thread!

mpkocher · 2021-02-05T05:21:08Z

Looks like the PEP the first version of the PEP has been posted the typing-sig mailing list.

https://mail.python.org/archives/list/typing-sig@python.org/thread/3NJZS7VCHSF54MD465SO7AF3AXGBGEDO/

It looks like the current version of the spec, typing.Required can only be used in the context of TypedDict.

It is an error to use Required[...] in any location that is not an item of a TypedDict:

PrettyWood · 2021-02-05T06:51:40Z

Thanks for the update! I hope this is something that we'll be able to leverage.
But I'm happy the community is putting some effort into making this easier to write and more explicit, which was the whole purpose of this issue.

PrettyWood added feedback wanted change Suggested alteration to Pydantic, not a new feature nor a bug labels Jan 27, 2021

PrettyWood closed this as completed Feb 12, 2021

pydantic locked and limited conversation to collaborators Feb 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

This issue was moved to a discussion.

Change of API in v2 to declare required and optional fields #2295

Change of API in v2 to declare required and optional fields #2295

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This issue was move 4A20 d to a discussion.

Uh oh!

This issue was moved to a discussion.

Change of API in v2 to declare required and optional fields #2295

Change of API in v2 to declare required and optional fields #2295

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This issue was move 4A20 d to a discussion.