PEP 749: Add section on metaclasses #3847
Conversation
Co-authored-by: Carl Meyer <carl@oddbird.net>
| and replaces itself with the result. The descriptor also behaves like a mapping, | ||
| so that code that uses ``cls.__dict__["__annotations__"]`` will still usually | ||
| work: treating the object as a mapping will evaluate the annotations and behave | ||
| as if the descriptor itself was the annotations dictionary. (Code that assumes |
There was a problem hiding this comment.
Will the implicit mapping access also update the cls.__dict__ entry? (thanks to the owner information passed to __set_name__, it could).
If it doesn't, the specification should be explicit that the underlying dict will be cached on the descriptor instance, so it can be added to cls.__dict__ later and any changes made via the descriptor's mapping API will still be visible.
There was a problem hiding this comment.
I have a pure-Python proof of concept for the idea here: https://gist.github.com/AlexWaygood/29e386e092377fb2e288620df1765ed5
The PoC does not currently update the __dict__ entry -- it just caches the materialized annotations internally in the descriptor instance -- but I think you're right that it could, potentially.
There was a problem hiding this comment.
Adding a sentence that makes my proposal more explicit.
peps/pep-0749.rst
Outdated
| retrieve annotations. | ||
|
|
||
| Alex Waygood has suggested an approach that avoids these problems. On class | ||
| creation, ``cls.__dict__["__annotations__"]`` is set to a special descriptor. |
There was a problem hiding this comment.
Do type.__annotate__ and type.__annotations__ still exist in this approach? (I don't believe they're needed, but best to be explicit)
There was a problem hiding this comment.
I agree that I don't think they'd be needed with this approach
There was a problem hiding this comment.
I believe they may be needed for non-heap types. I will experiment with this when I implement the idea.
peps/pep-0749.rst
Outdated
| class object returns ``None`` if the class has no annotations, and an annotation | ||
| function that returns the class's annotations if it does have annotations. | ||
|
|
||
| Classes always contain ``__annotations__`` and ``__annotate__`` keys in their |
There was a problem hiding this comment.
Does "classes" here mean all type instances? Or specifically heap types created with a class statement? What about the equivalent Python and C APIs?
I suspect only types created via a class statement getting these descriptors implicitly will make the most sense, with static types and types created via the dynamic Python and C APIs getting __annotate__ = None and __annotations__ = {} if nothing else is specified in the supplied class dicts.
There was a problem hiding this comment.
All classes that have >=1 class with annotations in their MRO must have "__annotations__" and "__annotate__" keys in their class dictionaries under this proposal. Theoretically I think it would be fine to omit these keys from the class dictionary if you could reliably determine that no classes in the MRO have any annotations. I don't know if that's worth the complexity, though.
There was a problem hiding this comment.
All classes that have >=1 class with annotations in their MRO
Or in the MRO of a metaclass.
Does "classes" here mean all type instances? Or specifically heap types created with a class statement? What about the equivalent Python and C APIs?
I think you're right and this should apply to classes created through the class statement, and classes created in ways that may inherit from Python classes should simply set the fields to None and {}.
There was a problem hiding this comment.
All classes that have >=1 class with annotations in their MRO
Or in the MRO of a metaclass.
right, yeah
peps/pep-0749.rst
Outdated
| class object returns ``None`` if the class has no annotations, and an annotation | ||
| function that returns the class's annotations if it does have annotations. | ||
|
|
||
| Classes always contain ``__annotations__`` and ``__annotate__`` keys in their |
There was a problem hiding this comment.
All classes that have >=1 class with annotations in their MRO must have "__annotations__" and "__annotate__" keys in their class dictionaries under this proposal. Theoretically I think it would be fine to omit these keys from the class dictionary if you could reliably determine that no classes in the MRO have any annotations. I don't know if that's worth the complexity, though.
|
After sleeping on this and looking at the proof of concept in pure Python, I'm a bit concerned about the performance implications. Classes are already large, but adding a new object instance with several fields to every class that exists could still have serious memory consequences for large Python codebases. I think we need to consider whether the bugs that are being fixed here are serious enough in practice to justify this overhead, or whether there are simpler solutions that don't fix everything perfectly but address the biggest problems, and don't have this overhead. |
|
@carljm it would be a small proportion of the overall size of a class object. In the prototype I'm currently working on, the size of the descriptor object is just 48 bytes, only 3% of the size of the class object: |
|
python/cpython#120719 has a very partial prototype now (I haven't gotten around to testing it yet or integrating descriptor behavior). |
|
What if we never set Or alternatively, if we used the "cache annotations under a different name than |
|
The concern with the "cache under a different name strategy" is that it isn't fully backwards compatible: retrieving the annotations directly from the class dict wouldn't work anymore. |
That's also true for the original text of PEP 649. It says:
But that's not really true under PEP 649 as written: This makes me think we should either use a strategy where It also occurred to me that we may want to avoid using the class dict as storage for
We could do something like this, but I'd rather have all classes work similarly, instead of special-casing some categories of classes. |
|
In the proof-of-concept gist currently, @carljm is correct that every class -- even if it has no annotations -- gets a unique descriptor instance in its class dictionary: class Object:
"""Pretend this is how builtins.object would work"""
__annotate__ = None
annotations = {}
def __init_subclass__(cls):
cls.annotations = _AnnotationsDescriptor(cls)
if "__annotate__" not in cls.__dict__:
cls.__annotate__ = NoneBut this does indeed seem unnecessary for classes that have no annotations. Instead, we could have a single descriptor instance in the class dictionary for class Object:
"""Pretend this is how builtins.object would work"""
__annotate__ = None
def __init_subclass__(cls):
if "__annotate__" in cls.__dict__:
cls.annotations = _AnnotationsDescriptor(cls)
else:
cls.__annotate__ = None
cls.annotations = Object.__dict__["annotations"]
Object.annotations = _AnnotationsDescriptor(Object)This should result in dramatically fewer descriptor instances being produced at class creation time. We would still create a fresh descriptor instance for every class that does have annotations and insert that instance into the class dictionary; but this is not so different to the way that an |
|
@AlexWaygood that becomes problematic if the
|
This is a really good point. That optimization was already implemented in the prototype implementation of PEP 649 that was available at the time of its acceptance, and although the optimization wasn't specified in the text of PEP 649, it was discussed publicly in advocating for the PEP prior to its acceptance, and I think it may turn out to be a valuable optimization. So it would be great if we can avoid closing the door to it. |
|
Coming back to this after a month, I'm now leaning towards the approach that the current version of this PR rejects:
This approach has the disadvantage that odd things may still happen if you access The details would be that we'd use this trick in This also implies we should add an |
Co-authored-by: Carl Meyer <carl@oddbird.net>
* Include three comments from @Carreau (drop two bullets, *args in example, explain *args.) * Lambda-wrapped expressions use annotation scope * Clarify use of annotation scope * Mention what happens to named unicodes followed by text * Use DecodedConcrete in assertion * Rewrite why annotation scope is needed (#4) * Rewrite why annotation scope is needed * Minor copyediting * PEP 747: Fix rules related to UnionType (T1 | T2). Contrast TypeExpr with TypeAlias. Apply other feedback. (python#3856) * PEP 694: Fix typo (python#3859) * PEP 2026: Update following discussion (python#3860) Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> * PEP 101: Remove outdated info and add new info (python#3863) * PEP 101: Remove outdated info * PEP 101: Update make command for running tests * PEP 101: Replace '#python-dev and/or python-committers' with 'Discord and/or Discourse * PEP 101: Add Hugo as 3.14 RM * PEP 101: Add to PSRT * PEP 11: Add Russell as an iOS contact (python#3865) * Meta: Document the PEPs API (python#3864) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> * PEP 719: Update for today's release of 3.13.0b4 (python#3868) * PEP 740: Mark as Provisional (python#3848) Signed-off-by: William Woodruff <william@yossarian.net> * PEP 749: Add section on metaclasses (python#3847) Co-authored-by: Carl Meyer <carl@oddbird.net> * PEP 8: Update a Wikipedia link (python#3552) * PEP 635: Minor typo fix in code sample (python#3871) Looks like an unclosed f-string. * PEP 751: A file format to list Python dependencies for installation reproducibility (python#3870) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Carol Willing <carolcode@willingconsulting.com> * PEP 743: Rewrite to hide (soft-)deprecated API (pythonGH-3869) Co-authored-by: Victor Stinner <vstinner@python.org> * PEP 751: Add Discussions-To and Post-History (python#3872) * PEP 639: Incorporate the latest discussion feedback (python#3866) * Remove the requirement of license-files defaults * Cover all rejected subkeysideas in one paragraph * Change the deprecation policy around classifiers * Flatten the value of the license-files key, only globs are specified * Update the Rejected ideas to match the current license-files proposal --------- Co-authored-by: Miro Hrončok <miro@hroncok.cz> * PEP 715: clarify what `[package.tool]` is (python#3873) * PEP 665: Superseded-By: 751 (python#3875) * PEP 751: update based on feedback (python#3877) * PEP 751: update based on feedback * Fix a section underline * Include three comments from @Carreau (drop two bullets, *args in example, explain *args.) * From Carol, move the point about import to the following paragraph. * Per Carol: Remove paragraph about lifecycles as that is about *a* DSL, not DSLs in general. --------- Signed-off-by: William Woodruff <william@yossarian.net> Co-authored-by: pauleveritt <paul.everitt@jetbrains.com> Co-authored-by: Jim Baker <jim.baker@python.org> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: David Foster <david@dafoster.net> Co-authored-by: Barry Warsaw <barry@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: T. Wouters <thomas@python.org> Co-authored-by: William Woodruff <william@yossarian.net> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Lavrentiy Rubtsov <rnbsov@gmail.com> Co-authored-by: Mariatta <Mariatta@users.noreply.github.com> Co-authored-by: Brett Cannon <brett@python.org> Co-authored-by: Carol Willing <carolcode@willingconsulting.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Karolina Surma <33810531+befeleme@users.noreply.github.com> Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Deriving from https://discuss.python.org/t/pep-749-implementing-pep-649/54974/28 and offline discussion with Alex.
📚 Documentation preview 📚: https://pep-previews--3847.org.readthedocs.build/