8000 Improve mypy typing performance by cdce8p · Pull Request #387 · pydantic/pydantic-core · GitHub
[go: up one dir, main page]
Skip to content

Improve mypy typing performance #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

Improve mypy typing performance #387

wants to merge 3 commits into from

Conversation

cdce8p
Copy link
Contributor
@cdce8p cdce8p commented Feb 8, 2023

Ref: python/mypy#14627

Trying to run mypy with pydantic 2.0.0.dev.0 hangs when trying to lint

# test.py
from pydantic import BaseModel

The likely reason for that is the CoreSchema type. Mypy can't handle unions with a lot of items that well. Especially for TypedDicts, it tries to find a common base type but when there is none, checks the compatibility of each keys individually.

That's even more of an issue as there are also some recursive annotations with CoreSchema.

pydantic-core/pydantic_core/core_schema.py

Lines 2547 to 2588 in bfb373c

CoreSchema = Union[
AnySchema,
NoneSchema,
BoolSchema,
IntSchema,
FloatSchema,
StringSchema,
BytesSchema,
DateSchema,
TimeSchema,
DatetimeSchema,
TimedeltaSchema,
LiteralSchema,
IsInstanceSchema,
IsSubclassSchema,
CallableSchema,
ListSchema,
TuplePositionalSchema,
TupleVariableSchema,
SetSchema,
FrozenSetSchema,
GeneratorSchema,
DictSchema,
FunctionSchema,
FunctionWrapSchema,
FunctionPlainSchema,
WithDefaultSchema,
NullableSchema,
UnionSchema,
TaggedUnionSchema,
ChainSchema,
LaxOrStrictSchema,
TypedDictSchema,
ModelSchema,
ArgumentsSchema,
CallSchema,
RecursiveReferenceSchema,
CustomErrorSchema,
JsonSchema,
UrlSchema,
MultiHostUrlSchema,
]

pydantic-core/pydantic_core/core_schema.py

Lines 1724 to 1726 in bfb373c

class UnionSchema(TypedDict, total=False):
type: Required[Literal['union']]
choices: Required[List[CoreSchema]]

To improve mypy performance, I added an empty CoreSchemaBase TypedDict type from which all CoreSchema types can inherit and replaced the recursive CoreSchema types. With that a mypy run is back down to 15-20s.

time mypy --no-incremental test.py

@cdce8p cdce8p mentioned this pull request Feb 8, 2023
Closed
@cdce8p
Copy link
Contributor Author
cdce8p commented Feb 8, 2023

CI issues seems to be related to cargo. The branch probably needs to be changed to just release-0.18.

pydantic-core/Cargo.toml

Line 30 in bfb373c

pyo3 = { git = "https://github.com/PyO3/pyo3", branch = "release-0.18.1" }

pydantic-core/Cargo.toml

Line 71 in bfb373c

pyo3-build-config = { git = "https://github.com/PyO3/pyo3", branch = "release-0.18.1" }

@samuelcolvin
Copy link
Member

See also python/mypy#14034.

V0.18.1 of pyo3 was released late last night, so we need to.update cargo.

But the bigger question is: does this worth generate_core_schema.py - remember CoreSchema is consumed and sure to build a validator.

@codecov-commenter
Copy link
codecov-commenter commented Feb 8, 2023 •
edited
Loading

Codecov Report

Merging #387 (54f4a48) into main (bfb373c) will increase coverage by 0.00%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #387   +/-   ##
=======================================
  Coverage   95.31%   95.31%           
=======================================
  Files          90       90           
  Lines       10510    10512    +2     
  Branches        8        8           
=======================================
+ Hits        10018    10020    +2     
  Misses        492      492
Impacted Files Coverage Δ
pydantic_core/core_schema.py 98.78% <100.00%> (+<0.01%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfb373c...54f4a48. Read the comment docs.

@codspeed-hq
Copy link
codspeed-hq bot commented Feb 8, 2023

CodSpeed Performance Report

Merging #387 improve-typing-performance (54f4a48) will not alter performances.

Summary

🔥 0 improvements
❌ 0 regressions
✅ 85 untouched benchmarks

🆕 0 new benchmarks
⁉️ 0 dropped benchmarks

@cdce8p
Copy link
Contributor Author
cdce8p commented Feb 8, 2023

Seems like this won't type check, unfortunately 😕
It was worth a shot.

@cdce8p cdce8p closed this Feb 8, 2023
@cdce8p cdce8p deleted the improve-typing-performance branch February 8, 2023 12:51
@samuelcolvin samuelcolvin mentioned this pull request Feb 9, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cdce8p @samuelcolvin @codecov-commenter
0