PERF: faster constructors from ea scalars #45854
Conversation
pandas/core/dtypes/cast.py
Outdated
| cls = dtype.construct_array_type() | ||
| subarr = cls._from_sequence([value] * length, dtype=dtype) | ||
| if isinstance(dtype, CategoricalDtype): | ||
| subarr = cls._from_sequence([value] * length, dtype=dtype) |
There was a problem hiding this comment.
what happens if you don't special case Categorical?
There was a problem hiding this comment.
Calling take on the empty categorical would error because the categories were not defined.
I just pushed another commit which first sets the categories. Categorical now has a similar improvement:
import pandas as pd
N = 1_000_000
%timeit pd.Series(1, index=range(N), dtype='category')
415 ms ± 91.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <- main
4.5 ms ± 1.57 ms per loop (mean ± std. dev. of 7 runs, 100 loops each) <- PR
There was a problem hiding this comment.
Calling take on the empty categorical would error because the categories were not defined.
Hmm i think this might be a problem for other not-fully-initialized dtype objects. IIRC IntervalDtype without 'closed' set is another one, not sure if we have a way to detect the general case
There was a problem hiding this comment.
latest update uses .repeat per @jreback's suggestion below.
pandas/core/dtypes/cast.py
Outdated
| cls = dtype.construct_array_type() | ||
| subarr = cls._from_sequence([value] * length, dtype=dtype) | ||
| subarr = cls._from_sequence([value], dtype=dtype) | ||
| taker = np.broadcast_to(np.intp(0), length) |
There was a problem hiding this comment.
can we just use .repeat()? (I think we define that generally on EA)
There was a problem hiding this comment.
Yes, that's even better - just updated. Thanks for pointing it out.
|
very nice @lukemanley keep em coming! |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.Perf improvement when constructing a DataFrame/Series from a scalar EA value.
Examples: