8000 Optimize `ffill`, `bfill` with dask when `limit` is specified by josephnowak · Pull Request #9771 · pydata/xarray · GitHub
[go: up one dir, main page]
Skip to content

Optimize ffill, bfill with dask when limit is specified #9771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Nov 14, 2024
Merged

Optimize ffill, bfill with dask when limit is specified #9771

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f4fb252
Reduce the number of tasks when the limit parameter is set on the pus…
josephnowak Nov 12, 2024
40232dc
Reduce the number of tasks when the limit parameter is set on the pus…
josephnowak Nov 12, 2024
fde767b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2024
0d61d74
Update xarray/core/dask_array_ops.py
josephnowak Nov 12, 2024
b3d3307
Merge branch 'main' into feature/reduce-tasks-on-push-method-with-limit
josephnowak Nov 12, 2024
8d99dd0
Use last instead of creating a custom function, and add a keepdims pa…
josephnowak Nov 12, 2024
4fc115c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2024
297afdd
Remove the keepdims on the last and first method and use the nanlast …
josephnowak Nov 12, 2024
656c508
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2024
100369e
Include the optimization of ffill and bfill on the whats-new.rst
josephnowak Nov 13, 2024
b528a4b
Use map_overlap when the n is smaller than all the chunks
josephnowak Nov 14, 2024
6ae5e1f
Avoid creating a numpy array to check if all the chunks are bigger th…
josephnowak Nov 14, 2024
aad5ecb
Updating the whats-new.rst
josephnowak Nov 14, 2024
61d6574
Merge branch 'main' into feature/reduce-tasks-on-push-method-with-limit
dcherian Nov 14, 2024
c30ed51
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use map_overlap when the n is smaller than all the chunks
  • Loading branch information
@josephnowak
josephnowak committed Nov 14, 2024
commit b528a4b9d472021e4b11e5a0aef5e371cfc268d1
6 changes: 4 additions & 2 deletions xarray/core/dask_array_ops.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,11 @@ def push(array, n, axis, method="blelloch"):
from xarray.core.duck_array_ops import _push
from xarray.core.nputils import nanlast

if n is not None and np.all(n <= np.array(array.chunks[axis])):
return array.map_overlap(_push, depth={axis: (n, 0)}, n=n, axis=axis)

# TODO: Replace all this function
# once https://github.com/pydata/xarray/issues/9229 is implemented
# once https://github.com/pydata/xarray/issues/9229 being implemented

def _fill_with_last_one(a, b):
# cumreduction apply the push func over all the blocks first so,
Expand All @@ -90,7 +93,6 @@ def _dtype_push(a, axis, dtype=None):
)

if n is not None and 0 < n < array.shape[axis] - 1:

def _reset_cumsum(a, axis, dtype=None):
cumsum = np.cumsum(a, axis=axis)
reset_points = np.maximum.accumulate(np.where(a == 0, cumsum, 0), axis=axis)
Expand Down
0