8000 [DataPipe] Basic snapshotting with IterableWrapper by NivekT · Pull Request #70373 · pytorch/pytorch · GitHub
[go: up one dir, main page]

Skip to content

[DataPipe] Basic snapshotting with IterableWrapper #70373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 49 commits into from
Closed
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
daa86d7
[DataPipe] Basic snapshotting with IterableWrapper
NivekT Dec 23, 2021
29e6eeb
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Dec 29, 2021
55787b4
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Dec 29, 2021
e83b8fd
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Dec 30, 2021
1abbc8e
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jan 18, 2022
a155137
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jan 19, 2022
488cd16
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Apr 15, 2022
73c3fcd
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Apr 18, 2022
e6f9065
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Apr 21, 2022
f48ab43
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Apr 28, 2022
dd5c3de
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 3, 2022
8694365
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 3, 2022
cc110c6
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 5, 2022
6adfc75
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 5, 2022
8db0723
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 6, 2022
018d524
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 6, 2022
0959b9e
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 9, 2022
5bc7f65
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 12, 2022
6a03285
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
8000 NivekT May 13, 2022
1c4b5a0
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 16, 2022
53bffc9
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 17, 2022
b774c42
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 17, 2022
fe12e57
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 18, 2022
f2a3266
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 18, 2022
58fc834
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 19, 2022
141eb09
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 19, 2022
13214e3
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 19, 2022
52365f1
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT May 19, 2022
9ddbac7
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 13, 2022
aedfd1c
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 15, 2022
895bca6
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 15, 2022
8411ceb
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 16, 2022
da514b4
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 16, 2022
8eec787
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 17, 2022
b5d1abd
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 20, 2022
91c37c0
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 21, 2022
931f80a
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 24, 2022
1b68e79
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 24, 2022
428d149
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 27, 2022
a87853e
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 28, 2022
b5639d3
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 28, 2022
e8b4ef5
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 28, 2022
bca6b64
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 28, 2022
478635e
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 28, 2022
fe90144
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jun 29, 2022
9f32ccf
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jul 13, 2022
9ec1a65
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jul 18, 2022
2f5d188
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jul 19, 2022
8f9f541
Update on "[DataPipe] Basic snapshotting with IterableWrapper"
NivekT Jul 20, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions test/test_datapipe.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,21 @@ def reset_after_n_next_calls(datapipe: Union[IterDataPipe[T_co], MapDataPipe[T_c
return res_before_reset, list(datapipe)


def snapshot_test_helper(source_datapipe: IterDataPipe, n_elements_to_advance: int) -> IterDataPipe:
"""
Given an IterDataPipe and an integer `n`, advance the source_datapipe `n` times,
then get the state and snapshot from the DataPipe, recreates it by passing the
state and snapshot into the given new_datapipe and returns it.
"""
p = pickle.dumps(source_datapipe)
for _ in range(n_elements_to_advance):
next(source_datapipe)
snapshot = source_datapipe.save_snapshot()
new_datapipe = pickle.loads(p)
new_datapipe.restore_snapshot(snapshot)
return new_datapipe


def odd_or_even(x: int) -> int:
return x % 2

Expand Down Expand Up @@ -828,6 +843,13 @@ def test_iterable_wrapper_datapipe(self):
# __len__ Test: inherits length from sequence
self.assertEqual(len(input_ls), len(input_dp))

# Snapshot Test:
input_dp = dp.iter.IterableWrapper(input_ls)
n_elements_to_advance = 5
new_dp = snapshot_test_helper(input_dp, n_elements_to_advance)
for old_ele, new_ele in zip(input_dp, new_dp):
self.assertEqual(old_ele, new_ele)

def test_concat_iterdatapipe(self):
input_dp1 = dp.iter.IterableWrapper(range(10))
input_dp2 = dp.iter.IterableWrapper(range(5))
Expand Down
0