Description
🚀 The feature
We have serialization tests in PyTorch Core https://github.com/pytorch/pytorch/blob/b56ba296b1cb5d65a3fe2e33cc1d910481baa644/test/test_datapipe.py#L431
There are a few more things needed:
- Cover all DataPipe
-
- Dill test for DataPipes that accept function as input
- Cover DataPipe serialization after iteration starts
- Cover serialization for factory DataPipe (1-to-N) like
demuxandfork, especially when one of children is not depleted
Motivation, pitch
These tests are going to make sure DataLoader graph can be created. And, for forward compatibility, snapshotting also requires DataPipe serializable.
Alternatives
No response
Additional context
No response