torch.save writes 0s for tensors >2GB #717Description
t = torch.Tensor(600000000).fill_(1)
torch.save(t, 'foo.pt')
u = torch.load('foo.pt')
u
5F84
[-5:]
0
0
0
0
0
[torch.FloatTensor of size 5]
The problem is here:
https://github.com/pytorch/pytorch/blob/master/torch/csrc/generic/serialization.cpp#L39
You can't write an arbitrary number of bytes. See https://linux.die.net/man/2/write . On my system the limit seems to be 2GB, YMMV.
To be safe, you probably want to fix the read call as well at https://github.com/pytorch/pytorch/blob/master/torch/csrc/generic/serialization.cpp#L85 , because there's an SSIZE_MAX limit.