Feature/local sparse #8619
Feature/local sparse #8619
Conversation
|
We can't use CGO in rclone - it complicates the multiplatform build too much. Is there another way of doing this? Perhaps we should just check for zero bytes in the writer and create sparse files then? I'm pretty sure that is what rsync does with That would be a much smaller patch. |
Yeah, but let's admit, it's a little less cool. That's probably a good thing though. I'll get to work on that |
08fe956 to
1673acc
Compare
|
@ncw 1673acc implements rsync-style sparse preservation. This thing depends on openChunkWriterfromOpenWriterAt for multithreaded copies and WriterAt in general. For single threaded copies, I've only gone in and implemented this for the local backend by changing the backend-specific update function. Is this alright? I think the usage of fs/operations in backend/local might create a circular dependency in the tests (again). IMHO, it would be the smoothest to force multithreaded transfers (default 1 thread) when --sparse is specified. Second, tests for sparseness may be hard to do as sparseness is very fs/os version dependent. How do we want to proceed given this? |
…lone into feature/local-sparse
What is the purpose of this change?
Ensure sparse blocks are maintained as well as possible in local to local transfers.
To do so, I use the FS_IOC_FIEMAP ioctl on linux to list out all of the non-sparse extents, which are then passed to the multi threaded transfer system, which has been slightly modified to accept non-constant size chunks.
https://forum.rclone.org/t/possible-to-use-rclone-with-sparse/45189
Checklist