Update run_language_modeling.py to handle writes on networked filesystem better #3356
Conversation
…ystem during distributed training better
|
For some reference, check out pytorch/pytorch#12042 and facebookresearch/maskrcnn-benchmark#40. These address the same issue. Also, one of the checks that failed, |
There was a problem hiding this comment.
This looks good to me, thanks for looking into it @Genius1237!
There's a current issue with the code quality test so don't worry about it.
We should probably do this modification to all scripts.
|
I did think about the other scripts. Are those already setup with Also, I have one or 2 more fixes along the lines of this one for distributed training. I was wondering if I should rename this PR and add those in, or create a new one for each of those fixes. One of them is about loading checkpoints (of the optimizer and scheduler) while resuming training. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Closing this as |
In the case of multi-node distributed training, reads and writes typically happen to a common networked filesystem.
In the current version of the
run_language_modeling.pyscript, processes that havelocal_rankas 0 perform the writes to disk (tensorboard, dataset cache and model checkpointing). In the case of multi-node distributed training, there ends up being one process per node havinglocal_rankas 0, hence multiple processes try writing to the filesystem at one point, resulting on errors being thrown depending on the filesystem.This pull request updates the script such that only the process having a
global_rankof 0 does the writing.global_rankisn't a variable directly accessible in the script, it is obtained by callingtorch.distributed.get_rank().I've tested the script in 4 different cases and they work without any error in these cases: multi-node training with DDP, single-node training with DDP, single-node training with DP and single gpu training.