fix codecache write_atomic path issue on Windows. #138331
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138331
Note: Links to docs will display an error until the docs builds have been completed. ❌ 5 New Failures, 1 Unrelated FailureAs of commit 92ee32e with merge base aa3ae50 ( NEW FAILURES - The following jobs have failed:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
94b3999 to
6113e3e
Compare
6ff4335 to
ac8408a
Compare
ac8408a to
31470a5
Compare
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
1. handle FileExistsError on Windows. 2. Breakdown this code into equal Windows code.
|
Successfully rebased |
31470a5 to
76ff6f1
Compare
malfet
approved these changes
Oct 18, 2024
malfet
reviewed
Oct 18, 2024
|
I tested the new code both on Windows/Linux. And confirmed they are works. @malfet import os
import sys
from pathlib import Path
import shutil
_IS_WINDOWS = sys.platform == "win32"
def test_case():
cwd = os.getcwd()
path1 = os.path.join(cwd, "haha1.txt")
path2 = Path(os.path.join(cwd, "haha2.txt"))
try:
path2.rename(target=path1)
except FileExistsError as e_file_exist:
if not _IS_WINDOWS:
raise
# On Windows file exist is expected: https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename
# Below two lines code is equal to `tmp_path.rename(path)` on non-Windows OS.
# 1. Copy tmp_file to Target(Dst) file.
shutil.copy2(src=path2, dst=path1)
# 2. Delete tmp_file.
os.remove(path2)
print("run here.")
if __name__ == "__main__":
test_case() |
|
@pytorchmergebot merge -f "skip unrelated error." |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
|
2.5.1 is an emergency release to address large regression, so fixes to a new features do not meet the criteria to be included into it. |
8 tasks
xuhancn
added a commit
to xuhancn/pytorch
that referenced
this pull request
Oct 26, 2024
Fixes pytorch#138211 `Path.rename` function has Windows OS specific behavior, that will raise `FileExistsError` when the target file existing. This behavior is not happened on Linux, so I write a small repoduce code to figure out what happened. After stepping trace the repo code: ```python import os import sys from pathlib import Path _IS_WINDOWS = sys.platform == "win32" def test_case(): cwd = os.getcwd() path1 = os.path.join(cwd, "haha1.txt") path2 = Path(os.path.join(cwd, "haha2.txt")) try: path2.rename(path1) except FileExistsError as e_file_exist: if _IS_WINDOWS: # on Windows file exist is expected: https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename shutil.copy2(path2, path1) os.remove(path2) else: raise e_file_exist except BaseException as e: raise e print("run here.") if __name__ == "__main__": test_case() ``` We found the code `path2.rename(path1)` can breakdown into: 1. copy file2's content to file1. 2. delete file2. So, we can implemented equal code on Windows path: ```python shutil.copy2(src=tmp_path, dst=path) os.remove(tmp_path) ``` So, we can get current PR. TODO: need cherry-pick to release/2.5 branch, CC: @atalman . Pull Request resolved: pytorch#138331 Approved by: https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
73 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #138211
Path.renamefunction has Windows OS specific behavior, that will raiseFileExistsErrorwhen the target file existing.This behavior is not happened on Linux, so I write a small repoduce code to figure out what happened.
After stepping trace the repo code:
We found the code
path2.rename(path1)can breakdown into:So, we can implemented equal code on Windows path:
So, we can get current PR.
TODO: need cherry-pick to release/2.5 branch, CC: @atalman .
cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @iremyux @Blackhex @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov