Switch to per-file locking. #10596
Merged
Switch to per-file locking. #10596
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5a47356 to
9034b0f
Compare
|
Interesting that it has a race condition, this was adapted from the conda source iirc.... |
tacaswell
approved these changes
Feb 25, 2018
|
there's even a conda license for it (in the license subdirectory :)) |
Replace the Locked contextmanager (which locks a directory, preventing
other (cooperative) processes to access it) by a private _lock_path
contextmanager, which locks a single file (or directory).
- The finer grained lock avoids locking out the entire tex cache when
handling usetex, which is useful when running multiple processes at
once.
- Python3 implements the `"x"` ("exclusive") mode to open, which we can
use instead of relying on `makedirs` to achieve a race-free operation
on the filesystem.
- The previous implementation allowed multiple threads of a single
process to acquire the same lock, but (for the use cases here, e.g.
running a tex subprocess) this is actually undesirable. Removing this
behavior also simplifies the implementation.
- As far as I can tell, the previous implementation was actually racy:
in
retries = 50
sleeptime = 0.1
while retries:
files = glob.glob(self.pattern)
if files and not files[0].endswith(self.end):
time.sleep(sleeptime)
retries -= 1
else:
break
else:
err_str = _lockstr.format(self.pattern)
raise self.TimeoutError(err_str)
# <----- HERE
if not files:
try:
os.makedirs(self.lock_path)
except OSError:
pass
else: # PID lock already here --- someone else will remove it.
self.remove = False
multiple processes can reach "HERE" at the same time and each
successfully create their own lock.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replace the Locked contextmanager (which locks a directory, preventing
other (cooperative) processes to access it) by a private _lock_path
contextmanager, which locks a single file (or directory).
The finer grained lock avoids locking out the entire tex cache when
handling usetex, which is useful when running multiple processes at
once.
Python3 implements the
"x"("exclusive") mode to open, which we canuse instead of relying on
makedirsto achieve a race-free operationon the filesystem.
The previous implementation allowed multiple threads of a single
process to acquire the same lock, but (for the use cases here, e.g.
running a tex subprocess) this is actually undesirable. Removing this
behavior also simplifies the implementation.
As far as I can tell, the previous implementation was actually racy:
in
multiple processes can reach "HERE" at the same time and each
successfully create their own lock.
This should mostly mitigate out #7776.
PR Summary
PR Checklist