-
Notifications
You must be signed in to change notification settings - Fork 58
Concurrency errors from multiple TUF Updaters #1403
New issue
Have a question abo 8000 ut this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Hi @spencerschrock, thanks for opening an issue!
Yeah, I think this is outside of our intended support: we use the python-tuf APIs directly, so we inherit whatever concurrency constraints they impose. I'll look at opening a docs PR to clarify this 🙂 |
I'm not sure if python-tuf can 100% guarantee safe concurrency but we could do a lot better (just a lock file per repository would handle most issues) it's just not been a priority. I'll file an issue in any case. @spencerschrock can you clarify what your design looks like: do you have multiple processes or multiple threads or just multiple signingcontexts in a single thread? |
Currently the issue was exposed in our test suite, where In terms of real usage, our project's CLI creates one signingcontext but nothing stops users from running multiple CLI processes. Similarly, I have no immediate insight into how people are using our library in parallel. |
We might actually have multiple processes try to sign various models in threads, using the same signing context. I have not run an experiment with that, though. But I imagine some model hub wanting to sign all past models in a pipeline, might perform signing in parallel. I'll run some experiments and be back |
I ran this experiment in two python interpreters:
Sooner or later one of them trows an exception, oftentimes one related to that existing file.
I don't think the error will happen very often in practice... but it should have some sort of serialization. Let's say we wanted to put a lock-file into Full trace:
It also can be reproduced with this:
|
Yeah other software touching things under
|
The first errors in the test suite actually occurred on Windows test machines in github actions...
Considering that I was able to reproduce the issue on the python-sigstore level here, which is part of the model signing code path, I would say that it should also be solved on this level. It would resolve it for all callers.
|
I believe @jku is saying that this should be solved in (I'm hesitant to have |
Uh oh!
There was an error while loading. Please reload this page.
Description
Hello, Sigstore's
model-signing
library usessigstore-python
, which in turn usespython-tuf
.We've had some intermittent test failures (sigstore/model-transparency#465) when we have multiple Sigstore signers, which launch multiple trusted roots and lead to some known concurrency errors. TUF's documentation says not to have multiple
Updater
sThe callstack is roughly this:
This scenario is in violation of TUF's concurrency support, but does/should Sigstore python support this scenario? If not, can you update your documentation to clarify this?
I've only seen it on our Windows runners, but haven't looked into it too much yet.
https://github.com/sigstore/model-transparency/actions/runs/15144309667/job/42576120558#step:4:100
https://github.com/sigstore/model-transparency/actions/runs/15165373394/job/42641686030#step:4:102
Version
sigstore 3.6.2
The text was updated successfully, but these errors were encountered: