Description
I'm seeing frequent 503 and 504 responses using the blob.upload_from_filename() method. An example trace looks like this:
503 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=multipart: {
"error": {
"code": 503,
"message": "Backend Error",
"errors": [
{
"message": "Backend Error",
"domain": "global",
"reason": "backendError"
}
]
}
}
: (u'Request failed with status code', 503, u'Expected one of', 200)' - Calling rollback() ...
Traceback (most recent call last):
File "/my/app/module.py", line 112, in upload
blob.upload_from_filename(file_path)
File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 2471, in upload_from_filename
checksum=checksum,
File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 2359, in upload_from_file
_raise_from_invalid_response(exc)
File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 3886, in _raise_from_invalid_response
raise exceptions.from_http_status(response.status_code, message, response=response)
ServiceUnavailable: 503 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=multipart: {
"error": {
"code": 503,
"message": "Backend Error",
"errors": [
{
"message": "Backend Error",
"domain": "global",
"reason": "backendError"
}
]
}
}
: (u'Request failed with status code', 503, u'Expected one of', 200)
It looks like this method calls blob.upload_from_file() with num_retries=None, so there is no retry strategy applied. I also see that the num_retries argument is documented as deprecated.
Does this library provide an alternative way to retry blob.upload_from_filename() or blob.upload_from_file()? I can wrap these calls in retry logic of my own, but I see that retries are enabled by default for other functions.
I'm using google-cloud-storage==1.34.0. I've already pulled in google-resumable-media==1.20.0 so I should be getting the recent resumable media retry fixes that came with #342.
Thanks for any tips on retrying uploads.