8000 fix: avoid unnecessary call to jobs.get in case of retriable query failure by tswast · Pull Request #1797 · googleapis/python-bigquery · GitHub
[go: up one dir, main page]
Skip to content

fix: avoid unnecessary call to jobs.get in case of retriable query failure #1797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

fix: avoid unnecessary call to jobs.get in case of retriable query failure #1797

wants to merge 2 commits into from
+8 −2

Conversation

tswast
Copy link
Contributor
@tswast tswast commented Jan 24, 2024

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes issue identified here: https://github.com/googleapis/python-bigquery/pull/1794/files#r1465230014
🦕

@product-auto-label product-auto-label bot added size: s Pull request size is small. api: bigquery Issues related to the googleapis/python-bigquery API. labels Jan 24, 2024
@tswast tswast mentioned this pull request Jan 24, 2024
Merged
4 tasks
@kiraksi kiraksi mentioned this pull request Jan 24, 2024
Closed
@kiraksi kiraksi linked an issue Jan 24, 2024 that may be closed by this pull request
bug: client doesn't retry "Job exceeded rate limits" for DDL query jobs that exceed quota for table update operations #1790
Closed
@tswast
Copy link
Contributor Author
tswast commented Jan 26, 2024

I was just reminded on #1122 today. We need to be careful with this change. The jobs.getQueryResults() API can fail in ambiguous ways where the underlying job may or may not have failed. Might actually be best to revert this change.

@tswast tswast added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jan 26, 2024
@tswast
Copy link
Contributor Author
tswast commented Jan 29, 2024

An example of where doing jobs.getQueryResults can lead to ambiguity is the rateLimitExceeded error. With jobs.getQueryResults, if we get such an error, we don't really know if it's because of the general API-level rate limit or if it's because we hit the max number of concurrent queries and need to retry the whole query with a new job ID.

By doing jobs.get, which doesn't raise on query job failures, we remove this ambiguity. A rateLimitExceeded on jobs.get will be that API-level error and never mean the query job failed for that reason.

@tswast tswast closed this Jan 29, 2024
@tswast tswast deleted the getQueryResults-before-jobs-get branch January 29, 2024 21:45
@kiraksi kiraksi mentioned this pull request Jan 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

api: bigquery Issues related to the googleapis/python-bigquery API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. size: s Pull request size is small.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug: client doesn't retry "Job exceeded rate limits" for DDL query jobs that exceed quota for table update operations

2 participants

@tswast @kiraksi
0