Extend rate limit window and retry on Wayback 429s#826
Extend rate limit window and retry on Wayback 429s#826northeastprince wants to merge 1 commit intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR extends the existing job rate-limiting mechanism to support an “extended wait/cooldown” after specific Faraday errors (e.g., Wayback 429/5xx), and consolidates Wayback-related jobs to share the same rate-limit behavior.
Changes:
- Add
extend_wait:support toRateLimitableand introduceRateLimitWindow.block/remaining_wait_time. - Configure Wayback archival job to extend the wait window and retry on 429/5xx errors.
- Update and expand rate limiting tests, and refactor the follow-up archival job to inherit from the shared Wayback job.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
test/jobs/rate_limitable_test.rb |
Adds cleanup and a new test for extended-wait retry behavior. |
app/models/rate_limit_window.rb |
Adds APIs to “block” a key and query remaining wait time. |
app/models/hackathon/website/archivable.rb |
Refactors FollowUpJob to inherit from shared archival job. |
app/models/concerns/rate_limitable.rb |
Adds extend_wait: option, cooldown rescue behavior, and pre-admit wait enforcement. |
app/jobs/hackathons/website_archival_job.rb |
Enables extended wait + retry on Faraday 429/5xx for Wayback calls. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
1edcb6a to
cb1ed31
Compare
There was a problem hiding this comment.
Pull request overview
Adds an “extended wait” mechanism to the existing job rate-limiting system so Wayback/InternetArchive jobs back off longer (and auto-retry) after 429s / server errors.
Changes:
- Add
extend_waitsupport toRateLimitableand integrate it with newRateLimitWindow.block/remaining_wait_time. - Configure
Hackathons::WebsiteArchivalJobto extend/back off on Faraday 429/server errors, and make the follow-up job inherit this behavior. - Extend test coverage for the new “extended wait” behavior and stabilize tests by clearing
RateLimitWindow.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
app/models/rate_limit_window.rb |
Adds blocking + remaining-wait-time helpers and tweaks join logic for windows. |
app/models/concerns/rate_limitable.rb |
Adds extend_wait option to rate limiting, blocks execution during extended waits, and retries on configured errors. |
app/jobs/hackathons/website_archival_job.rb |
Enables extend_wait backoff/retry for Wayback-related Faraday errors. |
app/models/hackathon/website/archivable.rb |
Makes the follow-up archival job inherit the Wayback rate limiting/backoff behavior. |
test/jobs/rate_limitable_test.rb |
Adds a test for extended wait behavior and clears RateLimitWindow between tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
cb1ed31 to
8c912b5
Compare
There was a problem hiding this comment.
Pull request overview
Adds support for extending a rate-limit “cooldown” window when specific external errors occur (e.g., Wayback Machine 429/5xx), and applies that behavior to hackathon website archival jobs.
Changes:
- Extend
RateLimitablewith anextend_waitoption that blocks and retries jobs on configured exception types. - Add
RateLimitWindow.blockandremaining_wait_time, and adjust window-join logic to support blocked windows. - Apply the new behavior to Wayback Machine archival jobs and add a targeted test.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
test/jobs/rate_limitable_test.rb |
Adds coverage for the new extend_wait behavior and resets RateLimitWindow state in setup/teardown. |
app/models/rate_limit_window.rb |
Introduces “blocked window” support and exposes remaining wait time for enforcement. |
app/models/hackathon/website/archivable.rb |
Makes FollowUpJob inherit the shared Wayback archival job configuration. |
app/models/concerns/rate_limitable.rb |
Adds extend_wait configuration, blocking behavior, and enforcement before admitting jobs. |
app/jobs/hackathons/website_archival_job.rb |
Configures Wayback Machine rate limiting to extend waits and retry on 429/5xx Faraday errors. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| end | ||
|
|
||
| def remaining_wait_time(key) | ||
| expiration = active.where(key:).pick :expiration |
| class ExtendedWaitJob < ApplicationJob | ||
| rate_limit to: 3, within: 1.minute, | ||
| extend_wait: {on: [Faraday::TimeoutError], for: 2.minutes} |
No description provided.