8000 Attempt to fix actions hang by FoamyGuy · Pull Request #397 · adafruit/adabot · GitHub
[go: up one dir, main page]

Skip to content

Attempt to fix actions hang #397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 22, 2025
Merged

Conversation

FoamyGuy
Copy link
Contributor
@FoamyGuy FoamyGuy commented Apr 18, 2025

This is an attempt to fix the issue causing the actions to hang sometimes such as: https://github.com/adafruit/circuitpython-org/actions/runs/14512434132

My hypothesis is that this issue is caused by different parts of the code handling of waiting for the rate limit reset time. I think there is a situation where some parts of the code end up sleeping for a very long time due to a "negative overflow" from subtracting dates.

I found this section of the log at the end of the failed / stalled actions run:

2025-04-17T09:57:55.2025426Z GitHub API Rate Limit reached. Pausing until Rate Limit reset.
2025-04-17T09:57:55.2025809Z Rate Limit will reset at: 2025-04-17 10:23:13
2025-04-17T09:57:55.2026084Z Sleeping 1517 seconds
2025-04-17T10:23:13.3062291Z Resetting dropped connection: api.github.com
2025-04-17T10:23:13.4537153Z https://api.github.com:443 "GET /repos/adafruit/Adafruit_CircuitPython_Hue HTTP/1.1" 200 None
2025-04-17T10:23:13.4579740Z GET https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 200 {'date': 'Thu, 17 Apr 2025 10:23:13 GMT', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'private, max-age=60, s-maxage=60', 'vary': 'Accept, Authorization, Cookie, X-GitHub-OTP,Accept-Encoding, Accept, X-Requested-With', 'etag': 'W/"dff493f5c44d84e6d33a7c7bb77ff4287fa8128fdd2f5fe8cf1aa509754696a0"', 'last-modified': 'Thu, 27 Feb 2025 02:10:53 GMT', 'x-oauth-scopes': 'repo', 'x-accepted-oauth-scopes': 'repo', 'github-authentication-token-expiration': '2026-03-18 04:00:00 UTC', 'x-github-media-type': 'github.v3; format=json', 'x-github-api-version-selected': '2022-11-28', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '1744885393', 'x-ratelimit-used': '5000', 'x-ratelimit-resource': 'core', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '0', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'content-encoding': 'gzip', 'transfer-encoding': 'chunked', 'x-github-request-id': 'AC00:18777F:38DEEA8:70FDC4B:6800D691', 'server': 'github.com'} {"id":181008398,"node_id":"MDEwOlJlcG9zaXRvcnkxODEwMDgzOTg=","name":"Adafruit_CircuitPython_Hue","full_name":"adafruit/Adafruit_CircuitPython_Hue","private":false,"owner":{"login":"adafruit","id":181069,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE4MTA2OQ==","avatar_url":"https://avatars.githubusercontent.com/u/181069?v=4","gravatar_id":"","url":"https://api.github.com/users/adafruit","html_url":"https://github.com/adafruit","followers_url":"https://api.github.com/users/adafruit/followers","following_url":"https://api.github.com/users/adafruit/following{/other_user}","gists_url":"https://api.github.com/users/adafruit/gists{/gist_id}","starred_url":"https://api.github.com/users/adafruit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adafruit/subscriptions","organizations_url":"https://api.github.com/users/adafruit/orgs","repos_url":"https://api.github.com/users/adafruit/repos","events_url":"https://api.github.com/users/adafruit/events{/privacy}","received_events_url":"https://api.github.com/users/adafruit/received_events","type":"Organization","user_view_type":"public","site_admin":false},"html_url":"https://github.com/adafruit/Adafruit_CircuitPython_Hue","description":"Philips Hue Lighting Helper for CircuitPython","fork":false,"url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue","forks_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/forks","keys_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/keys{/key_id}","collaborators_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/teams","hooks_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/hooks","issue_events_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/issues/events{/number}","events_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/events","assignees_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/assignees{/user}","branches_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/branches{/branch}","tags_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/tags","blobs_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/git/refs{/sha}","trees_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/git/trees{/sha}","statuses_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/statuses/{sha}","languages_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/languages","stargazers_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/stargazers","contributors_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/contributors","subscribers_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/subscribers","subscription_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/subscription","commits_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/commits{/sha}","git_commits_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/git/commits{/sha}","comments_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/comments{/number}","issue_comment_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/issues/comments{/number}","contents_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/contents/{+path}","compare_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/compare/{base}...{head}","merges_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/merges","archive_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/downloads","issues_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/issues{/number}","pulls_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/pulls{/number}","milestones_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/milestones{/number}","notifications_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/labels{/name}","releases_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/releases{/id}","deployments_url":"https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/deployments","created_at":"2019-04-12T12:56:49Z","updated_at":"2025-02-27T02:10:53Z","pushed_at":"2025-02-27T02:22:37Z","git_url":"git://github.com/adafruit/Adafruit_CircuitPython_Hue.git","ssh_url":"git@github.com:adafruit/Adafruit_CircuitPython_Hue.git","clone_url":"https://github.com/adafruit/Adafruit_CircuitPython_Hue.git","svn_url":"https://github.com/adafruit/Adafruit_CircuitPython_Hue","homepage":null,"size":109,"stargazers_count":7,"watchers_count":7,"language":"Python","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"has_discussions":false,"forks_count":10,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":1,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"allow_forking":true,"is_template":false,"web_commit_signoff_required":false,"topics":["hacktoberfest"],"visibility":"public","forks":10,"open_issues":1,"watchers":7,"default_branch":"main","permissions":{"admin":false,"maintain":false,"push":true,"triage":true,"pull":true},"temp_clone_token":"","allow_squash_merge":false,"allow_merge_commit":true,"allow_rebase_merge":false,"allow_auto_merge":false,"delete_branch_on_merge":false,"allow_update_branch":false,"use_squash_pr_title_as_default":false,"squash_merge_commit_message":"COMMIT_MESSAGES","squash_merge_commit_title":"COMMIT_OR_PR_TITLE","merge_commit_message":"PR_TITLE","merge_commit_title":"MERGE_MESSAGE","custom_properties":{},"organization":{"login":"adafruit","id":181069,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE4MTA2OQ==","avatar_url":"https://avatars.githubusercontent.com/u/181069?v=4","gravatar_id":"","url":"https://api.github.com/users/adafruit","html_url":"https://github.com/adafruit","followers_url":"https://api.github.com/users/adafruit/followers","following_url":"https://api.github.com/users/adafruit/following{/other_user}","gists_url":"https://api.github.com/users/adafruit/gists{/gist_id}","starred_url":"https://api.github.com/users/adafruit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adafruit/subscriptions","organizations_url":"https://api.github.com/users/adafruit/orgs","repos_url":"https://api.github.com/users/adafruit/repos","events_url":"https://api.github.com/users/adafruit/events{/privacy}","received_events_url":"https://api.github.com/users/adafruit/received_events","type":"Organization","user_view_type":"public","site_admin":false},"network_count":10,"subscribers_count":17}
2025-04-17T10:23:13.5037907Z https://api.github.com:443 "GET /repos/adafruit/Adafruit_CircuitPython_Hue/actions/workflows/build.yml HTTP/1.1" 403 None
2025-04-17T10:23:13.5053071Z GET https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue/actions/workflows/build.yml {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 403 {'date': 'Thu, 17 Apr 2025 10:23:13 GMT', 'content-type': 'application/json; charset=utf-8', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '1744885393', 'x-ratelimit-used': '5000', 'x-ratelimit-resource': 'core', 'x-oauth-scopes': 'repo', 'x-accepted-oauth-scopes': 'repo', 'github-authentication-token-expiration': '2026-03-18 04:00:00 UTC', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '0', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'vary': 'Accept-Encoding, Accept, X-Requested-With', 'content-encoding': 'gzip', 'transfer-encoding': 'chunked', 'x-github-request-id': 'AC00:18777F:38DEF3D:70FDD6F:6800D691', 'server': 'github.com'} {"message":"API rate limit exceeded for user ID 33701428. If you reach out to GitHub Support for help, please include the request ID AC00:18777F:38DEF3D:70FDD6F:6800D691 and timestamp 2025-04-17 10:23:13 UTC.","documentation_url":"https://docs.github.com/rest/overview/rate-limits-for-the-rest-api","status":"403"}
2025-04-17T10:23:13.5063075Z Starting new HTTPS connection (1): api.github.com:443
2025-04-17T10:23:13.5580091Z https://api.github.com:443 "GET /repos/adafruit/Adafruit_CircuitPython_Hue/contents/ HTTP/1.1" 403 None
2025-04-17T10:23:13.5583288Z GET /repos/adafruit/Adafruit_CircuitPython_Hue/contents/ (0 remaining) status=403
2025-04-17T10:23:13.5584406Z 0 requests remaining this hour
2025-04-17T10:23:13.5585524Z GitHub API Rate Limit reached. Pausing until Rate Limit reset.
2025-04-17T10:23:13.5596925Z Starting new HTTPS connection (1): pypi.org:443
2025-04-17T10:23:13.5691257Z https://pypi.org:443 "GET /pypi/Adafruit_CircuitPython_Hue/json HTTP/1.1" 200 7580
2025-04-17T10:23:13.5706289Z Starting new HTTPS connection (1): api.github.com:443
2025-04-17T10:23:13.6129317Z https://api.github.com:443 "GET /repos/adafruit/Adafruit_CircuitPython_Hue/labels HTTP/1.1" 403 None
2025-04-17T10:23:13.6132685Z GET /repos/adafruit/Adafruit_CircuitPython_Hue/labels (0 remaining) status=403
2025-04-17T10:23:13.6133904Z 0 requests remaining this hour
2025-04-17T10:23:13.6135149Z GitHub API Rate Limit reached. Pausing until Rate Limit reset.
2025-04-17T10:23:13.6563616Z https://api.github.com:443 "GET /repos/adafruit/Adafruit_CircuitPython_Hue HTTP/1.1" 403 None
2025-04-17T10:23:13.6578072Z GET https://api.github.com/repos/adafruit/Adafruit_CircuitPython_Hue {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 403 {'date': 'Thu, 17 Apr 2025 10:23:13 GMT', 'content-type': 'application/json; charset=utf-8', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '1744885393', 'x-ratelimit-used': '5000', 'x-ratelimit-resource': 'core', 'x-oauth-scopes': 'repo', 'x-accepted-oauth-scopes': 'repo', 'github-authentication-token-expiration': '2026-03-18 04:00:00 UTC', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '0', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'vary': 'Accept-Encoding, Accept, X-Requested-With', 'content-encoding': 'gzip', 'transfer-encoding': 'chunked', 'x-github-request-id': 'AC00:18777F:38DEFE6:70FDED4:6800D691', 'server': 'github.com'} {"message":"API rate limit exceeded for user ID 33701428. If you reach out to GitHub Support for help, please include the request ID AC00:18777F:38DEFE6:70FDED4:6800D691 and timestamp 2025-04-17 10:23:13 UTC.","documentation_url":"https://docs.github.com/rest/overview/rate-limits-for-the-rest-api","status":"403"}
2025-04-17T10:23:13.7026590Z https://api.github.com:443 "GET /rate_limit HTTP/1.1" 200 None
2025-04-17T10:23:13.7038476Z GET https://api.github.com/rate_limit {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 200 {'date': 'Thu, 17 Apr 2025 10:23:13 GMT', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache', 'x-oauth-scopes': 'repo', 'x-accepted-oauth-scopes': '', 'github-authentication-token-expiration': '2026-03-18 04:00:00 UTC', 'x-github-media-type': 'github.v3; format=json', 'x-github-api-version-selected': '2022-11-28', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '1744885393', 'x-ratelimit-used': '5000', 'x-ratelimit-resource': 'core', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '0', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'vary': 'Accept-Encoding, Accept, X-Requested-With', 'content-encoding': 'gzip', 'transfer-encoding': 'chunked', 'x-github-request-id': 'AC00:18777F:38DF012:70FDF47:6800D691', 'server': 'github.com'} {"resources":{"core":{"limit":5000,"used":5000,"remaining":0,"reset":1744885393},"search":{"limit":30,"used":0,"remaining":30,"reset":1744885453},"graphql":{"limit":5000,"used":0,"remaining":5000,"reset":1744888993},"integration_manifest":{"limit":5000,"used":0,"remaining":5000,"reset":1744888993},"source_import":{"limit":100,"used":0,"remaining":100,"reset":1744885453},"code_scanning_upload":{"limit":1000,"used":0,"remaining":1000,"reset":1744888993},"code_scanning_autofix":{"limit":10,"used":0,"remaining":10,"reset":1744885453},"actions_runner_registration":{"limit":10000,"used":0,"remaining":10000,"reset":1744888993},"scim":{"limit":15000,"used":0,"remaining":15000,"reset":1744888993},"dependency_snapshots":{"limit":100,"used":0,"remaining":100,"reset":1744885453},"audit_log":{"limit":1750,"used":0,"remaining":1750,"reset":1744888993},"audit_log_streaming":{"limit":15,"used":0,"remaining":15,"reset":1744888993},"code_search":{"limit":10,"used":0,"remaining":10,"reset":1744885453}},"rate":{"limit":5000,"used":5000,"remaining":0,"reset":1744885393}}
2025-04-17T10:23:13.7046364Z Rate Limit will reset at: 2025-04-17 10:23:13
2025-04-17T15:22:47.2450588Z ##[error]The operation was canceled.
2025-04-17T15:22:47.2542261Z Post job cleanup.

At 09:57:55 the code prints that it will sleep for 1517 seconds, I believe the print came from here: https://github.com/adafruit/adabot/blob/main/adabot/github_requests.py#L106. It also prints that reset is at 10:23:13

Then at 10:23:13 it wakes up and starts to do more things. But it seems that the rate limit is not actually (fully?) reset yet because it is still indicating 1 and then 0 for remaining requests.

This is the last line before the job seems to start getting cancelled / cleaned up by Github:

2025-04-17T10:23:13.7046364Z Rate Limit will reset at: 2025-04-17 10:23:13

It seems to be saying that the rate limit will reset at the time that it currently is. I believe maybe the times "collided" or that the current time is actually slightly in front of the reset time.

I think that print must have come from one of these places:
https://github.com/adafruit/adabot/blob/main/adabot/lib/circuitpython_library_validators.py#L937
https://github.com/adafruit/adabot/blob/main/adabot/lib/circuitpython_library_validators.py#L1297

In the logic next to it to determin how long to sleep it subtracts datetimes:

sleep_time = core_rate_limit_reset - datetime.datetime.utcnow()

If datetime.datetime.utcnow() were actually bigger than core_rate_limit_reset even by a little bit this subtraction basically overflows into negative which then causes the seconds value to be a very large (positive) value that the code then sleeps for.

I tested this theory in the python repl with this:

>>> dt = datetime.datetime.fromtimestamp(1683854400)
>>> dt1 = datetime.datetime.fro
8000
mtimestamp(1683854400.001)
>>> dt - dt1
datetime.timedelta(days=-1, seconds=86399, microseconds=999000)

In this PR I have tried to address the issue in two ways:

  1. sleep for a full extra 60 seconds inside of github_requests.py to try to avoid a situation where the code in circuitpython_library_validators.py is reached at a time when the current time could be past the rate limit reset time already but not by very much.
  2. cap the maximum time that these sections of code will sleep to 62 minutes using min() so that if it comes up with a large value like 86399 it will instead only wait for just over an hour. Theoretically it could get by waiting less in some some scenarios, but I figure that waiting for just over an hour is safest to ensure the limit reset is reached no matter what.

Copy link
Member
@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please factor out the 62 minute timeout so we can change it in one place. Thanks!

@FoamyGuy
Copy link
Contributor Author

Thank you, the latest commit refactors that into a variable and uses it from the various places instead.

Copy link
Member
@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks!

@tannewt tannewt merged commit 80adf57 into adafruit:main Apr 22, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
0