8000 PR failure reporter has stopped working · Issue #548 · python/buildmaster-config · GitHub
[go: up one dir, main page]
Skip to content

PR failure reporter has stopped working #548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mhsmith opened this issue Oct 26, 2024 · 13 comments
Closed

PR failure reporter has stopped working #548

mhsmith opened this issue Oct 26, 2024 · 13 comments
Labels
bug

Comments

@mhsmith
Copy link
Member
mhsmith commented Oct 26, 2024

If I understand the code correctly, it's supposed to post a comment on a PR if a stable builder fails, and there has been exactly one commit to the branch since the previous success.

The last such comment was around the end of September. Here are a couple of more recent failures which should have caused a comment but didn't:
@mhsmith mhsmith added the bug label Oct 26, 2024
@vstinner
Copy link
Member

I didn't find anything relevant in Twisted logs. For example, grep pr_reporter master/twistd.log* returns nothing, and Traceback in the latest 3 log files are not related to pr_reporter.py.

@vstinner
Copy link
Member

cc @pablogsal @zware

@pablogsal
Copy link
Member

Yeah these two PRs should have triggered and I just confirmed that those builders are in this list:

buildmaster-config/master/master.cfg

Line 288 in d5e7bc5

github_status_builders.append(buildername)

So not sure what is going on :(

@pablogsal
Copy link
Member

I am adding more logging for now #549

@pablogsal
Copy link
Member

We need to wait until it fails again to investigate why is not happening I am afaid

@pablogsal
Copy link
Member

CC: @ambv @encukou If you see a buildbot failure with just one change that was not reported ping here or check the logs to see wha 8000 t's going on

@pablogsal
Copy link
Member

I looked today into why there was no comment for https://buildbot.python.org/all/#/builders/1594/builds/600.

Seems the build id for this is: 1701133. You can check it using the API:

https://buildbot.python.org/api/v2/builds/1701133

Checking the logs:

$grep 1701133 twistd.log.3
2024-11-18 14:55:56+0000 [-] Considering reporting build :1701133
2024-11-18 14:55:59+0000 [-] Attempting to issue a PR comment for failed build for build 1701133
...
[HTTP11ClientProtocol (BufferingTLSTransport),client] Issued a dicord comment for python/cpython at b0fcc2c47a34a69c35c1a8031cd0589d3747c1af, issue 126502.

but there is no other trace

@pablogsal
Copy link
Member

So looks like

buildmaster-config/master/custom/discord_reporter.py

Line 122 in f2bc68a

"Issued a dicord comment for {repoOwner}/{repoName} "
was hitted but not

buildmaster-config/master/custom/pr_reporter.py

Line 166 in f2bc68a

"Issued a Pull Request comment for {repoOwner}/{repoName} "

@pablogsal
Copy link
Member

There is a lot of this:

2024-11-18 15:08:31+0000 [Broker,217389,8.43.85.223] invalid login from unknown user 'cstratak-RHEL7-x86_64'
2024-11-18 15:08:31+0000 [Broker,217389,8.43.85.223] Peer will receive following PB traceback:
2024-11-18 15:08:31+0000 [Broker,217389,8.43.85.223] Unhandled Error
        Traceback (most recent call last):
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/cred/portal.py", line 144, in login
            return maybeDeferred(
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 212, in maybeDeferred
            result = f(*args, **kwargs)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 2287, in unwindGenerator
            return _cancellableInlineCallbacks(gen)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 2197, in _cancellableInlineCallbacks
            _inlineCallbacks(None, gen, status, _copy_context())
        --- <exception caught here> ---
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 2014, in _inlineCallbacks
            result = context.run(gen.send, result)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/worker/protocols/manager/pb.py", line 83, in requestAvatarId
            raise error.UnauthorizedLogin()
        twisted.cred.error.UnauthorizedLogin:

but that happens all the time and looks unrelated. An looks like is for fetching an avatar??? (requestAvatarId)?

@python python deleted a comment from bedevere-bot Nov 19, 2024
@pablogsal
Copy link
Member

Ah I found the problem. Turns out upstream changed how auth is passed down to the service in commit buildbot/buildbot@6b75ddf

I will create a fix

@pablogsal
Copy link
Member

I think this is fixed will reopen if we keep having the problem

@vstinner
Copy link
Member

It works again: python/cpython#127086 (comment) :-) (the failure is unrelated to the change, but the reporter works as expected.)

@pablogsal
Copy link
Member

Nice 👌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mhsmith @vstinner @pablogsal
0