8000 [Bug] Binding server sometimes fails to properly respond · Issue #1975 · typesafegithub/github-workflows-kt · GitHub
[go: up one dir, main page]
Skip to content

[Bug] Binding server sometimes fails to properly respond #1975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
1 of 3 tasks
Vampire opened this issue May 21, 2025 · 7 comments
Open
1 of 3 tasks

[Bug] Binding server sometimes fails to properly respond #1975

Vampire opened this issue May 21, 2025 · 7 comments
Assignees
@krzema12
Labels
bug Something isn't working

Comments

@Vampire
Copy link
Collaborator
Vampire commented May 21, 2025

Component

Action

for i in {01..100}; do echo $i; rm -rf ~/.m2/repository/actions/checkout*; .github/workflows/build.main.kts; done

Expected

No failures happening.

Actual

Occasionally download fails and retry mechanics have to kick in.
For example

[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (org.jetbrains.kotlin.org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://bindings.krzeminski.it:443: The target server failed to respond
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->https://bindings.krzeminski.it:443

on 5th and 24th try.
Or

[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://bindings.krzeminski.it:443: Connection reset
[main] INFO org.jetbrains.kotlin.org.apache.http.impl.execchain.RetryExec - Retrying request to {s}->https://bindings.krzeminski.it:443

on 12th try.
@Vampire Vampire added the bug Something isn't working label May 21, 2025
@krzema12
Copy link
Member

I failed to reproduce it with a local server.

@krzema12
Copy link
Member

When running the script mentioned in the issue's description for more iterations, I don't see anything significantly off in the metrics. This one is going to be tricky.

Ideas for the next steps: have logs and/or metrics to catch a case where a request for an artifact is received, but ktor doesn't respond. Strawman: have two metrics, one increased when a request is received, and one increased when a response is sent. We should ideally see them both at the same values. If they diverge, it means we managed to capture the problem in the metrics, and it's a step forward since we'll be able to quantify the problem.

@krzema12 krzema12 self-assigned this May 22, 2025
@krzema12
Copy link
Member

Another idea: make ktor fail if no response is sent. I'm discussing with the community what's the best way to achieve it: https://kotlinlang.slack.com/archives/C0A974TJ9/p1747947460471269?thread_ts=1747947460.471269&cid=C0A974TJ9

@krzema12
Copy link
Member

We have a concrete next step advised by the ktor support: https://kotlinlang.slack.com/archives/C0A974TJ9/p1747988699947299?thread_ts=1747947460.471269&cid=C0A974TJ9
I think we need to tweak log4j2.xml to set logging level for Netty to DEBUG somehow.

@krzema12
Copy link
Member
krzema12 commented May 28, 2025 •
edited 8000
Loading

Let me add this to the logging config:

    <Loggers>
        ...
        <Logger name="io.netty" level="debug">
            <AppenderRef ref="Console Appender"/>
            <AppenderRef ref="Rolling File Appender"/>
        </Logger>
    </Loggers>

@Vampire
Copy link
Collaborator Author
Vampire commented May 28, 2025

Just <Logger name="io.netty" level="DEBUG"/> is enough, the appenders are inherited

@Vampire
Copy link
Collaborator Author
Vampire commented May 28, 2025

Or with TRACE or ALL if it logs below DEBUG and DEBUG is not enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krzema12 krzema12

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Vampire @krzema12
0