8000 guestagent: ticker: watch sys_exit_bind with eBPF by AkihiroSuda · Pull Request #4066 · lima-vm/lima · GitHub
[go: up one dir, main page]
Skip to content

guestagent: ticker: watch sys_exit_bind with eBPF #4066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 23, 2025
Merged

guestagent: ticker: watch sys_exit_bind with eBPF #4066

merged 1 commit into from
Sep 23, 2025

Conversation

AkihiroSuda
Copy link
Member

The event watcher is now triggered immediately on sys_exit_bind, not waiting for the next 3-second tick.

This commit resolves the long-standing TODO since the initial commit:

lima/cmd/lima-guestagent/daemon_linux.go

Lines 57 to 63 in 7459f45

newTicker := func() (<-chan time.Time, func()) {
// TODO: use an equivalent of `bpftrace -e 'tracepoint:syscalls:sys_*_bind { printf("tick\n"); }')`,
// without depending on `bpftrace` binary.
// The agent binary will need CAP_BPF file cap.
ticker := time.NewTicker(tick)
return ticker.C, ticker.Stop
}

Close #3067
Close #3766
Close #4021

@AkihiroSuda AkihiroSuda added this to the v2.0.0 milestone Sep 22, 2025
@AkihiroSuda AkihiroSuda mentioned this pull request Sep 22, 2025
Closed
@balajiv113
Copy link
Member

Does this work for UDP as well ??

@jandubois
Copy link
Member

I don't have time to test right now, but I think this will not trigger when the port is closed, right?

One common problem with our polling is that test suites create a container with exposed ports, and after stopping the container create another one again in the next test that exposes the same ports. Due to the 3 second polling the old forwards may not have been removed yet, blocking the new container from exposing the same ports again.

Maybe there is another trace point to also kick the ticker, like sys_exit_close or tcp:tcp_close, if that works reliably?

@AkihiroSuda
Copy link
Member Author
AkihiroSuda commented Sep 23, 2025 •
edited
Loading

Does this work for UDP as well ??

Yes

sys_exit_close

Seems too frequent

$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_close { printf("%d %s %s\n", elapsed / 1000 / 1000 / 1000, probe, comm); }'
Attaching 1 probe...                                   
0 tracepoint:syscalls:sys_exit_close tmux: server      
0 tracepoint:syscalls:sys_exit_close tmux: server      
0 tracepoint:syscalls:sys_exit_close tmux: server      
1 tracepoint:syscalls:sys_exit_close tmux: server      
1 tracepoint:syscalls:sys_exit_close tmux: server      
2 tracepoint:syscalls:sys_exit_close tmux: server      
2 tracepoint:syscalls:sys_exit_close tmux: server      
2 tracepoint:syscalls:sys_exit_close containerd        
2 tracepoint:syscalls:sys_exit_close containerd        
2 tracepoint:syscalls:sys_exit_close containerd        
2 tracepoint:syscalls:sys_exit_close containerd 
...

tcp:tcp_close

Doesn't seem to exist in /sys/kernel/debug/tracing/available_events

@AkihiroSuda
guestagent: ticker: watch sys_exit_bind with eBPF
4962543 
The event watcher is now triggered immediately on `sys_exit_bind`,
not waiting for the next 3-second tick.

This commit resolves the long-standing TODO since the initial commit:
https://github.com/lima-vm/lima/blob/7459f4587987ed014c372f17b82de1817feffa2e/cmd/lima-guestagent/daemon_linux.go#L57-L63

Close PR 3067, 3766, 4021

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@AkihiroSuda
Copy link
Member Author

One common problem with our polling is that test suites create a container with exposed ports, and after stopping the container create another one again in the next test that exposes the same ports. Due to the 3 second polling the old forwards may not have been removed yet, blocking the new container from exposing the same ports again.

How does this happen? The guest kernel isn't aware of the usage of the host port, so it doesn't block bind() in the guest.
After bind() in the guest succeeds, the ticker event is triggered, and the hostagent updates the host port status accordingly.

balajiv113
balajiv113 approved these changes 8000 Sep 23, 2025
View reviewed changes
Copy link
Member
@balajiv113 balajiv113 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Looks like i overcomplicated by thinking of getting a real event with open ports.

@jandubois
Copy link
Member

How does this happen? The guest kernel isn't aware of the usage of the host port, so it doesn't block bind() in the guest.
After bind() in the guest succeeds, the ticker event is triggered, and the hostagent updates the host port status accordingly.

I guess the issue used to be that the test would connect to the dangling port on the host. So maybe just looking for bind events is enough; will need to write a test to verify.

jandubois added a commit to jandubois/lima that referenced this pull request Sep 23, 2025
@jandubois
Add port monitoring test script
814c0dc 
It verifies that when a container is destroyed, its ports can be reused
immediately and are not subject to being freed by a polling loop. See lima-vm#4066.

Signed-off-by: Jan Dubois <jan.dubois@suse.com>
@jandubois jandubois mentioned this pull request Sep 23, 2025
Merged
jandubois
jandubois approved these changes Sep 23, 2025
View reviewed changes
Copy link
Member
@jandubois jandubois left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't reviewed the code (@balajiv113 seems to have done so already), but I wrote a test (#4077) to verify it works properly, which is seems to do.

This seems like the most straightforward fix to the issue!

@jandubois jandubois merged commit 45c9527 into lima-vm:master Sep 23, 2025
140 of 144 checks passed
jandubois added a commit to jandubois/lima that referenced this pull request Sep 23, 2025
@jandubois
Add port monitoring test script
a5c15b6 
It verifies that when a container is destroyed, its ports can be reused
immediately and are not subject to being freed by a polling loop. See lima-vm#4066.

Signed-off-by: Jan Dubois <jan.dubois@suse.com>
@norio-nomura
Copy link
Contributor
norio-nomura commented Sep 25, 2025 •
edited
Loading

After this is merged, the CPU utilization rate of lima-guestagent is always close to 90%.

screenshot 2025-09-25 9 36 07

I confirmed that it will be resolved with reverted.

@norio-nomura norio-nomura mentioned this pull request Sep 25, 2025
Closed
jandubois added a commit to jandubois/lima that referenced this pull request Sep 28, 2025
@jandubois
Add port monitoring test script
3234d11 
It verifies that when a container is destroyed, its ports can be reused
immediately and are not subject to being freed by a polling loop. See lima-vm#4066.

Signed-off-by: Jan Dubois <jan.dubois@suse.com>
@abiosoft abiosoft mentioned this pull request Oct 1, 2025
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jandubois jandubois jandubois approved these changes

@balajiv113 balajiv113 balajiv113 approved these changes

Assignees
No one assigned
Labels
area/portfwd
Projects
None yet
Milestone
v2.0.0
Development

Successfully merging this pull request may close these issues.
4 participants
@AkihiroSuda @balajiv113 @jandubois @norio-nomura
0