eBPF Implementation for FreeBSD
Yutaro Hayakawa
Mail: yhayakawa3720@gmail.com
Twitter: @YutaroHayakawa
1
About me
Name: Yutaro Hayakawa
Affiliation: Keio University, Japan (Master student)
Research topic: Network (SDN/NFV), Operating Systems
Misc: Now on GSoC for FreeBSD and job hunting
Yutaro Hayakawa | eBPF implementation for FreeBSD 2
Agenda
1. Linux eBPF the Basic
2. eBPF implementation for FreeBSD
3. Usecase: VALE-BPF
Yutaro Hayakawa | eBPF implementation for FreeBSD 3
Agenda
1. Linux eBPF the Basic
2. eBPF implementation for FreeBSD
3. VALE-BPF
Yutaro Hayakawa | eBPF implementation for FreeBSD 4
What’s eBPF?
Extended general perpose BPF virtual machine ISA
- Closer to modern CPU ISA (64bit registers * 11, 64bit wide instructions...)
- C calling convention and LLVM backend
- Call instruction
- Maps (in-kernel key-value store shared with user space program)
- Write data to tracing buffer
- etc…
More performance optimization (JIT, static code analysis)
bpf(2) for loading program, creating maps, manipulating maps ...
Yutaro Hayakawa | eBPF implementation for FreeBSD 5
Use cases?
Use cases: Dynamic tracing
Use eBPF as a backend of dynamic tracing (like DTrace)
https://github.com/iovisor/bcc
http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html
Yutaro Hayakawa | eBPF implementation for FreeBSD 7
Use cases: XDP (eXpress Data Path)
No “kernel bypass” (e.g. DPDK,
netmap)
Hook and process packet right after
reception inside the driver by eBPF
- DDos mitigation: Droplet
- Load balancing: Katran
- IDS/IPS backend: Surikata
Hardware offloading https://www.iovisor.org/technology/xdp
- Netronome Agilio
Yutaro Hayakawa | eBPF implementation for FreeBSD 8
Tooling?
eBPF Tooling
Linux kernel provides only very premitive API to users
- bpf(2)
- Program loader (e.g. Netlink, setsockopt, ioctl... )
- Some useful libraries (but very primitive)
Need tooling for better utilization
Yutaro Hayakawa | eBPF implementation for FreeBSD 10
Tooling: BCC (BPF Compiler Collection)
Compiler driver and useful libraries for eBPF
- Deal with restricted C, call clang/llvm
- Compiler frontend for various languages (C, P4)
- ELF parsing, Map libraries
- Language bindings (Python, C++, Lua…)
Source: https://github.com/iovisor/bcc
Yutaro Hayakawa | eBPF implementation for FreeBSD 11
Embedded C
Embedded C
Interact with Map
Output
Embedded C
Interact with Map
Tooling: PLY
Tracing frontend which is heavily
inspired by DTrace
dtrace -n syscall:::entry'{@syscalls[probefunc] = count();}'
Source: https://github.com/iovisor/ply
Yutaro Hayakawa | eBPF implementation for FreeBSD 16
Tooling: PLY
Tracing frontend which is heavily
inspired by DTrace
dtrace -n syscall:::entry'{@syscalls[probefunc] = count();}'
Source: https://github.com/iovisor/ply
Yutaro Hayakawa | eBPF implementation for FreeBSD 17
Tooling: bpfilter
iptables (Linux’s ipfw or pf) which uses XDP as a backend
Transparently accerelates existing iptables
RFC patch: https://www.mail-archive.com/netdev@vger.kernel.org/msg217095.html
https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/
Yutaro Hayakawa | eBPF implementation for FreeBSD 18
Conclusion for this section
Recent Linux implements a lot of interesting features using eBPF
- Dynamic tracing
- Very fast packet processing framework
- etc ...
The community also introduces a lot of interesting tools
- BCC, PLY, bpfilter
More information
- https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/
- Really useful collection of links
Yutaro Hayakawa | eBPF implementation for FreeBSD 19
Agenda
1. Linux eBPF the Basic
2. eBPF implementation for FreeBSD
3. VALE-BPF
Yutaro Hayakawa | eBPF implementation for FreeBSD 20
generic-ebpf
Generalized multi-platform eBPF implementation
- Currently supports FreeBSD user/kernel, Linux user/kernel and macOS user
- About 200 lines of glue code for each platform
- Shares most of the code (easy to test in userspace)
- Interpreter and JIT compiler for x86-64 based on ubpf
- Maps which uses tommyds as a backend
- Verifier is not yet implemented...
Source: https://github.com/YutaroHayakawa/generic-ebpf
Yutaro Hayakawa | eBPF implementation for FreeBSD 21
Current status
/dev/ebpf + ioctl(2) interface (Linux bpf(2))
- load program, create and manipulate maps, run simple test
Interpreter and JIT compiler for x86-64
- Most of the instructions are implemented
- atomic operations are missing
Array, Hashtable maps
Yutaro Hayakawa | eBPF implementation for FreeBSD 22
Hashtable map benchmark
For more details: https://github.com/YutaroHayakawa/generic-ebpf/tree/master/benchmark
Yutaro Hayakawa | eBPF implementation for FreeBSD 23
Why is FreeBSD case so slow?
FreeBSD Linux
Experiment
- Simply returns immediately from ioctl ioctl(2) ioctl(2)
handler
struct cdevsw struct file_operations
- See latency of ioctl
ioctl handler ioctl handler
ebpf_dev_ioctl
Map operations
(update/delete/lookup)
Yutaro Hayakawa | eBPF implementation for FreeBSD 24
Why is FreeBSD case so slow?
FreeBSD Linux
Experiment
- Simply returns immediately from ioctl ioctl(2) ioctl(2)
handler
struct cdevsw struct file_operations
- See latency of ioctl
ioctl handler ioctl handler
ebpf_dev_ioctl
About 85% of the difference comes from ioctl
Map operations
Need more precise analysis... (update/delete/lookup)
Yutaro Hayakawa | eBPF implementation for FreeBSD 25
Agenda
1. Linux eBPF the Basic
2. eBPF implementation for FreeBSD
3. VALE-BPF
Yutaro Hayakawa | eBPF implementation for FreeBSD 26
VALE (Virtual Local Ethernet)
Fast and modular software switch (a.k.a mSwitch)
uint32_t
mylookup(struct nm_bdg_fwd *ft, uint8_t *dst_ring, mymodule.ko
struct netmap_vp_adapter *na, void *private_data) User
{ netmap API netmap API
struct ip *iph; Kernel
iph = (struct ip)(buf + ETHER_HDR_LEN);
if (iph - ft->ft_buf > ft->ft_len) {
return NM_BDG_DROP; Modular Lookup
Logic
}
VALE
return ntohl(iph->ip_dst) & 0xff;
}
Yutaro Hayakawa | eBPF implementation for FreeBSD 27
VALE-BPF
VALE module which enhances eBPF programmability to VALE
uint32_t
vale_bpf_lookup(struct vale_bpf_md *md) vale-bpf.ko
{
User
struct ip iph; netmap API netmap API
Kernel
iph = (struct ip)(md->buf + ETHER_HDR_LEN);
if (iph > md->buf_end) {
return VALE_BPF_DROP;
eBPF lookup logic
}
return ntohl(iph->ip_dst) & 0xff; VALE
}
Source: https://github.com/YutaroHayakawa/vale-bpf
Yutaro Hayakawa | eBPF implementation for FreeBSD 28
Performance evaluation
Forward packets between two virtual ports with different logic
- Learning bridge
- No logic
Learning Bridge [Mpps] No Logic [Mpps]
VALE 17.74 27.71
VALE-BPF 8.52 23.66
For more details: https://docs.google.com/document/d/1rdrHIeap8gYRh3es4yCnuWkuA6zDDot4UDFgEyiuG3E/edit?usp=sharing
Yutaro Hayakawa | eBPF implementation for FreeBSD 29
Demo
Miscellaneous ideas
Networking
- ng_ebpf: Netgraph module for eBPF
- XDP emulator: Compatibility with XDP program
- Hardware offloading
Security
- Systemcall filtering like seccomp
Yutaro Hayakawa | eBPF implementation for FreeBSD 31
Sammary
1. eBPF is a hot technology among Linux community and they introduce
a lot of interesting features and useful tools around that
2. eBPF implementation for FreeBSD is going on
3. VALE-BPF, a extension module which enhances eBPF
programmability to VALE switch improves the programmability of
VALE switch
Yutaro Hayakawa | eBPF implementation for FreeBSD 32
Questions?
