DTrace provides dynamic instrumentation capabilities on Solaris by inserting tracepoints in the kernel and user processes. It uses three main providers - FBT for function boundary tracing, fasttrap for process tracing, and SDT for static probes. The key aspects are inserting traps via binary rewriting, and intercepting the traps in the kernel to correlate probes with user-level programs and actions.
DTrace provides dynamic instrumentation capabilities on Solaris by inserting tracepoints in the kernel and user processes. It uses three main providers - FBT for function boundary tracing, fasttrap for process tracing, and SDT for static probes. The key aspects are inserting traps via binary rewriting, and intercepting the traps in the kernel to correlate probes with user-level programs and actions.
DTrace provides dynamic instrumentation capabilities on Solaris by inserting tracepoints in the kernel and user processes. It uses three main providers - FBT for function boundary tracing, fasttrap for process tracing, and SDT for static probes. The key aspects are inserting traps via binary rewriting, and intercepting the traps in the kernel to correlate probes with user-level programs and actions.
DTrace provides dynamic instrumentation capabilities on Solaris by inserting tracepoints in the kernel and user processes. It uses three main providers - FBT for function boundary tracing, fasttrap for process tracing, and SDT for static probes. The key aspects are inserting traps via binary rewriting, and intercepting the traps in the kernel to correlate probes with user-level programs and actions.
Copyright:
Attribution Non-Commercial (BY-NC)
Available Formats
Download as PDF, TXT or read online from Scribd
Download as pdf or txt
You are on page 1/ 22
The DTrace backend on
Solaris for x86/x64
Frank Hofmann OP/N1 Released Products Engineering Sun Microsystems UK Overview • General overview on DTrace capabilities • DTrace providers • Architecture-specifics: Dynamic Instrumentation • Implementation of DTrace providers: > fbt(7d) – kernel function boundary tracing > fasttrap(7d) – the PID provider > sdt(7d) – statically-defined tracing • DTrace architecture overview dtrace(1M), others – DTrace consumers Event specifications, actions (D program source) Results User Mode libdtrace(3lib) – D compiler
Event specifications, actions
(DOF – D Object Format) ioctl() Results (DIF – D Intermediate Format) Kernel builtin dtrace(7d) – Framework, D interpreter providers ECB – Enabler Control Block
fbt(7d) enabling/disabling probes
(provider specific) fasttrap(7d) 'Ye greate sdt(7d) probe hits Solaris kernel other providers Architecture Dependence in DTrace • Most parts of DTrace (including many providers) are fully generic • Architecture Dependencies found in: > Safe access to kernel memory (stray pointer detection) > Areas that differ via ABI: > Function argument retrieval > Stacktracing > Providers with “tracepoint”-style probes • The devil is in the detail ... DTrace Sourcecode structure Check OpenSolaris source tree: http://cvs.opensolaris.org/source/xref/on/usr/src/ • Generic sourcecode organization: > “top half” - generic interfaces, common to all architectures > “bottom half” - platform-specific backend. DTrace Sourcecode structure Program generic source platform-specific dtrace(1m) cmd/dtrace/ cmd/dtrace/amd64/ cmd/dtrace/i386/ cmd/dtrace/sparc/ libdtrace(3lib) lib/libdtrace/common/ lib/libdtrace/amd64/ lib/libdtrace/i386/ lib/libdtrace/sparc/ dtrace(7d) uts/common/dtrace uts/intel/dtrace/ uts/sparc/dtrace/ Tracepoints – heart of dynamic tracing • Allow to instrument everything ... • Zero overhead if tracing is inactive
What DTrace does:
• Actively manipulate binary code (program text) • Insert instructions that cause traps • Interpose on the trap handler(s). Traced vs. non-traced: PID provider machine code, tracing inactive: traced: main: pushl %ebp int $0x3 main+1: movl %esp,%ebp int $0x3 {junk} main+3: andl $0xfffffff0,%esp int $0x3 {junk} main+6: pushl %ebx int $0x3 main+7: pushl %esi int $0x3 main+8: pushl %edi int $0x3 main+9: pushl $0x8050a4c int $0x3 {junk} main+0xe: pushl $0x6 int $0x3 {junk} main+0x10: call -0x19f int $0x3 {junk} libc.so.1`setlocale main+0x15: addl $0x8,%esp int $0x3 {junk} main+0x18: pushl $0x8050a3c int $0x3 {junk} main+0x1d: call -0x19c int $0x3 {junk} libc.so.1`textdomain main+0x22: ... ... Traced vs. non-traced: FBT provider machine code, tracing inactive: traced: ufs_mount: pushq %rbp int $0x3 ufs_mount+1: movq %rsp,%rbp movq %rsp,%rbp ufs_mount+4: subq $0x88,%rsp subq $0x88,%rsp ufs_mount+0xb: pushq %rbx pushq %rbx [ ... ] [ ... ] [ ... ] ufs_mount+0x3f3: popq %rbx popq %rbx ufs_mount+0x3f4: movq %rbp,%rsp movq %rbp,%rsp ufs_mount+0x3f7: popq %rbp popq %rbp ufs_mount+0x3f8: ret int $0x3
Invalid operation causes #UD trap How SDT works • Sourcecode: Name of statically-defined probe DTRACE_PROBE4(squeue__enqueuechain, squeue_t *, sqp, \ mblk_t *, mp, mblk_t *, tail, int, cnt); \
• ELF object: Symbol Table Section: .symtab index value size type bind oth ver shndx name [2561] 0x000be0d1 0x000000000965 FUNC GLOB D 0 .text squeue_enter_chain Relocation Section: .rela.eh_frame type offset addend section with respect to R_AMD64_PC32 0xbe283 0xfffffffffffffffc .rela.text __dtrace_probe_squeue__enqueuechain
Zero overhead ! How SDT works (continued) • Kernel runtime linker, krtld: usr/src/uts/intel/amd64/krtld/kobj_reloc.c #define SDT_NOP 0x90 #define SDT_NOPS 5
static int sdt_reloc_resolve(struct module *mp, char *symname, uint8_t *instr) { [ ... ] /* * The "statically defined tracing" (SDT) provider for DTrace uses * a mechanism similar to TNF, but somewhat simpler. (Surprise, * surprise.) The SDT mechanism works by replacing calls to the * undefined routine __dtrace_probe_[name] with nop instructions. * The relocations are logged, and SDT itself will later patch the * running binary appropriately. */ [ ... ] for (i = 0; i < SDT_NOPS; i++) instr[i - 1] = SDT_NOP; [ ... ] Tracepoint insertion – DTracing DTrace • Quick idea: # dtrace -n "fbt::fasttrap_tracepoint_install:entry { stack();ustack();exit(0) }"
• Doesn't work – no tracepoints in providers. Use a trick:
uts/intel/dtrace/fbt.c The core of DTrace – trap interposition /* uts/intel/ia32/ml/exception.s * #BP */ ENTRY_NP(brktrap) #if defined(__amd64) cmpw $KCS_SEL, 8(%rsp) Usermode tracepoint hook je bp_jmpud #endif TRAP_NOERR(T_BPTFLT) /* $3 */ jmp dtrace_trap #if defined(__amd64) bp_jmpud: /* * This is a breakpoint in the kernel -- it is very likely that this * is DTrace-induced. To unify DTrace handling, we spoof this as an * invalid opcode (#UD) fault. Note that #BP is a trap, not a fault -- * we must decrement the trapping %rip to make it appear as a fault. * We then push a non-zero error code to indicate that this is coming * from #BP. */ decq (%rsp) push $1 /* error code -- non-zero for #BP */ jmp ud_kernel #endif SET_SIZE(brktrap) The core of DTrace – trap interposition ENTRY_NP(invoptrap) uts/intel/ia32/ml/exception.s cmpw $KCS_SEL, 8(%rsp) jne ud_user
/* * We must first check if DTrace has set its NOFAULT bit. This * regrettably must happen before the TRAPTRACE data is recorded, * because recording the TRAPTRACE data includes obtaining a stack * trace -- which requires a call to getpcstack() and may induce * recursion if an fbt::getpcstack: enabling is inducing the bad load. */ movl %gs:CPU_ID, %eax shlq $CPU_CORE_SHIFT, %rax leaq cpu_core(%rip), %r8 addq %r8, %rax movw CPUC_DTRACE_FLAGS(%rax), %cx testw $CPU_DTRACE_NOFAULT, %cx jnz .dtrace_induced [ ... ]
Check/Catch DTrace-caused faults
References • DTrace OpenSolaris community: http://www.opensolaris.org/os/community/dtrace/ • Blogs of the DTrace authors: > Bryan Cantrill: http://blogs.sun.com/bmc/ > Adam Leventhal: http://blogs.sun.com/ahl/ > Mike Shapiro: http://blogs.sun.com/mws/ • OpenSolaris sourcecode archive: http://cvs.opensolaris.org/source/ • Solaris/x86 Internals and Crashdump analysis: http://www.genunix.org/gen/crashdump/index.html “I am thirsty.”
Jesus John 19:28-29 The DTrace backend on Solaris for x86/x64 Frank Hofmann Frank.Hofmann@sun.com
