1 Kprobe-based Event Tracing
2 ==========================
4 Documentation is written by Masami Hiramatsu
9 These events are similar to tracepoint based events. Instead of Tracepoint,
10 this is based on kprobes (kprobe and kretprobe). So it can probe wherever
11 kprobes can probe (this means, all functions body except for __kprobes
12 functions). Unlike the Tracepoint based event, this can be added and removed
13 dynamically, on the fly.
15 To enable this feature, build your kernel with CONFIG_KPROBE_TRACING=y.
17 Similar to the events tracer, this doesn't need to be activated via
18 current_tracer. Instead of that, add probe points via
19 /sys/kernel/debug/tracing/kprobe_events, and enable it via
20 /sys/kernel/debug/tracing/events/kprobes/<EVENT>/enabled.
23 Synopsis of kprobe_events
24 -------------------------
25 p[:[GRP/]EVENT] SYMBOL[+offs]|MEMADDR [FETCHARGS] : Set a probe
26 r[:[GRP/]EVENT] SYMBOL[+0] [FETCHARGS] : Set a return probe
28 GRP : Group name. If omitted, use "kprobes" for it.
29 EVENT : Event name. If omitted, the event name is generated
30 based on SYMBOL+offs or MEMADDR.
31 SYMBOL[+offs] : Symbol+offset where the probe is inserted.
32 MEMADDR : Address where the probe is inserted.
34 FETCHARGS : Arguments. Each probe can have up to 128 args.
35 %REG : Fetch register REG
36 @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
37 @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
38 $stackN : Fetch Nth entry of stack (N >= 0)
39 $stack : Fetch stack address.
40 $argN : Fetch function argument. (N >= 0)(*)
41 $retval : Fetch return value.(**)
42 +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(***)
43 NAME=FETCHARG: Set NAME as the argument name of FETCHARG.
45 (*) aN may not correct on asmlinkaged functions and at the middle of
47 (**) only for return probe.
48 (***) this is useful for fetching a field of data structures.
51 Per-Probe Event Filtering
52 -------------------------
53 Per-probe event filtering feature allows you to set different filter on each
54 probe and gives you what arguments will be shown in trace buffer. If an event
55 name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event
56 under tracing/events/kprobes/<EVENT>, at the directory you can see 'id',
57 'enabled', 'format' and 'filter'.
60 You can enable/disable the probe by writing 1 or 0 on it.
63 This shows the format of this probe event.
66 You can write filtering rules of this event.
69 This shows the id of this probe event.
74 You can check the total number of probe hits and probe miss-hits via
75 /sys/kernel/debug/tracing/kprobe_profile.
76 The first column is event name, the second is the number of probe hits,
77 the third is the number of probe miss-hits.
82 To add a probe as a new event, write a new definition to kprobe_events
85 echo p:myprobe do_sys_open dfd=$arg0 filename=$arg1 flags=$arg2 mode=$arg3 > /sys/kernel/debug/tracing/kprobe_events
87 This sets a kprobe on the top of do_sys_open() function with recording
88 1st to 4th arguments as "myprobe" event. As this example shows, users can
89 choose more familiar names for each arguments.
91 echo r:myretprobe do_sys_open $retval >> /sys/kernel/debug/tracing/kprobe_events
93 This sets a kretprobe on the return point of do_sys_open() function with
94 recording return value as "myretprobe" event.
95 You can see the format of these events via
96 /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
98 cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format
102 field:unsigned short common_type; offset:0; size:2; signed:0;
103 field:unsigned char common_flags; offset:2; size:1; signed:0;
104 field:unsigned char common_preempt_count; offset:3; size:1;signed:0;
105 field:int common_pid; offset:4; size:4; signed:1;
106 field:int common_lock_depth; offset:8; size:4; signed:1;
108 field:unsigned long __probe_ip; offset:12; size:4; signed:0;
109 field:int __probe_nargs; offset:16; size:4; signed:1;
110 field:unsigned long dfd; offset:20; size:4; signed:0;
111 field:unsigned long filename; offset:24; size:4; signed:0;
112 field:unsigned long flags; offset:28; size:4; signed:0;
113 field:unsigned long mode; offset:32; size:4; signed:0;
116 print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->__probe_ip,
117 REC->dfd, REC->filename, REC->flags, REC->mode
119 You can see that the event has 4 arguments as in the expressions you specified.
121 echo > /sys/kernel/debug/tracing/kprobe_events
123 This clears all probe points.
125 Right after definition, each event is disabled by default. For tracing these
126 events, you need to enable it.
128 echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
129 echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable
131 And you can see the traced information via /sys/kernel/debug/tracing/trace.
133 cat /sys/kernel/debug/tracing/trace
136 # TASK-PID CPU# TIMESTAMP FUNCTION
138 <...>-1447 [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0
139 <...>-1447 [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe
140 <...>-1447 [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6
141 <...>-1447 [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
142 <...>-1447 [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10
143 <...>-1447 [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
146 Each line shows when the kernel hits an event, and <- SYMBOL means kernel
147 returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel
148 returns from do_sys_open to sys_open+0x1b).