strace Command: Tutorial & Examples

The x-ray that shows you every word a program says to the kernel.

What It Is

strace is the tool that makes Linux transparent. It attaches to any running program and prints, line by line, every single system call that program makes to the kernel — every file it opens, every byte it reads, every network packet it sends, every signal it receives. If a program does anything at all that touches the outside world — a file, a socket, a clock, a lock, a screen — it must ask the kernel to do it, and strace shows you the conversation.

That conversation is shorter than you'd think. There are only about 330 system calls in the entire Linux interface, and most programs use a few dozen of them in a tight loop: openat, read, write, close, stat, mmap, brk, futex, clone, exit. Once you've watched the trace scroll past for a minute, "what is this program doing?" stops being a mystery. The operating system becomes a thing you can see through. That's the whole magic of strace, and it's why veteran admins reach for it the moment a program "doesn't work" with no useful logs — the kernel can't lie about what was asked of it.

strace usually isn't preinstalled — sudo apt install strace on Debian/Ubuntu, sudo dnf install strace on Fedora/RHEL. One package, no daemons, no config.

Your First Look

Prepend strace to any command and watch what it does. The output goes to stderr — usually noisy, so redirect with 2>&1 and pipe through head:

strace ls /tmp 2>&1 | head -20

execve("/usr/bin/ls", ["ls", "/tmp"], 0x7ffd2a8e3b80 /* 56 vars */) = 0
brk(NULL)                               = 0x55a8c2f3e000
arch_prctl(ARCH_SET_FS, 0x7f3a8d2e4740) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3a8d2e2000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=89234, ...}) = 0
mmap(NULL, 89234, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3a8d2cc000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\243\0\0\0\0\0\0"..., 832) = 832
...

Every line is one syscall: the name, the arguments in parentheses, an =, and the return value (a file descriptor, a byte count, or -1 plus an error code like ENOENT). What you're watching is ls being born — the kernel loads the binary (execve), then the program asks for memory (brk, mmap), opens its shared libraries (openat on libselinux.so.1, libc.so.6, …), reads them in, and only then gets around to listing /tmp. The first 50 lines of any program are almost always the same boring loader dance; the interesting calls are at the bottom.

How I Read It

The unfiltered firehose is unreadable on purpose — that's why every real session of strace starts by filtering. Here's the order of moves in my head.

First, I ask: do I want to start the program, or attach to one already running? A failing command goes through strace command args. A stuck or runaway process gets strace -p PID — find the PID with pgrep, ps, top, or htop. Attaching reveals what a hung program is doing right now, which is almost always "blocked in a single syscall" — usually read from a socket waiting for data that never came, or futex on a lock another thread is holding.

Second, I filter to the syscalls that matter. Nobody reads raw strace. The flag is -e trace=... and the categories pay you back instantly: -e trace=openat,stat to see every file the program looks for, -e trace=%network for every socket call, -e trace=%file for anything touching the filesystem, -e trace=%signal for signals. Combine them with commas.

Third — and this is the move that's saved me more hours than any other — I grep for ENOENT. When a program "doesn't work" and the logs are useless, nine times out of ten it's looking for a config file in a place you didn't expect:

strace -e openat myprogram 2>&1 | grep ENOENT

You'll see the fourteen paths it tried, in order, before giving up — /etc/myprog.conf, ~/.config/myprog/config, ./myprog.conf, /usr/local/etc/myprog.conf, … and the one you actually edited isn't in the list. The bug is found in five seconds; no source code, no docs, no support ticket. The kernel just tells you where the program looked. This trick alone justifies learning strace.

Fourth, if I just want a budget — what's this program spending its time on? — I use -c. It runs silently, then prints a summary table when the program exits.

Warning

strace pauses the traced process on every single syscall using ptrace — a busy program can run 10-100× slower under strace. Never wrap a production web server or database in a bare strace -p PID during traffic — you'll trigger the very timeout you're trying to debug. Scope tightly: attach for a few seconds (-p PID then Ctrl-C), or use -e trace=... to filter, or use the per-call summary -c which is far cheaper than printing every line.

The Output, Explained

Read one line and you can read them all. Every strace line has the same five-part shape:

openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3

Syscall name — openat here. The full catalog (man 2 syscalls) is the Linux kernel's entire API to userspace.
Arguments in parens — exactly the C function call signature from the man page. Flags like O_RDONLY|O_CLOEXEC are bitwise OR'd constants; strace decodes them back to names for you. Strings are quoted and printable; binary blobs show as escaped bytes like "\177ELF\2\1\1\0..." with a length suffix.
= — the return-value separator. Always there, even for syscalls that "don't return" anything meaningful.
Return value — usually a small non-negative integer (file descriptor, byte count, PID) on success.
Error tail — on failure, -1 followed by the symbolic error name (ENOENT, EAGAIN, EACCES, EPERM, ECONNREFUSED) and its plain-English meaning in parens. This is the column you grep.

When you pass -f (follow forks), each line is prefixed with the PID that made the call, so you can untangle a multi-process program. Pass -tt and each line gets a microsecond timestamp; pass -T and each line gets <0.000123> at the end showing how long the syscall took. Those three flags together — -ftt -T — are the "real debugging" stance.

A short tour of the calls you'll see on every page of every trace:

openat, close, read, write — file I/O. The whole filesystem is just these four, plus cousins (pread, pwrite).
stat, fstat, newfstatat — "tell me about this file without opening it" (size, permissions, mtime). Programs love stating things.
mmap, munmap, brk, mprotect — memory. mmap maps a file or fresh pages into the process's address space; brk grows the heap.
clone, execve, wait4, exit_group — process lifecycle: spawning children, replacing programs, dying.
socket, connect, accept, sendto, recvfrom — networking. The whole TCP/UDP stack as syscalls.
futex — fast userspace mutex. A hung multi-threaded program is almost always blocked here on a lock.
rt_sigaction, kill, tgkill — signal handling (see kill). ioctl — the catch-all "do something device-specific" call.

Reading It by Example

The four moves you'll keep coming back to.

Find where a program looks for its config (the ENOENT trick):

strace -e openat -f mytool 2>&1 | grep ENOENT

openat(AT_FDCWD, "/etc/mytool.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/mytool/mytool.conf", O_RDONLY) = -1 ENOENT
openat(AT_FDCWD, "/usr/local/etc/mytool.conf", O_RDONLY) = -1 ENOENT
openat(AT_FDCWD, "/home/holger/.config/mytool/config", O_RDONLY) = -1 ENOENT

There's your search path, ranked. Put your file in the first location it tried and the bug is fixed.

Attach to a hung process and see what it's stuck on:

strace -p $(pgrep -n mytool)

read(7,

…and nothing else for thirty seconds. The program is blocked in read() on file descriptor 7, waiting for bytes that aren't coming. Pair with lsof to learn what fd 7 is (lsof -p PID | grep '7u') and you'll know whether it's a stuck socket, a dead NFS mount, or a pipe with no writer. Combine in one shot with -y (decode descriptors to filenames inline) — see the Pro Tip.

Get a syscall budget — what's this program spending its life on?

strace -c -p 2323

Wait a few seconds, Ctrl-C, and you get a sorted table:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ---------------
 87.42    1.402311          14    100165       412 read
  8.11    0.130091           4     32540           write
  3.02    0.048470          14      3461           futex
  ...
------ ----------- ----------- --------- --------- ---------------
100.00    1.604812                136166       412 total

If 80% of the time is in read on a socket, this is a network-bound program. If it's in futex, you have lock contention. If it's in stat, it's hammering the filesystem looking for files. Different bottlenecks, different fixes — and now you know which.

Trace a forking program — strace -f -e trace=execve mybuild.sh 2>&1 gives you the full call tree of every command a script ran. -f (follow forks) attaches to every child; without it, strace silently ignores the children and the wrapper looks like it does nothing.

Pro Tip

add -y to decode file descriptors to their actual paths inline (read(3</etc/passwd>, ...) instead of read(3, ...)), and -yy to decode socket fds to their endpoints (read(7<TCP:[10.0.0.5:443->1.2.3.4:34521]>, ...)). The pair -fyy is the modern "I want everything legible" stance; you trade a tiny bit of overhead for output you can actually read without consulting lsof in another window.

Cheat Sheet

The flags worth knowing by heart:

strace cmd args — trace from launch.
strace -p PID — attach to a running process. Detach with Ctrl-C.
strace -f cmd — follow forks; trace child processes too. Almost always what you want.
strace -e trace=openat,stat cmd — only these syscalls. Comma-separated.
strace -e trace=%file cmd — all file-touching calls (also %network, %signal, %process, %memory, %ipc, %desc).
strace -e trace=!futex cmd — exclude futex (the ! is "everything except").
strace -c cmd — silent run, summary table on exit. The cheapest, lowest-overhead view.
strace -y cmd — decode fds to filenames; -yy decode socket fds to endpoints.
strace -tt cmd — microsecond timestamps. -T time spent in each syscall.
strace -o trace.log cmd — output to a file instead of stderr. Faster, and lets the program's own stderr through.
strace -s 4096 cmd — print strings up to 4096 bytes (default 32, which truncates everything interesting).
strace -e inject=openat:error=ENOENT cmd — inject an error to test how the program handles it.

Anatomy of a trace line — openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = -1 ENOENT (No such file or directory):

Part	Meaning
Syscall name	The call being made (`openat`)
Arguments (in parens)	The call's arguments, matching its C signature
`=`	Separator before the return value
Return value	Result on success (fd, byte count, PID)
errno + message	On failure: `-1` plus error name and meaning (e.g. `ENOENT`)

Common syscalls you'll see:

Syscall	Meaning
openat	Open a file (returns a file descriptor)
read	Read bytes from a file descriptor
write	Write bytes to a file descriptor
stat / fstat	Get file metadata (size, permissions, mtime) without reading it
mmap	Map a file or fresh pages into the process's address space
futex	Fast userspace mutex — where blocked threads wait on a lock
connect	Open a network connection to a remote endpoint

How You'll Actually Use It

In practice, strace lives in three moments. One: a program fails silently or with a useless error — you run strace -e openat,stat -f program 2>&1 | grep ENOENT and find the missing file in seconds. Two: a process is hung and you want to know why before you kill it — strace -p PID shows the syscall it's stuck in, lsof -p PID tells you what the file descriptor is, and now you can decide if the fix is a restart, a network repair, or a code bug. Three: a program is mysteriously slow — strace -c -p PID for ten seconds shows you the time budget, and the answer is almost always "it's making 100,000 stat calls per second on a directory it could cache."

If you want library-level (not syscall-level) tracing — to see which C-library functions a program calls before they hit the kernel — that's ltrace instead. If you want a snapshot of what files a process has open right now, that's lsof. If you want a stack trace because the program crashed, that's gdb. strace is the in-between view: not "what's the code doing" and not "what's open," but "what is this program asking the kernel to do, right now, in order."

The other beautiful trick is pause-inspect-resume: from another shell, kill -STOP PID freezes the process (zero CPU, no state lost), you attach with strace -p PID, then kill -CONT PID from a third shell to let it run again. You see the very next syscall it makes after waking up. See kill for the signal recipe.

Gotchas

The performance hit is real. Every syscall is paused, copied to strace, printed, then resumed — 100k syscalls/sec may slow to 1-10k/sec. Use -c (summary) or -e trace=... filters; never bare strace on a live database under load.
Output goes to stderr, not stdout. Redirect with 2>&1 | less or -o file — beginners pipe to grep and wonder why nothing comes through.
Strings are truncated to 32 bytes by default. A read(3, "very long tha"..., 4096) = 1834 hides the interesting part. Use -s 4096.
You can only trace processes you own. ptrace needs the same permission as kill; use sudo strace -p PID for root-owned ones. On hardened systems with yama's ptrace_scope, even sudo can be blocked — check cat /proc/sys/kernel/yama/ptrace_scope.
-f is almost always what you want. Without it, strace shows the parent and silently ignores the children — so a wrapper script appears to do almost nothing.
Errors in the trace are not bugs. Programs expect ENOENT from probing config paths and EAGAIN from non-blocking sockets — they're normal control flow. Don't panic at every -1 you see; only the unexpected ones matter.
Ctrl-C on strace -p PID only detaches. The traced process keeps running, undisturbed — surprises people the first time.

History & Philosophy

strace was written in 1991 by Paul Kranenburg for SunOS, ported to Linux shortly after, and Dmitry Levin steers it today. The underlying mechanism, ptrace, is older still: it shipped in Version 6 Unix in 1975 as the kernel's primitive for debuggers. Every interactive debugger — gdb, lldb, the Java debugger — uses the same syscall.

The deeper idea is the one that makes Linux lovable: the boundary between userspace and the kernel is narrow, well-defined, and observable. A program can do absolutely anything it wants inside its own address space — compute, mutate memory, run any algorithm — but the instant it wants to talk to the outside world (read a file, open a socket, get the time, allocate memory, spawn a process), it must cross one tiny, documented interface: the system call table. Around 330 entries. The whole operating system is reachable through 330 numbered doors, and strace sits at the doorway with a clipboard logging every visit.

Once that lands, you stop seeing programs as black boxes and start seeing them as citizens having conversations with the kernel. A web server is accept → read → write → close in a loop. A compiler is openat → read → close ten thousand times. A database is pread → pwrite → fsync → futex forever. The same way top and ps reveal the live state of the machine as files in /proc, strace reveals the live behavior of every program as a stream of syscalls. The kernel can't lie — the trace is always the ground truth.