ltrace Command: Tutorial & Examples

Watch a program call its libraries in real time — every malloc, every strcmp, live.

What It Is

ltrace is a tap on the wire inside a program. You hand it any command, and as that command runs, ltrace prints — live, one line each — every call it makes into a shared library: every malloc, every strcmp, every fopen, with the arguments going in and the value coming back. It's an x-ray of a running program's conversation with the code it borrowed, and you don't need the source, a debugger, or even to know what language it was written in.

If you've ever stared at a binary doing something wrong and wished you could just see what it's doing without recompiling it with a thousand printfs, this is that wish granted. ltrace is the close cousin of strace: where strace watches calls into the kernel (the system-call boundary), ltrace watches calls into libraries (the library boundary) — one rung up the ladder, in user space, where the program talks to libc and friends. By the end of this page you'll read its output fluently, know every flag worth knowing, and understand the genuinely clever trick it pulls to spy on a function call it was never invited to.

Your First Look

Point it at any dynamically linked program. Here's ls on an empty directory — trimmed, because the real thing scrolls:

ltrace ls
__libc_start_main(0x401d60, 1, 0x7ffd3e9c2b18, 0x4046e0 <unfinished ...>
setlocale(LC_ALL, "")                            = "C.UTF-8"
bindtextdomain("coreutils", "/usr/share/locale") = "/usr/share/locale"
textdomain("coreutils")                          = "coreutils"
__cxa_atexit(0x404170, 0, 0x4f0000, 0)           = 0
getenv("QUOTING_STYLE")                          = nil
isatty(1)                                        = 1
getenv("COLUMNS")                                = nil
ioctl(1, 21523, 0x7ffd3e9c2960)                  = 0
malloc(80)                                       = 0x1c4e2a0
opendir(".")                                     = 0x1c4e2d0
readdir(0x1c4e2d0)                               = 0x1c4e300
readdir(0x1c4e2d0)                               = 0x1c4e330
readdir(0x1c4e2d0)                               = nil
closedir(0x1c4e2d0)                              = 0
free(0x1c4e2a0)                                  = 0
strcmp("file_a", "file_b")                       = -4
fwrite("file_a\n", 7, 1, 0x7f...)                = 1
+++ exited (status 0) ===

Read one line and you've read them all. It's function(arguments) = return value, exactly the call you'd write in C, captured the instant it happened. You can already see the whole story of ls: it sets the locale, checks whether stdout is a terminal (isatty(1) returned 1, so yes), asks the terminal its width via ioctl, then opendir / readdir / closedir to walk the directory, strcmp to sort the names, and fwrite to print them. That +++ exited (status 0) === at the end is ltrace telling you the program finished cleanly. You just watched a compiled binary think out loud.

How I Read It

Here's the order my eyes actually move, because raw ltrace output can be a firehose and 95% of it is noise around the one line that matters.

First, I scroll straight to the bottom. When a program crashes or hangs, the last library call is almost always the scene of the crime — the thing it was doing when it died. A program that ends on +++ killed by SIGSEGV +++ right after a strcpy into a buffer it malloc'd too small has just confessed. I read bottom-up.

Second, I look at return values, not arguments. A library call that returns nil (a null pointer) or -1 is a function saying "I failed." fopen("/etc/config", "r") = nil means the file open failed — and the very next thing the program does with that nil pointer is usually where it falls over. The return column is where bugs announce themselves.

Third, I narrow. Unfiltered ltrace shows everything, and on a real program that's thousands of malloc/free/strlen calls burying the three that matter. So almost the first thing I ever type is a filter — -e to whitelist the calls I care about. I don't read full ltrace output; I interrogate it with -e malloc+free to hunt a leak, or -e 'fopen+open*' to see which files it touches. Treat the unfiltered run as a menu, then order off it.

The move that took me embarrassingly long to learn: add -S. By default ltrace shows only library calls and hides system calls — but -S folds them back in, interleaved in true time order, so you see the program call fopen (a library function) and then watch it call the kernel's open (a system call) underneath. Suddenly you've got both rungs of the ladder on one screen, and you can see exactly where the library hands off to the kernel. That single flag turns ltrace into ltrace and strace at once.

The Output, Explained

Every part of an ltrace line, so nothing is mysterious:

The function name — the library function being called: malloc, strcmp, fopen. These are real symbols from real shared libraries; ltrace resolves them by name so you read getenv, not a raw address.
The arguments, in parentheses — ltrace prints them the way you'd hope: strings as "text" (quoted, escaped, truncated at a sensible length), numbers as numbers, pointers as hex like 0x1c4e2d0, and a null pointer as the friendly word nil. It's guessing at the types (more on that gotcha below), but for the common libc functions it knows them cold.
= return value — what the function handed back. nil and -1 mean failure; a hex pointer means success (it allocated something); a number is whatever the function returns. This column is where you debug.
<unfinished ...> / <... resumed> — a call that hadn't returned yet when something else happened (a signal, or in a multithreaded program another thread cutting in). ltrace parks the line, prints the interruption, then resumes it later. You'll see __libc_start_main sit <unfinished ...> for the program's entire life, because it only "returns" when main does.
+++ exited (status N) +++ — the program ended normally, returning exit code N to the shell.
--- SIGSEGV --- / +++ killed by SIGSEGV +++ — a signal arrived (the program tripped over a bad pointer, got killed, etc.). The triple-dash form is "a signal was delivered"; the triple-plus killed by form is "and it died from it."
PID: prefix — only appears with -f (follow children). When ltrace is watching a process that forks, every line gets stamped with which process said it, so you can untangle parent from child.

The Flags, Explained

The flags you'll actually reach for, grouped by job. ltrace has a lot of them; these are the load-bearing set.

Choosing what to trace:

-e FILTER — the most important flag, full stop. A filter expression of which calls to show. -e malloc shows only malloc; -e malloc+free+realloc shows the memory trio; -e 'str*' globs all the string functions; -e '!printf' shows everything except printf. You'll live in this flag.
-l LIBRARY — restrict tracing to calls into one specific library, e.g. -l libssl.so.* to watch only the crypto. Gold when a program links a dozen libraries and you care about one.
-S — also trace system calls (the strace job), interleaved with the library calls in time order. The flag that gives you both boundaries at once.
-x SYMBOL — trace a specific internal symbol by name, not just the library-boundary ones (see the magic section for why that distinction matters).

Following the program:

-f — follow forked children. Without it you trace only the parent and lose sight of anything it spawns; with it every child is traced too, each line stamped with its PID. Essential for shells, daemons, anything that forks.
-p PID — attach to an already-running process instead of launching a new one. ltrace -p 4242 taps a live program mid-flight; Ctrl-C to detach and leave it running.

Counting instead of listing (the underrated mode):

-c — don't print every call; print a summary table at the end. Count, total time, and time-per-call for each function, sorted by cost. This turns ltrace from a log into a profiler — one glance and you know the program called strlen four million times, which is your whole performance problem right there.
-T — print the time spent inside each individual call, in the margin. Pairs with reading a slow call live.
-t / -tt / -ttt — prefix each line with a timestamp (-t seconds, -tt microseconds, -ttt raw epoch). For "what happened at 14:03:07?" forensics.

Taming the output:

-o FILE — write the trace to a file instead of stderr. Do this for anything real — ltrace output and the program's own output otherwise tangle together on your terminal.
-s SIZE — max string length before truncation (default is short, ~32). Bump it with -s 200 when a string argument you care about is getting cut off mid-word.
-n SPACES / -i / -A — indent nested calls, show the instruction pointer of each call, and set how many array elements to print. Cosmetic, occasionally clutch.

Pro Tip

Reach for -c before you read a single line of trace. On any non-trivial program the unfiltered firehose is unreadable, but ltrace -c ./prog ends with a tidy table ranking every library function by how much time it ate. That ranking usually points straight at the bug or the bottleneck — and then you know which -e FILTER to drill in with.

The Magic: How It Spies on a Call It Wasn't Invited To

Here's the part that's genuinely lovely, and it explains both how ltrace works and one frustrating limit you'll eventually hit.

When you compile a program that calls malloc, the compiler doesn't know where malloc will live in memory — that's decided later, when the dynamic linker loads libc at runtime. So instead of a direct call, the compiler inserts a little hop through two tables baked into every dynamically linked binary: the PLT (Procedure Linkage Table) and the GOT (Global Offset Table). The first time the program calls malloc, control bounces through the PLT, the linker looks up where malloc actually landed, writes that address into the GOT, and from then on every call goes straight there. It's a tiny, beautiful piece of lazy plumbing that makes shared libraries possible at all — run readelf -d /bin/ls and you'll see the PLTGOT entry pointing right at it.

And that hop is the doorway ltrace walks through. It uses ptrace — the same kernel debugging facility behind gdb and strace — to set a breakpoint on each PLT entry. Every time the program crosses the library boundary, it trips the breakpoint, ltrace wakes up, reads the arguments out of the registers and stack, prints the line, then sets the program running again until it returns — at which point it reads the return value too. The program never knows. It thinks it's just calling malloc; in reality every single library call is being intercepted at that PLT hop, photographed, and waved through.

Which hands you the one limitation worth understanding up front. ltrace can only see calls that cross the library boundary — calls that go through the PLT. A function that libc calls internally (one library function calling another within the same .so) never crosses that boundary, so it never trips a breakpoint, so ltrace is blind to it. That's why ltrace shows you the program's conversation with its libraries but not the libraries talking among themselves. (The -x SYMBOL flag lets you hand-place a breakpoint on a named internal symbol when you really need to peek inside — but the easy, automatic magic is strictly at the boundary.) Once you know it's "breakpoints on the PLT," every quirk of ltrace suddenly makes sense.

Reading It by Example

Build instinct fast — readout on the left, verdict on the right.

fopen("/etc/app.conf", "r") = nil → the program couldn't open its config file (returned a null pointer). The bug isn't in ltrace's output — it's that the file is missing, misnamed, or unreadable, and whatever the program does next with that nil is where it'll misbehave. This one line solves a startling number of "it won't start" mysteries.

A wall of malloc(...) with very few matching free(...) → a memory leak in slow motion. Run ltrace -e malloc+free+realloc ./prog, watch the allocations pile up with no frees, and you've found it without a single source edit.

strcmp("admin", "admin") = 0 → a string compare that matched (strcmp returns 0 for equal). Watching the strcmp calls of a program checking a password or a license key is a small education in why you don't compare secrets with strcmp.

... <unfinished ...> as the very last line, then nothing → the program is hung inside that call. If it's read(...) or a lock function, it's blocked waiting on something that's never coming — a dead socket, a held mutex. The unfinished line names exactly what it's stuck on.

+++ killed by SIGSEGV +++ immediately after a strcpy or memcpy → a buffer overrun. The program copied more bytes than the destination could hold and walked off the end of its memory. The call right before the crash is the smoking gun.

ltrace -c shows strlen called 4,000,000 times, eating 60% of runtime → an algorithmic bug, usually calling strlen inside a loop on the same unchanging string. Pure profiling win; you'd never have guessed without the count.

How You'll Actually Use It

Honestly, most real ltrace sessions are one targeted question, not a full read-through:

# Why won't this thing start? Show me every file it tries to open.
ltrace -e 'fopen+open*' -S ./myapp 2>trace.log

# Is it leaking memory? Watch the alloc/free balance.
ltrace -e malloc+free+realloc -o leak.log ./myapp

# It's slow — which library function is the hog?
ltrace -c ./myapp arg1 arg2

# It's already running and stuck — attach and look.
ltrace -p "$(pgrep -n myapp)"

Note the 2>trace.log and -o: ltrace writes to stderr, which tangles with the program's own output if you don't redirect one of them. Separate them every time.

Gotchas

The performance cost is brutal — never on production. Every library call now means a context switch out to ltrace, a read of registers, a print, and a switch back. A program that calls malloc a million times will run tens to hundreds of times slower under ltrace. It's a debugging tool for a test box, not something you leave attached to a live service. (This is also why ltrace is a foot-gun under heavy load — see the tie-in.)
Static binaries are invisible. No shared libraries means no PLT means nothing for ltrace to hook. A statically linked Go binary, or anything built with -static, shows you essentially nothing. Check with ldd ./prog first — "not a dynamic executable" means ltrace won't help.
ltrace is guessing argument types. For well-known libc functions it knows the signatures and prints them perfectly. For a random third-party library it doesn't recognize, it falls back to printing arguments as plain integers — so a string shows up as a meaningless hex pointer instead of "hello". The fix is a .conf prototype file teaching it the signatures, which almost nobody does; just know that ugly integer args mean "ltrace doesn't have the prototype."
It needs ptrace permission. On hardened systems ptrace of non-child processes is restricted (kernel.yama.ptrace_scope), so ltrace -p on something you don't own may need root. Inside a container, ptrace is often blocked entirely without --cap-add=SYS_PTRACE.
Threads and forks scramble the output. A multithreaded program interleaves calls from every thread, and without -f you silently lose every forked child. Reach for -f early on anything that spawns.

ltrace vs strace vs gdb

Three tools, three different windows into the same running program — and knowing which to grab is half the battle:

ltrace watches the library boundary — your program calling malloc, fopen, strcmp. Use it when the bug is in how your code talks to the libraries it uses.
strace watches the kernel boundary — your program (or the libraries beneath it) calling open, read, write, connect. Use it when the question is "what is it asking the operating system to do?" — files, sockets, permissions. When a program can't open a file and you want to know why (ENOENT? EACCES?), strace gives you the exact errno; ltrace only shows you a nil came back.
gdb is the full debugger — stop anywhere, inspect any variable, step line by line. Heavier, interactive, source-aware. Use it when tracing isn't enough and you need to poke around inside.

The mental model: ltrace and strace are the same idea (ptrace breakpoints, watch the calls fly by) aimed at two different boundaries one rung apart. Many veterans reach for strace first because the kernel boundary is where files, networks, and permissions live — but for a program misbehaving in its own logic, ltrace is the sharper knife. And the lovely thing: ltrace -S gives you both at once, so you needn't always choose.

History & Philosophy

ltrace was written in the late 1990s by Juan Cespedes as the obvious companion to strace — if one tool watches the syscall boundary, surely something should watch the library boundary right above it. It leaned on the same foundation: ptrace, the venerable Unix system call whose entire job is to let one process puppeteer another — read its memory, read its registers, stop it, single-step it. Every tracer and debugger you've ever used — strace, gdb, ltrace — is ultimately a creative use of that one kernel facility. (ptrace is also, BTW, the reason a debugger can't attach to your shell without permission, and the reason hardened kernels lock down ptrace_scope: a tool this powerful is exactly as dangerous as it is useful.)

There's a deeper idea hiding here worth carrying away. The whole reason ltrace can exist is that shared libraries are linked late — resolved at load time through the PLT and GOT rather than baked in at compile time. That late-binding indirection is what lets a hundred programs share one copy of libc in memory, what lets you patch a security hole in OpenSSL once and fix every program that uses it, and — almost as a side effect — what leaves a convenient seam for a tracer to slip a breakpoint into. The same mechanism that makes Linux efficient makes it observable. Pull on that thread — late binding, the dynamic linker, position-independent code — and you end up understanding why a Linux box is the gloriously transparent, pokeable thing it is, where almost nothing about a running program is truly hidden from someone curious enough to look.